New World Notes

Watch: Awesome AR App Copies Real World Objects Then Pastes Them as 3D Files Directly Into Photoshop

4/10 - Cut & paste your surroundings to Photoshop

Code: https://t.co/cVddH3u3ik

Book: @HOLOmagazine
Garment: SS17 by @thekarentopacio
Type: Sainte Colombe by @MinetYoann @ProductionType
Technical Insights: ↓#ML #AR #AI #AIUX #Adobe #Photoshop pic.twitter.com/LkTBe0t0rF

— Cyril Diagne (@cyrildiagne) May 3, 2020

Cyril Diagne, an artist, designer and programmer currently in residency at Google Arts, created this utterly awesome augmented reality app which seamlessly captures objects in the real world... and then "pastes" them as 3D files directly into Photoshop. (Watch the video above, you'll immediately see what I mean.) I've seen similar AR apps before, but nothing quite as seamless or as magical.

Perhaps as amazing: Cyril also made the code publicly available on GitHub here.

"The code was given away because it's a research prototype," Cyril tells me. "I don't think it giving that code away necessarily take away commercial potential. There is much more that can be done with that approach."

His goal is to make AR reality capture as intuitive on a smartphone as it is in Magic Leap -- except without needing an expensive HMD.

"Also the phone has such a perfect form factor it's hard to beat," he adds. "[T]here is no UI, there's no need to move the thumb or care about how we're holding the phone, which happens to be quite pleasant and intuitive... Because we're able to delegate some of the cognitive tasks (like deciding which object is the one being pointed at in an image, selecting and removing the background, .etc) we can reduce the number of steps required to perform a task."

As for how he did it, he goes into great technical detail on Twitter, which I've excerpted for the technically minded below:

Magic Leap's Massive Layoffs Today Telegraphed by Tiny Headset Sales

It's the worst possible news to happen to the VR/AR industry right now, but also the most unsurprising:

Bloomberg reports that half of Magic Leap’s employees were laid off, roughly 1,000 in total. Magic Leap was in the process of exploring a sale, Bloomberg had also reported, one that could value the company at $10 billion. Magic Leap is one of the most well-capitalized consumer hardware startups ever, having raised more than $2.6 billion from investors, including Google, Alibaba and Saudi Arabia’s Public Investment Fund.

Citing COVID-19, CEO Rony Abovitz wrote in a blog post that the company needed to shift focus to enterprise applications of its technology. Early-on, Magic Leap had long emphasized that it was a consumer device company.

I've been expecting this shoe to drop since late last year, when I first heard insider whispers that Magic Leap was selling shockingly tiny numbers of its augmented reality headset -- something more or less confirmed by The Information last December:

VR Developer Trapped in Quarantine Maps His Real Life Home, Turns It Into a Treehouse Escape in VR

After being stuck in the same space for so long, I decided to try mapping it to a virtual treehouse for a more relaxing atmosphere. I think it would be great new feature if Oculus could make the home environments mapped to your real home pic.twitter.com/dL7XCOtXEe

— Emanuel Tomozei (@eman_insilico) April 4, 2020

Like most of us, developer Emanuel Tomozei has been largely trapped at home under quarantine these last few weeks. Unlike most of us, Tomozei is an AR/VR developer by specialty, and so was able to turn his home into a virtual world he could escape into:

"After being stuck in the same space for so long, I decided to try mapping it to a virtual treehouse for a more relaxing atmosphere," he explains on Twitter. (See above.) And so when he's wearing his Oculus Quest, he can still walk over to and sit in his couch -- it's just that instead of staring at the blank wall in front of him, in VR it looks as if his couch is on top of a tree house with a view of the lush forest below.

Creating a VR world that maps exactly to his real life apartment, as you might imagine, took some effort:

Best SuperData AR/VR Datapoints for 2019: Pokémon GO is the AR Market, Standalone VR Games Earned $171 Million Last Year

Game/XR analyst firm SuperData* recently released its 2019 year in review report, and was nice enough to give me a preview. I love when a single datapoint can tell you quite a lot about the larger ecosystem, and in this case, the report has (for me) two. First up, a telling one for AR:

Pokémon GO is 81% of the augmented reality game market -- which effectively means Pokémon GO is basically the entire AR game market.  

And one for VR:

Xiola Linden's Farewell Address to the Second Life Community: Be Open To Change, Be Informed, Be Welcoming

I told you last week that the tech worker formerly known as "Xiola Linden" recently left her role as Community Manager at Linden Lab to take a similar role with mixed reality startup Ubiquity 6. As she did, I asked her if she had any last message for the Second Life community she loves so much. I put it to Xiola this way:

If the SL community asked you the best way to improve itself and make it stronger, what would you suggest?

And this is what she told me -- her farewell address as a Linden, so to speak. 

"Be open to change, but never stop questioning it. Constructive criticism is always going to be received more enthusiastically than attacks. It is very easy to get emotional about something that means so much to us. I know, from my own experiences with Second Life, that it's sometimes difficult to keep calm and Second Life on -- but I also witnessed that sort of emotional response grow into a lot of misinformation and unnecessary panic. So, it's a balance. But it is important and key, I think."

The community should also strive to be more of a community in the best sense of the word, because there's still so much potential in Second Life:

Watch: Oculus Quest Demo Virtually Recreates Developer's Apartment at 1:1 Scale, Adds Interactive Surfaces

Unity Labs developer Greg Madison created this seriously impressive Oculus Quest demo in which he mapped out* his real life apartment with a virtual, 1:1 scale recreation of that apartment -- including the flat surfaces. That way, if you're wearing the Quest, any surface can instantly become an interface for games or other interactive apps:

Why Linden Lab's Beloved Community Manager Joined Mixed Reality Startup Ubiquity6

What you're looking at above is a demo of Display.Land, a mixed reality app from San Francisco-based startup Ubiquity 6, which just hired a new Community Lead: The tech worker formerly known "Xiola Linden", who was Linden Lab's much-loved Community Manager for many years. Just settling into her new office now, Xiola sees much possibility to create unique varieties of virtual social experiences with this technology:

 

"In my experiences with the communities I've had the opportunity to work with," as she puts it to me, "I've talked to a lot of creators who have explained to me the pain points of doing photogrammetry the old way. Seeing how the Displayland app can create a 3D representation of physical spaces and items with the use of a smartphone is so exciting to me.

 

"I think about the possibilities for social experiences within those spaces and I get a little giddy. When I saw the demo [above], I was struck by the idea that artists and professionals can revitalize their workflows with this app. When I got to play with the app, I started to think about what it could mean for creative and creator communities. You can already export your captures from Display.land as a 3D model and use them in compatible 3D software -- which is very exciting for creators of all levels.

 

"Now I am obsessed with the idea of capturing vignettes of my life and how amazing it will be to experience those memories in a three dimensional way... I've already seen some community people from SL checking it out -- which is such a delight to me. I'm really looking forward to so much to come!"

 

Speaking of which, she sees much overlap between the corporate culture of Linden Lab and this much younger startup:

 

Amber Case: Augmented Reality is Hampered By a Zeno's Paradox of Ever-Growing Computational Power

"What’s Holding Augmented Reality Back?" is a new Modus essay by my colleague Amber Case, who makes some novel points that help explain why Magic Leap has, despite $2.6 billion in funding, only sold some 6000 units. It's not just the cost or discomfort with headsets, she argues, but the computing power required to recognize hand gestures and incorporate them into the AR display:

AR advocates often assume gesture control will be the next iteration in user interfaces since it seems so intuitive and natural to human expression. But hand-gesture models and libraries are not uniform. The ambiguous input produced by humans forces the computer to process far more information than a controller with comparatively limited function, like a touchscreen. This makes the interaction prone to a wide range of errors. The user’s background could be too dark, they could be wearing gloves, or they could have hands smaller than those that the device was tested with. This interaction model also likely requires having to train someone to use gestures they’re not yet familiar with, and not everyone will make the gestures in the same way.

... AR devices like Magic Leap and HoloLens struggle with detecting the intersection between hand movements and objects. Awash in the effluvia of reality, the headset cannot always discern that the user is, say, trying to pick up a block, and it forces them to grab it multiple times... Most augmented reality headsets were launched to early adopter enthusiasts and content creators, but even these users quickly found using these devices on a daily basis to be difficult.

I.E., it's not so much that Magic Leap only sold 6000 units, but that those 6000 Magic Leap owners find their own evangelical enthusiasm dampened by the consistent frustration of using it themselves. 

The usual response to this complaint is to say greater computing power will improve the AR user interface over time. But Amber argues that this is an ever-elusive goal:

Despite $2.6 Billion in Funding, Magic Leap Sold Less AR Headsets Than Jeri Ellsworth's Tilt 5 Kickstarter

I've heard rumors to this effect for months, but when The Information reports on it, you can pretty much take it to the bank -- even if it's not much of a deposit:

Magic Leap managed to sell just 6,000 units of its $2,300 Magic Leap One headset in its first six months on sale, a figure made worse by CEO Rony Abovitz’s internal claims that he wanted the startup to sell at least one million units of the device in the first year, a goal the report states he was later convinced to rethink — Abovitz later projected the company would sell 100,000 units in the first year... The company has now raised around $2.6 billion in venture funding from firms like Google, Alibaba and a slew of other investors. 

To put that 6,000 in perspective, Jeri Ellsworth's Tilt 5 Kickstarter recently sold over 6,000 Augmented Reality HMDs in 30 days.

Obviously that's not a completely fair comparison, since Magic Leap's headset is selling for $2,300, as opposed to Tilt 5's $299 entry level price. Then again, you can also say the comparison is unfair in the other direction, since Jeri and her team made these sales as a bootstrapped startup working with a tiny six figure budget, versus Magic Leap working with $2.6 billion dollars from some of the world's top tech companies.

In any case, the repercussions in both directions should be pretty significant:

Why VR/AR Devs Should Stop Referencing Minority Report-ish Gesture Control

"Why Sci-Fi Is Bad for Design" is a new Modus essay (reg. req.) by my colleague Amber Case (a fellow at the Institute for the Future) delving into some classic science fiction movie and TV tropes -- such as Jarvis from Iron Man and Star Trek-esque voice control AI -- that have infected the tech industry for the worse. Another trope she discusses is seriously significant for AR and VR developers: Minority Report-esque gesture control. Here's the problem with that, as she sees it:

Since its debut in 2002, Steven Spielberg’s Minority Report has been an archetypal reference point for technologists enthralled by the way Tom Cruise could manipulate complex data with dramatic swipes of his hands.

But what looks amazing for a few minutes of screentime doesn’t necessarily perform as well offscreen. Can you imagine how sore your muscles would be after using this system for eight hours a day?

And as it happens, she can: