New World Notes

Why Apple Owns the AR Industry Even Without Announcing AR Glasses (Comment of the Week)

"Joey1058" has an excellent rant responding to my speculation on just when (probably not any time soon) Apple will announce the launch of its much-awaited augmented reality glasses. To which he replies:  

HMDs do exist in the wild. A company called Vuzix has been tackling hardware issues for decades. Their flagship product is called the Vuzix Blade. [Pictured here] In my humble opinion, their glasses are every bit as good as Magic Leap, Hololens, or the current darling that is sparkling everyone's eyes, Nreal. The hardware isn't an issue. My issue is Apple.

For some ungodly reason, the industry has taken a "wait-and-see" approach in development that centers around Apple. We all know that Apple has been working on developing AR for some number of years. At first, buying up companies and patents. Then releasing ARKit. And now the buzz is all about its U1 chip and Ultra Wideband tech. But somehow, they've been dragging their butts in releasing an HMD. So in the meantime, smaller companies are developing on what already exists.

That's not to ignore the big tech company other than Google (LOL Glass) which jumped into AR early:

Apple Apparently Testing AR Glasses, But Don't Expect Them Anytime Soon

Despite some insider rumors that Apple would announce its highly anticipated augmented reality glasses this week, nothing is so far forthcoming. Yesterday,  however, a developer more or less confirmed that those AR glasses do indeed exist, and that they're being tested -- because there's mention of a device that can be "worn" in the notes of its next operating system:

It comes to us in the form of this screenshot from iOS developer and reliable leak-finder Steve Troughton-Smith...

The screenshot pretty much says it all: Apple is testing an internal device known as “Garta,” and if you want to test your stereo AR apps with that device, you’ll want to set them to “worn” mode so that it can apply different “distortions” than if the app is being “held” instead.

Which implies that Garta is a stereoscopic AR headset — because that’s the kind of device you would wear, and which would adjust the way an augmented reality app would look to match lenses in front of your eyes. (AR and VR headsets typically adjust their computer graphics “projections” to properly stretch across your field of view while using as little processing power as possible.)

Click to embiggenate Troughton-Smith's screengrab of the OS notes. I'm far from an Apple Kremlinologist, but barring a miracle (more on that later) I wouldn't expect any mass market AR glasses announcement from Apple in the near future -- certainly not next year. A few reasons why:

Watch: Real World Location Virtually Recreated to Scale in Minutes

This is a pretty impressive tech demonstration from 6D.AI, a startup that's creating a 3D mirror world of real life from the input of standard smartphones.

"Building a 3D model of the world means fusing crowdsourced scans into huge models and here's a demo of that in action," 6D CEO Matt Miesnieks explains. "The model is a SLAM map, it improves [with] every scan and map coordinates are real world coordinates." In this particular demo, he goes on, "The ~5000 square foot space took 3 minutes for 4 people to scan.... these are regular phones, no depth cameras involved. Scans can be from one phone doing partial scans over many days, or lots of people scanning at once. The fused model will figure itself out and assemble based on any new scan that overlaps a previous scanned area."

The implications are pretty amazing and open-ended: This potentially means that, say, a whole city block could be slurped up and imported into a virtual world/MMO in a matter of days or weeks. And if the smartphone capture works as well as presented, the potential is far more staggering:

Watch: Piloting A Drone in Real Life While in Virtual Reality

This latest mixed reality demo from Toronto's mix reality mad scientist Lee Vermeulen is an update from a recent one from June: He is flying a Tello drone with a Unity-based controller that is hooked up to his Vive VR headset and Knuckles hand controllers. So the inset video you see is the stream from the drone's camera. This time, however, the heads-up display which shows what the drone is seeing is transparent -- that way, you can still pilot the drone even when the HUD display used to navigate seems to temporarily block the drone in real life.

Oddly enough, this really feels like a killer app for VR/AR headsets. To the extent that drones have multiple killer apps (and they do), they need a way to be controlled with a high degree of precision. And what better way to do that than being able to see what the drone is seeing (so to speak) while also able to see what the drone pilot is seeing?

Watch: Creating and Sculpting Physics-Enabled Objects in AR/VR

Creating and interacting with physics-enabled objects that seem to exist in the real world -- another dazzling feature to Modbox, a multiplayer sandbox game from Toronto developer Lee Vermeulen, who's rigged his Vive VR headset to display an AR experience. (Read all the details on his blog.) 

"This 'Hull Brush' idea was added to TiltBrush by Jeremy Cowles back in July 2018," he explains. "Pretty brilliant way to sculpt 'game ready' assets! Low performance cost and creates low poly objects with convex physics (which also means easy shattering in Modbox).

Watch: Piloting a Drone With a VR Rig While Still Seeing the Drone in AR

This latest brain-expanding mixed reality hack from Toronto developer Lee Vermeulen is a bit difficult to process visually, but it's basically this: He is flying a Tello drone with a Unity-based controller that is hooked up to his Vive VR headset and Knuckles hand controllers. So the inset video you see is the stream from the drone's camera.

However, he's not navigating the drone completely through the VR headset, because he has its AR passthrough function enabled. That way, he can still see the drone in real life while also seeing through the drone's video camera:

Modbox, Sandbox Game Development Platform With Amazing Mixed Reality Features, Coming to Magic Leap

Anytime I blog a VR/AR demo created in Modbox, the multiplayer sandbox game platform from Toronto dev Lee Vermeulen, the brains of mixed reality fans proceed to melt. (Like say this one, and also this one.) So their brains should also liquify at this video showing Vermeulen hard at work converting Modbox to Magic Leap:

Do You Want a Future Where Family Members All Wear AR/VR HMDs, Alone Together in the Same Room?

In a family setting: one person may use a Magic Leap One to watch movies (giant screens, no atoms); another may play experiential games (like Dr. G), another may learn how to code (Unreal, Unity), and another may start building the volumetric internet with Helio.

— Rony Abovitz (@rabovitz) April 9, 2019

Fascinating tweet last night from the CEO of Magic Leap Rony Abovitz, which now has well over $2 billion in funding to bring augmented reality to the mass market; in it, Abovitz sketches out a future when AR has become mainstream:

In a family setting: one person may use a Magic Leap One to watch movies (giant screens, no atoms); another may play experiential games (like Dr. G), another may learn how to code (Unreal, Unity), and another may start building the volumetric internet with Helio.

Which strikes me as a fascinating vision for the future. Abovitz may well be right -- and in a decade or so, it's likely that the price of AR headsets will come down enough that each member of a middle class family could feasibly own a pair. But I suspect this forecast makes a common but false assumption that many technologists make: Assessing the value of a product only by its technical features, while not giving any consideration to how the product might change the social dynamics of its users.

Because direct eye contact, especially among close friends, lovers, and family members, seems to be a deeply ingrained human need or reflex:

Our sensitivity to eye contact begins incredibly early. Infants of just two days of age prefer looking at faces that gaze back at them... Whether or not other people make eye contact with us changes the way that we think about them and their feelings. For example, we are more likely to remember faces with which we’ve experienced mutual gaze, and we consider displays of anger and joy to be more intense when shown by a person making eye contact. In fact, when a person or human-like entity (such as a human face morphed with a doll) makes eye contact with us, we assume that he/she/it has a more sophisticated mind and a greater ability to act in the world, such as to show self-control and act morally, and a greater desire for social contact.

That's quite a lot to override with a new technology. And of course, it's true that we spend far too much of our time on the digital devices we have now. But that brings up another feature, best expressed in visual terms:

Watch: Modbox Sandbox Game Includes Mind-Bending "Mirror World" Mixed Reality Feature

This demo video makes my brain slightly melt:

This was created in the latest update to Modbox, a multiplayer sandbox game from Toronto developer Lee Vermeulen. This particular demo uses a ZedMini camera with an HTC-Vive set-up, and while it may seem like a massive leap to incorporate a mirror's reflection into an augmented reality experience, so that real objects seem to move back and forth into a "mirror world", Lee tells me it's available as a default feature for Modbox: "Literally no special programming /setup at all. I am hoping within the next 2 months the update will be out."

And yes, this is the same platform that makes The Abyss/Terminator 2-style fluid physics possible in VR. More about Modbox on the homepage and on its Steam page. The feature list is insanely robust and exciting:

Jesse Damiani's Mixed Reality Trends for 2019: AI, Location, Theater

Three big trends I’m excited to see emerging in the XR community in 2019:

1. Artificial Intelligence (AI)
2. Physical Installation
3. Immersive Theatre

👇

— Jesse Damiani (@JesseDamiani) January 30, 2019

This is an excellent Twitter thread on 2019 AR/VR/AI trends from mixed reality thought leader* Jesse Damiani. I agree with him that immersive theater and art installations incorporating VR will enjoy a fairly strong wave of interest, because art patrons tend to have greater patience for new/novel experiences, and are a smaller audience that's easier to on-ramp. (As anyone who's tried out a VR demo in a gallery, the set-up time is inevitably irksome and time consuming.)

I'm more skeptical VR/AR physical installations for a mass market will gain traction, notwithstanding the huge VR investment in Sandbox VR -- again, the basic friction of setting up thousands of people to experience VR a day will likely kill the novelty factor quick, especially for a price point of $20-40. (And once again, even in Los Angeles, name brand IMAX couldn't attract an audience to its VR experiences that was larger than the audience for a single small movie theater.)