New World Notes

Watch: Creating and Sculpting Physics-Enabled Objects in AR/VR

Creating and interacting with physics-enabled objects that seem to exist in the real world -- another dazzling feature to Modbox, a multiplayer sandbox game from Toronto developer Lee Vermeulen, who's rigged his Vive VR headset to display an AR experience. (Read all the details on his blog.) 

"This 'Hull Brush' idea was added to TiltBrush by Jeremy Cowles back in July 2018," he explains. "Pretty brilliant way to sculpt 'game ready' assets! Low performance cost and creates low poly objects with convex physics (which also means easy shattering in Modbox).

Watch: Piloting a Drone With a VR Rig While Still Seeing the Drone in AR

This latest brain-expanding mixed reality hack from Toronto developer Lee Vermeulen is a bit difficult to process visually, but it's basically this: He is flying a Tello drone with a Unity-based controller that is hooked up to his Vive VR headset and Knuckles hand controllers. So the inset video you see is the stream from the drone's camera.

However, he's not navigating the drone completely through the VR headset, because he has its AR passthrough function enabled. That way, he can still see the drone in real life while also seeing through the drone's video camera:

Modbox, Sandbox Game Development Platform With Amazing Mixed Reality Features, Coming to Magic Leap

Anytime I blog a VR/AR demo created in Modbox, the multiplayer sandbox game platform from Toronto dev Lee Vermeulen, the brains of mixed reality fans proceed to melt. (Like say this one, and also this one.) So their brains should also liquify at this video showing Vermeulen hard at work converting Modbox to Magic Leap:

Do You Want a Future Where Family Members All Wear AR/VR HMDs, Alone Together in the Same Room?

In a family setting: one person may use a Magic Leap One to watch movies (giant screens, no atoms); another may play experiential games (like Dr. G), another may learn how to code (Unreal, Unity), and another may start building the volumetric internet with Helio.

— Rony Abovitz (@rabovitz) April 9, 2019

Fascinating tweet last night from the CEO of Magic Leap Rony Abovitz, which now has well over $2 billion in funding to bring augmented reality to the mass market; in it, Abovitz sketches out a future when AR has become mainstream:

In a family setting: one person may use a Magic Leap One to watch movies (giant screens, no atoms); another may play experiential games (like Dr. G), another may learn how to code (Unreal, Unity), and another may start building the volumetric internet with Helio.

Which strikes me as a fascinating vision for the future. Abovitz may well be right -- and in a decade or so, it's likely that the price of AR headsets will come down enough that each member of a middle class family could feasibly own a pair. But I suspect this forecast makes a common but false assumption that many technologists make: Assessing the value of a product only by its technical features, while not giving any consideration to how the product might change the social dynamics of its users.

Because direct eye contact, especially among close friends, lovers, and family members, seems to be a deeply ingrained human need or reflex:

Our sensitivity to eye contact begins incredibly early. Infants of just two days of age prefer looking at faces that gaze back at them... Whether or not other people make eye contact with us changes the way that we think about them and their feelings. For example, we are more likely to remember faces with which we’ve experienced mutual gaze, and we consider displays of anger and joy to be more intense when shown by a person making eye contact. In fact, when a person or human-like entity (such as a human face morphed with a doll) makes eye contact with us, we assume that he/she/it has a more sophisticated mind and a greater ability to act in the world, such as to show self-control and act morally, and a greater desire for social contact.

That's quite a lot to override with a new technology. And of course, it's true that we spend far too much of our time on the digital devices we have now. But that brings up another feature, best expressed in visual terms:

Watch: Modbox Sandbox Game Includes Mind-Bending "Mirror World" Mixed Reality Feature

This demo video makes my brain slightly melt:

This was created in the latest update to Modbox, a multiplayer sandbox game from Toronto developer Lee Vermeulen. This particular demo uses a ZedMini camera with an HTC-Vive set-up, and while it may seem like a massive leap to incorporate a mirror's reflection into an augmented reality experience, so that real objects seem to move back and forth into a "mirror world", Lee tells me it's available as a default feature for Modbox: "Literally no special programming /setup at all. I am hoping within the next 2 months the update will be out."

And yes, this is the same platform that makes The Abyss/Terminator 2-style fluid physics possible in VR. More about Modbox on the homepage and on its Steam page. The feature list is insanely robust and exciting:

Jesse Damiani's Mixed Reality Trends for 2019: AI, Location, Theater

Three big trends I’m excited to see emerging in the XR community in 2019:

1. Artificial Intelligence (AI)
2. Physical Installation
3. Immersive Theatre

👇

— Jesse Damiani (@JesseDamiani) January 30, 2019

This is an excellent Twitter thread on 2019 AR/VR/AI trends from mixed reality thought leader* Jesse Damiani. I agree with him that immersive theater and art installations incorporating VR will enjoy a fairly strong wave of interest, because art patrons tend to have greater patience for new/novel experiences, and are a smaller audience that's easier to on-ramp. (As anyone who's tried out a VR demo in a gallery, the set-up time is inevitably irksome and time consuming.)

I'm more skeptical VR/AR physical installations for a mass market will gain traction, notwithstanding the huge VR investment in Sandbox VR -- again, the basic friction of setting up thousands of people to experience VR a day will likely kill the novelty factor quick, especially for a price point of $20-40. (And once again, even in Los Angeles, name brand IMAX couldn't attract an audience to its VR experiences that was larger than the audience for a single small movie theater.)

Watch a Single 2D Image Become 3D Animation: Amazing University of Washington Project Sponsored by NSF, Intel, Facebook, Google & Huawei

Photo Wake-Up is a pretty impressive (if poorly named) project created by researchers at the University of Washington Reality Lab, with big gun sponsorship/assistance from the National Science Foundation, Intel, Facebook, Google and Huawei:

Given a single photo as input, we create a 3D animatable version of the subject, which can now walk towards the viewer. The 3D result can be experienced in augmented reality; in the result [in the video] the user has virtually hung the artwork with a HoloLens headset and can watch the character run out of the painting from different views.

More on the project (and academic paper) at this link. I'd be a bit more skeptical this process works as smoothly as it seems in the demo video, but the major corporate backing suggests we'll see a consumer-ready version of this relatively soon.

Top 12 AR, VR & Avatar-Based AI Projects & Innovations of 2018 Featured in New World Notes

Watch An AI-Controlled Avatar "Learn" How To Dribble In Real Time - Huge Implications For Games, AR/VR, And Real Life Robotics

• HoleLenz Turns HoloLens Into A Literal (Virtual) Portal Into Other Worlds
• We Heard You Like AR & VR, So We Put AR In Your VR
• Project H.E.A.R.T: A Virtual Reality War Game Where Soldier Morale Is Controlled By The Player’s Real Emotions
• VR Pioneer Helping Advise $10 Million ANA Avatar XPRIZE, The Contest To Merge Robotics With VR/AR-Driven Human Expertise

IOS' ARKit Enables Reflection Of Real Objects On Virtual Surfaces

Many more beyond the break!

Instagram Game by an Augmented Reality Expert: Guess If an Image Posted to Her Stories is Real or AR UPDATE: Answer Revealed!

Here's a fun, weekly mini-game on the Instagram of Dr. Helen Papagiannis, an influential author and speaker on the topic of Augmented Reality. As suggested by the screengrab here, you have to guess whether the image posted every week to her Stories is a real photo or an Augmented Reality screenshot. After 24 hours, she reveals which category it actually is; good chance you'll guess wrong.

"My game Real or AR was inspired by just how good Augmented Reality is getting and the double takes I found myself making, as well as the influence AR and the digital was having on the real physical world," she tells me. "The game became a way to share my research and love for AR — as well as art — with a larger audience in a fun game. It’s been awesome to watch the game grow and to play it with people around the globe both on Instagram and on stage in my keynotes."

As with the real or virtual human web quiz I posted awhile back, aggregate answers to previous Real or AR games from Dr. Papagiannis have been wildly off base. 77% of people guessed that this image was AR (but it's real), and while many thought this image of a mesh building was AR, it's made out of actual wire mesh.

I'll update this post when Helen reveals this photo's ontological substrate! I'm personally guessing Real, but a high tech Evil Deceiver may be flimflamming me.

Update, 12/19: Answer is now available below - it may surprise you!

With Slow Sales, VR Headset Market Still Not Having Meaningful "Rebound"

IDC, an analyst firm which was once so bullish about virtual reality, the company predicted (as recently as 2016) that VR headset adoption rates would be as fast the iPhone -- yes, seriously -- is now touting a "rebound" of sales:

The VR headset market returned to growth in 3Q18 after four consecutive quarters of decline and now makes up 97% of the combined [AR/VR] market. Global shipments for VR headsets reached 1.9 million units in the quarter, up 8.2% from 3Q17, as extensive discounts on existing products and interest in new ones led to shipment gains in both the consumer and commercial markets. "The VR market is finally starting to come into its own," said Jitesh Ubrani senior research analyst for IDC Mobile Device Trackers. "On the consumer front, the combination of lower prices and increased content is beginning to resonate with users. Meanwhile, commercial adoption is also on the rise for a range of use cases, including training, design, and showcasing."

If there's commercial adoption, why are there so few palpable VR hits besides Beat Saber (still a modest hit at best, in terms of the overall market) while even the once-viral VRChat has seen a usage decline? But the real problem with this analysis is that "shipments " are not the same as sales or usage  -- and more key, 8.2% growth in shipments isn't much of a "rebound" when you compare the growth to other competing platforms. For instance: