Matthew Ball’s epic, nine part metaverse primer is a must-read for anyone who wants to understand the technological components to make the Metaverse possible. Former global Head of Strategy at Amazon Studios, the widely-sought analyst and VC is most recently the founder of the Metaverse investment fund, the first of its kind.
I’m planning to interview Matthew later this Fall; so over this Summer, I’ll be blogging each part, highlighting the key themes that stand out most to me, then following up with commentary of my own.
My take on Part 1 is here; today I delve into Part 2: Hardware and the Metaverse:
Advances in image capture technology are making mirror world concepts feasible:
Long sought by Metaverse advocates and investors -- over a decade ago, founding Linden Lab investor Mitch Kapor mentioned it to me as a major necessity to drive adoption -- the rise of smartphones and their component technology has made it possible to convert our real world into the virtual one. Most crucial of all, perhaps: Our very selves, into our avatars.
You can see how far we’ve come in this demo (watch above) of Epic’s Live Link Face app, which uses the latest smartphone-based facial capture to map a user’s expression onto high fidelity avatars.
Similar capture technology is also being used to convert the entire real world into high fidelity 3D models that can be shared in the Metaverse. (See this iPhone-driven mirror world demo I wrote about back in 2019.) The technology has vastly evolved since then, as Matthew writes:
Apple’s Object Capture, meanwhile, enables users to create high-fidelity virtual objects using photos from their standard-issue iPhone in a matter of minutes. That all of this is possible through standard consumer-grade hardware is astonishing.
Underlying technology like this, he goes on, can power some unexpected use cases far beyond enabling your Fortnite character to share your expressions:
In the future, this sort of tracking system will be used to reproduce these users, in real-time, in a virtual mirrorworld. Technologies such as Google’s Starline will then allow remote workers to be ‘present’ in the store (or a museum, or a DMV, or a casino) from a sort of offshore ‘Metaverse call center’ — or perhaps at home in front of their iPhones.
Hardware advances may help drive consumer adoption:
Motion sickness is often fingered as a barrier to mainstream adoption of VR, a key access point for the Metaverse. Matthew notes that recent improvements may soon address those:
The Oculus Rift also peaked at a 72hz refresh rate, while the most recent edition achieves 90hz, and up to 120hz when connected to a gaming PC via Oculus Link. Many believe 120hz is the minimum threshold for avoiding disorientation and nausea in some users. And ideally this would be achieved without needing a gaming-level PC and tether.
My own thoughts on Part 2:
The consumer market for mirror world technology remains unproven:
It’s quite possible I’m missing a counter-example, but I’ve yet to see any mirror world app gain strong, sustained mass market usage. For instance, while there have been numerous apps which offer consumers a chance to map their real face or expressions onto an avatar, they seem to fall into the “briefly fun toy” category.
"Hopefully everyone's downloaded it,” the Epic engineer announcing the Live Link Face app says at one point during his demo -- but in a year since it's been available, less than 5,000 people have downloaded it (according to Sensor Tower).
One possible exception are Metaverse platforms with a heavy emphasis on personal expression. A key reason for VRChat’s solid growth as the most-used VR app is that many members of its community turn their real expressions and body gestures into an emotionally engaging part of the world, as avatar-based entertainers -- as above, originally blogged here. But actual usage seems mainly relegated to dedicated content creators like them, while VRChat itself is still a relatively small virtual world.
VR hardware developers may be missing cultural issues that actually hurt mass growth:
For the moment I remain convinced, as veteran game designer Warren Spector first brought up to me years ago (when many were utterly convinced VR would take off), that the real barriers to VR growth are social and cultural, and not technological. I don’t just mean from a consumer standpoint, but among the hardware developers themselves.
Years ago, for instance, pioneering technologist dana boyd published a report suggesting that virtual reality-based nausea may be attributable to biological sex. I.E., VR tends to make females in particula, well, puke. More recent studies seem to confirm this. Something seems to be at work other than low refresh rate.
However, when I’ve brought this up to members of the VR industry -- well, to other men in the VR industry -- the concern seems for their perspective to be irrelevant. Most notably, for instance:
I sent a quick e-mail to someone far more qualified than me to talk about that topic: John Carmack, CTO of Oculus VR and an evangelist of virtual reality for some twenty years:
"Since several studies suggest women are much more likely to get sick using VR than men," I asked him, "has Facebook or Oculus ever extensively studied VR and nausea by gender?"
"I’m not involved with any of our user studies," Carmack told me, "so I don’t have any insight there." He pointed me to Oculus' media contact, who I'm following up with now.
Narrator’s voice: Oculus’ media contact did not reply to the follow-up.
My initial take to Part II, which may very well evolve as we move into the rest of the primer (and I get a chance to chat with Matthew)!
Next week (hopefully): Blogging Part 3, Networking and the Metaverse.
Good, good. Although I think there's too much emphasis on VR. VR while maybe more true to a fictional representation of a metaverse isn't something that is practical for immersion.
VR requires a disconnect from reality, which while great in fiction doesn't help to integrate a metaverse into your life. I think what people want in a metaverse (if they even know what they want) is integration, not separation and VR doesn't fit that bill. You need to completely halt your immersion for the smallest of RL tasks, rather than put it on hold mentally.
Where VR might shine is mobile (I know that's a long way off), because mobile is the least immersive experience so it can equalize mobile with a higher immersion experience of sitting at a desktop.
Our limitations are graphics cards and computing power, size of screens and types of display screens. I'm excited about Unreal Engine 5 with Nanite, which helps us down the road of being able to render more rich environments without the cost of so many draw calls, as well as world partitioning for larger worlds, etc.
Looking forward to see the rest of the articles for this primer and the upcoming interview with Ball. Feels like we might be moving forward finally outside of these walled gardens like SL and friends.
Posted by: Kyz | Wednesday, July 21, 2021 at 10:48 AM
We are actually using Unreal Engine with the Live Link app+Video AI motion capture for an upcoming project. This technology is simply magical.
Posted by: Zatch Ixchel | Wednesday, July 21, 2021 at 11:28 AM
@Zatch: Sounds pretty cool.
Side note, this from an announcement today from Epic's new acquisition of Sketchfab:
"By joining forces, Epic and Sketchfab will be able to make 3D, AR and VR content more accessible and grow the creator ecosystem, which are critical to an open and interconnected Metaverse."
Epic is definitely looking at the Metaverse as a thing and they get that it's more than just them.
Posted by: Kyz | Wednesday, July 21, 2021 at 01:39 PM
Yes and on that point about SketchFab …some context for where it might fit into the Metaverse:
https://outofscope.bureauofbrightideas.com/sketchfab-epic-and-the-metaverse/
Posted by: Dusan Writer | Wednesday, July 21, 2021 at 03:40 PM
@Dusan Thanks, great read and I think spot on, on all points. I also hadn't thought of Sketchfab as a global distribution for all things Metaverse and the ties to photogrammetry in the scheme of things.
Posted by: Kyz | Wednesday, July 21, 2021 at 06:15 PM