New World Notes

My Take On Matthew Ball’s Metaverse Primer, Part 3: Networking, Streaming, and the Einstein Barrier

Above: One of these pics is reality, the other is Microsoft Flight Simulator -- but which? 

I'm blogging Matthew Ball’s must-read, nine part metaverse primer this Summer; my take on Part 1 is here, and my coverage of Part 2 is here.

Part 3 of Matthew's Metaverse Primer, Networking and the Metaverse, touches on similar territory as the paper I wrote for Samsung Next last year,  but with a big difference. (More on that below.) This section explores all the networking technology needed to make highly immersive, highly multiplayer applications possible. As an example of that, Matthew touches on how Microsoft's Flight Simulator (current version) uses streaming to display a mirror world:

Microsoft Flight Simulator is the most realistic and expansive consumer simulation in history. It includes 2 trillion individually rendered trees, 1.5 billion buildings and nearly every road, mountain, city and airport globally… all of which look like the ‘real thing’, because they’re based on high-quality scans of the real thing. But to pull this off, Microsoft Flight Simulator requires over 2.5 petabytes of data — or 2,500,000 gigabytes... Microsoft Flight Simulator works by storing a core amount of data on your local device (which also runs the game, like any console game and unlike cloud-based game-streaming services like Stadia). But when users are online, Microsoft then streams immense volumes of data to the local player’s device on an as-needed basis.

This becomes even more complicated when other users, not to mention new content they create, are part of your virtual world. But while broadband penetration and bandwidth speeds continue to increase, any multi-user Metaverse worth the name will run up against the simple fact that our broadband can't ever run faster than the speed of light:

[W]hile the Metaverse isn’t a fast-twitch AAA game, its social nature and desired importance means it will require low latency. Slight facial movements are incredibly important to human conversation — and we’re incredibly sensitive to slight mistakes and synchronization issues (hence the uncanny valley problem in CGI)...

Unfortunately, latency is the hardest and slowest to fix of all network attributes. Part of the issue stems from, as mentioned above, how few services and applications need ultra-low latency delivery. This constrains the business case for any network operator or latency-focused content-delivery network (CDN) — and the business case here is already challenged and in contention with the fundamental laws of physics.

At 11,000–12,500km, it takes light 40–45ms to travel from NYC to Tokyo or Mumbai. This meets all low-latency thresholds. Yet while most of the internet backbone is fiber optics, fiber-optic cable falls ~30% short of the speed of light as it’s rarely in a vacuum (+ loss is typically 3.5 dB/km)... Furthermore, network congestion can result in traffic being routed even less directly in order to ensure reliable and continuous delivery, rather than minimizing latency. This is why average latency from NYC to Tokyo is over 3× the time it takes light to travel between the two cities, and 4–5× from NYC to Mumbai.

Emphasis mine. Or to put it another way: The metaverse of our dreams will always be hobbled by the speed of light.

In fact, the way Matthew lays out all the bandwidth and latency hurdles facing metaverse applications leaves me significantly less confident than I was while writing the Samsung Next article. Which takes me to my own thoughts on Part 3:

Nanite for Unreal 5 May Be Key to Making the Metaverse -- VR, Not So Much

Great comment from reader "Kyz", reflecting on my analysis of Matthew Ball's metaverse primer thus far:

Our limitations are graphics cards and computing power, size of screens and types of display screens. I'm excited about Unreal Engine 5 with Nanite, which helps us down the road of being able to render more rich environments without the cost of so many draw calls, as well as world partitioning for larger worlds, etc.

Watch above to share Kyz's excitement! However, Kyz also adds, in a possible demur to Matthew, VR should not be the metaverse's focus:

Facebook Moving Oculus to Support Industry Standard OpenXR

I usually blog any Facebook/Oculus news with this irresistible pic of Zuckerberg striding like a dictator through the opiated masses in VR, but I'll give him some credit in this case:

My Take on Matthew Ball’s Metaverse Primer, Part 2: Hardware, Mirror Worlds, and Mainstream Adoption

Matthew Ball’s epic, nine part metaverse primer is a must-read for anyone who wants to understand the technological components to make the Metaverse possible. Former global Head of Strategy at Amazon Studios, the widely-sought analyst and VC is most recently the founder of the Metaverse investment fund, the first of its kind.

I’m planning to interview Matthew later this Fall; so over this Summer, I’ll be blogging each part, highlighting the key themes that stand out most to me, then following up with commentary of my own. 

My take on Part 1 is here; today I delve into Part 2: Hardware and the Metaverse:

Advances in image capture technology are making mirror world concepts feasible:

Long sought by Metaverse advocates and investors -- over a decade ago, founding Linden Lab investor Mitch Kapor mentioned it to me as a major necessity to drive adoption -- the rise of smartphones and their component technology has made it possible to convert our real world into the virtual one. Most crucial of all, perhaps: Our very selves, into our avatars.

You can see how far we’ve come in this demo (watch above) of Epic’s Live Link Face app, which uses the latest smartphone-based facial capture to map a user’s expression onto high fidelity avatars. 

Similar capture technology is also being used to convert the entire real world into high fidelity 3D models that can be shared in the Metaverse. (See this iPhone-driven mirror world demo I wrote about back in 2019.) The technology has vastly evolved since then, as Matthew writes:

Apple’s Object Capture, meanwhile, enables users to create high-fidelity virtual objects using photos from their standard-issue iPhone in a matter of minutes. That all of this is possible through standard consumer-grade hardware is astonishing.  

Underlying technology like this, he goes on, can power some unexpected use cases far beyond enabling your Fortnite character to share your expressions:

In the future, this sort of tracking system will be used to reproduce these users, in real-time, in a virtual mirrorworld. Technologies such as Google’s Starline will then allow remote workers to be ‘present’ in the store (or a museum, or a DMV, or a casino) from a sort of offshore ‘Metaverse call center’ — or perhaps at home in front of their iPhones.

Hardware advances may help drive consumer adoption:

Motion sickness is often fingered as a barrier to mainstream adoption of VR, a key access point for the Metaverse. Matthew notes that recent improvements may soon address those:

The Oculus Rift also peaked at a 72hz refresh rate, while the most recent edition achieves 90hz, and up to 120hz when connected to a gaming PC via Oculus Link. Many believe 120hz is the minimum threshold for avoiding disorientation and nausea in some users. And ideally this would be achieved without needing a gaming-level PC and tether.

My own thoughts on Part 2:

30 Years Later, "The Metaverse" is Still Struggling to Find Its Elevator Pitch

I started reading this recent New York Times feature on the Metaverse with a rising sense of "Looks like we finally made the mainstream!" triumph... which rapidly decayed into frustration. Because the overall editorial tone struck me as: "We're not quite sure what the Metaverse is but some people sure are talking about it a lot, so I guess that's something?" For instance:

In fiction, a utopian metaverse may be portrayed as a new frontier where social norms and value systems can be written anew, freed from cultural and economic sclerosis. But more often metaverses are a bit dystopian — virtual refuges from a fallen world. As a buzzword, the metaverse refers to a variety of virtual experiences, environments and assets that gained momentum during the online-everything shift of the pandemic. Together, these new technologies hint at what the internet will become next.

While there is a lot of corporate interest in the metaverse, skeptics abound.

Strauss Zelnick, the chief executive of the game publisher Take-Two, said in a May earnings call that he was “allergic to buzzwords,” and suggested the metaverse could be all hype. “If you take metaverse, SPAC and cryptocurrency, in five years, will any of this matter? I’m not sure it will,” he said. Then there are those who wonder whether interest from the tech sector is simply opportunistic, or missing the point entirely.

Unsurprisingly, no one in the Metaverse business interviewed for the Times article is able to cleanly articulate a concept of the Metaverse that explains what it's for, and why it's better than the platforms we already have now. 

Having pitched the concept of the Metaverse in various contexts to many different kinds of people, I understand the suspicion that it's only a buzzword, liable to crash and burn when a skeptic's follow-up question is, "OK, but why should I care?"

In fact, I've learned over many years to only mention the word "Metaverse" to people (almost entirely in the tech industry) who are likely to know and love Snow Crash or Ready Player One. I had hoped the recent Spielberg adaptation of the latter novel would help mainstream the concept, but it was a modest hit at best. And in either novels' case, the concept still only makes sense and seems compelling within their dystopian context: The real world is so horrible in the future, most everyone spends life in the Metaverse.

Or to put it in contemporary terms: The Metaverse as an escape from the real world seemed pretty compelling during a global pandemic, but now as Summer vacation trips are overbooked (my home town in Hawaii is currently teeming with ransacking tourists), the power of that use case rapidly erodes.

My own solution to the elevator pitch problem? I'd rather be non-utopian about it and reference something that is working now. For instance:

Watch: Talking the Metaverse & Demoing the Breakroom to Brian Solis

Starting at around 27:20, going into a demo of Breakroom at about 46:19, futurist and digital marketing guru Brian Solis chatted earlier today about the metaverse, virtual worlds, and related topics with Sine Wave Entertainment CEO Rohan Freeman and myself:

With Amazon's Lumberyard Going Open Source, The Time is Now for a Metaverse Framework

I was right about to blog a good reader comment on my post about Matthew Ball's new metaverse framework for succeeding the mobile web, when news dropped about Amazon Lumberyard (previously on NWN here) which enforced its points even further:

Amazon is contributing its Lumberyard game engine to open source, and it will be known as the Open 3D Engine.

The Linux Foundation will oversee the project and form the Open 3D Foundation to accelerate collaboration with game developers to enhance the triple-A game engine. This shift could bode well for future projects like an open metaverse, the universe of virtual worlds that are all interconnected, such as in novels such as Snow Crash and Ready Player One.

More on O3DE here. Reader "Kyz" writes about how platforms like these press the need for Matthew's big picture perspective:

Matthew Ball's Metaverse Framework for Succeeding the Mobile Internet

Strategist/investor Matthew Ball has a must-read primer to a new understanding of the Metaverse which is rather mind-melting, because it compares its coming emergence to the industrialization of electricity over a century ago. From my current perspective, I see many competing platforms we casually call Metaverses (ROBLOX, Fortnite, etc), and assume that several of them will evolve and mature over time, replacing and/or subsuming 3D gaming delivery platforms like Steam, while absorbing more and more non-gaming applications beyond entertainment.

Matthew's fairly brilliant vision, however, is much broader:

Tomorrow: Join Philip Rosedale, Adam Frisby & Me for a Fireside Chat About the Metaverse & Imagination Inside the Metaverse of Sinespace's Breakroom

Click here to RSVP for a literal fireside chat with Philip Rosedale and Adam Frisby tomorrow at Noon PT inside Breakroom from Sinespace (a proud sponsoring partner to this blog), which I am thrilled to moderate. (And yes, that's Philip's classic SL avatar, updated for today with a glittering codpiece.) Event info:

Since its inception in the early 90s, the Metaverse has inspired our wildest imagination -- even if the technology to fully realize that dream has seemed just out of our grasp. But now, with advances in edge computing, VR, and 3D graphics/audio, nothing seems impossible. Given such powerful tools at our fingertips - what could this look like?

The fireside will also showcase Philip's High Fidelity spatial audio, which was just integrated into Breakroom, while the world itself will be streamed onto Chrome browsers (no install needed). If you can't make it, no worries: The Breakroom team is recording the event and will be broadcasting in coming days.

NeosVR Can Now Import Minecraft Worlds (Though I'm Not Sure How)

Hopefully I'll hear back soon from Frooxius, lead developer of the crowd-funded social VR project NeosVR, but apparently, he's figured out how to reverse engineer a version of Minecraft and import it into his world. (And apparently did so as a three day project.) My go-to VR expert Adeon speculates that Frooxius may be parsing a Minecraft save file and making a voxel renderer to interpret it internally, separate from Minecraft. (Or something like that!)

While it only has a few hundred daily users (at least on Steam), this is yet another example of how NeosVR seems to be on the vanguard of innovation in social VR. More example below: