How That Amazing Unity Demo Dissolving Reality Into Pixels Was Made

Last week's post on Ruben Fro's amazing "melting reality" Unity demo dazzled many. Fortunately, he recently shared with me some of his creative process that made it possible:

"So, I’m using 360 cameras to record streets and daily life, then I use a series of scripts to analyze all the frames, choose the best ones and do further optimizations before sending everything to the photogrammetry pipeline that recreates a point cloud model. After that I’m running it in Unity using a geometry shader I developed to render point clouds in a more cinematic way."

The effect isn't merely meant to be visually impressive, but also evocative of how we experience reality:

Hanoi 2 - The two men were discussing over large bags of clothes. The quality was quite good, but there was still haggling to be done. #Photogrammetry capture running real-time in #Unity3D.
Vimeo: https://t.co/BZxR48R5SX #vfx #cgi #madewithunity #shaderlab #hanoi #vietnam pic.twitter.com/cdVK9jY8Gv

— RubenFro (@Ruben_Fro) January 17, 2020

"I’m really fascinated by the creative uses of volumetric capture, especially its potential to be used as a new medium to document reality," as Ruben puts it. "We had photos and videos to record our history, but now in the era of AR/VR, we have access to extra dimensions, and I think we’ll soon be able to easily capture locations, events, people in 3D.

Here's the source #madeinreality :) pic.twitter.com/oLL5nxwgVZ

— RubenFro (@Ruben_Fro) January 17, 2020

"In the visualizations with the shader I developed I embrace the fuzzy, imprecise nature of raw point data instead of trying to get close to photorealism. I find this more authentic to reality, especially when capturing people on the streets... I play a lot with the concept of fading memories too in the visual. Something so close you can almost touch but instantly disintegrate as you get close. Or deconstructing, taking apart scenes to then go back to their original state."

As you might have guessed, he's planning to incorporate his approach into a full-fledged VR app:

"My idea was to create a narrative based experience, letting people also wander in and interact with the actual real world scans. Possibly something like this could be used to 'revive' old memories (maybe adding 4D volumetric capture), or allow people to explore and experience places they, for any reasons, can’t reach in real life."

Follow him on Twitter for updates.

Comments

[TonyVT Skarredghost]

Wow, this experience is incredibly cool! :O