Mixed Reality Project Uses Multiple Kinects to Convert People Into Volumetric Avatars and "Teleport" Them into a Virtual World

JMoF2024で「3D動画技術でリアルのきぐるみをVRの世界とつなぐ企画」 きぐるMixedReality やったよ！！
今までにない試みでいろんな試行錯誤があったけど、VR世界からも一緒に盛り上がれて最高だった！！

来て遊んでくれた方、本当にありがとうございました！！！！！#JMoF #VR_JMoF #Resonite pic.twitter.com/NZpRaZBcDU

— ととがんま (@totegamma) January 10, 2024

This demo video is somewhat confusing because just about everyone in it is a furry and it's from a Japanese-language site, but it's a mixed reality project that uses a battery of Kinect motion detectors to convert a real person into a volumetric avatar and then "teleport" them into a virtual world. (Resonite, in this case.)

Called Kiguru MixedReality. here's some (hopefully accurate) explanation translated by DeepL:

Kiguru MixedReality is a 3D video calling project that displays a 3D Kigurumi [i.e. a pajama cosplay furry] in the VR world by real-time distribution of depth-captured camera images obtained from three AzureKinect units.

This allows users to view the kigurumi from the VR world, greet them, communicate with them over the air, and take pictures with the VR avatar!

In recent years, with the growth of technologies such as Generative AI, more and more people are developing technologies to capture and play back higher definition volumetric video, but viewing them inevitably requires specialized software. The good thing about Spatial Clip is that, although the quality is not that high, it can be experienced immediately with Resonite, software that many people already have installed and use on their PCs.

Spatial Clip uses Azure Kinect, which is also used by TFPortal to capture video. This is a fairly accurate depth camera that can get video with depth.

However, with only one camera, only a small part of the image is captured. Therefore, Kiguru MixedReality uses three Kinect cameras and combines the images from each camera.

Translated with DeepL.com (free version)

Since it does require three Kinect devices, it's not necessarily something we'll see for general consumer use, but it looks pretty fun (as the video suggests) for people at real world conferences who want to "visit" a virtual world.

Hat tip: Sanny!