This is an awesome hack that links Kinect to Second Life in an incredibly innovative way: Kinect reads the head movement and facial expressions of someone in real life, and those movements are translated into data which then dynamically alters a face sculpture in SL, composed of thousands of cubes. Created by an artist known in SL as Glyph Graves, he narrates this video above, featuring a friend who volunteered to demonstrate. This mixed reality sculpture has already been avidly blogged by a number of SLers starting with Chestnut Rau, but I begged Glyph to shoot this video, so we could see what was happening on the other side.
“It seemed like an obvious thing to do,” Glyph tells me, when I ask what inspired this project. “[Kinect] does face depth. I thought I could do it in SL. So I did.” The reaction in-world when he’s showed this to avatars has been pretty stellar: “Shock, amazement, some disbelief. Part of it is the potential it suggests for SL.”
To make it possible, he had to create a fairly complex interaction between Kinect and SL, read on:
“There are four programs: one on my computer, one on my hosted site, and two different ones in SL. I learned C# for this project (and sockets, about ports, etc.)” He also used Microsoft’s software development kit for Kinect, but heavily modified it so he could extract the 3D facial data and add server code.
“Microsoft released the SDK on June 17... took me until a little before the opening on July 7th, but then I had to learn C# and about server and client ports etc, and build a sim with a whole lot of other stuff as well.” (Coder Miki Gymnast helped him quite a bit in this effort.) Not bad for a evolutionary genetics scientist, which is Mr. Graves’ real life background.
The final result from an SL persective looks like this, an eerie perturbation of reality into Second Life, as if someone was pressing his face into the veil that separates both worlds:
(Video by ColeMarie Soleil for her SL arts blog)
Here’s even cooler news: He’ll open source the code that makes this Kinect hack possible, so others can do projects like this, and perhaps others even more ambitious: “Soon,” Glyph Graves tells me. “I want to finish a few more projects with it first... it was a lot of sleep lost making this, a large number of man hours. So I’ll do the projects I intend to do first.”
To get announcements of future demos, join the in-world group ArtNation. This weekend on Saturday and Sunday at 5pm and 2am Pacific (SLT), you can see Glyph demonstrate this in-world: Click here for a direct teleport.
Much thanks to Bettina Tizzy for the tip!