Kinect Integrated Into Snowstorm Second Life Viewer by Merov Linden (Who Also Worked on a Kinect Predecessor)!

Another metaverse milestone: Kinect interactivity has been integrated into Snowstorm, Linden Lab's open source Second Life viewer, by engineer Philippe Bossut, known in SL as Merov Linden. Watch:

As it happens, this is actually Kinect returning to its roots, so to speak. Back in 2008, Bossut was working with Linden board member Mitch Kapor to create a hands-free 3D camera interace for Second Life. "It compares pretty well since it's the same code really," as Bossut explains to me. "The Kinect hardware is provided by Primesense, and Mitch and me did work with a Primesense camera at the time." As Kapor told me then, this was part of his plan to help make Second Life mass market, but the project was discontinued due to the difficulty of mass producing the 3D camera hardware. With Kinect very much mass market, however, it may be that Microsoft has solved that problem for them. And us.

More from Bossut after the break, on making this code open source, and the fascinating challenges of integrating it into Second Life usage:

"We don't have a clear plan right now as to when we'll put that code out in an open source repo on bitbucket but we will eventually," Merov tells me. He's not sure if this Kinect code will ever be part of the official Second Life viewer: "For the moment, it's at the level of experimental hacks as in, 'can we do something fun and usable with that thing in SL?'" As for the actual code, "Most of my work for the demo you see today was to get the libfreenect (openkinect) library, derive my camera class to support that new camera, implement a simple tracking system on the depth field, and port my viewer 1.18 code to the 2.6 Snowstorm viewer code base."

In Merov's demo, simple avatar actions are triggered by specific physical pantomimes, so the avatar isn't really replicating the user's movements in real time. I asked him how difficult it would be to directly control a Second Life avatar through Kinect's motion capture camera.

"That's a whole different level of complexity as you can imagine. As you know, we had code for puppeteering client side a few years ago, but we never implemented the protocol to broadcast live joint positions from clients to other clients through the server.

"Beyond that, I am not personally convinced it'll provide the best animation experience. One can fall into the Uncanny Valley in no time with something like this. I'd rather first enrich the number of available animations and use a gesture-triggering system as we have today, except it'll be triggered by camera capture, and we'll be transparent to a user who so wishes to animate his/her avatar. You could dance and see you avatar dance without it dancing awkwardly. In my opinion, that will give a much more natural and pleasing experience."

And now with Microsoft releasing its own SDK for Kinect, we're likely to see many more SL viewer developers experimenting in Merov's motion-tracked footsteps.

Hat tip: Opensource Obscure.

Comments

[Toxic Menges]

Now we are cooking with gas - this is where things get very interesting! Congrats on doing this!

[Samm Florian]

I'd get so worn out if I had to keep dancing for as long as my avi does! I'm not sure I'd want a Kinect interface by itself, but combine it with monitors built into glasses (whatever you'd call them) and that could be really awesome.

[Ann Otoole InSL]

I think I saw a statue of Merov under construction in Zindra. :)

Where does this guy find the time lol?

[Bryon Ruxton]

“Hands free” for avatar motion as demonstrated (standing up in front of your laptop or desktop monitor) is DEFINITELY not a viable direction for SL as a whole. “Hand Free 3D” is the wrong term to use for virtual worlds; it’s misleading in my opinion...

Kinect is great for games and a light/simple touch interface like the Xbox. But SL and virtual worlds are much more complex to handle with the body alone. For one, the body need to always face the screen, and in most case sitting down is favorable. Therefore body joins can’t match that of one’s avatar really. Avatar puppeteering is NOT the way to go either in that sense... It would be somewhat contradictory. The body needs to be combined with traditional controls like perhaps the Wii controllers. So it’s not hand free and never will, until we have goggles on our heads with mind control for the interface.

The important factor for SL is how to enhance and facilitate expression of real-time human emotion (via animations) while trying to finding a good alternative to joysticks or arrow keys to move around, “point at”, and for mouselook mode.

Like Philippe/Merov suggests a gesture-triggering system by camera capture of the upper torso and head while sitting down for animations is the best way forward in my view. And combined with a traditional controller for in-world motion and camera look-at/point-at, like this: http://www.youtube.com/watch?v=as4SlcAExwc

If animations, and perhaps some navigation, can be triggered by motion of the upper torso and head, that would be a good first step. Some sort of Wii controller is needed to handle the rest.

I think the difficult part, is to assess whether the controller or the body should be the one used to move around (for turning left and right, walking or flying). It seems to me that the torso is best to assume that role for gaming (inSL), while a controller is best for the normal casual SL experience, unless we use our feet somewhere…

In the not-so-near future, if the few essential face emotions that are needed to convey universal feelings can be tied to animations with face recognition that would be fantastic. But let’s not fall into the “Uncanny Valley”, to paraphrase. LL has gone there too often already.

Let’s assess the technology available, and how to best apply it to SL to actually enhance the experience on its own merits, instead of trying to copy others. Second Life is not Facebook, it’s not Xbox either, it’s Second Life!!! Comparing it or trying to flat replicate others platforms, is only going to alienate those who liked SL for what it was in the first place.

As Pathfinder well said, "Virtual Worlds are like participatory theater". "The trick, of course, is for someone to make participatory theater a lot easier to jump into."

The tricks is NOT, however, to promote SL as the movie that it it not, or trying to make it so, for all the wrong reasons.

[Dandilyon Jinx]

Amazing what is going on with sl. You can already do so much and the platform is an amazing opportunity to experiment. I can see the day coming when the regular world and the virtual one melds together seamlessly and mutually beneficial ^.^

[Hitomi Tiponi]

All cred to Merov - he is a real star. He will know what is best to do with it - let's hope that LL management listen to him.

[Michael Sitarzewski]

Now all we need is 3D VR goggles, and we're off to the races. Nice work!

[Arcadia Codesmith]

I don't see this becoming the dominant interface option any time soon, so the chair potatos can relax.

On the other hand, having the option to stand up and dance, as opposed to sitting and watching your avatar dance, provides an interesting fitness opportunity for people like me who are utterly bored with traditional exercise options.

[Graham Mills]

I'm more interested in using the Kinect for prim manipulation. Does that work too?

[Ciaran Laval]

This is certainly a good development, more options are good. Whatever happened to Mitch Kapor's developments about controlling avatars?

[Donald Schwartz]

I never understood why the earlier Kapor effort was abandoned. This could be a significant game changer for SL if Linden Labs acts quickly and incorporates into the official viewer.

Albeit, I am not optimistic Linden Labs will act with sufficient dispatch. Reminds me of Compuserve & Mosaic.

The possibilities for gamers, eds, Machinima...