New World Newsfeed: Microsoft's Natal Will Help Drive Adoption of Hands Free Virtual Worlds (Updated)

BBC: Microsoft unveils new controller

Post bumped up to highlight the many updates below, including comments from Mitch Kapor and Philip Linden - Hamlet Au

Yesterday at E3, Microsoft unveiled Natal, a 3D interaction system for the Xbox 360 which purports to recognize full-body gestures (including facial expressions), and voice recognition. This immediately reminded me of Handsfree 3D, a similar system under development with backing from original Second Life investor (and Linden board member) Mitch Kapor. [Update: It's using the same hardware system, which Microsoft purchased last year, see below.]

Last year, Kapor unveiled the system, and told Tara5 Oh of UgoTrade that they expected to have a version available to Second Life users this holiday season. (I'm double checking this now.) [See Update below] If it works as well as promised, Natal should propel growth of the 360 system, and just as important, create mass market awareness of and demand for gesture-driven interaction in games and other 3D mediums, like Second Life. I'm reminded of an interview I had with Kapor a few months ago, when I pointed out that SL's complicated and confusing user interface was a huge impediment to mainstream adoption.

Rather than note the Lindens' current efforts to improve the UI, Mitch replied to that point with a more long-term prediction: in the future of virtual worlds like Second Life, he told me, "The interaction metaphor is not going to be a mouse and keyboard." That was an interesting point then, and with Microsoft's new product, seems even more plausible now. But it would not come without cost: a Second Life where voice-based communication and gesture-driven interaction is the standard way we interact will be quite a different Second Life than what we're familiar with now. (Article link courtesy Tonjampae Amat.)

Update, 12:15pm: Just heard back from Mitch Kapor, who says Handsfree 3D has been discontinued: "We closed down the Handsfree project as it became apparent 3D cameras for PCs (as opposed to gaming consoles) weren't coming to market soon enough to justify continuing to fund Handsfree," he tells me by email. However, of Natal, he says: "[I]t looks like an endorsement of the 3D camera approach, and if so, I like that."

Update 2, 1:15pm: Philippe Bossut, who headed the Handsfree 3D project, has since become a Linden developer (known in-world as Merov Linden.) He points to this Gesture Recognition project on the SL developer's wiki, which aims to incorporate similar technology into PC webcams. He points out that Natal is using a 3DV camera (the same one he used to develop Handsfree 3D), which Microsoft acquired in 2008. More here.

Update 2, 3:35pm: Philip Linden emails me to say, "[W]e're actively looking right now at how to use commodity (2D) cameras to make the SL avatar move and gesture naturally. We think (and both Merov and LL have worked with these 3D cameras since they first were in prototype) that the 3D cameras are incredibly cool and seem very likely to be an important part of the future of computing interfaces, but we also think that a large set of amazing improvements can be achieved using the normal 2D cameras that are now on tens of millions of machines."

Comments

[Echelon]

While I have to admit the technology holds a lot of promise, that demo video was so painfully awkward I felt second-hand embarrasment for the team who put it together. I know they were going for natural-feeling "real life usage examples" but it just came across as horribly contrived. It's going to be interesting to see how this tech evolves, considering there's already a lot of the components for these kinds of systems already freely available to developers.

[Ordinal Malaprop]

Facial expression is one potentially interesting thing but I really can't see how full-body gesture recognition is important for virtual worlds, particularly at the moment. If it was required to stand up and move about for the entire time of being in SL, the group of active builders and residents would be very small. (They would however be very fit.)

For that matter, even knowing that a camera is scanning one's appearance constantly is quite exhausting, like the difference between being in a video call and being on audio; it means one has to be aware of one extra issue.

Spielberg's comment that

> "the main barrier stopping people getting into video games was the complexity of a games controller,"

is of course utter nonsense; quite apart from the fact that, er, people _have_ gotten into video games very extensively, the controller is not the issue, rather what one has to do with it. That is like saying that he main barrier stopping people getting into C++ development is the complexity of the keyboard.

[Hamlet Au]

@Ordinal: I didn't see the full context of Spielberg's comments, but it's definitely fair to say the emergence of intuitive, non-joystick interfaces have greatly expanded the audience for games. Consoles were once almost the exclusive domain of young males, but they started appealing much more to older men/women and girls with the Wii, the Wii Balance Board, Rock Band, SingStar Karaoke, DDR, etc. etc.

[Ordinal Malaprop]

Hamlet: Actually, I don't think it is to do with the interfaces at all, rather the general topics. Nintendo has always had quite high general appeal across gender and age, and is also relatively adventurous in terms of interfaces, but the fact that, say, the DS has a far wider player base than the PSP is not I would say to do with it being touchscreen, more that it doesn't rely on racing and shooting games, which are extremely old hat and have a poor reputation.

I will grant that innovation in control mechanisms also encourages innovative game design to an extent, but there is nothing particularly innovative about the controls of many of the most popular games.

[Hamlet Au]

It's true Nintendo has always been the more family friendly console, Ordinal, but the addition of intuitive interfaces is definitely a factor in growing the gaming market. Wii Fit is now vastly outselling GTA IV, for example, and the main buyers are women in their 30s. The appeal of Wii Fit is almost entirely the interface, i.e. the balance board.

[Ordinal Malaprop]

It could also be because GTA IV is a competent but tired addition to a fairly predictable series.

And in any case - I do not think that the interface is the major factor in the appeal of something like Wii Fit (even if it is central to how the game _works_), more the subject matter i.e. "you can play games and it makes you fit too". See also the various "Brain Trainer" type games. If all that the Wii interface was used for was bog-standard fighting games, say, it would not make much difference at all.

These days the whole family is familiar with keyboards and mice and so on and this has been the case for a while. The Sims, for instance, in all manifestations has been an _immense_ seller ever since it was released.

[Hamlet Au]

"you can play games and it makes you fit too"

That's not how Nintendo promoted Wii Fit, though, they promoted it as a full body exercise device and emphasized the yoga and aerobics training as much as the mini games, if not more so, even hiring fitness celebrities etc. to demonstrate and push it. Now that it's sold so well, they're emphasizing games developed to play with the balance board, for snowboarding, etc.

[Arcadia Codesmith]

For some of us, the keyboard is so much a part of our daily lives that we're no longer consciously aware of it... we basically think words and they appear on the screen with no real awareness of the intricate finger motions required to produce them.

We have to keep in mind that many people, perhaps most, never achieve this level of fluency. For them, the keyboard is an omnipresent barrier between them and the virtual world. Remove that barrier and the world becomes much more accessible.

That said, I doubt that voice and gesture will replace keyboard and mouse anytime soon. They might be a useful and interesting alternative, and they have the potential to expand participation, but there is much that is achievable with the written word and the (relatively) fine motor control of a mouse that you just can't accomplish with a microphone and a cam.

[Ordinal Malaprop]

I don't know anybody who thought it was a serious yoga/aerobics training device. I do however know lots of people who thought "this isn't just some silly computer game, it's good for me too, therefore it is better than silly computer games". Hiring celebrities is just part of promoting that side of it.

Wii Fit is not a very good example though because the content is pretty inseperable from the control mechanism. I could say that it was the content, and you could say that it was the control mechanism, and neither of us would really have any other examples to make the point with. I can say that DDR-type games are popular regardless of whether they involve actual dancing or tapping buttons on an iPhone or DS, but there isn't anything similar to Wii Fit with a different control mechanism that I'm aware of.

[Ann Otoole]

This technology will make a difference when the UI controls are projected as a hologram your hands directly interact with. Nice to see some movement but the 3D world is still not really 3D yet is it? I mean if it is projected onto a flat surface (monitor screen) then how can people say it is really 3D?

[Connie Sec]

Sorry..I think this is just another side road on the way to what I feel is the real deal. A non invasive brain-computer interface (BCI)using EEG's and neural nets to move the learning stage away from the human. Simple BCI's are already being developed for the handicapped with neuroprosthetics applications that aim at restoring damaged hearing, sight and movement. To think and go is the way. All other interfaces pall beside that when it comes to controlling an avatar's movements in a virtual space.