Grow the Metaverse By Improving Body Language and Facial Recognition Technology, Says Ex-Linden John Lester

While we often talk about how the Metaverse of Snowcrash influenced Second Life, ex-Linden Lab staffer John Lester (known then as Pathfinder Linden) points out that in the novel, the Metaverse didn't grow because you could have cool sword fights or ride motorcycles in it. Instead, notes Pathfinder, it was something seemingly more prosaic:

What made the Metaverse in Snow Crash insanely successful was technology that could read Real Life facial expressions and body language. Then the information was instantaneously and realistically reflected on avatars. Where other people could see and understand.

Trouble is, too many developers are not focused on this aspect of the Metaverse, and by extension, nor are they on that aspect of Second Life. Rather:

In my experience, most programmers and developers of virtual worlds tend to ignore “soft” scientific concepts like empathy, emotions and the squishy human nuances of interpersonal communication. They primarily want to build tools that allow people to create and exchange things.

Instead, he points to Microsoft's Kinetc project (formerly known as Natal) and the work of Anton Bogdanovych (who I wrote about here, video embedded below) as pointing the way to the future.

I agree with my pal John that people and how they interact virtually is way more important than what they do with their stuff. So keep following the path of Pathfinder's thinking here.

Comments

[jjccc]

very cool i want one

[Melissa Yeuxdoux]

Not to mention that Deaf SL residents should be able to sign to one another!

[John "Pathfinder" Lester]

Thanks Hamlet! :D

And Melissa, that's a very important point. There are so many deaf people in SL, and sign language is such a subtle and beautiful language. It would be amazing to be able to accurately replicate it on avatars.

At the same time, I know there are a lot of blind people who use SL. While avatar facial expressions and body language would not be visible to them, I can imagine a system that could remap those signals and give useful audio cues. An avatar frowns, and perhaps you hear a faint sad melody?

[Medhue]

Although it is cool stuff. There are all kinds of issues with it.
First- sensors are old technology, and I've already seen camera motion capture for moving in SL and I use it to make mocaps.
2nd- A good amount of people don't come into SL to be themselves, they come to be creatures, monsters, furries.
3rd- Most people are multi-tasking, in some way or another.
4th- Many are handicapped and this won't help, or they won't want to use it.
5th- Personally, I don't like wearing headphones, the last thing I'm going to do is put a suit on or wire myself up.
Oh, I could name dozens more reasons why this has many, many years before any part of this will be widely used in SL. Again, camera tracking real motion will very quickly surpass sensors, just because of the freedom. We already see xbox and ps going in that direction. Like I said tho, it is cool.

[Medhue]

Oh, and the only feasible way to do real facial expressions is with a camera, which I think this will come into SL way before any kind of body movements.

[shaqq]

Camera based techniques are not quite there yet to replace the sensors, but they sure will be one day. The reason why we chose to start with the sensors - is because it's the best and most precise technology today... every sensor can be tracked in real time with millimetre precision. That's the perfect testbed for the technology we develop to show the world its possibilities, test the limitations and get the industry interested in the topic. Our work is not going to be limited to just sensors, nor will our technology only work with this particular suit. Any camera-based solution that is precise enough and is capable of sending the motion data through a socket to our software will work just fine.

I'm also fully aware that in many circumstances people don't want to be stuck with their imperfect self and will want to multitask. We know how to address these problems, but I can't reveal the details yet. Just be patient.. the answers are coming :)

[shaqq]

Also want to add that I really appreciate your interest, ideas and concerns. Will keep reading your comments and will take everything constructive on board.

[Extropia DaSilva]

There is a RL device called VOICe. This consists of cellular phone equipped with a camera capable of recording video, and software that converts visual images into 'soundscapes'.

After using this device for some time, and learning to associate different sounds with visual cues such as height and width, the wearer trains their brain to 'see' in sound. fMRI imaging shows this is happens on a physical level- well-practiced users of VOICE activate areas of the visual cortex as their brain processes information contained in the soundcapes. The brain rewires itself so that audio signals are translated into visual images (not sure if this works for people who were blind from birth).

I would imagine this system could work in SL, if there could be some way to have objects and environments emit sounds that form soundscapes the mind can be trained to interpret as cube prim, cone prim, square prim...shirts...houses...

[John "Pathfinder" Lester]

@Extropia, VOICe is a fascinating technology. For folks who want to learn more, here's a link: http://www.seeingwithsound.com/

I remember someone in SL (their name escapes me right now) who simply took the output of the SL viewer and fed it to the VOICe software where it could be interpreted. Really cool stuff.

[Renmiri]

I'm sorry but I must be the downer:

It is soo typical of a Linden to think a new "shiny" will bring all the glory and subscribers!

Look facebook has none of it and gets a lot more subscribers. WoW has none of it and gets 12 million.

SL doesn't nee more gadgets, it needs to work smoother with the gadgets it has. And it needs to be cheaper, a lot cheaper. I spend a lot less on WoW and I have 5 accounts i pay. I own a sim on SL and spend 3x of what my 5 wow accounts spend.

In my corporate job we used to call those gadgets "airplane horns". Is it useful and a good idea ? sure. Is it relevant to the plane's "mission" ? Is it worth the time and money when the plane has a bad landing gear that needs work ?

Please LL and ex-LL get into your thick head: A stable cheaper paltform will do much more to grow sl than all the bells and whistles in the world.

[Hamlet Au]

Those are all good points, Renmiri, though for myself, it's not an either-or. Definitely make the existing platform stable now, but in 2-3 years, I think it'll be important to think about how stuff that Anton Bogdanovych can become mass market. Second Life's existing platform depends on laptops to work, but laptops are becoming less and less how we experience computer-driven entertainment/communication.

[Viorel Daviau]

The thing is in 2-3 years we'll still be looking at the same exact thing, just with a little more shiny.

LL needs to improve on the basics. The very keystone of SL is the avatar. Update the mesh (which will piss off developers, but at the same time will also stir the economy like sculpt maps did for prims).

Once the avatar is updated, THEN update the rest. Seriously, we're looking at the backside of '04 right now..and it's ugly.

As far as physically impaired folks, if there is ANYTHING that would help them with SL, I say do it. From what I understand though, avatar fingers with the current mesh are grossly difficult to work with.

I'll leave that note for the experts to expand on.

[Medhue]

You know who this would probably be a big hit with? Machinima makers would love to have something like this, and also the SL theatre acting groups. I've done some works with both of them and it is just too expensive and time consuming to make mocaps for every actor and every scene.

[Arcadia Codesmith]

Real-time facial expression via webcam ought to be possible with tech similar to voice/mouth sync (granted, that system is far from perfect).

I'd say face mapping would be adopted a lot faster than full-body puppeteering. There are a number of issues with getting people out of their computer chairs and moving, not least of which is many or even most of them just don't wanna.

But I could be wrong. People loved the Wii...

[Adeon Writer]

I'm not so sure if motion capture via cameras is the future. Sensors are much more accurate and they're becoming less and less intrusive with every year. Although personally I've been wearing a headset (studio headphones with a mic piece built in, best way to avoid echos on voice) since I started in SL, so wearing gear to use SL is nothing new to me.

[Adeon Writer]

@Arcadia Even in the Metaverse, you could have an epic sword battle while sitting in your chair. Although to be sure in SnowCrash it was a bit more matrix-esque because they never went into details about HOW you controlled your avatar. It did mention it took getting used to. (Hiro noticed a newbie by the fact that they stumbled around unsure of how to walk properly.)

[Renmiri]

Motion capture via webcam will be great for the Adult sims :p

If LL wants a new shiny that will bring more subscribers, then they should make it easier to share webcam sessions in world.

I hope they don't do it though, I shudder to think of the lag it will bring for us not on the SL sex industry!

Hamlet, I agree is not an either / or thing, but in my experience LL has always been chasing the next big shiny while it's world is laggy and expensive to play, and gives nothing to content creators. We make SL what it is and for our troubles we get to pay tier. That simply isn't right.