Watch: NEOS Social VR Adds Highly Detailed Mouth Tracking With Reported Latency Under 100 Milliseconds

Here's the latest update from NeosVR, an impressive indie social VR metaverse project backed by Patreon funders that we featured last year: Eye and lip tracking using the HTC Vive (along with HTC's Lip Tracking module). As demonstrated by lead developer "Frooxius", this is among the most detailed variation of mouth tracking I've seen in any VR app, not just tracking lip movement but tracking tongue, teeth, and inside the mouth.

In the video you definitely see an occasional delay between real life mouth movement and the avatar's mouth, but Frooxius tells me that's a feature of audio processing that wouldn't show up in the social VR world of NEOS:

"The videos aren't perfectly synchronized in all places," as he puts it. "It's also using a combination of voice analysis and lip tracking for some avatars and the voice has inherent processing delay as well, so that could be what you're seeing."

In-world, however, he says: "It's pretty low! I find it hard to estimate by eye, but I'd say less than 60-100 milliseconds. It feels nearly instant in-game." (Less than 100 milliseconds, by the way, is Philip Rosedale's benchmark for immersive audio.)

More updates soon, hopefully: Follow Frooxius for further.

Comments

[Maxwell Graf]

I have to say, out of all the things that that LL tried to accomplish with Sansar, one of the things they actually got exactly right (and even pushed the boundaries of the technology in general) was the VR voice and facial symmetry they developed, very similar to what Neos has done here.

I only got to experience it a few times in there with an actual VR rig on, but when I did this was one of the things I specifically wanted to check out, knowing how much they put into it. It was not just the mouth moving when sound was present, it adapted to variations in phonetics, volume and emphasis to change the facial mesh in a very accurate way. When people near you spoke it was not readily apparent beyond the locality and volume of the sound, but then it sortof dawns on you that the mouth is moving accordingly and you accept it without notice. Its a social behavior that easily falls in between the virtual and the human elements that suddenly come together with full VR immersion, the most interesting part of virtual worlds to me, the grey area where machine ends and humanity begins. It's where the curtains part and you see the wizard.

I mean, its one of those things that when it's done right you don't even really notice it specifically, but when you do focus on it you realize how different and advanced it is because it just feels oddly normal, realistic to the point of blending in all uncanny-valley like, even a little disconcerting. The sound team did some amazing work there to make it work. I think Draxtor helped them work some of the stuff out as well.

[Joey1058]

This is some amazing interfacing! Both simple to use, and can get as complex as needed. Very cool!

[Jason Moore]

I'm an active content developer on Neos, having first spent almost three years in High Fidelity. Neos is THE place to make content in social VR right now, hands-down. The engine is so powerful it feels like magic. Frooxius is not only a genius, he's also incredibly connected to the community, so the relationship between the makers of Neos and users of Neos is very, very close (not so on most other social vr platforms). To anyone who hasn't dropped in, you should give Neos a look.

[Theanine3D]

Too bad Neos is practically unusable on desktop. The interface is clearly VR-first at the expense of everything else. No love for desktop users, who will always outnumber VR users.

[Patrice Rey]

I am a total newbie in VR, plus I am a total newbie in PC (I have been a mac user for 30 years). Yet, bought my first pc a month ago ($2K), bought an Oculus Quest, got into NEOS and within hours I was building stuff. In my opinion, NEOS is totally usable on a desktop, but you need an appropriate graphics card. Cheers.