Designing Good Mixed Reality Social Experiences With Vision Pro's Ghostly Personas

XR developer and blogger Skarredghost has a good long look at the challenges of creating effective mixed reality social experiences with Vision Pro. The challenge is the Vision Pro doesn't use traditional virtual world avatars, but "Personas" which are ultra-realistic maps of the users real life face. As I've written, there's already challenges with real life avatars -- for many reasons, people generally prefer avatars which do not look like them in a virtual space -- and they're made even more challenging when these personas don't even have arms or legs. 

Though these ghostly personas do have some advantages which avoid the usual Uncanny Valley problem, as Skarred points out:

The Apple Vision Pro uses a trick of this kind, too, to make social MR more believable. The social personas do not have arms, do not have legs, feet, and such (strange that no one is backlashing Apple for it as they did for Meta). They float, which is unrealistic, but is also a good way to escape problems: the avatar of someone who is with you in a room may float on a chair, a sofa, or the floor… in any case, it looks ok-ish. If he’s in the walls it’s a bit stranger, but still, it’s just a ghost.

The problem with this approach is that geographically separate Vision Pro users don't inhabit the same virtual space while connecting:

As much as it is cool seeing someone appearing inside your room (I tried it and it’s great, it is like a friend of yours came to visit you), it is pretty useless, because he doesn’t see himself inside your room, but inside his room. So you have the sensation that he came to your home, but actually, he has the sensation of having stayed at his home, and this creates a critical difference in how both of you approach the experience. All of this made me realize that the naive approach of social shared mixed reality just does not work.

Mixed Reality is about me living an experience in my own place and this is the same for the other players… making this shared does not work.

Much more here. Based on his thoughts, my strong guess is that Apple will need to create to create a basic, mixed reality virtual world where multiple users can arrange simple aspects of their real world environment -- a table, chairs, maybe a small room -- where more effective social Vision Pro experiences can happen. 

Comments

[Martin K.]

> Based on his thoughts, my strong guess is that Apple will need to create to create a basic, mixed reality virtual world where multiple users can arrange simple aspects of their real world environment -- a table, chairs, maybe a small room -- where more effective social Vision Pro experiences can happen.

What each app needs to do is to establish what is known in the research on "Computer-Supported Cooperative Work" (CSCW) as a "shared workspace", e.g. a shared virtual 3D model, a shared virtual movie screen, a shared virtual projection screen for a shared slide show, a shared whiteboard, a shared virtual tabletop game, etc. It's not a new topic; some people have spent decades researching it in CSCW.

If an app allows to align the shared virtual screen with a real wall in your room or the shared virtual tabletop game with the surface of a real table in your room, then the app gets extra points for immersion, but that's not necessary. (And if the app requires this alignment, it might introduce additional friction for users.)

This scenario is what Skarredghost labels "You want to have a social AR experience about an element in particular"; according to them, it "was the theme of 80% of the social videos of the Vision Pro on X." I'm not surprised and would assume that an even higher percentage of successful MR apps will have some kind of "shared workspace" - at least that's what you would expect from CSCW research.

Maybe some words about some of the other scenario's by Skarredghost:

> You need just a bit of sociality

The question you have to answer here is: if it is like a discord call, why would users not use discord? Maybe because of convenience, but then users have to use your app for other reasons. (An example might be the multiplayer lobby of "Escape Simulator".)

> You want people to have a social MR experience at the place of a person in particular

Some people are obsessed with this scenario. Here are my issues with it:

1) If 10 people meet this way, for 9 of them it will be basically a VR experience of a virtual reconstruction of someone's room. Thus, for the majority of users it won't be mixed reality.

2) The real magic would start when you could turn your whole room into a shared workspace, i.e. you would not only share a 3D reconstruction of your room, but track the dynamic position and rotation of interesting objects in your room and you would share the dynamic contents of interesting screens in your room. (One fun and cheap way of tracking an object is to attach a VR controller to it.) If you don't do that, it's somewhat lame.

3) It's basically as lame as a shared 360 photo (or video) of your room. It's like a zoom call where someone rotates their device around to show their environment. It might be useful, but not very exciting.

[Luther Weymann]

Hehe, I told you so.

13 November, 2024
Apple is significantly scaling back production of its initially popular Vision Pro headsets and may even cease making the item by the end of the year, The Information reports.

"Right now, it's an early-adopter product. People who want to have tomorrow's technology today — that's who it's for." - CEO Tim Cook.

Tim baby, it will never be "tomorrow's technology." Come out of your little bubble and get out here in the real world with people who will never wear something on their face for hours and hours just so it makes them sick and you can make money. We are not like that, but you would never know from the bubble you live inside your ivory tower.