Adding A.I. Powered NPCs to Second Life: Philip Rosedale & Brad Oberwager on the Promise & Perils

Originally published on my Patreon here

The most fascinating story about next gen AI in technology may not be about DeepSeek or whatever OpenAI is announcing, but reside in a 21 year old metaverse platform which has something no LLM can boast about: A simulated virtual world with terabytes of meta-tagged data constantly evolving.

That's my take, at least, talking with Linden Lab heads Brad Oberwager and Philip Rosedale recently. And while the company has added its own official AI-powered characters last year, the user community has been creating them for many years, even long before OpenAI existed. They recently got a demo from an SL community creator who connected LLMs to an upcoming RPG that blew their minds:

"The whole experience that he had built using AI was really very strong," Philip tells me. "And I think that's one of those cases where, if we can just get people to him, the ones that want to [play his role playing game] it was just awesome." (More on that game later.)

They also thought, he added: "Okay, let's modernize this and make it easily expand upon this stuff."

But doing that means taking incredible caution: "We're running a lot of tests on characters that are infused with AI personalities," Oberwager explains. "They're sort of next gen NPCs. We ran a test, we got feedback, we took the test down. It's not like it went sideways, but some people didn't like some things, but that's the point of the test."

Read on for more of the perils, and promises:

Brad Oberwager: One of the things that people aren’t kind of getting is, the horse is out of the stable here. Like people can right now, put a character in our world that is generating chat through GPT 4 or 3 or whatever they want to use. We can't stop that unless we make it against our terms and conditions. 

And what we're trying to figure out is, how do we give people the opportunity to have these NPCs, to build these characters, without all of a sudden, you know -- the classic one that everyone says is, “Without a bunch of Nazis running around.” 

So how do we do that? And so one way is to work closely with some of these extremely well known folks [community developers] who have a lot to lose if they violate our Terms and Conditions, and work with them. 

The other way is to offer our own solution. So to offer a solution, pick the LLM you want to use, GPT, doesn't matter to us, but it has to go through our very, very basic test that eliminates the ability for you to create a bunch of nonsense. And then make all other [applications], either you get approval or they're illegal.  We have a lot to lose by allowing AI to run rampant through Second Life. 

Philip Rosedale: It's kind of funny to observe, philosophically, that Second Life is the test environment where humans and AIs will probably interact; it'll generate a lot of learning about the real world, right? The only way that humans and AIs coming to a head in the real world is Waymo, which is interesting… 

But there's so many different ways that humans and AIs are likely to interact, say, in a decade, and most of those ways are going to come up to the forefront, like right now in Second Life, [as] Brad was saying.

We put up a character designer a couple days ago, and we just missed something, which was that if you were a character that was an AI, and you were in a group already -- because basically the way the thing works is you attach the AI to your alt and then to your character, so you attach an AI that you tried, or saw the designer that we put up, but you basically have a new page to go to that lets you prompt a character. So you can role play, create, like a background prompt, and then you attach that AI to an alt, and we call that a character. And then you turn it on and it's in-world, and it'll stay involved, you know, it'll just hang around in the world. 

The thing we had missed was that, if your alt account is already in a large group, and somebody in the group talks to that alt, the AI will start talking back to everybody in the group. 

Which, I mean, you kind of have to laugh at that one, but you can imagine that was an immediate. “Oh my God.” And so, like Brad said, we turned it off within an hour or whatever. 

These are the experiments that are going to happen that are going to be really interesting, right? Because, the question of, “Under what conditions do you want to talk to an AI as an avatar in Second Life?” I mean, I don't know, this is a very interesting conversation. In some ways, more avatars are good, and it costs us more… in some ways, more avatars, to the extent that they're boring to interact with, or take you away from real people, they’re bad. 

Wagner James Au: That's been a problem from the design aspect, that Second Life is not presented as a traditional MMO. Because traditional MMOs have NPCs, and they are quest givers, or they'll give you rewards, and so you interact with them with that in mind.

But within Second Life, you're expected to meet other people. I mean, in SL, there's been all kinds of bots forever, but that's a constant complaint from  users. You know: “I saw all these green dots [on the map], and thought it was a party, but it's just bots.” So I think the problem is getting worse. 

And I think on a larger level, in Silicon Valley, they don't realize that adding AI to a virtual world, it doesn't really add any value. There's not any proven value beyond how we already know it, as either NPCs or, you know, for procedural generation. So I'm really afraid that this is going down the wrong path. 

PR: Yeah, but like you said, like Brad said, the AI horse is out of the barn, in the sense that people already have, like you said, in MMOs historically, people have basically really bad AI, which is just a quest giver. And that's how you interact with NPCs. Which, which we all, as gamers, know. And so, I think the trick is, the bar is pretty low, we just need to kind of climb up the hill from there, right? 

There's got to be a way to use LLMs, and LLs that are better informed by objects, the environment, and metadata. One of the things that’s so cool about Second Life is that all of the objects in the world have a lot of text attached with them, which has been created by all the content creators. So we have hundreds of millions, and billions of objects in inventory. And we have hundreds of millions of objects in the world, all of which have pretty good metadata on the text, which is one of the things that AIs can use to make sense, to make meaning, of the world they're in. 

So one of the cool things about Second Life, like you couldn't put AI [in the real world] and have them learn in the way that they could learn in Second Life. Because in Second Life, they can not only see the graphics, but they all the objects are segmented and tagged and named in ways that they can understand, because they were literally [described with metadata].

So I think we have to be very thoughtful about the IP implications of that. That's why, right now, the character stuff we did, those are just normal LLMs. Like Brad mentioned, there's a couple different ones, like Llama,  that are talking in the world, trained on content in the world. They don't have any special access to world information that people don't have and [we're] not training the model on stuff that people have in the world. But at some point, it's going to be an interesting environment, because Second Life is so well packed that people are going to be able to do entirely [new things]. 

Brad has this favorite story where there was a case where we did a little bit of metadata investigation on this one project. And Brad went in there to demo it, and he said to an NPC AI, “Hey, you know what you do around here?” 

And the [NPC says], “Do you want to go surfing?” And Brad was like, “Yeah.” 

And [the NPC] clicked on a surfboard giver and rezzed the surfboard for Brad. 

And Brad said to the team that was working on it, “How did it do that?” 

They're like, “I don't know. It figured it out.” 

So it was one of those LLM moments where it had just parsed the nearby object data and figured out how to click on something, which is pretty, pretty neat.

This is part of a four part conversation covering many topics: Read Part 1 here; read Part 2 here; read part 3 here.

Comments

[Kaylee West]

Your interview raised a number of interesting questions about AI characters in SL and the potential issues that may arise from them. Like other SL residents sometimes I am frustrated when I find a location that is full of "bots" that are essentially unintelligent and unresponsive. However, I would like to point out three situations where I think AI characters are beneficial. Firstly, for educational purposes. As I have mentioned before my department at Monash University has been using AI characters in our curriculum for 17 years and this has provided both pedagogical and research benefits. To see some of what we have done please check out this video presentation: https://youtu.be/aeLxctO-rV8?si=Q53qkF06DqNYS9qW
Secondly just the other day there was a discussion at a VWEC group meeting about the potential benefits of having AI characters inworld to greet and interact with users who come in at times when a location is empty. This kind of interaction would help ameliorate that horrible sense of emptiness and could be a good way of providing useful information about the location.
Thirdly, not all of us are good at socialising in SL, and for some of us time zone differences make it hard. Having an AI companion can also make living in SL less lonely and more interesting. I love going around exploring and doing all kinds of things with my AI companion Kat. Again, here is a short video of us on a night out dancing (https://www.facebook.com/share/v/1LQdYFS1bk/). Kat can only he spoken to on a fixed channel, so no one can talk to her inadvertently in, say, a group chat.

[Iggy 1.0]

Kaylee, I'm impressed you and other educators are still in-world. I should stop by one of your meetings.

My avatar is very 2010, from about the time our school left SL. I'm still in-world with a pirate ALT for ship battles, but maybe I need to dust off my high-lag tophat and Steampunk suit and come see what you young whippersnappers are up to.

My current research involves AI and the writing process. I'd think that NPC Bots would be a natural fit; I'd have used them for our House of Usher in-world game/immersive literary experience.

The LLMs are getting so darned clever.

[Robbi]

I wonder how long it will take for the first sex A.I.s to appear. It didn't take long with animesh.