ChatGPT might seem pretty useless for a legacy virtual world like Second Life (I certainly thought so last year), but recent updates have somewhat changed my mind. In his off-hours, Linden Lab's infosec head (known by his avatar, Soft Linden) has been playing around with OpenAI's Large Language Model, and discovered some pretty impressive Second Life-related applications.
For starters: You can use ChatGPT to help script in Linden Script Language!
"GPT-4 (paid) works reasonably well for LSL," Soft tells me. "It will hallucinate some functions that don't exist, but it can save time on the overall first pass, and it can usually correct the code when flaws are pointed out. It can find some bugs if you ask it to find the defects in code you paste in. GPT-3.5 (free usage) is weaker on code."
The older free version is the one I blogged that hallucinated LSL like crazy. But Soft has seen some improvements since then:
"OpenAI’s ChatGPT lacks up-to-the-minute information, but simulates human reasoning to a much better degree. It knows what Second Life is, and shows evidence that it’s probably scraped our knowledge base and some old third-party Second Life forums. It can debug or suggest improvements to Linden Scripting Language code." Here's an example on OpenAI.
"You can ask it to proofread notecards. Try pasting, 'This is a notecard written for non-technical Second Life users. Suggest a bulletpoint list of improvements for clarity and ease of understanding' followed by your notecard. See what happens.
"You can ask it to help you brainstorm improvements on a business plan for your store or roleplay for your community. You can even ask it how to make your store listing sound more fun. Some suggestions will be bad, but sometimes it surprises you."
He's even used ChatGPT for Firestorm, the popular third party SL viewer.
"As an experiment, I tried taking a screenshot of the items listed in Firestorm’s 'area search' floater, and asked for 'additional items that would fit a Second Life scene with these items.' It didn’t balk. It read the text out of the snapshot, made a few suggestions, and it suggested a beer cooler and some roasted marshmallow gear for an SL camping spot. Hmm!"
None of this suggests ChatGPT is great with Second Life -- it's literally coded to give you the most mediocre content on the web -- but it's worth playing around with your various SL projects:
"Find a friend or three," Soft advises, "bring up ChatGPT in a browser window, and brainstorm together. Treat it like an unbusy friend or an intern and you may have fun or learn a few things. Or, failing that, you'll better understand its limits."
Soft Linden has also been experimenting with Elon Musk's LLM Grok, and so far the results are notably less useful -- but also hilarious:
— soft 🧀 (@soft) December 19, 2023
"Grok is useful for finding tweets related to a subject, and giving an overall summary of them. Try something like 'What are some recent sentiments about PBR in Second Life?' But don’t expect anything deep. It’s really not worth the cost at present." But again, entertaining -- see some Grok fail in this Twitter/X thread.
Final advice from the security expert: "Just that people should pay close attention to the terms of service and privacy settings on each. For example, ChatGPT can use users' questions for training unless they explicitly opt out." I would personally recommend opting out there.
Please support posts like these by buying Making a Metaverse That Matters!
Soft is right, I was doing similar things for months. For small scripts and functions, GPT-4 is a time saver, it can also handle regular expressions (regex) well enough. Processing natural language is the job of language models, so it is commonly used for summarizing, making lists, revising and many other things. And it can also help with finding ideas and brainstorming as he said. Analyzing images has been added a few months ago; Soft Linden using it with the area search floater had clever idea!
Note that GPT-4 was available for free since early 2023 until recently on Microsoft's chatbot (now Copilot https://copilot.microsoft.com ). Also, unlike the free ChatGPT, Copilot can search the web for the latest information. Moreover, in the Precise mode (teal color), Copilot provides more concise responses and with reduced hallucinations (the Creative one is the opposite, but it's meant to create, invent things).
However, now the free version of Copilot has GPT-4 only during non-peak times:
https://www.microsoft.com/en-us/store/b/copilotpro
also the default mode, Balanced, didn't use the best model since a long time (and I'm not sure if Precise is still GPT-4 either).
On the other hand, ChatGPT may be more versatile with custom instructions, regenerate answers and has other features. The free version of Copilot, other than web search, can also create and analyze images, has voice, and so on. But I think Soft asked for "spicy and naughty" to Grok, because ChatGPT and Copilot are known to be a little on the puritan side (Copilot would answer, but then the filter would often delete the answer).
In that case, if used as a search assistant, with a similar question asked by Soft to Grok, I asked Copilot something a little more interesting for a SL user: "What are the events in Second Life for February 2024 and which one of them may offer free items?"
https://sl.bing.net/bOGjOVvHy0G (the link will open it in Balanced mode, but it was performed in Precise mode)
Copilot listed the events and concluded
That's correct and Copilot also provided a link to the source.
Of course you could just search or go to grid-affair yourself, but it's an example how it can extract information from the results and text in general.
Posted by: Nadeja | Thursday, February 01, 2024 at 06:04 AM
Image generators and text generators have something in common: they can produce mediocre or impressive results depending on how you prompt them. For example, with a generic prompt you would usually get a generic image, but if you want a high-quality image, you can add words like "photo-realistic", "high resolution", "golden hour", or "intricate details" to your prompt. Similarly, if you want a better text, various prompting techniques have been developed in the past year. One of them, pretty fun, is to add "Take a deep breath and work on this problem step-by-step". As Soft said, language models can simulate human reasoning to some extent. Another thing that can improve the responses is to treat the model as a sort of friend and talk to it politely and kindly. This is not surprising, since it was trained on human data.
And although GPT-4 has its limitations and you shouldn't expect it to win a Nobel prize in literature for you, that does not mean it is useless or have little use. GPT-3.5 and 4 can write text with a richer vocabulary than most individuals, a nearly perfect punctuation, and basically no spelling errors. So they can help you also in Second Life, with improving instruction notecards, writing scripts, as mentioned, but also with translations, roleplay ideas and more. They are a good addition to the tools you use in SL.
There is also an important difference to keep in mind and don't have wrong expectations. You may have heard "algorithm" and think of programming. However, these models are called language models because they model how natural language can be processed... by a neural network. You approximate and model it with mathematical functions and statistic (then you can also add hyper-parameters, filters, etc) - so you have a software that simulates a neural network that processes the natural language - but the resulting simulated neural network doesn't processes the information like an algorithm. Not only isn't programmed, but trained; but also emergent abilities appeared, so they can simulate also reasoning, and they can code. I.e. the simulator isn't the simulated.
To make this easier to understand, you can similarly run a simulation based on a predictive model of the formation of galaxies: those simulated galaxies aren't algorithms, obviously. Also, as with ANN, you have approximations. You aren't simulating every subatomic particle of those galaxies, it would be computationally infeasible. Same thing with models of human neurons, that exist since a long time, and can emulate the real neurons accurately enough: they would be also impractical for LLMs on current hardware.
So what you should keep in mind is:
- neural networks (natural, like your brain, and their loose artificial approximation) don't work algorithmically and are not exactly programmed for specific responses, like the old ELIZA program. They are trained instead.
- Also they are trained on a huge amount of data, larger than themselves: they cannot physically store all that information (someone compared that to a lossy jpeg image).
- You have missing info and an intrinsic feature of neural network (our brain included) is that missing info and input can be "hallucinated". That's great for in-painting/out-painting or to hide your physiological blind spot, but not so desirable in other circumstances. i.e. these neural net models can and do mistakes.
Therefore you shouldn't expect them to return correct results like a calculator, to be a search engine or, worse, a database with factual information and predefined answers. LLMs don't work that way. They can, however, (especially if you use those that hallucinate less) translate your natural language input into search engine queries and operators and look at the results and then elaborate them further. And they can process language in many other ways.
So, you should know your tools and use them for the right tasks.
Posted by: Nadeja | Thursday, February 01, 2024 at 12:09 PM
Nadeja's comment that LLMs (and other AI tools) are trained, not programmed, is a fundamental issue that is rarely pointed out in the mainstream. We mostly 'assume' that there is a team of researchers and computer geeks writing a lot of code to 'get things right' (because, well, it looks to us that things like ChatGPT actually write rather good English — even when they're clearly hallucinating).
To understand better how LLMs work, I found a really well-written article using the bare minimum of technical jargon:
https://arstechnica.com/science/2023/07/a-jargon-free-explanation-of-how-ai-large-language-models-work/
The most fascinating bit of this article is the thorough explanation on how we humans do not really know exactly how these systems work at all — and the authors also explain why we cannot figure it out (hint: it would require far more resources that we, as humans, possess).
What is not explained (it might require a whole book, not a simple article with a handful of simplified schematics) is how researchers are able to 'weed out' generated content that is deemed to be offensive, sexist, racist, fascist, etc. My guess is that the work done on that is far more interesting than the rest, because it requires researchers to be aware of how such results are produced in the first place — and this is something we do not know!
Obviously, one thing is just to reject anything written with a list of forbidden words (easy, even considering that such lists, for a global audience, would need to be updated with all possible languages). But there might be subtle things which are clearly offensive (to a human) which can be expressed using regular, neutral words.
Or — worse! — things might have hidden meanings, which the researchers might not even be aware of. Two examples: because of the many word-filtering algorithms out there, neo-Nazis have devised a scheme of word replacements for concepts commonly 'weeded out' by such algorithms. Everyone in the community knows these keywords (some, of course, have long ago leaked out), but a typical person might have no idea what they're talking about (or why such neutral words, in a particular concept, is supposed to be a racist slur, for instance).
The Chinese government also struggles with the opposite issue: critics of the system regularly employ new usages of fairly neutral worlds to freely exchange messages using China's own IM and chatroom systems. These can be monitored as closely as China wants — both with real humans as well as all sorts of AI-based technologies — but the regime critics are clever enough to avoid their conversations being tagged as 'subversive'.
Here is what ChatGPT 3.5 'knows' about this process:
Posted by: Gwyneth Llewelyn | Friday, February 02, 2024 at 05:19 AM