WHAT GWYN KNOWS

Screenshot by Moon Adamant

Somewhere above the quiet streets of Portugal is a woman who appears in Second Life as a beguiling redhead in stiletto heels and a Nicole Kidman topography, and somehow, though I know nothing else about her, I trust her most to explain how SL is developing as a metaverse-- and what will need to happen, to get it there.  "Upgrading Second Life, and Why It Is So Hard To Do So" is Gwyneth Llewelyn's latest epic essay, and if you've been wondering about that very question, this is the blog entry you need to read.  I should be running another story from her on this blog this week.

Comments

[Blah blah]

Nothing against Gwyn mind you, but here is my take on her article.

Its a wall of text that the average user doesn't care about, nor will take the time to try and understand.

Bottom line, I can play games X, Y and Z which are graphically superior MMO's with VERY few hiccups.

SL lags like I'm on dial up anytime I get near 15 people and LL manages to f-up almost every update with weeks of f-ing people without rolling back to a stable build.

That is all that matters.

[SignpostMarv Martin]

That's kinda the point of the article Blah- it's an analysis of those kind of unbalanced, ignorant comments about ZOMG SL SUCKS AND LL ARE INCOMPETENTZORZ

your rant itself is flawed:

1) Second Life is not a game.
2) How many MMOs do you know that have been running since 2003 where the visuals aren't the most important part of the software.
3) You're ignoring one of the key points in the article explaining why everything "f-up"s.

[Gwyneth Llewelyn]

Hammie, thanks for the thumbs up — you're definitely embarassing me publicly with all this free publicity — but naturally, I thank you for it and do appreciate your very kind words.

And Blah blah — a skill that I definitely lack is how to explain highly complex environments in a very simple way. They are, indeed, "rocket science". I totally agree that understanding how SL works is not for the faint of heart. I even admit that while I can fathom its complexity, and understand *some* of the ways it works (just because Computer Science is called a "science" it doesn't necessary follow that it is an exact one!), it's well beyond my grasp to explain, in very simple words, *why* it is such a complex system.

While I cannot fully convey to the ordinary user why some things work like they do, the least I can struggle to do is to explain, at least, that:

a) This is a complex system. Complex systems are very often regulated by chaotic models, not statistical ones. (Chaotic models in the sense that tiny changes at one point can have huge repercussions all over the system, and it's impossible to pinpoint exactly why and how this happens) It means that you cannot say simple, linear things, like that you need 10 times more machines to deal with 10 times as many users. Chaotic models can be approximated (ex. weather systems) and simulated, but often they are not *fully predictable*, which is the issue we have with SL.
b) Linden engineers, system administrators, and software developers are not "incompetent fools", "stubborn, autistic parasites" or "out of touch with reality". A well-known science fiction author always said that "Sufficiently advanced technology will look like magic to someone who doesn't understand how it works." That is certainly the case with SL — SL will look like magic, because it *almost* seems to look like your average run-of-the-mill MMO (which are developed a dozen a week these days, by semi-professional programmers, and just take then several months to get the *content*) but it's an entirely different beast. The SL developers are magicians — they provide us the illusion that SL "is almost like a game" (ie. it has 3D graphics... and the avatars look cute :) ) when in fact it's a hugely complex environment. But all this is "hidden under the hood" so to speak — you don't need to be a rocket scientist to *use* SL, just one to understand how it works :)
c) Some things are not done by the developers (no matter how loudly people yell at them!) not because they're lazy, uncaring, ignorant, or incompetent. They are not done because — they might very well be *impossible*. This is something very very hard to understand for most people. When one is used to have "magic" that works, one tends to expect that the wizards can do whatever they want — and if they don't do whatever *we* want, it's because they're mean or lazy. Well. The sad truth is that SL borders on the miraculous — pushing the edge of technology beyond anything people thought it would be possible to do.

Again, this doesn't mean that SL is done the only possible way — or that there aren't other ways to accomplish the same — it's just that the required know-how to make it work like it does is not so commonly available as most people tend to think it is (ie. because there are millions of games and MMOGs available).

A good example seems to be WoW. Blizzard Entertainment has extraordinarily good developers, both in software and content design. They applied to WoW the old rule of television. Anyone taking a snapshot ("still frame") from a VHS movie or a broadcasted movie will be amazed at the lack of detail it actually has. In fact, the way our brain works, when we see *moving pictures* at over 25 fps, we can get away with lower resolution and millions of colours. We don't need high resolution when watching movies — our brain copes differently with moving images than still ones (to read more about the subject, I can only recommend Pinker's "How The Mind Works" or any of his or Antonio Damasio's books).

If you don't trust any of the experts — well, get a video capture board and take a still frame. It'll be under 800x600 resolution, and you'll be shocked at how "bad" it looks. But get these "bad" images running through your eyes at 25fps, and they seem to be "perfect". It's a nice optical illusion.

WoW and many of these high-profile games "get away" with 800x600 because with that resolution you can get over 25 fps easily. And why is that so?

Well, mostly because of avatars. Common avatars have around 10 or 15 thousand polygons. With very good 3D modellers, you can do incredible things with low-polygon counts, mostly by replacing textures — if you can have them on your disk and pre-render scenes on the fly, by simply loading them from your disk. Add this to lower resolution, and you can get flawless motion.

Second Life has none of these. First, people tend to use much higher resolutions — 1024x768 is common, but more is always used, if you have it. SL avatars have 150,000 polygons or thereabouts. If you're in a place with 30 or so avatars, that means around 5 million polygons to render every 1/25th second. Now in 2000 (when SL was under development), an nVidia card was able to render, say, 25 million polygons per second, thus was barely able to give you 5 fps on busy areas with 30 avatars! While newer, state-of-the-art nVidia cards are able to render up to 1.5 billion polygons per second, most users don't have that kind of hardware in their computers — yet — but they will!

Also, SL retrieves all textures remotely. There is no way to "pre-render" scenes. Everything is done "on the fly". This means not only rendering 5 million polygons per frame — it also means loading all those textures first! Since most content creators in SL are amateurs (in relative terms, at least — there are lots of professionals, but they're a tiny minority), these tend to use 1024x1024 textures, which take 3 MBytes uncompressed, but very likely a few hundred KBytes compressed. And how many textures will you need to load? Well, one sim can have 15,000 prims, and prims have, on average, 7 faces. That's 100,000 textures to load for a single sim. And with, say, 100 KB per texture, you have — 10 GBytes of textures to load. Per sim. And your local disk cache will only hold 1 GByte, remember. To make matters worse — your average graphics card will just have around 128 MBytes of space to hold textures — so only around 1% of all those textures in the sim can effectively be put into the GPU for processing.

Thus, while your GPU is busily rendering polygons, your CPU is decompressing textures on the fly all the time — swapping them from disk into memory, and from there into the graphics card — while your Ethernet/wireless adapter is constantly dealing with the GBytes of textures arriving all the time. And — wow — on a computer built in 2002 or so you can still get 5 fps on several areas, even with a dozen or so avatars around!

Magic? Most certainly. Remember, all the games you know do NOT work that way. Content creators and 3D modellers are aware of these limitations. First, they'll have all the textures stored locally — no need to constantly decompress them on-the-fly, keeping your CPU ready for other tasks. Secondly, since on everything but SL you have the privilege of pre-rendering scenes, you can know *exactly* how many polygons need to be rendered. You can also tailor your textures to be much smaller than what people in SL use — even 16x16 textures, properly used, can go a *long* way to add realism to scenes. Now do some trade-offs — lower resolution, lower polygon count on avatars, fitting a scene and all the textures exactly in the 128 MBytes your graphics card has, and have professional 3D modellers developing content so that it looks *awesomely good* — and there is the secret of the success of thing with fluid "gameplay" even on slower machines.

SL simply cannot do the same, not by far. It's still *amazing* how LL is able to come *near* to the same performance.

Engineering, like politics, is the art of compromise. SL, however, due to the way it works, is *very hard* to compromise. There is only one thing that can, indeed, help on a lag-free zone — use as little textures as possible, and have all of those textures be as small as you can get away with. A professionally done environment in SL will use similar tricks — you can have indeed lots of textures if you keep in mind what people are able to view each time. A building made up of separate, closed rooms (that have solid walls, so that the occlusion algorithm can get rid of the non-viewable prims), where you just use a handful of low-sized (but highly detailed!) textures will look awesomely and run quite fast, compared to an open-air building (or one with lots of windows!) that can sometimes have thousand of prims in sight, all of them with huge textures... and professional 3D modellers are being successfully employed in SL to apply all these tricks to cut down on lag.

Of course, your major source of lag will always be avatars — but just reducing the textures will go a long, long way to improve overall performance.

SL, however, is not targeted to the professional designer, that knows all these tricks. It's being marketed to amateurs who have no clue about these tricks. The way it still manages to cope with the whole load is something that will never cease to amaze me.

Yes, it does also look like magic to me. :)

[Blah Blah]

1. Second Life is for all purposes a game, sorry it is, hate to burst your bubble. So you can build in it, so you can script in it, you can do all of that with Neverwinter Nights also, so whats the difference? Oh yeah, virtual cocks and vaginas.

2. I didn't mean the graphics were important, I mean that I can play games with far superior graphics with no issues, yet SL chokes on my machine. Do I care that it is an issue with the server pushing the graphics, no, I log on to these worlds for fun and entertainment. SL could be fun, but as it stands it isn't, And MMO's running since then, lets see off the top of my head, Meridian 59 is still going, UO, EQ, City of Heroes since 2004, sure there are more but I don't care, I don't see the point. Hell Neverwinter Nights, 03, I would say had more going for it in a creative MMO than SL, it was far more stable per server, some groups had linked servers, if you built in that world you could actually learn real world useful skills like 3dStudio/Maya without having to kludge it into SL. And if visuals aren't important to people in SL Mr Signpost why is sexually related content the biggest draw outside bling and realistic skins and the latest ho-wear? Hmmm?

3. I honestly didn't read all the keypoints. We the customer don't care why things f-up, all I need to know is I pay for it, it's fucked up. I'm not paying LL to learn about their system, I play these GAMES to blow off steam and have a good time. SL crashes, runs like shit on my machine that can handle every other MMO out, has issues out the wazoo, etc. What does my learning about why its fucked up help? So I can sit back while I can't log in and play armchair quaterback? The f-ing system should run right, its what the customer pays for, not detailed lessons in the subtle art of coding a MMO.

[Blah Blah]

It's nothing against you Gwyn, and kudos for typing it out, but honestly to me its just a wall of text, I phase out in the first paragraph.

I'm sure some people love to read it, but I haven't the time nor the patience, nor the desire to read about why SL is scrod. It's scrod, that all I need to know.

[Barney Boomslang]

Sorry to burst _your_ bubble, blah blah - but SL _is_ fun for thousands of ppl every day. You might notice a common theme inworld - ppl run around, dance, build, chat, and *gasp* have fun. You might either recalibrate your perception (to notice that not everybody is thinking like you) or rewrite your formulations :)

_You_ as the customer might not care for details, but please refrain from talking about "us" the customers, as there is one thing for sure in SL: there is no common consensus of customers about it.

If other 3D games suit you - fine. Go there. Have fun. You see, I as a builder can't get my kind of fun in other environments, since no other 3D environment gives me the same freedom of building right now, with the same large mix of users visiting it. Sure, I could bang VRML worlds together - and be alone in them (same for the current open source 3D environments). Or I could go to there or activeworlds, which both heavily restrict what I can do.

Am I pissed off by problems in SL? For sure. But I try to find ways around it, because it's the only current 3D environment where I can get my kind of fun. And despite all problems, SL runs damn good.

[IAmEric]

I'm stating the obvious, but the only solution to these problems is to open the platform. For each build, there should be a system admin. How many builds are there in SL and how many admins are there at LL? Only once I've got my build hosted on my own server and I'm responsible for its upkeep will this monster even begin to be manageable. Any word on when that will happen? Is it inevitable or is there a desire to keep SL on LL servers only?

I'm sure this issue has been covered to death, so I apologize for being too lazy (it's almost 12am on Sunday and I'm still in the office so not sure how lazy I really am) to find the relevant discussions. Pointers and references would be welcome.

Cheers!
Eric

PS: Hamlet, comparing "topographies"??? :)

[SignpostMarv Martin]

Blah Blah, you're a very closed minded, ignorant person if you think SL is a game.

1) Games have goals. SL has no goals other than those you create for yourself (part of the reason why it's called Second Life)

2) Most games have artificial conflict- MMOs especially; Kill the monster to get the bigger weapon to kill the bigger monster to get the bigger weapon ad nauseum.

3) All games have resolution. In a game of chess, either one person wins or it is a draw.

Second Life is a platform. You can run games on it, but SL itself is not a game. You can run games on Windows, but the OS is not a game. You can make games in Visual Studio, but it is not a game.

With regards to the graphics issue, you're completely ignoring the fact that Gwyn pointed out- games can pre-render their scenes. Neverwinter Nights content is not downloaded on the fly. You can't just connect to a random server and start playing, you have to download the module for it.

segments of this comment were paraphrased from comments made by Philip Rosedale (from a SecondCast interview) and Cory Ondrejka (from a Lang.NET presentation)

[Grace McDunnough]

At the risk of sounding disgustingly diplomatic, from my perspective:
1) Gwyneth is correct: this is a complex system.
2) Blah Blah is correct: SL performance is not meeting expectations that are based on *games*, not emerging technology platforms.
3) Signpost is correct: this is no game, architecturally or otherwise.

After reading the comments to the LL release announcement, Gwyn's post, Hamlet's post and these comments - the point is clear to me. We have reached the Tipping Point and no longer will Linden Lab have the luxury of addressing a handful of early adopter archetypes. And as Barney said - "as there is one thing for sure in SL: there is no common consensus of customers about it."

It's time for some serious and *actual* Product Management at LL. Right now we have marketing and technical planning, but the real essence of Product Management is to not only understand the technical aspects as Gwyn outlined but also must be able to translate a meaningful plan to the residents, in a way that everyone, including Blah Blah, can appreciate. A Product Manager would have ensured that Jeska Linden's post was explicit and more transparent, something to the effect: "We are working hard to resolve the bugs introduced by the last release. This is what we think is most important, and why."

Given the recent population explosion, this must be industrial strength Product Management if the platform is to grow during what will feel like an unbearable drought of release planning, road mapping and inevitable second guessing.

[Blah Blah]

Your right Signpost, SL is not a game, its a whole brand new golly gee whiz way of life. I applaud you for living it.

*golf clap*

And to think those of us playing other MMO's live such shallow, full resolution, zero lag real lives. Pity us...

[Melissa Yeuxdoux]

OK. Perhaps the thing to do is to go at it from the other end. We all know that as things stand, SL can be really slow.

What I'd be interested in knowing is this: what's the fastest SL can possibly be? For example, one can prove that on a single processor, a comparison sort can't possibly run faster than O(n log(n)) where n is the number of things being sorted, so once you have an O(n log(n)) algorithm for sorting, you know that while you may be able to fiddle with the constant, you won't ever make a major improvement without doing something very different.

Can SL be analyzed in that way? To some extent, no, because people can write arbitrarily time complex scripts (hey, let's sort things by shuffling them at random and then checking whether they're in order!), but ignoring that, what can one determine about the best SL can possibly be? Just curious.

[Aenea Nori]

I just had to say something here. Gwyn's "off-the-cuff" comment here was almost as good as the original post referenced :) I don't think I'd ever understood the real complexities of the system like that. Thanks Gwyn! :)

[Harvey]

Actually, the first 3 definitions for game on dictionary.com are :

1. an amusement or pastime: children's games.
2. the material or equipment used in playing certain games: a store selling toys and games.
3. a competitive activity involving skill, chance, or endurance on the part of two or more persons who play according to a set of rules, usually for their own amusement or for that of spectators.

To some SL is a game, to others it isnt but I'd guess to the majority it is an amusement or pastime and thus a game. But it's kind of irrelevant.

Generally speaking SL shares a lot more with MMO's than any other system which is why the comparison is made so much.

[Gando Thurston]

It's good to understand why SL can't currently compare directly to games.

Perhaps some of our problems in SL could be fixed by interested folks teaching others how to take advantage of occlusion culling, smaller textures, and other performance enhancing creation tips.

I would think that hitting up the largest content creators and giving them these tips would go a long way to helping performance.

-G

[Gwyneth Llewelyn]

These are tough choices for LL. I do agree that it might well be next to impossible to explain to the common user — who will have their Xbox, PS3, or Wii as a comparison — why SL cannot beat that performance using current technology.

This is like trying to explain why you cannot use your cellular phone to use a satellite uplink. "Why not?" says Regular User. "I've seen people with satellite phones using it all the time!" Following with a detailed explanation on how cellular phone technology has almost nothing to do with satellite communications except in two aspects: both use electromagnetic fields for communication; and both use a transmitter/receiver that is a cute box with numbers on it and that you can hold in your hand :)

When we're in the realm of explaining how "magic" works — and why SL's magic is different from the "magic" inside an Xbox, for example — it's way hard to get the message across.

I totally subscribe Grace's opinion — what LL does actually need is a good product manager with a solid background of marketing :) After all, companies are able to successfully market cellular phones (which won't work on satellite uplinks) and the customers are happy to understand what the limitations are on their "magic" box without needing to understand how the magic works!

Gando, I agree that these "tips" for improving the user's viewing experience should be given/explained to content creators. I just feel personally uncomfortable to explain to them these things — after all, SL is "promoted" as a platform targetted for amateurs, who don't know — or don't care — how professionals squeeze so much performance with cool tricks. They just want to upload a texture and expect SL to work as fast as any other "game" :)

Believe me, some professional content creators are well aware of this — and they let the market decide. They're available for hire, for dozens of US$ per hour. And they guarantee low-lag buildings with extraordinary detail, realism, complexity — which also happen to look very good. These professionals are very expensive, however — and the question is naturally: it's worth the price? That, ultimately, is for the market to decide. In my mind, the answer is "yes, there are people willing to pay premium for high-quality, low-lag content", just not many. Yet :) When SL hits 100 million users, this might change totally :)

[starcomber Vig]

The current flow of comments reminds me of the longstanding issue that manifests always and everywhere between seasoned users (citizens) and new users (consumers).
I won’t go thru this distinction again, it is actual, clear and known.

Gwyn’s essay is good and very technical, even going into proposing the splitting of UUIDs in pairs and all the way thru mentioning clearly the advantages (server locality, distributing of the overhead i.e. sub-grids, dynamic updates) and issues of said choice (backward compatibility). During the next year we may be able to tell if this is indeed the direction that SL in to employ and if LL will have the resources and time to walk such an ambitious path and reach before Google and/or the conglomerate Disney/Apple/Pixar.

I thank Grace for saving me a long walk. All that she says is accurate and shared:
SL is a complex system
Blah Blah is correct: SL performance is not meeting expectations […]
Signpost is correct: this is no game

In Grace’s words: “Given the recent population explosion, there must be industrial strength Product Management if the platform is to grow”

Is there Product Management at LL for SL? The answer is no, there is not, at least to a level that is visible to 2M registered accounts. The company seems surprised that what used to work for a niche of 200K does not for a larger base (of 2M). Surprised why?

Gwyn can manage to envision and even detail a growth path to the infrastructure that has not, on LL’s side, a robust planning of resources, communication and vision for the platform. And without that flip side there’s no SL winning Blah Blah to stay onboard.

LL and Phil personally put a huge effort in the past 6 months into generate media frenzy, one that is historically noticeable, successfully. But neither LL nor Phil were ready for the big wave of gamers and consumers for which a standard usable UI counts, immediate performance comparison with apparently similar MMOG counts, help counts.

These aspects were underestimated and/or left unmanaged with visible huge repercussions throughout SL.

I hope that LL can go thru the changes and optimizations that Gwyn detailed, if not those any that make the platform more stable, more captivating. But the target and expectations of this target, these things may take years to shape without proper planning. Years that LL does not have for granted in the pocket.

P.S.: Stop hammering newcomers and their expectations. They are as respectable as any seasoned user's.

[Melissa Yeuxdoux]

"SL avatars have 150,000 polygons or thereabouts."

I'm no graphics expert, but is that really the case? To my very untutored eye, the Project Human Open Source LoRes Female figure looks very good in comparison with the SL female avatar, and it's listed as having 16,828 polygons, about a tenth as many. Is there something I'm unaware of that makes it not a valid comparison?

[On High in Blue Tomorrows]

Sorry Gwyn, come again - 'when SL hits a 100 million users' ..huh? As far as I am concerned, I'd be extremely worried whether LL will be able to keep their ship upright another million registrations further down the line.

Also, I'm not sure just what could be done to more successfully manage a product that, while brilliant and inspiring, is also deeply flawed in so many ways. Perhaps the two most egregious design mistakes seem to be

numero um) the fact that it doesn't talk to the internet at all. How on earth, dear Gwyn, with all your experience working for ISPs and such, can you say, with any confidence, that any networked technology will be widely adopted, scale massively, and even, perhaps, one day, become a cash nexus for business when it doesn't even talk to the internet? Help me understand.

numero dois) the fact that it is still nearly impossible to import or export any content into or from SL - apparently following a decision to specifically restrict content flows of this kind - according to what I have read about it at least.

With restrictions to content (deliberately or otherwise) built into the architecture of SL and without being able to tap the amazing vitality and relevance of the internet as it is today, I believe there is a huge relevance gap for virtual worlds like SL.

I don't think it has to be this way at all, but unless someone, somewhere, some time soon, manages to build the equivalent of a LAMP stack of robust, secure, scalable, open-source technologies that will allow virtual worlds to spruce up anywhere and everywhere, this technology is going to continue to be far away from ever becoming relevant to a more or less generously defined constituency of early adopters.

[On High in Blue Tomorrows]

s/ to more than a more or less..

[Pablo Andalso]

On High in Blue Tomorrows–I imagine you haven't read the NWN article The Engines of Offline Creation or heard of the Second Life Creation Engine.

The first is an example of SL "talking to the internet," albeit talking to a database (which bear to mention that SL does, indeed, use the internet, just not the World Wide Web, a set of interconnected flat documents).

The second is an example of importing content... which, of course, the other application was already doing as well. You see, Second Life in Mono is the planned conversion from the proprietary LSL to an open language with which many, many developers are already familiar.

[Pablo Andalso]

...

I assume, though, that you are referring to some sort of in-world email client or network terminal. By "importing content" you mean from SketchUp or from Blender, and by "exporting content" you mean saving the cache to the hard drive in a usable form.

If Second Life is really so closed and backward, it's bound to sink.

[Hypatia Callisto]

"To my very untutored eye, the Project Human Open Source LoRes Female figure looks very good in comparison with the SL female avatar, and it's listed as having 16,828 polygons, about a tenth as many. Is there something I'm unaware of that makes it not a valid comparison?"

Melissa,

The SL avatar has only 3912 polygons. (I opened her up in a modelling program and pulled the statistics). That "open source" figure you talk about, is not an avatar, its a figure for Poser, and is WAY too many polys for a game character.

And don't look too close at her... that character is not very attractive at all. There's a reason most Poser users pay for professionally modelled human characters from Daz3d or efrontier... I think I even like Ruth more than that character. :P

[Hypatia Callisto]

... if one adds prim hair, then there are more polys, but even with prim accessories, I can't see any avatars reaching that astronomical number of 150k. Statistics need to be checked here and there. :)

I'm not an expert in how Linden has their prims programmed, IIRC they are parametric models with mathematical descriptions, so they would have less lag.

However game cards render everything in triangles... So as an experiment, I triangulated the SL avatar (Ruth dear) and she's still only 7186 polygons rendered as triangles. Maybe some twisty tori hair and cute shoes will bump that up to 10k or so, but I cant see it being more than that. But most female avs seem to prefer straight hair in SL (these are also easier to model in prims nicely, no coincidence there), so I wouldnt even count on that. Straight hair is certainly lower in polygons.

'Nuff from me on this subject.