Linden Lab War on Lag Feels Like US War on Drugs to Me

Last Friday Linden Lab announced "Project Shining", billed as a major effort to combat avatar and object streaming performance problems, or what's more colloquially called lag, and what makes Second Life look like the gray undifferentiated mass you see above, many minutes after logging in. Sample:

This year, Linden Lab is making the single largest capital investment in new server hardware upgrades in the history of the company... As a result of the Shining investigation, the project has been split into three larger performance projects: Project Sunshine, Object Caching, and HTTP Library. All together, the hardware upgrades and the Shining projects should improve avatar and object streaming speeds significantly.

Which sounds awesome, and may very well lead to amazing improvements, but if this feels familiar, that's no surprise. Here is what Linden Lab announced in 2010:

In August [2010], we made some changes that improve the speed with which textures are loaded in a scene. Our release of HTTP Textures changed the way textures are delivered to, and loaded in, Viewer 2, resulting in less waiting around for the scene in front of you to come into focus. This change will also serve as the foundation for a series of bigger improvements, including the deployment of a new asset system, which will improve texture loading and object rez times even more.

And here is what Linden Lab announced in 2011:

We’re planning significant grid infrastructure enhancements throughout the year including technologies to speed server-side rendering (SSR) and server virtualization (web and simulator services). We are also exploring new storage and asset delivery systems. Some of the benefits will not always be noticeable, but they are foundational platform changes that set the stage for rapid performance and scalability improvements. We will continue to keep you updated as we roll out these systems.

Notably, both the 2010 and 2011 posts were written by former Senior VP Frank "FJ Linden" Ambrose, who left the company last December, while last Friday's post was written anonymously. (This may reflects some management shuffling within the engineering division.) And while many SLers have reported performance improvements from the 2010 and 2011 initiatives, others have not. (My own viewer performance has been degrading since then.) That the company is making such a prominent announcement now suggests that lag is still a recurring, widespread problem for a lot of SLers.

All this skepticism to one side, maybe this time the improvements will truly be improving, and I remain hopeful. At the same time, a part of me feels like Linden Lab's war on lag is sort of like my government's ill-begotten "war on drugs" -- a battle successive administrations try to fight, which never seems winnable, and never seems to address the underlying problems. (But maybe that's just oldie grousing on Monday morning.)

Tweet

Comments

[Botgirl Questi]

Coincidentally, I addressed the idea of technical debt in a bogpost last week:

"In a world unconstrained by time, money and competitive pressure a software product's underlying code would be consistently refined and refactored. In reality, companies usually focus their resources on short-term tactical requirements instead of the less urgent task of optimizing code and updating architecture. Over time, a product's codebase can become so kludgy, convoluted and incompatible with current technology that significant improvement is too costly and complex to be practical. Second Life sits on a codebase with a decade's accumulation of technical debt."

So it was good to hear them announcing a serious commitment to paying down the technical debt, as well as how concrete they were about what they have planned. As usual, only time will tell.

[Ajax Manatiso]

Just as meaningless as when household products say they are "New and Improved". It looks like once again LL is trying to roll out a software solution to a hardware problem. No matter what kind of code they dish out, it will not really do much for performance and might actually decrease performance. Pry open your wallet, LL, and get some more server memory -- like every other company on the planet!

[Ehrman Digfoot]

Well if I'm understanding the object caching aspect of the project announced, this should be something we will notice immediately. In fact it sounds like something advocated for in a NWN post not too long ago, (maybe about skyrim?) where Hamlet suggested the ability to download specific sims so they would rez immediately. This object caching should provide essentially the same functionality. It will check if the UUID's have changed and only update those that have, everything else in our cache will rez immediately.

[Jon Brouchoud]

I guess I'm relieved, maybe even a bit surprised, that they're still investing capital in Second Life at all. It felt to me as if they had gone into indefinite idling mode some time back.

I'd sure love to see SL pick up again.. another bubble would be nice. ;-)

[froggie]

When people wish for bubbles you know all sanity has left the building. Slow and steady used to win the race.

[Metacam Oh]

Makes you wonder what they actually have been working on these past years if not for this. It just makes me shake my head that a company as profitable and with as much staff as Linden Lab has such a big problem of working on multiple things at once. They also take years to roll out anything that they announce. I don't know what has to change to fix this, but developing wise, they are falling extremely behind.

[Kadah Coba]

The 2010 stuff was under M, so none of that even counts.

2011 did see very good improvements to TP and region crossing reliability and performance. Remember when ever time someone would enter and leave a region, the sim would stall for a minute?

The server side 2012 projects are still in research and data gathering stages. The short answer is way many of the services were setup was not a good idea and just reconfiguring them correct those issues would cause far worse problems. The long answer is that the problems are going to be much bigger than they think.

[Hamlet Au]

"The 2010 stuff was under M, so none of that even counts."

Actually, Philip mentioned it after he briefly replaced M in 2010:

http://nwn.blogs.com/nwn/2010/08/open-tech-forum.html

During his in-world strategy update, Philip Rosedale said Linden Lab's first priority under his interim CEO management is "a big attack on lag and crashes, clearly things that very negatively impact all users."

[shockwave yareach]

One improvement would be to actually GET images into the cache folder, and USE the cached images instead of retransmitting them over and over and over again. If you have something which changes textures (like a photo shower) then every picture change is accompanied by the texture, all over again. How many terrabytes are wasted in repeated transmission of data?

And how many other problems exist where data is sent over and over and over again?

But the biggest issue is certainly to be optimization of the server code. Optimize it so make maximum effect of the cache. Do all of one function to all the datasets first so all that is in cache. Then the second function. And so on. Time to optimize for speed.

And if more ram is necessary to do this, then buy and install the ram. Nobody is using the same amount of ram OR drivespace they were in 2002... well, nobody successful anyways.

[Pussycat Catnap]

I'm in the camp of folks for whome the grid has experienced -massive- improvements to issues like lag and stability.

If you're able to run the newest viewer - or one based on it, its going to be a lot better. If not, you're still working with old code - 2009 at best, 2003 at worst, is sitting under the hood for you. And for those folks I can see it getting steadily worse as they encounter more and more of an SL tailored to a different tech base.

The war on drugs is an epic fail. If you assume it was meant to combat drugs at all:
http://www.youtube.com/watch?v=wtspQWOZ1ns
- for what it was really meant to do, its been a massive success.

But the 'war on lag' seems to be more or less working. It has some causalities though. That's how these things go. In any tech advance, choices have to be made and you pick a spot on the line and cut away everything on one side.

This advance could be great. Could be the 'best thing since sliced bread'. But only if its not coupled to the new 'texture compression' on-by-default "feature". That feature eliminated lag by making no-lag appear to be massive lag: rendering everything fully but blurry. :)

[froggie]

Lets legalize drugs tomorrow. It'll bankrupt Mexico and turn all of California into an immigrant no mans land of people looking for any type of survival. Phillip and his cronies may have to share their SL built/paid for houses with displaced Mexican families.

[Kadah Coba]

"Actually, Philip mentioned it after he briefly replaced M in 2010"

That was near the end of 2010 when the effort was more on saving the company financially from several years of poor decisions to focus on the "business aspects" instead of the actual platform. This killed off or shelved a lot of internal projects for the next several years.

[Hamlet Au]

Actually it was the middle of 2010, in July:

http://nwn.blogs.com/nwn/2010/07/philip-on-the-back-to-basics-second-life.html

Whenever he said it, this just fits a larger pattern of Lindens publicly announcing a war on lag, again and again, year after year. If the improvements were broadly substantial, there'd be no need to make these kinds of announcements, so it's likely that it's still a very common user complaint.

[Metacam Oh]

"I'm in the camp of folks for whome the grid has experienced -massive- improvements to issues like lag and stability."

@Pussycat

I'd hope so, even if they did nothing, lag would improve since bandwidth speeds have increased tremendously in the past 5 years. Still can't cross a sim smoothly though.

[Salazar Jack]

Don’t Do Lag! Just Say Go!

[Salazar Jack]

I would like to think that this gets referred to as “The Shining” from now on.

[Hamlet Au]

All work and no place makes Salazar Jack a dull boy.

[elizabeth (16)]

"(But maybe that's just oldie grousing on Monday morning.)"

yes. jejejejjee (:

if cant give the real deal then give hope

[Salazar Jack]

Oh I still have a place, Mr. Au. My lookout in Rodeo on the Mainland still stands. But if you are referring to a web site then, sadly, yes no place. My mobileme Kahruvel site is no more as Apple has closed their web hosting space of old. But fear not, my kahruvel.com domain will hopefully be resolving properly soon. Good G”al that IS dull.

[Salazar Jack]

I know precisely how dull I am as that shining reference has just now congealed in my brain. I need a drink.

[Dave Bell]

Will somebody please tell Linden Lab that a project in beta is not finished.

[Caliburn Susanto]

You'll never be entirely rid of lag because you'll never be rid of the bozos walking around covered in high resolution textures and loaded down with scripts -- or running laggy badly-written scripts on their property 24/7/365, whether or not they are in-world (which usually, they aren't).

And why venue owners bother to put out those resource usage boards I have no idea, because none of them have the balls or commitment to kick heavy laggers off the property or out of events.

My point, therefore, is just accept the lag. It's much more relaxing. :-) I spend a lot of time roaming the Grid, and travel all over the Mainland regions by car, motorcycle, train, plane, and boat and manage fine. I've made trips spanning over 50+ contiguous regions without any mishaps (rubber-banding at border crossings, yes; viewer crashes or being wrenched off the vehicle, no). This would not have been possible back when I first entered SL, so things are astronomically better already. Whatever improvement this "Shining" project contributes can only be an additional plus.

[Hitomi Tiponi]

Hamlet - I think your opinion on this is rather coloured by the performance of your fairly decent Alienware laptop, which does not reflect most people's experiences with lesser kit.

There are still major issues involving lag that can be and are being addressed - improved sim crossing code is currently being rolled out. Rod announced that 2012 would be the year of server side improvements in SL (2011 was mainly viewer and restructuring) and Shining is just the latest of many, though most go unpublicised by LL. Incidentally LL's policy of pushing more server side is very sound - it is the area it can control - and is clearly a good policy if they adopt cloud streaming in future.

Try running new kid on the block Cloud Party on a basic netbook and do the same with a scene in SL rendering many times the triangles and you will see that lag in SL is not that bad.

[Iggy]

Sim crossings have improved recently. But overall, I'm reminded of this bit from the Onion:

"War On String May Be Unwinnable, Says Cat General"

[foneco zuzu]

As soon as I started using Niran viewer i got a much better preformance overall.
As a nascar racer, i need at least 30 fps minimum, i was getting 35 with phoenix, at full setts but no shadows.
With Niran v 1.43 i got same settings 80 Fps!
With Shadows and Dr (as now i use all time) i still got 40 fps (draw distance 450 on both)!
So as a long long user of pre mesh viewrs i can say that the latest LL code made a big improvment overall (i can even cross 100 sims straight on my bike, with 217 scripts on me, lol, even if reguraly i wear only a ao and open collarm and on the races not a single one)!
Yes, a user top computer helps, a Optical fiber connection of 200 mega download sec as well, but Nvidia latest drivers helped a lot more, as i could use 16AA and 16AF via hardware and finnaly use a V3 viewer based (Niran for me all the way!)

[Meursault]

Wow - it is tragic that LL keeps repeating itself. It's like every new exec team thinks Lag is the killer variable. I bet this is coming from the new guy, "Danger Linden." [sic]

Lag does not fundamentally change the value of SL. Sadly, they didn't get the memo in 2008, 2009, 2010, 2011, 2012.... It is heart breaking.

Low tier, different land products, better discoverability, improved concurrency, better accessibility... all showed more promise than a few FPS.

Those who do not study history are doomed to repeat it.

[foneco zuzu]

I cant agree, lag was making me lost faith and interest in sl, cause the things i wanted to do, like exploring mainland by sea, land or air, was not possible!
Now that those problems are sorted (fingers crossed, we never know, lol) I can accept that tiers are as ever, as even if i had 10 sims of my own, i could not have so much fun as after crossing 100, straight, on bike, watching all around (some amazing places, some really bad taste, but nevertheless a lot of continents that i didn't even knew they existed, as i was a homestead user since i got premium and used to to all above, but on private regions, like the Cobra's sims, the paradise islands and so on!
Yes, lower tiers is a must but not mandatory anymore if users can enjoy mainland as now!