Can OpenSim Worlds Scale Better Than Second Life?

SL blogger Radar Masukami points to this smart and tech-heavy post from Adam Frisby, one of the lead developers of OpenSimulator, the open source spinoff of Second Life.  In it, Adam describes the troubles besetting OSGrid, among the largest OpenSim worlds, now with nearly 12,000 registered users and 2500 monthly active users.  At that size, however, the OSGrid is already having trouble with its Asset Server, where all the clothing, vehicles, notecards, and other content avatars store in their inventory is actually kept.  Trouble is, it already has "several million asset entries and a 75GB table file" and with less than several thousand users "is nearly full on disk space".  Frisby outlines his strategies for replacing it, and if you have heavy geek kung fu, I suspect you'll find it a fascinating read-- check it out here. 

For Mr. Masukami, however, it provokes a larger question: "You habitual complainers out there starting to understand the magnitude of the difficulty in scaling this stuff yet?"  It's a good query.  Last week I noted the intellectual property challenges inherent in OpenSim worlds, and this week, maybe it's a time to think about the scalability hurdles.  I've read many complaints about Second Life's intricate server architecture, and to be sure, it seems to cause no end of downtimes and lag, especially as it strains under rapid user growth/activity.  But are those woes endemic to all SL-based worlds, or will OpenSim developers find themselves unable to come up with a better solution?

Comments

[Torrid Luna]

In said article, Adam Frisby not only describes the current problems of the osgrid asset, but also a possible solution. (And a neat trick to make the asset faster, at least on MySQL backed platforms).

[radar]

I don't think they face any challenges that SL doesn't, but as they note, it's expensive to choose the solutions that LL has implemented to date. And honestly, if LL was using something a little more scaleable than MySql, they'd be spending a lot more too.

I just think people think this stuff is like throwing a blog up on the web or something, and it's not. It's a massive challenge to get a 3d world with changing, user-created content and as many regular users as SL has to not come crashing down in a heap. LL has certainly made mistakes in the past, and no doubt will again, but they actually are doing a much better job than the (largely non-technical) critics give them credit for.

[Dirk Krause]

The worst thing that could happen is that the whole grid thought (as defined by LL/Rosedale) inherently wouldn't scale.

This would be really bad news. But I don't see that, really. Having said that, neither LL nor the OpenSim core developers pushed the envelope yet on how to treat the asset storage problem. The OpenSims just found out that they get a problem if they treat assets like SL does. This doesn't really come as a surprise, does it?

The mailing list is full with ideas on how to treat this, and my feeling is that the best ideas are about to be born yet.

[Carter Liveoak]

Hamlet, OpenSim is a work in progress (FULL DISCLOSURE: I work for Intel and Intel participates in OpenSim development) and scalability is one area where thoughts and work are starting to gel in the effort.

The great thing about an active open source effort like OpenSim is that lots of people bring ideas to the mix and there's a chance to try things and find solutions that wouldn't arise from a closed effort. That's not guaranteed with an open effort, but it is more likely than when the community is enclosed or ignored.

[g]

75GB is a fairly small database. InnoDB when properly setup will rock it; this is a pretty typical situation insofar as people tend to start trying to re-architect the service model when performance becomes and issue instead of looking where they need to first, which is:

1. Hardware

2. Schema

And not necessarily in that order. A proper schema with indexes tuned by someone who knows how MySQL works (and how it doesn't) is important. And unless you have the right hardware configuration for your very application specific workload and resource consumption profile, you are just wasting time monkeying with different architectural solutions that will all be to some extent crippled if the RAID configuration sucks, or the drives are slow, or there aren't enough physical drives in the array, or the filesystem choice is wrong and/or not mounted properly relative to the RAID controller's cache, or the controller card uses sub-par components, isn't setup correctly relative to the host OS & kernel strengths & weaknesses, etc.

Are they really running on Windows, as the article seems to indicate toward the end? That's hilarious.

[FlipperPA Peregrine]

There are some very misinformed statements in the comments here. To clarify:

(1) MySQL is a very robust, scalable database. Look at the benchmarks, even from a few years ago. It is also now feature rich; stored procedures, transactions, etc. MySQL may have been more of a toy DB a decade ago, when 3 was the production version. That is simply not true any more, and anyone who tells you otherwise is grossly misinformed. I'm Oracle certified and have been using MySQL for a decade.

(2) Keep in mind, as well, that OpenSim does not require you to use MySQL. You can currently use MySQL, SqLite, MS SQL Server, or PostgreSQL.

(3) OpenSim is currently programmed in C# on the .NET framework. It will run through the Mono virtual machine on Linux, however, it runs more efficiently and with more stability under Windows. While I tend to lean towards the Linux side for my servers (but refuse to become a blind zealot), the facts are that you'll get more bang for your buck out of OpenSim on a Windows machine now.

(4) A lot of Adam's post is being missed. Yes, 75 GB is not that big a table / database. But read more carefully: OSGrid currently lives on donations, and a major thrust of his article is being able to do this for no (or very low) cost. Buying a NetApp isn't in the cards here.

I also agree with Carter: we're starting to see some very cool new ideas in OpenSim, such as the HyperGrid concept, which will allow various connections between multiple grids.

Regards,

-Flip

[Mo Hax]

OpenSim will eventually be able to distribute assets over the entire internet using URI schemes and per server authentication. Assets will eventually just be another resource files, like images and HTML pages, available from the internet. The traditional, centralized asset server model will not sustain growth in either OpenSim or Second Life as the number of assets continues to grow exponentially. It also does address gross sim/grid asset transfer although the HyperGrid has somewhat addressed that.

In fact, the current OpenSim asset server size limitations may lead to a more granular approach with the HyperGrid linking grids and brokering asset content.

My personal concern is for the eventual exhaustion of all possible UUIDs without any thought for recycling them. I know the number is astronomically big, but if this is to grow to internet proportions over the coming years exhausting them is a real possibility.

[FlipperPA Peregrine]

While recycling is not the worst idea, the math involved here is epic in scale. UUIDs have approximately 3.4 x 10 to the 38th power unique combos. A quick Google brings this good analogy: "This means that 1 trillion UUIDs would have to be created every nanosecond for slightly more than 10 billion years to exhaust the number of UUIDs." (Wikipedia)

That's a REALLY, REALLY big number. While good for being a unique identifier, I still think OpenSim could optimize the database better: UUID for unique retrieval, but integers for joins.

Regards,

-Flip

[Adam Frisby]

As Flipper said - the key here is doing it in low cost. InnoDB does scale nicely into the many millions of rows, although it does begin to taper off once the table size gets larger - certainly once you break about 500GB or so things run off the rails - Indexes also do begin to break down after a certain point.

Ultimately, a relational database is not the worlds best solution for storing asset-style data. The solution I implemented over the weekend (which we'll probably switch to in the next month after sufficient testing) is based on the Java version of Oracle's Berkeley DB combined with some very nifty automatic routing so we can spread it across more than one server - this means we get primary key searches against the hash very very quickly and also allows us to maximise storage hardware donations by making every official box capable of storing assets if we need to.

Metadata such as access times are stored seperately in an InnoDB based MySQL table, some of these have been tweaked since their OpenSim versions to allow faster indexing. For example, switching from access_time being a unix timestamp, to a count of the days since 2000, this divides the distinct index size by up to 86,400 and doesnt remove the fundemental function of that data (garbage collection).

This all being said - Assets are not a weak point. There exist a multitude of solutions for scaling what is effectively, 'document storage' We can use a combination of things such as web proxies and load balancers to handle this - the main catch is just that OSGrid operates on donations, so we need to maximise the efficiency of donated hardware.

For the person who asked - OSGrid runs on a very heterogenous set of hardware, however the core User, Grid, Asset, Inventory and Message servers are running on Fedora 9 on some fairly high performance gear.

I might make a post in a little while explaining the backend setup a bit more since we're about to make some large changes to it.

[Hamlet Au]

Thanks much for the background, Adam! So to get down to brass tacks, let me ask: do you think OpenSim worlds can scale better than Second Life?

[Adam Frisby]

If I drop the word 'can' from your question, to simply 'Does OpenSim scale better than SL?', the short answer is - yes in some ways, and only if you set it up in a certain manner.

Right now, it's possible to use the current hypergrid systems to break the grid up into thousands and thousands of tiny linked fragements - each of those has it's own scaling properties, the smaller the fragments, the easier it is to deal with.

Now, the catch here is - OSGrid, LCO, OLG, etc all rely on monolithic systems to a degree - those right now dont scale as effectively as Linden Lab's systems. There are some reasons for this ranging from quality of hardware to configuration/implementation details, to software issues.

If they needed to scale in the long term however, it would be possible to start breaking up the grids into 'sections' each running as hypergrid linked areas - in theory this would give some pretty decent scaling properties to the grid (bandwidth being the biggest limitation then), similar to how the OGP/AWG work will provide Second Life with excellent long term scaling properties (if they dont screw it up).

There is a lot that goes into planning for 100,000+ users, and certainly OSGrid is looking into making sure we're ahead of the growth curve by switching to things in advance of issues showing up, hence the current asset server expansion. We'll probably do a bit of that this year - if growth continues at the current crazy rate - we will have potentially up to 7TB of assets to deal with by the end of the year, so some further tweaks will be no doubt required.

[Maggie Darwin]

Server written in c++ == major fail

Server written in c# == fail

Why build your server in a weak, proprietary clone of Java? Don't even try to tell me it isn't ("it runs more efficiently and with more stability under Windows...you'll get more bang for your buck out of OpenSim on a Windows machine now").

Ah well, it's already legacy code.

Oh, just for completeness:

viewer written in c++ == epic fail

[radar]

aren't we past the java hype now, Maggie? i work in a fortune 500 company where vendors of many, many companies come in and help us customize their stuff. NONE of them are in java. ALL of them are in some form of C.

get past that, it's a red herring and it's bullshit.

[Maggie Darwin]

"Some form of C"? Sorry to hear that.

Hopefully it's at least C#, which is actually no more C than Java is. In modern times, a non-trivial application written in unmanaged code is a crash waiting to happen.

I note that you need your vendors' (no doubt very expensive...billed by the hour?) help to modify their applications. Will your fortune 500 company be asking for a bailout soon? Or have they already?

Java was hype in 1995. It's a world-class application platform in 2009, sorry you haven't noticed. I bet your competition has.

[Gaius Goodliffe]

Heh.

Bad programmers write unmaintainable code that's a crash waiting to happen, and no language you can name will change that.

Good programmers write maintainable code that works. They do so in whatever language they're given.

Young programmers obsess over what language is being used, as if it really makes a difference.

It does in some tasks, but the difference it makes is whether you finish a particular bit of code in three hours or six, depending on what functionality is available in the language and libraries. The ultimate quality and maintainability of the code at the end depends on the coders, not on the language...

[FlipperPA Peregrine]

Maggie, I've always been a firm believer that 99% of code reliability and quality is determined by an efficient plan with pencil and paper. The language is mostly irrelevant, and your Java zealotry is showing, quite blatantly. I'm with Gauis on this one: concept and logic trump language choice, always.

LanguageZealtry == Epic Fail

Use the right tool for the job. Arguing over which language is better is about as compelling a conversation as "Tastes great! Less filling!" or "PC versus Mac." C# is a fine language, in fact, it is quite like Java. It also learned from a lot of Java's mistakes.

Would you call Apache Web Server epic fail? *facepalm*

[Maggie Darwin]

Comparing Apache with Tomcat, its Java-based Apache Foundation cousin? Yes, I certainly would. Viewed from design coherence, stability, and resistance to vulns points of view. And if you don't like Tomcat, your Java-based web app is portable to Resin, Jetty or (ew) WebLogic.
Resin will even run your PHP apps. Speaking of epic fail. :-)

Choice of tools *does* matter, although it's true good tools won't fix bad design or bad coding. Conversely, bad tool choices create extra work or political problems.

Like "our server only runs good on Windows". Thats'a *severe* scalability problem, when you have to pay the Windows tax on every server in a farm. And it shows the dangers of the .Net Trojan horse. The "lessons .Net has learned from Java" seem to consist mostly of building ports of existing Java tools, because Java has many good ones that .Net lacks.

I've been developing software for over four decades, I think I know the good stuff when I see it. It's not zealotry when you're right.

[Wiz Nordberg]

> I've been developing software for over four decades, I think I know the good stuff when I see it. It's not zealotry when you're right.

Such arrogant claims are the last nail in the coffin of credibility.

All our software tools are imperfect to the extreme, and after 40 years I am surprised you don't know that. Our challenge as software engineers is to embrace techniques for delivering utility in the face of such a vast array of flawed tools and technologies. Rather than focus on a choice of tools, we must promote the use of provably-correct techniques across all toolsets.

It is reasonable to claim that some tools can assist us better than others but to equate the choices you suggest with unequivocal failure is nothing more than ignorance.