« Other World Notes: Why Lively Died So Fast | Main | Metaverse Consultancy Reports Light Traffic at Real World Company Sites in Second Life »

Monday, February 16, 2009

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Torrid Luna

In said article, Adam Frisby not only describes the current problems of the osgrid asset, but also a possible solution. (And a neat trick to make the asset faster, at least on MySQL backed platforms).

radar

I don't think they face any challenges that SL doesn't, but as they note, it's expensive to choose the solutions that LL has implemented to date. And honestly, if LL was using something a little more scaleable than MySql, they'd be spending a lot more too.

I just think people think this stuff is like throwing a blog up on the web or something, and it's not. It's a massive challenge to get a 3d world with changing, user-created content and as many regular users as SL has to not come crashing down in a heap. LL has certainly made mistakes in the past, and no doubt will again, but they actually are doing a much better job than the (largely non-technical) critics give them credit for.

Dirk Krause

The worst thing that could happen is that the whole grid thought (as defined by LL/Rosedale) inherently wouldn't scale.

This would be really bad news. But I don't see that, really. Having said that, neither LL nor the OpenSim core developers pushed the envelope yet on how to treat the asset storage problem. The OpenSims just found out that they get a problem if they treat assets like SL does. This doesn't really come as a surprise, does it?

The mailing list is full with ideas on how to treat this, and my feeling is that the best ideas are about to be born yet.

Carter Liveoak

Hamlet, OpenSim is a work in progress (FULL DISCLOSURE: I work for Intel and Intel participates in OpenSim development) and scalability is one area where thoughts and work are starting to gel in the effort.

The great thing about an active open source effort like OpenSim is that lots of people bring ideas to the mix and there's a chance to try things and find solutions that wouldn't arise from a closed effort. That's not guaranteed with an open effort, but it is more likely than when the community is enclosed or ignored.

g

75GB is a fairly small database. InnoDB when properly setup will rock it; this is a pretty typical situation insofar as people tend to start trying to re-architect the service model when performance becomes and issue instead of looking where they need to first, which is:

1. Hardware

2. Schema

And not necessarily in that order. A proper schema with indexes tuned by someone who knows how MySQL works (and how it doesn't) is important. And unless you have the right hardware configuration for your very application specific workload and resource consumption profile, you are just wasting time monkeying with different architectural solutions that will all be to some extent crippled if the RAID configuration sucks, or the drives are slow, or there aren't enough physical drives in the array, or the filesystem choice is wrong and/or not mounted properly relative to the RAID controller's cache, or the controller card uses sub-par components, isn't setup correctly relative to the host OS & kernel strengths & weaknesses, etc.


Are they really running on Windows, as the article seems to indicate toward the end? That's hilarious.

FlipperPA Peregrine

There are some very misinformed statements in the comments here. To clarify:

(1) MySQL is a very robust, scalable database. Look at the benchmarks, even from a few years ago. It is also now feature rich; stored procedures, transactions, etc. MySQL may have been more of a toy DB a decade ago, when 3 was the production version. That is simply not true any more, and anyone who tells you otherwise is grossly misinformed. I'm Oracle certified and have been using MySQL for a decade.

(2) Keep in mind, as well, that OpenSim does not require you to use MySQL. You can currently use MySQL, SqLite, MS SQL Server, or PostgreSQL.

(3) OpenSim is currently programmed in C# on the .NET framework. It will run through the Mono virtual machine on Linux, however, it runs more efficiently and with more stability under Windows. While I tend to lean towards the Linux side for my servers (but refuse to become a blind zealot), the facts are that you'll get more bang for your buck out of OpenSim on a Windows machine now.

(4) A lot of Adam's post is being missed. Yes, 75 GB is not that big a table / database. But read more carefully: OSGrid currently lives on donations, and a major thrust of his article is being able to do this for no (or very low) cost. Buying a NetApp isn't in the cards here.

I also agree with Carter: we're starting to see some very cool new ideas in OpenSim, such as the HyperGrid concept, which will allow various connections between multiple grids.

Regards,

-Flip

Mo Hax

OpenSim will eventually be able to distribute assets over the entire internet using URI schemes and per server authentication. Assets will eventually just be another resource files, like images and HTML pages, available from the internet. The traditional, centralized asset server model will not sustain growth in either OpenSim or Second Life as the number of assets continues to grow exponentially. It also does address gross sim/grid asset transfer although the HyperGrid has somewhat addressed that.

In fact, the current OpenSim asset server size limitations may lead to a more granular approach with the HyperGrid linking grids and brokering asset content.

My personal concern is for the eventual exhaustion of all possible UUIDs without any thought for recycling them. I know the number is astronomically big, but if this is to grow to internet proportions over the coming years exhausting them is a real possibility.

FlipperPA Peregrine

While recycling is not the worst idea, the math involved here is epic in scale. UUIDs have approximately 3.4 x 10 to the 38th power unique combos. A quick Google brings this good analogy: "This means that 1 trillion UUIDs would have to be created every nanosecond for slightly more than 10 billion years to exhaust the number of UUIDs." (Wikipedia)

That's a REALLY, REALLY big number. While good for being a unique identifier, I still think OpenSim could optimize the database better: UUID for unique retrieval, but integers for joins.

Regards,

-Flip

Adam Frisby

As Flipper said - the key here is doing it in low cost. InnoDB does scale nicely into the many millions of rows, although it does begin to taper off once the table size gets larger - certainly once you break about 500GB or so things run off the rails - Indexes also do begin to break down after a certain point.

Ultimately, a relational database is not the worlds best solution for storing asset-style data. The solution I implemented over the weekend (which we'll probably switch to in the next month after sufficient testing) is based on the Java version of Oracle's Berkeley DB combined with some very nifty automatic routing so we can spread it across more than one server - this means we get primary key searches against the hash very very quickly and also allows us to maximise storage hardware donations by making every official box capable of storing assets if we need to.

Metadata such as access times are stored seperately in an InnoDB based MySQL table, some of these have been tweaked since their OpenSim versions to allow faster indexing. For example, switching from access_time being a unix timestamp, to a count of the days since 2000, this divides the distinct index size by up to 86,400 and doesnt remove the fundemental function of that data (garbage collection).

This all being said - Assets are not a weak point. There exist a multitude of solutions for scaling what is effectively, 'document storage' We can use a combination of things such as web proxies and load balancers to handle this - the main catch is just that OSGrid operates on donations, so we need to maximise the efficiency of donated hardware.

For the person who asked - OSGrid runs on a very heterogenous set of hardware, however the core User, Grid, Asset, Inventory and Message servers are running on Fedora 9 on some fairly high performance gear.

I might make a post in a little while explaining the backend setup a bit more since we're about to make some large changes to it.

Hamlet Au

Thanks much for the background, Adam! So to get down to brass tacks, let me ask: do you think OpenSim worlds can scale better than Second Life?

Adam Frisby

If I drop the word 'can' from your question, to simply 'Does OpenSim scale better than SL?', the short answer is - yes in some ways, and only if you set it up in a certain manner.

Right now, it's possible to use the current hypergrid systems to break the grid up into thousands and thousands of tiny linked fragements - each of those has it's own scaling properties, the smaller the fragments, the easier it is to deal with.

Now, the catch here is - OSGrid, LCO, OLG, etc all rely on monolithic systems to a degree - those right now dont scale as effectively as Linden Lab's systems. There are some reasons for this ranging from quality of hardware to configuration/implementation details, to software issues.

If they needed to scale in the long term however, it would be possible to start breaking up the grids into 'sections' each running as hypergrid linked areas - in theory this would give some pretty decent scaling properties to the grid (bandwidth being the biggest limitation then), similar to how the OGP/AWG work will provide Second Life with excellent long term scaling properties (if they dont screw it up).

There is a lot that goes into planning for 100,000+ users, and certainly OSGrid is looking into making sure we're ahead of the growth curve by switching to things in advance of issues showing up, hence the current asset server expansion. We'll probably do a bit of that this year - if growth continues at the current crazy rate - we will have potentially up to 7TB of assets to deal with by the end of the year, so some further tweaks will be no doubt required.

Maggie Darwin

Server written in c++ == major fail

Server written in c# == fail

Why build your server in a weak, proprietary clone of Java? Don't even try to tell me it isn't ("it runs more efficiently and with more stability under Windows...you'll get more bang for your buck out of OpenSim on a Windows machine now").

Ah well, it's already legacy code.

Oh, just for completeness:

viewer written in c++ == epic fail

radar

aren't we past the java hype now, Maggie? i work in a fortune 500 company where vendors of many, many companies come in and help us customize their stuff. NONE of them are in java. ALL of them are in some form of C.

get past that, it's a red herring and it's bullshit.

Maggie Darwin

"Some form of C"? Sorry to hear that.

Hopefully it's at least C#, which is actually no more C than Java is. In modern times, a non-trivial application written in unmanaged code is a crash waiting to happen.

I note that you need your vendors' (no doubt very expensive...billed by the hour?) help to modify their applications. Will your fortune 500 company be asking for a bailout soon? Or have they already?

Java was hype in 1995. It's a world-class application platform in 2009, sorry you haven't noticed. I bet your competition has.

Gaius Goodliffe

Heh.

Bad programmers write unmaintainable code that's a crash waiting to happen, and no language you can name will change that.

Good programmers write maintainable code that works. They do so in whatever language they're given.

Young programmers obsess over what language is being used, as if it really makes a difference.

It does in some tasks, but the difference it makes is whether you finish a particular bit of code in three hours or six, depending on what functionality is available in the language and libraries. The ultimate quality and maintainability of the code at the end depends on the coders, not on the language...

FlipperPA Peregrine

Maggie, I've always been a firm believer that 99% of code reliability and quality is determined by an efficient plan with pencil and paper. The language is mostly irrelevant, and your Java zealotry is showing, quite blatantly. I'm with Gauis on this one: concept and logic trump language choice, always.

LanguageZealtry == Epic Fail

Use the right tool for the job. Arguing over which language is better is about as compelling a conversation as "Tastes great! Less filling!" or "PC versus Mac." C# is a fine language, in fact, it is quite like Java. It also learned from a lot of Java's mistakes.

Would you call Apache Web Server epic fail? *facepalm*

Maggie Darwin

Comparing Apache with Tomcat, its Java-based Apache Foundation cousin? Yes, I certainly would. Viewed from design coherence, stability, and resistance to vulns points of view. And if you don't like Tomcat, your Java-based web app is portable to Resin, Jetty or (ew) WebLogic.
Resin will even run your PHP apps. Speaking of epic fail. :-)

Choice of tools *does* matter, although it's true good tools won't fix bad design or bad coding. Conversely, bad tool choices create extra work or political problems.

Like "our server only runs good on Windows". Thats'a *severe* scalability problem, when you have to pay the Windows tax on every server in a farm. And it shows the dangers of the .Net Trojan horse. The "lessons .Net has learned from Java" seem to consist mostly of building ports of existing Java tools, because Java has many good ones that .Net lacks.

I've been developing software for over four decades, I think I know the good stuff when I see it. It's not zealotry when you're right.

Wiz Nordberg

> I've been developing software for over four decades, I think I know the good stuff when I see it. It's not zealotry when you're right.

Such arrogant claims are the last nail in the coffin of credibility.

All our software tools are imperfect to the extreme, and after 40 years I am surprised you don't know that. Our challenge as software engineers is to embrace techniques for delivering utility in the face of such a vast array of flawed tools and technologies. Rather than focus on a choice of tools, we must promote the use of provably-correct techniques across all toolsets.

It is reasonable to claim that some tools can assist us better than others but to equate the choices you suggest with unequivocal failure is nothing more than ignorance.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Your Information

(Name is required. Email address will not be displayed with the comment.)

Wagner James Au VR MMO blog New World Notes
Sinespace Unity MMO
Ample Avi  SL avatars
Tableau_SL_Nylon_pinkney
Click to visit Nylon Pinkey's many fashion brands in Second Life: Nylon Outfitters, Golden Years, Wrigglesworth Residence, Yummy, and Art Nails
my site ... ... ...