Enabling Player-Created Online Worlds with Grid Computing and Streaming (Second Life)

This webpage documents my presentation of "Enabling Player-Created Online Worlds with Grid Computing and Streaming" by Philip Rosedale and Cory Ondrejka for CPSC 538A: Topics in Time-sensitive Distributed Systems, taught by Charles 'Buck' Krasic.

Summary

"The hours and intensity with which players engage in games makes for very fast consumption of content"

As players participate in a game they need to interact with new content in order to stay interested and keep playing. Previously, game designers have used the following ways of providing users with a reason to keep playing:

Provide large amounts of content in the game itself (Final Fantasy)
Make the content reusable, enhancing replay value (Gran Turismo)
Make experience dependent on interaction with other human players (Quake)

These approaches do not work well for Massively Multiplayer Online Games (MMOG) where players pay a monthly subscription fee to participate in the game. Since it only takes a player around one month to consume 100 hours of content, MMOG developers have tried to frequently update the interactive content of their games to keep players subscribing. This approach is costly as new content must be developed on short time scales.

This paper describes the challenges behind creating the Second Life MMOG. Second Life makes the game players responsible for creating and modifying in-game content. The problem that the paper addresses is: How can we scale an online world to support tens of thousands of players and hundreds of thousands of objects at one time such that all of the players have a consistent view of the world around them, and can interact with each other by modifying the world collaboratively in real-time?

The authors address the above problem by using two techniques. First, the authors divide the world up into square tiles of 16 acres in area. Each tile represents a simulation (or 'sim') running on a single server that simulates the physics, manages all objects, behaviours, terrain and player actions in the tile. Each sim is targeted to manage around ten thousand objects. As players and objects move around the world their in-game representations are transfered from server to server as they cross tile boundaries. Servers only communicate with their four neighbouring servers. This solves the scalability problem for Second Life; in order to grow the world the developer only need to add more servers to the grid.

In order to solve the problem of distributing in-game content to the players in real time, the authors decide to use streaming. Players play the game using a small (~10MB) 'world viewer' application. All information about the world (audio, video, geometry, textures) is streamed to the viewer application; none of the content is actually stored on the players machine. As players move around the grid they maintain streaming connections only to the servers that they are near.

All data is sent over multiple UDP streams to allow for a more controllable stream than TCP. Reliability of a stream is built around loss detection and data correction instead of retransmission to allow for timely delivery of content. The authors decided to cap streaming bandwidth at 100kbps per client in order to avoid network congestion. However, there is a lot of game state that needs to be transmitted to clients at any one time. To fit all of the data into a 100kbps pipe, Second Life uses wavelet compression on textures, ogg vorbis compression on audio, and efficient compression of game geometry because of its simplicity (avoiding generalized meshes). In addition, players only receive information that is new to the clients (or has changed).

The main contributions of this paper are:

The use of a grid of simulators to solve the problem of scale when users are allowed to create and modify objects in a large online world.
Streaming data to game players over the internet in real-time allows users to modify the online world interactively and collaboratively with other players.

The Second Life world viewer can be downloaded from http://secondlife.com/download/. It can be played for free for the first 7 days.

Presentation Slides

The slides for my presentation (in ppt format) can be found here.

Group Discussion

Questions about the paper:

How does the game deal with diagonal movement? The game world is separated into square tiles. Each tile represents 16 acres of landscape; each tile is managed by a separate server. It is easy to see how player information is handed off from one server to another when a player moves horizontally or vertically across the world. The player maintains two streaming connections (one to each server) as he/she crosses the tile boundary. But suppose a player moves diagonally, perhaps walking across a corner of a tile instead of an edge. A corner of one tile is close to four different tiles. How does the game handle such a situation?
I believe that the player will maintain four streaming connections (one to each server) as he/she walks across the corners of the four tiles. This does not seem like a huge deal, but it did lead to an interesting discussion about dividing up worlds into tiles. Perhaps square tiles are not optimal for such a free roaming game. Maybe hexagonal tiles would be a better fit.
Compression is expensive. How did the authors of the game balance time and space complexity when using compression? In other words, how did they balance how effecient a compression algorithm is versus how long said algorithm takes to run? Did they make specific choices of compression algorithms to keep the real-time elements of the game?
For the most part the game designers used brute force. Essentially, they throw computing power at the problem by requiring client computers to have the following requirements:
- Graphics Cards: Nvidia Geforce 2 (32MB RAM) or higher, or ATI Radeon 8500 (32MB RAM) or higher.
- Computer: 800MHZ or higher, 256MB RAM or more.
With so much computing power, the time it takes to efficiently compress what they need is small enough not to matter.
How far ahead can you see in the game? Can you see through tile boundaries (ie. across server boundaries)?
Yes, you can. The developers tried to make the world as seamless as possible; the tile/server boundaries should go unnoticed unnoticed by a player as they walk over or look through them. The reason a player can be standing in one server while looking into another is because the player maintains two streaming connections. One connection would stream information from the server the player is located in while the other connection would stream information from the server the player is looking into.
It was noted that a lot of the problems mentioned above go away if you strategically put up walls across some of the tile boundaries. I am not sure if this is done very often in Second Life, but I do know that one of the goals cited by the developers was to make the world as free roaming as possible. Erecting walls between areas would go against this goal.
How does the grid deal with the failure of a machine? Does the 16 acres of land that was simulated by the failed server become unavailable? Do neighbouring servers contain enough redundent information to fill in for the failed server?
The paper does not discuss failure. It would seem that the inter-server communication present in Second Life has one purpose: dealing with object and player mobility. I do not believe that the servers exchange information for the purpose of failure recovery and robustness. It is much more likely that the area being simulated becomes unavailable while a replacement server is added to the grid. It is possible that there are backup machines ready to be swapped in quickly in the event of a failure.
How do the landscape designers distribute the load across the world? In other words, how can they be sure that one area is just as interesting as another so that everyone won't crowd into a single area and overload the server managing said area?
It is possible that there may be a hard limit on the number of objects that can exist in one area at any one time. It is also possible that there may be a hard limit on the number of players that can play in one area at any one time.
How realistic is it to push the task of generating content onto the users of the game?
First, the content generation tools provided with the game must be of high quality and easy enough for any player to use. This is a difficult thing to do. More importantly, how many users are going to be interested in authoring content? The content in Second Life can be audio, geometry, textures, or behaviour (via scripts). Is the average game player going to be interested in the time consuming task of creating such things when they could be playing the game? This is an open question.

Discussion of future work:
In what ways could the developers improve the game?

A TCP-friendly adaptive congestion control algorithm could be used instead of simply capping client bandwidth at 100kbps. This could allow the game to make better use of the clients available bandwidth, hopefully allowing for more detailed content to be streamed such as more detailed graphics.
Currently the game uses multiple UDP streams to send data from servers to clients. Firewalls between the server and clients cause problems for game players because most firewalls drop incoming UDP traffic by default. Perhaps the developers could work towards supporting TCP so that clients will be able to play from behind firewalls.

Links

Daniel Ferstay

Last modified: Sun Mar 28 18:18:43 PST 2004