The Effect of Latency on User Performance in Warcraft III

Presented by David Sprague

March 29, 2004

CPSC 538a Reading List: Paper # 49
Nathan Sheldon, Eric Girard, Seth Borg, Mark Claypool, and Emmanuel Agu (2003). The Effect of Latency on User Performance in Warcraft III. Proceedings of the 2nd workshop on Network and system support for games. Redwood City, California. pp. 3-14.

ACM Reference Paper

Summary:

Previously in this course we have examined the effects of network lag on first person shooter (FPS) games and examined how dead reckoning, cheat control, and tariffs are used to make such games fair and accessible to a wide range of users. Network lag in real-time strategy (RTS) games has not been thoroughly explored by the research community, and so the authors hoped to examine how user performance, network activity, and user preferences were affected by an increase in lag.

What are Real-Time Strategy Games? (In particular Warcraft III)

Real-time strategy games traditionally consist of resource collecting, unit construction (both building and character), and combat. Each command issued causes a unit to perform a series of predictable repeated actions (such as asking a peon to gather gold). Further, commands to units are given in advance of that unit's behaviour. Thus, results of a user's actions are not instantaneous. Performing repeated small time-frame user actions is referred to as micromanagement, and is considered a useful skill for RTS players to master. For network games, the authors also noticed that each client runs a complete copy of the game state and thus only user actions need to be communicated to other players. This also reduces the need for many prediction strategies found in first person shooters.

Figure 1: A screen shot of Warcraft III. Note the character selection (green circle), the options available to the user for that character (bottom part of the screen), and the map (bottom left corner) with occluded areas.

The Experiments

This study examined three major Warcraft III tasks: building, exploration, and combat. Three custom designed maps were used to isolate each of these activities (see figure 2) and lagged used performance was compared against regular users. Network lag was simulated using NIST Net. The experiment found no significant or substantial differences between lagged and non-lagged players for the three tasks and a line of best fit between network lag and game performance was almost zero. The one possible exception to this was exploration, which showed a greater (but still not considered substantial by the authors) best fit slope.

Figure 2: RTS Component Maps: Build (left), Explore (middle), and Combat (right).

The second part of the study explored network traffic due to lag for both Battle.net and LAN games. The average round-trip times for Battle.net was 100ms with less than 0.1% data loss, packet transmission was regular, and the median payload size was 9 bytes. The authors found that packet sizes increased with an increase in latency. Further, when the 6 bytes per packet header was taken into account, the sum of command payloads was similar regardless of the latency. This suggests that commands issued by users were similar and no commands were dropped irrespective of the latency rate.

Finally, the authors examined user opinion about lag. The three map experiments demonstrated that players find it relatively easy to adjust their strategies to compensate for latencies between 0 and 500ms. Players reported that delays greater than 800ms made the game appear erratic and the game experience degraded. This point of reduced usability ranged between 500ms and 800ms depending on the person. Finally, user reported that micromanaging battles was more frustrating than trying to micromanage towns and construction.

Conclusions:

Given the above results it was concluded that the overall user experience while playing RTS games was not affected by Internet latencies from hundreds to thousands of milliseconds. The authors attributed this finding to the fact that RTS games emphasize strategy more than interactive aspects and thus latency is much less of a problem. They wished to examine this topic further by examining the effects of latency on longer-term scenarios such as entire games. Also, other RTS games would need to be examined to ensure that the current findings can generalize to the entire genre, and are not just qualities of this particular game.

Questions:

Does Warcraft III timestamp the data transmitted?

It would seem that some form of time stamping would be required to ensure synchronization of each client's game state, and to ensure the order in which events occur.

Shouldn't the study have explored players playing against the computer (A.I.)? This way the quality of the opponent would be constant.

This brings up a larger point. The quality of the players is never evaluated. Playing against the A.I. would regulate or allow a direct comparison between no lag and lagged players. The problem with this is that against the A.I., players would have to perform less on-the-fly adaptations to their strategies (since presumably expect players will know exploitations for the A.I.). It would have been nice to see the quality of the players reported. If players can avoid micromanaging, it seems obvious that the playing abilities of the subjects may be limited (since for experts, micromanagement is crucial <see Steve's discussion point>).

Wouldn't [processor] lag cause performance problems?

Yes it would. Processor lag can and does presumably affect the player's performance. However the research was to examine network lag and its effects and so this type of lag was ignored.

If both players have the same starting army and they are on a flat plain, isn't the only major strategy to just attack the other team? There is less strategy possible with the given experiment.

Not only that, but since users become accustomed to particular armies and groups, they may not know successful micromanagement strategies for the provided armies. In this case, just attacking would be the most logical action.

Discussion (names were included to indicate how the conversation progressed):

Steve noted that in Age of Mythology, micromanagement is crucial to success. He also claims that users need to respond to game events in the order of 0.5 seconds (the lag presented to the user in this experiment would be too extreme).

Buck: In the paper, the authors show the results and say the fit is such that there is no statistical significance. Lack of statistical significance just shows that nothing was found, not that nothing was there. Although the fact that RTS show less lag than FPS and the network traffic information shown lends credibility to their position, they didn't run enough experiments.

David: I also found the study's examination of user opinion was rather rudimentary and did not really examine opinions. Users can learn to play the game with the lag and can work around the lag. This does not mean that they enjoyed playing the game as much. Maybe they were “good” subjects and tried to be positive. More experimentation is definitely required.

Buck: Another point about the effects of lag is that game designers can specifically work around this and design to reduce lag problems. It is conceivable that if they really wanted a game to test strategy rather than twitch response lag could be further reduced. Even if lag plays an effect right now it might be reduced later.

Mike: The three things tested in the study: combat, exploration and building. It seems like it would be interesting if they tested micromanaging tasks vs. macro managing tasks. It would be fairly obvious that perceived lag for micromanaging would be higher. This is shown on the exploration task but the way they rigged construction and combat is to “just go”. With two balanced armies there is less strategy.

Trent: The Path finding AI is pretty good. You can click on a target in the black and just let the character move. The exploration time should not have been effected by lag if players did that.

Warren: Variability could be due to the path finding AI. If it fails, then the exploration fails and that would show in the results.

David: That's a good point. With severe lag you might just rely on the AI to handle actions. It may not be perfect but it does a pretty good job.

PowerPoint Slides from the Presentation

References and Further Reading:

1. Warcraft III Home Page: http://www.blizzard.com/war3/

2. Mark Claypool's (one of the authors) Home Page: http://www.cs.wpi.edu/~claypool

3. IGN Message Board Discussion about Warcraft III Lag and the minimum requirements to play on-line: http://boards.ign.com/Warcraft/b5369/53547179/?13/

4. A Warcraft II Discussion about Lag (suggesting that lag may be important for other RTS games): http://artho.com/warcraft/lag.html

5. A series of slides and notes that may be interesting but were not used in the presentation

http://www.cs.wpi.edu/~claypool/papers/war3/