Wargaming Talks About Servers

Posted on by

Source: http://worldoftanks.ru/ru/news/pc-browser/1/data_center_world_of_tanks/

Hello everyone,

another interesting article, that did not make it to WG EU, appeared on WG RU server. In it, Wargaming explains how the servers work. Here’s what’s in it.

How it started

When the game started (back in 2011 or so), all the things related to the game (the game itself, forums, player info etc.) were processed by one big server cluster. A cluster of servers is a large group of computers, communicating with each other, acting as one single server (resource). The very first server cluster was in Munich and in 2011, it was moved to Russia. This reduced the ping for RU players significantly. It had one big disadvantage – since there was only one server cluster, that had to be restarted several times per week, when the cluster was down, not a single player could enter the game.

In order to fix that, in 0.7.0, the clusters were moved to multicluster technology. The structure changed and the cluster was split to “center” and “periphery”. “Center” is the database, in which all the user data is stored. “Periphery” is the rest, all the servers on which the players play actually (SS: for our purposes, a periphery is different word for various servers of the cluster, like EU1, EU2). At this point, all the games are running on 9 peripheries. All the players are playing on the periphery servers, nobody is playing on the center one, but it is the center that is operating the periphery. If the center is not working, the players cannot enter a game, but they can continue playing if they already were in it when it crashed.

Where are the servers

Servers are housed in specialized buildings called “data centers”. The closer a data center is to the player, the better. That’s why the WG servers are spread all across the game regions:

- 3 datacenters are in Moscow (one of them is running WoT, WoWs and WoWp at once)
- two other remaining Moscow sites are running four peripheries
- RU4 and RU8 are in Novosibirsk and Krasnoyarsk
- in Europe, there are three data centers – in Amsterdam, Frankfurt and Louxembourg
- US servers are on west and east coast
- in Asia, there is a Korean cluster and one cluster in Singapore
- Chinese cluster is not operated by Wargaming, but by partners

The server structure is very flexible and easy to change. From the administrator POV, it doesn’t matter what content the server is running (tanks, ships, planes).

How many computers are needed for everyone to play

When WoT started with just one cluster, the maximum amount of people online was 250-270k. The cluster consisted of cca 100 servers, that acted as one. This however did not strain just the servers, but the provider network as well. With 250k players, the server traffic is cca 6-7 Gbit/sec (which means that a normal movie 1,4 GB big travels there each 2 seconds).

In the current periphery system, there are no such limits and the server issues are being handled on individual basis. One single physical server can be used by cca 2,5 to 3 thousand users and has a peak traffic of 60-70 Mbit/s. The usual cluster has around 50 to 80 servers and some periferies allow the operators to add more servers to them if needed, but usually the server racks are used to their maximum capacity as it is.

How do the users recieve the game updates?

Patches are assembled by especially trained people, called “build engineers” for each game update. A patch is a file, that changes and updates whatever part of the software on user’s side. In order for the users to download the patch, it is moved to a place called CDN (“Content Delivery Network”) and then distributed to company servers. If the patch is larger than 30 MB, a torrent file is created in order to increase the efficiency of distribution – this torretn file also waits for the user’s query on CDN. Whenever the user runs the game launcher, the laucher asks the update server and if available, recieves the torrent file or a link to the patch (in case torrenting is disabled in the launcher).

How do the users get in the game?

Any user from any region can connect to any server he wants. If the user was authorized in the RU region, he can connect to any periphery. By default, the automatic selection of a periphery is enabled in the settings and during the authorization process, the request goes through a complicated algorhitm, that doesn’t take only the user ping into account, but also the overall load of the server. If the server X in Russia is loaded more than server Y, the user will be sent to server Y. In general, the system can send him to both Amsterdam and Novosibirsk, but the user can also select the server manually.

Where’s the hangar, where’s the battle and what happens if you press the “Battle” button?

There are servers, running the game processes, running the hangar and the login. And then there are servers, that do not work with users, but work with system information. For example, when the user starts the launcher, he gets into the login process. The authorization requests travels internally to the single authorization center, that confirms the username and password and that the player is genuine and can enter the game. After the authorization, the user gets transferred to the hangar and his data will be transferred to another process. This means that when a server crashes, a player might not even notice and can finish his battle. For example, when the servers are loaded a lot (more than 100k players online), a cluster can crash and in such cases, it requires a restart. But if this happens only to one physical server, the rest of the cluster will continue to work normally and thanks to that, the player can finish the game.

The worst that can happen is when the server that authorized the player himself crashes. Whenever that happens, when trying to enter the game, the player will recieve a message that the was transferred to another periphery.

This issue can only be fixed by restarting the server, as there’s a notice in the database that the player got stuck on a periphery in such and such game process, which is running on a server, that crashed. Until this notice gets removed, the player can’t move anywhere else. The trick is to do this in favour of the user.

It’s technically possible to “let” players move to another periphery when the account gets stuck, but this can lead to a loss of two hour previous progress on the “crashed” periphery, because the account data get saved to the database every two hours. In such a case, it’s better to wait than to lose some rare medal or achievement you just got.

In order to assure safe and smooth operation of the network and server equipment, both data redundancy and backups are widely used. For example, the doubled hard drives working as one have one advantage: if one fails, the server can still access the data and the game will keep on working. Every server has four network cards, paired in twos: one for connecting between the servers within the cluster, one for connecting to the players. New data centers have servers with redundant power sources, connected according to the data center means to energy supplies from different suppliers (whenever possible). Internet connection channels are provided by independent providers and are connected physically to the data center by dedicated pathways. Two – three years ago, it was impossible not to notice a technical failure, but right now, an issue might arise and be fixed without players noticing anything.

52 thoughts on “Wargaming Talks About Servers”

  4. “The very first server cluster was in Munich and in 2011, it was moved to Russia. This reduced the pink for RU players significantly.”

    Surely, that should be

    “The very first server cluster was in Munich and in 2011, it was moved to Russia. This reduced the pinG for RU players significantly.”

    Although “potting the pink” is more fun than “potting the brown”…..

  7. Wireshark showed me connected to an IP in Wichita, KS for USEast–anyone else? I wouldn’t exactly call that ‘east coast’ if that is indeed where the server is co-located. Explains why I get 80ms ping from the actual East Coast.

    • still think they should stick a server in Texas, Northern Mexico or New Mexico would be a decent idea.

    • The server is in Ashburn, Virginia – just to the west of Metro DC. I live about 7 miles (as the crow flies) from that server so I can tell you the line from above

      “The closer a data center is to the player, the better. ”

      Is complete bullshit. Peering agreements between yours and WG’s ISPs determine latency and packet loss.

  8. (The closer a data center is to the player, the better.)

    I have often wondered just How much of a factor this plays

    • Actually, a lot.
      I usually have 30-40 ping from Hungary on EU1. (Decent 20/20 cable net, gaming comp.)
      In a hotelroom (!) in Den Haag, Netherlands I had some unreal 4-6 ping using my mediocre laptop. I could barely believe it.

    • Last days I have way better ping to the amsterdam server than on Frankfurt server and I live in frankfurt…

      about 1-2 months ago all was okay but now my ping is higher on Frankfurt server.

  9. I play on the Russian server from America. I mostly use RU4, which is on the Amsterdam data center. I get an average ping of 130, which is good enough to not be an issue.

  10. “It’s technically possible to “let” players move to another periphery when the account gets stuck, but this can lead to a loss of two hour previous progress on the “crashed” periphery, because the account data get saved to the database every two hours.”

    If I play a few games on EU1 and then change to EU2 (without logging out), does that mean whatever game data on EU1 is ‘lost’ ?

  11. NA’s data centers are in the dumbest places tactically relevant to the NA market.

    We have people who play from Central and South America, as well as Alaska, Northern Canada, Hawaii, and some still play from SEA or Australia.

    Most MMO’s in this area have data centers hosted in Chicago, IL and Dallas, TX – you know… Geographically more centered to the North American Continent. By having them here, they’re less likely to get routed through Cambodia like some ping-plots have been showing going to NA West – and imagine that … average ping is more homogeneous across the United States. People playing in Central America/Mexico have tolerable pings connecting to Dallas’ data centers, and South America’s connection is slightly better.

    Fact of the matter is … WGNA were tactical-retards when it came to where to host their servers here in Murrca. They could have simply followed the successful network planning infrastructures of many popular games before it that are currently running today; but … as is with the Russian way … they had to *think* they could do it better, and instead stuck us with TWO retarded hosting services (Telia/Level 3).

  12. Ru Server = more 300k players online – stable
    EU Server = almost 300k players online – stable
    NA Server = 20k player in ES + 7k player in WS – Loss package and 300ms ping
    ….

  13. Fucking Korean Server Ping

    at peak times, The maximum number of concurrent users are about 6000 people.

    but sometimes goes up to several hundred ms ping.

    Most KR Server users are sympathetic to KR server does not have a stable ping.

    • They didn’t implement it because they knew we would tease their “deers” with this;

      “SLAVA UKRAINI!!!”

      It would have been translated to Russian, of course.

  15. Now I know based on previous experience why I can’t login after my game crashed after battle/loading the game but works pretty well on other players, well better to wait than roll back my experienced earned in past 2 hours.

  16. Their servers are always developing but my issues with ping are just really odd. Over the last couple of months my in game ping and random packet loss has made the game on EU pretty much unplayable (it works sort of hovering between 150-200, but frequent multi second hangups and the game’s automatic ‘carry on last command sent’ will kill you stupidly in more or less every game), yet when I manaully ping the servers I get virtually no loss and the same ping that I was used to (about 70).

  17. LOL ahh yes the superior Telia network. The company of last packages and Super bad Ping rates yet Wargaming continually use them like a drunken sailor on shore leave in Amsterdam.