I don’t usually repost WG stuff “as is”, but this shouldn’t “disappear”, because it’s a nice explanation of what happened and happens even now. So, what do you think about this?
The last couple of weekends have presented both you and us with some very high packet loss and latency issues while playing World of Tanks in the late afternoon and evening. In response to these problems, our engineers have been working around the clock and we have put in place several measures to increase the network performance at our end. We understand how disruptive these issues have been for your gameplay and we would like to apologise for the inconvenience caused to you!
We would like you to know more about the root causes we have identified and are currently addressing.
- During our investigations into how to optimise our internal network performance, we identified a piece of network hardware that has been causing some degradation (packet loss) during times of high traffic throughput. This issue was isolated and fixed last week.
- The second problem is unfortunately not fully under our control and it is also fairly technical. I apologise in advance if this description is a bit too technical, but we want to give you the full picture of the situation.
- Internet traffic volumes in Europe have been elevated during the same time period that we have monitored latency issues for a large number of our players. Partially, this traffic increase is coming from an unusually high volume of streaming content due to the Olympic Winter Games. This volume increase is placing high demand on ISP peering exchange point capacities. Internet Service Providers (ISPs) have peering agreements and connection points between each other, outside of the major Internet backbones. Usually ISP-to-ISP peering helps reduce the load faced by the major backbones and accordingly provides a better flow of traffic between all parties. That being said, the capacities available for ISP peering are regulated both by agreements between the ISPs and physical limitations at the exchange points. If the demand exceeds the capacity, as it has several times in the last weeks for many of the major ISP peering points, the traffic must be rerouted via lower priority routes, which results in higher latency and packet loss. Cascading effects when several peering capacities are simultaneously exceeded across different ISPs are making this issue a very tough problem to solve for the ISPs.
We have been and are continuing to work very closely together with our ISP to address capacity shortages on these peering agreements during peak times. Since these negotiations are taking place between the ISPs, we are able to exert our influence, but we cannot directly resolve the problem.
- Thirdly and unfortunately again outside of our control, the European network infrastructure had to cope with the largest DDoS (Distributed Denial of Service) attack in history last week, reaching more than 400 Gigabit/s of additional malicious traffic. This caused a severe overload on Internet backbones in Frankfurt, Amsterdam and London as well as additional stress on the peering capacities between providers.
Our promise: Going forward we will be continuing to monitor our internal network and server performance and will swiftly take action on everything that we can directly affect. We will also continue to monitor the overall network traffic conditions and work with our partners to find better alternative routing solutions and increase peering capacities. Additionally, we will increase our level of communication on all the issues we are detecting in our internal networks, as well as any impairments we observe on the major Internet routes to our data centres, so that you are aware of what is going on.
What we cannot promise: While we are confident that our actions to address these problems will help to reduce the severity of the issues, unfortunately at this time we are not yet able to say that you won’t experience additional latency and packet loss issues during peak times. We are hopeful that the ISPs in Europe will address their peering capacity shortages swiftly and that we can give you the “all clear” very soon.
Director of Customer Relations Europe