{"id":75970,"date":"2020-03-30T01:42:37","date_gmt":"2020-03-29T23:42:37","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/oblachnye-igry-stress-test-5-oblachnyh-igrovyh-servisov-plohim-internetom"},"modified":"2020-03-30T01:42:37","modified_gmt":"2020-03-29T23:42:37","slug":"oblachnye-igry-stress-test-5-oblachnyh-igrovyh-servisov-plohim-internetom","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/oblachnye-igry-stress-test-5-oblachnyh-igrovyh-servisov-plohim-internetom","title":{"rendered":"Cloud Gaming: Stress Test of 5 Cloud Gaming Services with Poor Internet","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Cloud
\n
\nAbout a year ago, I published an article \u2018Cloud Gaming: An Assessment of Service Capabilities for Playing on Weak PCs from First-Hand Experience\u2019<\/a><\/noindex>. It analyzed the pros and cons of different cloud gaming services for weak PCs. During my gaming sessions, I tested each service and ultimately shared my overall impressions.<\/p>\n

In the comments to that and other similar articles, readers often shared their impressions of various gaming services. Contradictory opinions about the same service were common. Some had a flawless experience, while others struggled with lags and freezes. This prompted me to evaluate the quality of these services under various conditions\u2014from ideal to terrible. The focus is on network quality, as not every user can boast of a fast and trouble-free connection, right? In general, below is an assessment of the services simulating different network quality.
\n<\/a><\/noindex><\/p>\n

What\u2019s the problem?<\/h3>\n

\nAs mentioned earlier, it\u2019s the quality of the connection. More specifically, it\u2019s about packet loss during gameplay. The higher the loss, the more problems gamers experience, and the less satisfied they are. Rarely does anyone have an ideal connection, such as fiber optics directly to their device, especially with a dedicated internet line rather than a shared one in an apartment building.<\/p>\n

For reference, at a connection speed of 25 Mbps, sending 1 frame requires 40-50 data packets. The more packets are lost, the poorer the image quality becomes, and the more noticeable the lags and freezes are. In particularly severe cases, playing becomes simply impossible.<\/p>\n

Naturally, the cloud service itself cannot influence the width and stability of the user\u2019s connection (although that would be fantastic, of course). However, different ways to mitigate connection issues can be anticipated. Which services handle this problem best will be revealed below.<\/p>\n

What exactly are we comparing?<\/h3>\n

\nA regular PC (Intel i3-8100, GTX 1060 6 GB, 8GB RAM), GeForce Now (its Russian version GFN<\/a><\/noindex> with servers in Moscow), Loudplay<\/a><\/noindex>, Vortex<\/a><\/noindex>, Playkey<\/a><\/noindex>, Stadia<\/a><\/noindex>. On all services, except for Stadia, we study the gaming quality in \u2018The Witcher\u2019. Google Stadia did not have this game at the time of writing, so I had to test another one\u2014\u2018Odyssey\u2019. <\/p>\n

What are the conditions and methodology for testing?<\/h3>\n

\nTesting from Moscow. Provider \u2014 MGTS, rate 500 Mbps, wired connection, not WiFi. We take quality settings in the services by default, resolution \u2014 FullHD. <\/p>\n

Using the program Clumsy<\/a><\/noindex> we model network issues, namely, packet loss of various types and volumes. <\/p>\n

Uniform single losses. <\/b>This occurs when only 1 packet is lost and losses are distributed fairly evenly. Thus, uniform losses of 10% mean that out of 100 packets, every 10th is lost, but always just 1 packet. The problem usually manifests with distortions (screening) on the channel from the client to the server. <\/p>\n

We test uniform losses of 5%, 10%, 25%.<\/p>\n

Non-uniform mass losses<\/b>, when at one point 40-70 packets are consecutively lost. Such losses most often occur due to problems with network equipment (routers, etc.) at the user\u2019s or provider\u2019s end. They can be related to buffer overflow of the network equipment on the communication line \"user \u2014 server\". WiFi with thick walls can also cause such losses. Wireless network congestion due to a large number of devices is another common reason, very characteristic for offices and apartment buildings.<\/p>\n

We test non-uniform losses of 0.01%, 0.1%, 0.5%.<\/p>\n

Below, I analyze all these cases and provide video comparisons for clarity. At the end of the article, I give a link to the raw, unedited gameplay videos from all services and cases \u2014 there you can take a closer look at the artifacts, as well as the technical information (in all services except Stadia, technical console data has been recorded; I couldn't find such for Stadia). <\/p>\n

Let's go!<\/h3>\n

\nBelow are 7 stress test scenarios and videos with timestamps (the videos are the same, for convenience, playback starts at the necessary moment in each point). At the very end of the post are the original clips for each of the services. A good friend helped me make the video, for which I am thankful!<\/p>\n

Scenario #1. Ideal conditions. Zero losses in the network<\/h4>\n

\nEverything is as it should be in an ideal world. There are no communication issues, no breaks, no interference, your access point is a beacon of the internet. In such greenhouse conditions, almost all participants in the testing perform admirably.<\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center>
\nPC<\/b><\/p>\n

In each scenario, we took frames from the PC game as a benchmark. It's clear that network quality does not affect it since the game runs locally on the PC. The presence of these frames answers the question of whether there is a noticeable difference when playing in the cloud compared to playing on one's own PC. Under ideal conditions in our case, most services do not show a difference. We won\u2019t write anything more about the PC; just remember that it exists.<\/p>\n

GeForce Now<\/b><\/p>\n

Everything is good, the picture is clear, and the process runs smoothly without any freezes. <\/p>\n

Vortex<\/b><\/p>\n

Here Vortex disrupts our perfect world. It immediately started having issues \u2014 the picture is worse than all the others, plus noticeable 'lags'. The possible problem is that the game servers are located far from Moscow and the hardware on the game servers seems to be weaker, which poorly handles Full HD. In all tests, Vortex performed poorly. If anyone has positive experiences playing with Vortex \u2014 please share in the comments where you played and how well it went.<\/p>\n

Playkey<\/b><\/p>\n

Everything is great, just like on a local PC. There are no visible issues like freezes or lags.<\/p>\n

Loudplay<\/b><\/p>\n

The service provides an excellent picture, with no visible problems.<\/p>\n

Stadia<\/b><\/p>\n

Google's gaming service works excellently despite having no servers in Russia, and officially Stadia doesn't operate in the country. Nevertheless, everything is fine. It's a pity that 'The Witcher' wasn't available on Stadia at the time of playing, but we opted for 'Odyssey' \u2014 also demanding, and also about a man who cuts down people and animals. <\/p>\n

Scenario #2. Uniform losses of 5%<\/h4>\n

\nIn this test, around every 20th packet is lost out of 100 packets. I remind you that it takes 40-50 packets to render one frame.<\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center>
\nGeForce Now<\/b><\/p>\n

Nvidia's service is good, with no issues. The picture is slightly more blurred than Playkey's, but 'The Witcher' remains playable.<\/p>\n

Vortex<\/b><\/p>\n

Things got even worse here. Why \u2014 it's not entirely clear, likely because redundancy is not provided for or is minimal. Redundancy is the error-correcting coding of transmitted data (FEC \u2014 Forward Error Correction). This technology restores data during partial loss due to network issues. It can be implemented and configured in various ways, and judging by the results, Vortex's creators have not succeeded in this. Even with minimal uniform losses, it won't be possible to play. In subsequent tests, Vortex simply 'died'.<\/p>\n

Playkey<\/b><\/p>\n

Everything is fine; there is no significant difference compared to ideal conditions. It may help that the company's servers are also located in Moscow, from where the tests were conducted. Moreover, the aforementioned redundancy might be better configured. <\/p>\n

Loudplay<\/b><\/p>\n

The service has suddenly become unplayable, even with relatively low packet loss. What could be the issue? I suspect that Loudplay operates with the TCP protocol. In this case, until packet receipt is confirmed, no other packets are sent; the system waits for a delivery confirmation. Accordingly, if a packet is lost, there will be no delivery confirmation, no new packets will be sent, the image will freeze, and that's the end of the story. <\/p>\n

However, if UDP is used, a receipt confirmation for the packet is not necessary. As far as I can tell, all other services, except for Loudplay, use the UDP protocol. If that's not the case, please correct me in the comments.<\/p>\n

Stadia<\/b><\/p>\n

Everything is playable. Sometimes the image pixelates, and there are minimal response delays. It seems that the error correction coding might not be performing ideally, leading to slight artifacts in an otherwise playable stream.<\/p>\n

Scenario #3. Uniform 10% losses<\/h4>\n

\nWe lose every 10th packet out of a hundred. This is already a challenge for the services. To manage such losses effectively, recovery technologists and\/or resend mechanisms for lost data are needed.<\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center>
\nGeForce Now<\/b><\/p>\n

Geforce experiences slight drops in the quality of the video stream. As far as I can tell, GFN responds to network issues, trying to mitigate them. The service reduces the bitrate, which is the amount of bits used for data transmission. This is how it attempts to lower the load on a network it deems insufficiently quality, while maintaining stable connectivity. And there are indeed no issues with stability, but the video quality suffers noticeably. We see significant pixelation in the image. Since the modeling assumes a constant loss of 10% packets, reducing the bitrate doesn't help much; the situation doesn't normalize. <\/p>\n

In real life, the image is likely to be not constantly bad, but fluctuating. When losses increase, the image becomes blurry; when losses decrease, the image returns to normal, and so on. This is certainly not beneficial for the gaming experience. <\/p>\n

Playkey<\/b><\/p>\n

There are no major issues. The algorithm probably detects network problems, assesses the level of packet loss, and focuses more on redundancy rather than on reducing bitrate. As a result, with 10% uniform loss, the image quality hardly changes, and users are unlikely to notice such losses.<\/p>\n

Loudplay<\/b><\/p>\n

Non-functional, it simply did not start. During further tests, the situation repeated itself. As far as can be judged, this service does not adapt to network issues. It may be due to the TCP protocol. Even slight losses completely paralyze the service. Not very practical for real life, of course.<\/p>\n

Vortex<\/b><\/p>\n

There are also significant problems. Playing under such conditions is impossible, although the image is still present and the character continues to move, albeit in jerks. I think the issue lies in poorly implemented or absent redundancy. Packets are frequently lost, and there is no recovery. Ultimately, the image quality degrades to an unplayable level.<\/p>\n

Stadia<\/b><\/p>\n

Unfortunately, everything is bad here. There are stream interruptions, causing events on the screen to occur in jerks, making it extremely difficult to play. It can be assumed that the problem arose, as in the case with Vortex, due to minimal redundancy or its absence. I consulted a couple of acquaintances who are 'in the know'; they stated that Stadia likely waits for the complete frame assembly. Unlike GFN, it does not attempt to salvage the situation with a total bitrate reduction. As a result, there are no artifacts, but freezes and lags appear (GFN, on the contrary, has fewer freezes\/lags, but due to low bitrate, the image is quite unappealing). <\/p>\n

Other services also seem not to wait for the complete frame assembly, replacing the lost part with a segment from the previous frame. This is a good solution; in most cases, the user won't notice the catch (30+ frames change per second), although artifacts may sometimes occur.<\/p>\n

Scenario #4. Uniform losses of 25%<\/h4>\n

\nEvery fourth packet is lost. It becomes increasingly scary and interesting. Generally, under such a 'leaky' connection, normal cloud gaming is hardly possible. Although some participants in the comparison manage, albeit not perfectly. <\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center>
\n GFN<\/b><\/p>\n

The issues are already quite noticeable. The image is pixelated and blurry. It\u2019s still possible to play, but it\u2019s definitely not what GFN offered at the beginning. And it\u2019s certainly not how you should play beautiful games. The beauty can no longer be appreciated. <\/p>\n

Playkey<\/b><\/p>\n

The gameplay is going reasonably well. There is some smoothness, although the image suffers a bit. By the way, in the top left corner \u2014 the numbers indicate how many lost packets are being recovered. As we can see, 96% of packets are being recovered.<\/p>\n

Loudplay<\/b><\/p>\n

Did not start.<\/p>\n

Vortex<\/b><\/p>\n

Playing is impossible even with a great desire; freezes (image pauses, resuming the video stream from a new fragment) are even more noticeable. <\/p>\n

Stadia<\/b><\/p>\n

The service is practically unplayable. The reasons have been mentioned above. It waits for frame assembly; the redundancy is minimal, and with such losses, it\u2019s insufficient. <\/p>\n

Scenario No. 5. Uneven loss of 0.01%.<\/h4>\n

\nOut of 10,000 packets, 40-70 packets are lost in a row once. This means we\u2019re losing about 1 out of 200 frames. This can happen if the network device buffer is full and all new packets are simply discarded (dropped) until the buffer is cleared. All comparison participants, except Loudplay, have managed such losses to some extent. <\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center>
\nGFN<\/b><\/p>\n

The image has lost a bit of quality, becoming somewhat hazy, but it\u2019s still quite playable.<\/p>\n

Playkey<\/b><\/p>\n

Everything is quite good. The image is smooth, and the quality is decent. You can play without issue.<\/p>\n

Loudplay<\/b><\/p>\n

In the first few seconds, the image was visible, and the hero even started running. But the connection to the server was almost immediately lost. Oh, this TCP protocol. The first loss completely crippled the service. <\/p>\n

Vortex<\/b><\/p>\n

Normal problems are observed. Freezes, lags, and all that. It would be very difficult to play under these conditions.<\/p>\n

Stadia<\/b><\/p>\n

Playable. There are minor dips, and the image sometimes pixelates.<\/p>\n

Scenario No. 6. Uneven loss of 0.1%<\/h4>\n

\nOut of 10,000 packets, 40-70 packets are lost in a row 10 times. So, we lose 10 out of 200 frames.<\/p>\n

I\u2019ll say right away that noticeable problems have arisen for most services. For example, the image stutters, so redundancy doesn\u2019t help here. That means there is a positive effect when using redundancy technology, but it\u2019s not significant.<\/p>\n

The thing is, the reaction time to user actions and the game itself is limited; the video stream must be continuous. It\u2019s impossible to restore the stream to an acceptable quality despite any efforts of the services. <\/p>\n

Artifacts appear (attempts to compensate for packet loss, not enough data) and there are image stutters.<\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center>
\nGFN<\/b><\/p>\n

The image quality has noticeably dropped, bit rate has clearly decreased, and quite significantly.<\/p>\n

Playkey<\/b><\/p>\n

It performs better \u2014 likely because redundancy is well-tuned, plus the bit rate algorithm considers losses not too high and doesn\u2019t turn the image into a pixelated mess.<\/p>\n

Loudplay<\/b><\/p>\n

Did not start.<\/p>\n

Vortex<\/b><\/p>\n

Launched, but with terrible image quality. Stutters and drops are very noticeable. Playing under such conditions is unlikely.<\/p>\n

Stadia<\/b><\/p>\n

Stutters are clearly visible, which indicates that there is not enough redundancy. The image freezes, then new frames appear, causing a video stream break. Playing is possible, but only if one really wants to and has a clinical tendency for self-torture.<\/p>\n

Scenario No. 7. Uneven losses of 0.5%.<\/h4>\n

\nOut of 10,000 packets, 40-70 packets are lost consecutively 50 times. We lose 50 frames out of 200.<\/p>\n

This is a situation of 'formal disaster'. Your router is glitching, the provider has an outage, wires have been chewed by mice, yet you still want to game in the cloud. What service should you choose? <\/p>\n

\r\n \"Play\r\n
<\/div>\r\n <\/div><\/center>
\nGFN<\/b><\/p>\n

Playing is already very difficult, if not impossible \u2014 the bit rate is significantly lower. Frames are lost, instead of a normal picture, we see 'mud'. Frames cannot be restored \u2014 there\u2019s not enough information to restore. If GFN even has recovery available at all. The aggressive way the service tries to manage the situation with the bit rate raises doubts about its readiness to handle redundancy. <\/p>\n

Playkey<\/b><\/p>\n

There\u2019s frame distortion, the image stutters, meaning elements from individual frames repeat. It's clear that a large part of the 'damaged' frame was restored from segments of previous ones. So new frames contain parts of old frames. But the image is fairly clear overall. Control is possible, but in dynamic scenes, like a fight where good reaction is needed \u2014 it\u2019s hard.<\/p>\n

Loudplay<\/b><\/p>\n

Did not start.<\/p>\n

Vortex<\/b><\/p>\n

Launched, but it would have been better not to launch at all \u2014 it\u2019s unplayable.<\/p>\n

Stadia<\/b><\/p>\n

The service is unplayable under such conditions. Reasons include the need to wait for frame assembly and weak redundancy. <\/p>\n

Who is the winner?<\/h3>\n

\nThe rating is, of course, subjective. You can argue in the comments. And the top spot, of course, goes to the local PC. This is primarily because cloud services are extremely sensitive to network quality, and that quality tends to be quite unstable in the real world, making a personal gaming PC unmatched. However, if for some reason you don't have one, check the rating.<\/p>\n

    \n
  1. Local PC. As expected.<\/li>\n
  2. Playkey<\/li>\n
  3. GeForce Now<\/li>\n
  4. Google Stadia<\/li>\n
  5. Vortex<\/li>\n
  6. Loudplay<\/li>\n<\/ol>\n

    \nAs a conclusion, let me remind you that the main factor in cloud gaming in terms of resilience to network issues is:<\/p>\n