About a year ago I published an article . It analyzed the pros and cons of different services for cloud gaming on weak PCs. During the game, I tested each service and shared my overall impression as a result.
In the comments to this and other similar articles, readers often shared their impressions about different gaming services. Often there were opposite opinions about the same thing. Everything is perfect for someone, but someone cannot play because of lags and friezes. Then I had the idea to evaluate the quality of these services in different conditions - from ideal to terrible. We are talking about the quality of networks, because not always the user can boast of a fast and hassle-free communication channel, right? In general, under the cut - an assessment of services with a simulation of different quality of network operation.
What is the problem at all?
As mentioned above - as a connection. Or rather, in the loss of packets during the game. The higher the losses, the more problems the gamer has, the less he is satisfied with the game. But rarely does anyone have an ideal communication channel like an optical fiber to the device, and with a dedicated, and not rummaged around for all the inhabitants of an apartment building, the Internet.
For reference, at a connection speed of 25 Mbps, 1-40 data packets are needed to transmit 50 frame / frame. The more packets are lost, the less quality the picture becomes, and the more noticeable lags and friezes. In especially severe cases, it becomes simply impossible to play.
Naturally, the cloud service itself cannot affect the width and stability of the user's channel in any way (although it would be great, of course). But on the other hand, different ways of leveling communication problems can be envisaged. Which services cope with the problem best of all - we will see below.
What exactly are we comparing?
Ordinary PC (Intel i3-8100, GTX 1060 6 GB, 8GB RAM), GeForce Now (its Russian version with servers in Moscow), , , , . On all services except Stadia, we study the quality of playing The Witcher. Google Stadia did not have this game at the time of writing, so I had to test another one - Odyssey.
What are the conditions and methods of testing?
Testing from Moscow. Provider - MGTS, tariff 500 Mbps, cable connection, not WiFi. We take the graphics quality settings in the services by default, the resolution is FullHD.
Using the program we simulate problems with the network, namely, packet losses of various types and volumes.
Uniform single losses. This is when only 1 packet is lost and the losses are more or less evenly distributed. So, a uniform loss of 10% means that for 100 packets, every 10th packet is lost, but always only 1 packet. The problem usually manifests itself in distortion (shielding) on the channel from the client to the server.
We test uniform losses of 5%, 10%, 25%.
Uneven mass loss, when at some one moment 40-70 packets are lost in a row. Such losses most often occur when there are problems with network equipment (routers, etc.) at the user or provider. May be related to buffer overflow of network equipment on the user-server communication line. Wi-Fi with thick walls can also cause such losses. Wireless network congestion due to the presence of a large number of devices is another reason that is very common in offices and apartment buildings.
We test uneven losses of 0,01%, 0,1%, 0,5%.
Below I analyze all these cases and attach a video comparison for clarity. And at the end of the article I give a link to raw, unedited gameplay videos from all services and cases - there you can see the artifacts in more detail, as well as technical information (in all services, except for Stadia, technical console data is recorded; Stadia did not find such).
Let's go!
Below are 7 stress test scenarios and videos with timestamps (the video is the same, for convenience, at each point, the viewing starts from the right moment). At the very end of the post are the source videos for each of the services. A good friend helped me make the video, for which I thank him!
Scenario #1. Ideal conditions. Zero network loss
Everything is as it should be in an ideal world. There are no communication problems, no breaks, no interference, your access point is the beacon of the Internet. In such greenhouse conditions, almost all test participants show themselves worthy.
PC
In each scenario, we took footage from the PC game as a reference. It is clear that the quality of the network does not affect it in any way, the game is launched locally on the PC. The presence of these frames answers the question “is there a difference when playing in the cloud compared to playing on your PC”. Under ideal conditions in our case, it is not felt in most services. We won’t write anything about the PC below, just remember that it exists.
GeForce Now
Everything is fine, the picture is clear, the process goes smoothly, without friezes.
V
Here Vortex spoils our ideal world. He immediately started having problems - the picture is worse than everyone else, plus the “brakes” are clearly visible. A possible problem is that the game servers are located far from Moscow, plus the hardware on the game servers seems to be weaker and FullHD exports poorly. In all tests, Vortex performed poorly. If someone has a positive experience of playing with Vortex - write in the comments, share what place you played from and how well everything turned out.
Playkey
Everything is fine, just like on a local PC. Visible problems like friezes, lags, etc. No.
loud play
The service shows an excellent picture, there are no visible problems.
Stadia
The game service from Google works perfectly despite the fact that it does not have servers in the Russian Federation, and in general Stadia does not officially work in Russia. However, all is well. It’s a pity, of course, that Stadia didn’t have The Witcher at the time of the game, but what to do, they took the Odyssey - also demanding, also about a man who cuts people and animals.
Scenario #2. Uniform loss 5%
In this test, out of 100 packets, approximately one in 20 is lost. Let me remind you that 40-50 packets are needed to render one frame.
GeForce Now
The service from Nvidia is doing well, there are no problems. The picture is a little more blurry than the Playkey, but The Witcher is still playable.
V
Here things got even worse. Why is not entirely clear, most likely, redundancy is not provided or it is minimal. Redundancy is Forward Error Correction (FEC). This technology recovers data in case of partial loss due to network problems. It can be implemented and configured in different ways, and judging by the results, the creators of Vortex did not succeed in this. Even with miserable uniform losses, it will not work to play. During subsequent tests, Vortex simply “died”.
Playkey
Everything is fine, there is not much difference with ideal conditions. Perhaps it helps that the company's servers are also located in Moscow, where the tests were carried out. Well, perhaps the aforementioned redundancy is better configured.
loud play
The service abruptly became unplayable despite relatively low packet loss. What can be wrong? I will assume that Loudplay works with the TCP protocol. In this case, while there is no confirmation of receipt of the package, no other packages are sent, the system is waiting for confirmation of delivery. Accordingly, if the package is lost, there will be no confirmation of its delivery, new packages will not be sent, the picture will stand up, the end of the story.
But if you use UDP, then confirmation of receipt of the packet is not required. As far as I can tell, all other services except Loudplay use the UDP protocol. If this is not the case, please correct me in the comments.
Stadia
Everything is playable. Sometimes the picture is pixelated, there are minimal response delays. Perhaps, it does not perfectly work out noise-correcting coding, hence minor artifacts when playable in the whole stream.
Scenario #3. Uniform loss 10%
We lose every 10th packet per hundred. This is already a call for services. To deal effectively with such losses, technologists are needed to recover and/or resend the lost data.
GeForce Now
Geforce is experiencing slight drops in the quality of the video stream. As far as one can tell, GFN is responding to problems in the network, trying to reduce them. The service reduces the bitrate, that is, the number of bits for data transmission. So he tries to reduce the load on the insufficiently high-quality, in his opinion, network and maintain a stable connection. And there really are no questions about stability, but the quality of the video suffers significantly. We see a significant pixelization of the image. Well, since the simulation assumes a constant loss of 10% of packets, then lowering the bitrate does not really help, the situation does not return to normal.
In real life, the picture, most likely, will not be consistently bad, but floating. Increased losses - the image blurred; losses were reduced - the image returned to normal, and so on. The gaming experience is not good, of course.
Playkey
There are no special problems. Probably, the algorithm detects problems on the network, determines the level of losses and focuses more on redundancy, and not on bitrate reduction. It turns out that with 10% uniform loss, the picture quality practically does not change, the user is unlikely to notice such losses.
loud play
Unusable, it just won't start. During further tests, the situation repeated itself. As far as you can tell, this service does not adapt to network problems in any way. Perhaps it is the TCP protocol that is to blame. The slightest loss paralyzes the service completely. Not very practical in real life, of course.
V
Also big problems. It is impossible to play in such conditions, although the picture is still there and the character continues to run, however, in jerks. I think it's all about the same poorly implemented or missing redundancy. Packets are lost frequently and never recovered. As a result, the image quality degrades to an unplayable level.
Stadia
Unfortunately, everything is bad here. There is a break in the flow, due to which events on the screen occur in jerks, it is extremely difficult to play. It can be assumed that the problem appeared, as in the case of Vortex, due to minimal redundancy or its absence. I consulted with a couple of acquaintances who are "in the know", they stated that Stadia is most likely waiting for the frame to be fully assembled. Unlike GFN, it doesn't try to save the day by dropping the bitrate all over the place. As a result, there are no artifacts, but friezes and lags appear (GFN, on the contrary, has fewer friezes / lags, but the picture is completely unattractive due to the low bitrate).
The rest of the services also don't seem to wait for the frame to be fully assembled, replacing the missing part with a fragment of the old frame. This is a good solution, in most cases the user will not notice the catch (30+ frames change per second), although sometimes artifacts may occur.
Scenario #4. Uniform loss 25%
Every fourth packet is lost. It's getting scarier and more interesting. In general, with such a “leaky” connection, a normal game in the cloud is hardly possible. Although some participants in the comparison cope, albeit not perfectly.
GFN
The problems are already very visible. The picture is pixelated and blurry. You can still play, but this is not at all what GFN offered at the very beginning. And definitely not the way to play beautiful games. Beauty is no longer appreciated.
Playkey
The gameplay is going well. There is smoothness, although the picture suffers a little. By the way, at the top left are the numbers of how many lost packets are restored. As you can see, 96% of packages are restored.
loud play
Didn't start.
V
You can’t play even with a very strong desire, friezes (stopping the image, resuming the video stream from a new fragment) are even more noticeable.
Stadia
The service is practically unplayable. The reasons have already been mentioned above. It is waiting for the frame to be assembled, the redundancy is minimal, with such losses it is not enough.
Scenario #5. Uneven loss 0,01%.
For 10 packets, 000-1 packets in a row are lost 40 time. That is, we lose about 70 out of 1 frames. It happens if the buffer of the network device is full and all new packets are simply discarded (dropped) until the buffer is freed. All participants in the comparison, except for Loudplay, worked out such losses to one degree or another.
GFN
The picture lost a little quality, became somewhat unclear, but everything is quite playable.
Playkey
Everything is very good. The picture is smooth, the image is good. You can play without problems.
loud play
The first few seconds the picture was, the hero even ran. But the connection with the server was lost almost immediately. Oh, this TCP protocol. The first loss cut down the service in the bud.
V
There are common problems. Friezes, lags and that's it. It would be very difficult to play under such conditions.
Stadia
Playable. Small drawdowns are noticeable, the picture is sometimes pixelated.
Scenario #6. Uneven loss 0,1%
For 10 packets, 000-10 packets are lost 40 times in a row. It turns out that we lose 70 out of 10 frames.
I must say right away that noticeable problems have appeared in most services. For example, the picture twitches, so redundancy does not help here. That is, there is a positive effect in the case of using redundancy technology, but it is small.
The fact is that the reaction time to the actions of the user and the game itself is limited, the video stream must be continuous. It is impossible to restore the stream to an acceptable quality despite any efforts of the services.
Artifacts appear (an attempt to compensate for packet loss, there is not enough data) and image jerks.
GFN
The quality of the picture has noticeably dropped, the bitrate has clearly been lowered, and quite significantly.
Playkey
Copes better - probably because the redundancy is well configured, plus the bitrate algorithm considers the losses not very high and does not turn the picture into a pixelated mess.
loud play
Didn't start.
V
It started, but with terrible picture quality. Very noticeable jerks and drawdowns. It is hardly possible to play under such conditions.
Stadia
Jerks are clearly visible, this is a clear indicator that there is not enough redundancy. The picture freezes, then other frames appear, the video stream breaks. In principle, you can play if you have a great desire and a clinical tendency to self-torture.
Scenario #7. Uneven loss 0,5%
For 10 packets, 000-50 packets in a row are lost 40 times. We lose 70 frames out of 50.
The situation of the class “shaped pi *** c”. Your router is sparking, your ISP is out of order, your wires are gnawed by mice, but you still want to play in the cloud. What service should you choose?
GFN
It is already very difficult to play, if at all possible - the bitrate is very much lowered. Frames are lost, instead of a normal picture we see “soap”. Frames are not restored - there is not enough information to restore. If GFN has recovery at all. The way the service aggressively tries to save the day with bitrate raises doubts about its readiness to work with redundancy.
Playkey
There is a distortion of the frame, the image twitches, that is, the elements of individual frames are repeated. It can be seen that most of the “broken” frame was restored from pieces of the previous one. That is, in the new frames there are parts of the old frames. But the image is more or less clear. You can manage, but in dynamic scenes, for example, in a fight where a good reaction is needed, it is difficult.
loud play
Didn't start.
V
It started, but it would be better if it didn’t start - you can’t play it.
Stadia
Service in such conditions is unplayable. The reasons are the need to wait for the frame assembly and weak redundancy.
Who is the winner?
The rating is, of course, subjective. You can argue in the comments. Well, the first place, of course, belongs to the local PC. It is precisely because cloud services are extremely sensitive to network quality, and the quality is rather unstable in the real world, that your own gaming PC remains out of competition. But if for some reason it is not there, then look at the rating.
- Local PC. expected.
- Playkey
- GeForce Now
- Google Stadia
- V
- loud play
As a conclusion, let me remind you once again what plays a major role in cloud gaming in terms of resilience to network problems:
- What network protocol is being used. It is best to use UDP for video streaming. I suspect that Loudplay uses TCP, although I don't know for sure. But you've seen the test results.
- Whether noise-correcting coding is implemented (FEC - Forward Error Correction, it is also redundancy). Also important is how it adjusts for packet loss. As we have seen, the quality of the picture depends significantly on the implementation.
- How bitrate adaptation is configured. If the service saves the situation primarily by bitrate, it has a stronger effect on the picture. The key to success is the delicate balance between bitrate manipulation and redundancy.
- How is post-processing set up? If problems arise, the frames are either dropped, or restored, or reassembled with fragments of old frames.
- The proximity of the servers to the gamer and the power of the iron also significantly affect the quality of the game, but this is also true for an ideal network. If the ping to the servers is too high, you won't be able to play comfortably even on an ideal network. We did not experiment with ping in this study.
As promised, here is the link to.
Source: habr.com