🥇Open-source cloud gaming on WebRTC: p2p, multiplayer, zero latency

Software as a service, infrastructure as a service, platform as a service, communication platform as a service, video conferencing as a service, but what about cloud gaming as a service? Several attempts have been made to create cloud gaming, for instance, Stadia, recently launched by Google. is no stranger to WebRTC, but can others use WebRTC in the same way?

Thanh Nguyen decided to explore this possibility in his open-source project CloudRetro. CloudRetro is based on Pion, a popular WebRTC library in Go (thanks to Shaun from the Pion development team for helping prepare this article). In this article, Thanh reviews the architecture of his project and shares insights he gained and the challenges he faced during work.

Introduction

Last year, when Google announced Stadia, I was blown away. The idea is so unique and innovative that I kept wondering how it was even possible with existing technologies. The desire to better understand this topic prompted me to create my version of open-source cloud gaming. The result was simply fantastic. Below, I would like to share my working process over the course of my year-long project.

TLDR: a brief slide version with key points

Why cloud gaming is the future

I believe that Cloud Gaming will soon become the next generation not only of gaming but also of other fields of computing. Cloud gaming is the pinnacle of the client/server model. This model maximizes backend control and minimizes frontend work by placing game logic on a remote server and streaming images/audio to the client. The server handles heavy processing, so the client is no longer constrained by hardware limitations.

Google Stadia essentially allows you to play AAA games (i.e., high-end blockbuster games) on an interface similar to YouTube. The same methodology can be applied to other heavy offline applications, such as operating systems or 2D/3D graphic design, etc., so we can consistently run them on low-spec devices across various platforms.

The future of this technology: imagine if Microsoft Windows 10 ran in the Chrome browser?

Cloud gaming is technically complex

Gaming is one of those rare fields where a constant quick response from the user is required. If we occasionally experience a 2-second delay when clicking on a page, that's acceptable. Live video streams typically lag behind by a few seconds, yet still offer sufficient usability. However, if a game frequently lags by 500 ms, playing becomes simply impossible. Our goal is to achieve extremely low latency, minimizing the gap between input and media as much as possible. Thus, the traditional approach to video streaming isn't applicable here.

General cloud gaming template

Open-source project CloudRetro

I decided to create a test sample of cloud gaming to check whether this is possible under such stringent network constraints. For the proof of concept, I chose Golang since it's the most familiar language to me and well-suited for this implementation for many other reasons that became apparent later. Go is simple and develops rapidly; channels in Go are excellent for managing concurrency.

Project CloudRetro.io – an open-source cloud gaming service for retro gaming. The project's goal is to bring the most comfortable gaming experience to traditional retro games and add multiplayer capabilities.
You can learn more about the project here: https://github.com/giongto35/cloud-game.

CloudRetro functionality

To demonstrate the full power of cloud gaming, retro games are used in CloudRetro. This allows for a multitude of unique gaming experiences.

Game portability
- Instant playback upon opening the page; no downloads or installations needed
- Works in mobile browsers, so no software installation is required to launch
Gaming sessions can be shared across multiple devices and stored in the cloud for the next login
The game can be streamed, or multiple users can play it instantly:
- Crowdplay similar to TwitchPlayPokemon, but more cross-platform and more real-time
- Offline games online. Many users can play without network setup. In Samurai Shodown, two players can now compete via CloudRetro
Demo version of multiplayer online game on different devices
Infrastructure
Requirements and tech stack
Below is a list of requirements that I set before starting the project.
1. One player
This requirement may seem unimportant and obvious here, but it is one of my key takeaways; it allows cloud gaming to stay as far away from traditional streaming services as possible. By focusing on single-player games, we can eliminate the need for a centralized server or CDN, because we don't need to stream to the masses. Instead of uploading streams to a consuming server or transmitting packets to a centralized WebSocket server, the service streams are delivered directly to the user via a peer-to-peer WebRTC connection.
2. Low-latency media stream
While reading about Stadia, I often come across mentions of WebRTC in some articles. I realized that WebRTC is an outstanding technology and it is perfectly suitable for use in cloud gaming. WebRTC is a project that enables real-time communication in web browsers and mobile applications through a simple API. It provides a peer-to-peer connection optimized for media and has built-in standard codecs like VP8 and H264.
I preferred ensuring maximum user comfort over maintaining high graphic quality. Some loss is permissible in the algorithm. Google Stadia includes an additional step to reduce image size on the server, and frames are scaled to higher quality before being transmitted to peer nodes.
3. Distributed infrastructure with geographic routing
Regardless of how optimized the compression algorithm and code are, the network remains the deciding factor that contributes most to latency. The architecture must have a mechanism for connecting to the nearest server to the user to reduce round-trip time (RTT). It should include one coordinator and several streaming servers distributed across the globe: West US, East US, Europe, Singapore, China. All streaming servers should be fully isolated. The system can adjust its distribution when a server joins or leaves the network. Thus, during heavy traffic, adding more servers allows for horizontal scaling.
4. Browser Compatibility
Cloud gaming shines best when it demands minimal from users. This means there’s the potential for browser-based play. Browsers help make the gaming experience as comfortable as possible for users by eliminating the need for software and hardware installations. They also assist in ensuring cross-platform compatibility for both mobile and desktop versions. Fortunately, WebRTC is well-supported across various browsers.
5. Clear Separation of Game Interface and Service
I view the cloud gaming service as a platform. Everyone should have the opportunity to plug anything into the platform. I have currently integrated LibRetro with the cloud gaming service because LibRetro offers a beautiful emulator interface for retro games such as SNES, GBA, and PS.
6. Rooms for Multiplayer, Crowd Play, and Deep-Linking to the Game
CloudRetro supports many new gameplay options, such as CrowdPlay and Online Multiplayer for retro games. If multiple users open the same deep link on different computers, they will see the same game running and can even join it.
Moreover, game states are stored in cloud storage. This allows users to continue their game at any time on any other device.
7. Horizontal Scaling
Like any SAAS today, cloud gaming must be designed to be horizontally scalable. The 'coordinator-worker' architecture allows for the addition of more workers to handle greater traffic.
8. No dependence on a single cloud
The CloudRetro infrastructure is hosted across various cloud providers (Digital Ocean, Alibaba, custom provider) for different regions. I activate the launch in a Docker container for the infrastructure and configure the network settings using a bash script to avoid dependency on a single cloud provider. By combining this with NAT Traversal in WebRTC, we gain the flexibility to deploy CloudRetro on any cloud platform and even on user machines.
Architectural Design
Worker: (or the streaming server mentioned above) multiplies games, runs the encoding pipeline, and delivers the encoded media to users. Worker instances are distributed globally, and each worker can handle multiple user sessions simultaneously.
Coordinator: is responsible for pairing new users with the most suitable worker for streaming. The coordinator interacts with workers via WebSocket.
Game State Storage: a central remote repository for all game states. This storage provides vital functions such as remote saving/loading.

High-level Architecture of CloudRetro
User Scenario
When a new user opens CloudRetro at steps 1 and 2 shown in the image below, the coordinator along with the list of available workers is requested on the first page. Next, at step 3, the client calculates the latencies for all candidates using an HTTP ping request. This latency list is then sent back to the coordinator, allowing it to determine the most suitable worker to service the user. At step 4 below, a game is created. A streaming connection using WebRTC is established between the user and the assigned worker.

User Scenario after Accessing
What's inside the worker
Gaming and streaming pipelines are stored isolated within the worker and exchange information there through an interface. Currently, this connection is achieved through data transfer in memory via Golang channels in the same process. The next goal is segregation, i.e., running the game independently in another process.

Interaction of worker components
Main components:
- WebRTC: the client component that receives user input and outputs encoded media from the server.
- Game emulator: the gaming component. Using the Libretro library, the system can run a game within the same process and internally intercept media and input streams.
- In-game frames are captured and sent to the encoder.
- Image/audio encoder: an encoding pipeline that takes media frames, encodes them in the background, and outputs encoded images/audio.
Implementation
CloudRetro relies on WebRTC as its backbone technology, so before delving into implementation details on Golang, I decided to talk about WebRTC itself. This amazing technology has greatly aided me in achieving a streaming latency of only a fraction of a second.
WebRTC
WebRTC is designed to provide high-quality peer-to-peer connections on native mobile applications and browsers using simple APIs.
NAT Traversal
WebRTC is known for its NAT Traversal functionality. WebRTC is intended for peer-to-peer communication. Its goal is to find the most suitable direct route, avoiding NAT gateways and firewalls for peer-to-peer connections through a process called ICE. Within this process, the WebRTC API discovers your public IP address using STUN servers and redirects it to a relay server (TURN), when a direct connection cannot be established.
However, CloudRetro does not fully utilize this capability. Its peer-to-peer connections exist not between users, but between users and cloud servers. The server side of the model has fewer restrictions on direct communication than a typical user device. This allows for pre-opening incoming ports or using public IP addresses directly, as the server is not behind NAT.
Previously, I wanted to transform the project into a game distribution platform for Cloud Gaming. The idea was to allow game creators to provide games and streaming resources, while users would interact directly with providers. In such a decentralized manner, CloudRetro serves merely as a medium to connect third-party streaming resources to users, making it more scalable since it no longer relies on hosting. The role of WebRTC NAT Traversal is crucial here to facilitate the initialization of peer-to-peer connections on third-party streaming resources, simplifying the connection of the creator to the network.
Video Compression
Video compression is an essential part of the pipeline that significantly contributes to stream smoothness. While it is not necessary to know all the details of encoding video in VP8/H264, understanding the concept helps to make sense of streaming video bitrate settings, troubleshoot unexpected behavior, and adjust latency.
Video compression for streaming services is a complex task because the algorithm must ensure that the total encoding time + network transmission time + decoding time is as low as possible. Additionally, the encoding process must be consistent and continuous. Certain trade-offs in encoding do not apply — for example, we cannot favor longer encoding times for smaller file sizes and decoding times, or use inconsistent compression.
The idea behind video compression is to eliminate unnecessary bits of information while maintaining an acceptable level of accuracy for users. Besides encoding individual static image frames, the algorithm derives the current frame from the previous and next frames, sending only their differences. As seen in the example with Pacman, only differential points are transmitted.

Comparison of Video Frames Featuring Pacman
Audio Compression
Similarly, the audio compression algorithm omits data that cannot be perceived by humans. Opus is currently the audio codec with the best performance. It is designed to transmit audio waveforms over a transport protocol such as RTP (Real-Time Transport Protocol). Its latency is lower than that of mp3 and aac, while offering higher quality. The latency typically ranges from about 5 to 66.5 ms.
Pion, WebRTC in Golang
Pion is an open-source project that brings WebRTC to Golang. Rather than simply wrapping native C++ WebRTC libraries, Pion is a native Golang implementation of WebRTC that offers better performance, integration with Go, and version control over WebRTC protocols.
The library also provides data streaming with a wealth of excellent built-in modules and latency of less than one second. It has its own implementations of STUN, DTLS, SCTP, etc., and some experiments with QUIC and WebAssembly. As an open-source library, it serves as a great learning resource with outstanding documentation, network protocol implementations, and cool examples.
The Pion community, led by a very passionate creator, is quite vibrant, with many quality discussions about WebRTC taking place. If you are interested in this technology, join the http://pion.ly/slack – you will learn a lot.
Writing CloudRetro in Golang

Implementing a worker in Go
Go channels in action
Thanks to the elegant design of Go channels, event streaming and concurrency issues are significantly simplified. As illustrated, multiple components operate in parallel across different GoRoutines. Each component manages its own state and communicates via channels. The Golang selective statement processes one atomic event at a time during each moment in the game (game tick). This means that locking is not required for this design. For instance, when a user saves, a complete snapshot of the game state is needed. This state must remain continuous while the input is processed until the save completes. During each game tick, the backend can only handle either the save operation or the input, making the process thread-safe.
```
func (e *gameEmulator) gameUpdate() {
for {
	select {
		case <-e.saveOperation:
			e.saveGameState()
		case key := <-e.input:
			e.updateGameState(key)
		case <-e.done:
			e.close()
			return
	}
    }
}
```
Fan-in / Fan-out
This Golang template is perfect for my use case of CrowdPlay and Multiple Player. Following this template, all user inputs in one room are embedded into a central input channel. Game media is then distributed to all users in one room. This way, we achieve game state separation across multiple gaming sessions of different users.

Synchronization between different sessions
Disadvantages of Golang
Golang is not perfect. The channel is slow. Compared to locking, the Go channel is just a simpler way to handle parallel and streaming events, but the channel does not provide the best performance. There is complex locking logic underneath the channel. Thus, I made some adjustments to the implementation, applying locks and atomic values again to replace channels for performance optimization.
Moreover, the garbage collector in Golang is unmanaged, resulting in occasionally suspicious long pauses. This greatly hinders the performance of real-time streaming applications.
CGO
The project uses an existing open-source VP8/H264 Golang library for media compression and Libretro for game emulators. All these libraries are merely wrappers of the C library in Go using CGO. Some of the disadvantages are listed in this post by Dave Cheney. The issues I encountered include:
- inability to catch crashes in CGO, even with Golang RecoveryCrash;
- inability to determine the performance bottleneck when detailed issues in CGO cannot be detected.
Conclusion
I achieved my goal – I learned about cloud gaming services and created a platform that helps play nostalgic retro games with my friends online. Creating this project would have been impossible without the Pion library and the support of the Pion community. I am extremely grateful for its intensive development. The simple APIs provided by WebRTC and Pion ensured smooth integration. My first proof of concept was released in the same week, even though I was not previously familiar with peer-to-peer (P2P) connections.
Despite the simplicity of integration, P2P streaming is indeed a very complex area in computer science. It has to deal with the intricacies of long-standing network architectures, such as IP and NAT, to create a peer-to-peer session. During the course of this project, I have gained a lot of valuable knowledge about networking and performance optimization, so I recommend everyone to try building P2P products using WebRTC.
CloudRetro serves all the use cases I expected from the perspective of a retro gamer. However, I believe there are many areas within the project that I can improve, such as making the network more reliable and efficient, ensuring higher game graphics quality, or enabling game sharing among users. I am working hard on this. Please stay tuned for project and support it if you like.