Introduction

Our company Leaning Technologies provides solutions for porting traditional desktop applications to the web. Our C++ compiler cheerp generates a combination of WebAssembly and JavaScript, which provides and easy browser interaction, and high performance.

As an example of its use, we decided to port a multiplayer game for the web and chose teeworlds. Teeworlds is a retro XNUMXD multiplayer game with a small but active community of players (including me!). It is small in terms of both downloadable resources and CPU and GPU requirements - an ideal candidate.

Running in the Teeworlds browser

We decided to use this project to experiment with general solutions for porting network code to the web. This is usually done in the following ways:

XMLHttpRequest/fetch, if the network part consists only of HTTP requests, or
WebSockets.

Both solutions require hosting a server component on the server side, and neither allows it to be used as a transport protocol. UDP. This is important for real-time applications such as videoconferencing and gaming software because the protocol's packet delivery and order guarantees TCP can interfere with low latency.

There is a third way - to use the network from the browser: WebRTC.

RTCDataChannel supports both reliable and unreliable transmission (in the latter case it attempts to use UDP as the transport protocol whenever possible), and can be used both with a remote server and between browsers. This means that we can port the entire application to the browser, including the server component!

However, this comes with an additional difficulty: before two WebRTC peers can communicate, they must perform a relatively complex handshake to connect, which requires multiple third-party entities (a signaling server and one or more STUN/TURN).

Ideally, we would like to create a networking API that uses WebRTC internally, but is as close as possible to the UDP Sockets interface, which does not need to establish a connection.

This will allow us to take advantage of WebRTC without having to expose complex details to the application code (which we wanted to change as little as possible in our project).

Minimum WebRTC

WebRTC is a set of APIs available in browsers for peer-to-peer transmission of audio, video, and arbitrary data.

The connection between peers is established (even if there is NAT on one or both sides) using STUN and/or TURN servers through a mechanism called ICE. Peers exchange ICE information and channel parameters through the offer and answer of the SDP protocol.

Wow! How many abbreviations at once. Let's briefly explain what these terms mean:

Session Traversal Utilities for NAT (STUN) - a protocol for bypassing NAT and obtaining a pair (IP, port) for communicating directly with the host. If he manages to complete his task, then the peers can independently exchange data with each other.
Traversal Using Relays around NAT (TURN) is also used to bypass NAT, but it does this by forwarding data through a proxy visible to both peers. It adds latency and is more expensive to execute than STUN (because it is applied throughout the session), but sometimes it is the only option.
Interactive Connectivity Establishment (ICE) is used to select the best possible way to connect two peers based on the information obtained by directly connecting peers, as well as information received by any number of STUN and TURN servers.
Session Description Protocol (SDP) - this is a format for describing connection channel parameters, for example, ICE candidates, multimedia codecs (in the case of an audio / video channel), etc. One of the peers sends an SDP Offer (“offer”), and the second responds with an SDP Answer (“response”) . After that, a channel is created.

To create such a connection, peers need to collect the information they receive from the STUN and TURN servers and exchange it with each other.

The problem is that they do not yet have the ability to exchange data directly, so an out-of-band mechanism must exist to exchange this data: a signaling server.

The signaling server can be very simple, because its only task is to forward data between peers during the "handshake" stage (as shown in the diagram below).

Simplified WebRTC Handshake Sequence

Overview of the Teeworlds network model

The network architecture of Teeworlds is very simple:

The client and server components are two different programs.
Clients enter the game by connecting to one of several servers, each hosting only one game at a time.
All data transfer in the game is carried out through the server.
A special master server is used to collect a list of all public servers that are displayed in the game client.

Thanks to the use of WebRTC for data exchange, we can transfer the server component of the game to the browser where the client is located. This gives us a great opportunity...

Get rid of the servers

The absence of server-side logic has a nice advantage: we can deploy the entire application as static content on Github Pages or on our own hardware behind Cloudflare, thus ensuring fast downloads and high uptime for free. In fact, it will be possible to forget about them, and if we are lucky and the game becomes popular, then the infrastructure will not have to be upgraded.

However, for the system to work, we still have to use an external architecture:

One or more STUN servers: We have several free options to choose from.
At least one TURN server: there are no free options here, so we can either set up our own or pay for the service. Luckily, most of the time it will be possible to connect via STUN servers (and provide true p2p), but TURN is needed as a fallback.
Signaling Server: Unlike the other two aspects, signaling is not standardized. What the signaling server will actually be responsible for depends somewhat on the application. In our case, before establishing a connection, it is necessary to exchange a small amount of data.
Teeworlds master server: it is used by other servers to announce its existence and by clients to find public servers. Although not required (clients can always connect to a server they know manually), it would be nice to have it so that players can play games with random people.

We decided to use Google's free STUN servers, and deployed one TURN server ourselves.

For the last two items, we used Firebase:

The Teeworlds master server is implemented very simply: as a list of objects containing information (name, IP, map, mode, ...) of each active server. Servers publish and update their own object, and clients take the entire list and display it to the player. We also display the list as HTML on the home page so that players can simply click on the server and jump right into the game.
Signaling is closely related to our socket implementation, described in the next section.

List of servers inside the game and on the home page

Socket Implementation

We want to create an API that is as close as possible to Posix UDP Sockets in order to minimize the number of changes required.

We also want to implement the necessary minimum required for the simplest data exchange over the network.

For example, we don't need real routing: all peers are on the same "virtual LAN" associated with a particular Firebase database instance.

Therefore, we do not need unique IP addresses: to uniquely identify peers, it is enough to use unique Firebase key values (similar to domain names), and each peer locally assigns "fake" IP addresses to each key that needs to be translated. This completely saves us from having to assign IP addresses globally, which is not a trivial task.

Here is the minimum API we need to implement:

// Create and destroy a socket
int socket();
int close(int fd);
// Bind a socket to a port, and publish it on Firebase
int bind(int fd, AddrInfo* addr);
// Send a packet. This lazily create a WebRTC connection to the 
// peer when necessary
int sendto(int fd, uint8_t* buf, int len, const AddrInfo* addr);
// Receive the packets destined to this socket
int recvfrom(int fd, uint8_t* buf, int len, AddrInfo* addr);
// Be notified when new packets arrived
int recvCallback(Callback cb);
// Obtain a local ip address for this peer key
uint32_t resolve(client::String* key);
// Get the peer key for this ip
String* reverseResolve(uint32_t addr);
// Get the local peer key
String* local_key();
// Initialize the library with the given Firebase database and 
// WebRTc connection options
void init(client::FirebaseConfig* fb, client::RTCConfiguration* ice);

The API is simple and similar to the Posix Sockets API, but with a few important differences: callback logging, local IP assignment, and lazy connection.

Registering callbacks

Even if the original program uses non-blocking I/O, the code must be refactored to run in a web browser.

The reason for this is that the event loop in the browser is hidden from the program (be it JavaScript or WebAssembly).

In a native environment, we can write code like this

while(running) {
  select(...); // wait for I/O events
  while(true) {
    int r = readfrom(...); // try to read
    if (r < 0 && errno == EWOULDBLOCK) // no more data available
      break;
    ...
  }
  ...
}

If the event loop is hidden for us, then we need to turn it into something like this:

auto cb = []() { // this will be called when new data is available
  while(true) {
    int r = readfrom(...); // try to read
    if (r < 0 && errno == EWOULDBLOCK) // no more data available
      break;
    ...
  }
  ...
};
recvCallback(cb); // register the callback

Local IP Assignment

The node IDs in our "network" are not IP addresses, but Firebase keys (these are strings that look like this: -LmEC50PYZLCiCP-vqde ).

This is convenient because we don't need a mechanism to assign IPs and check if they're unique (and dispose of them after a client disconnects), but it's often necessary to identify peers by numeric value.

This is what the functions are for. resolve и reverseResolve: The application somehow receives the string value of the key (via user input or via the master server) and can resolve it to an IP address for internal use. The rest of the API also receives this value instead of a string for simplicity.

This is similar to a DNS lookup, only done locally on the client.

That is, IP addresses cannot be shared between different clients, and if some kind of global identifier is needed, then it will have to be generated in a different way.

Lazy join

UDP does not need a connection, but as we have seen, WebRTC requires a lengthy connection process before it can start transferring data between two peers.

If we want to provide the same level of abstraction, (sendto/recvfrom with arbitrary peers without a pre-connection), then they must perform a “lazy” (delayed) connection inside the API.

Here is what happens in the normal communication between the "server" and the "client" in the case of using UDP, and what our library should do:

Server calls bind()to tell the operating system that it wants to receive packets on the specified port.

Instead, we will publish an open port to Firebase under the server key and listen for events in its subtree.

Server calls recvfrom(), accepting packets from any host on this port.

In our case, we need to check the incoming queue of packets sent to this port.

Each port has its own queue, and we add source and destination ports to the beginning of WebRTC datagrams so that we know which queue to redirect to when a new packet arrives.

The call is non-blocking, so if there are no packets, we simply return -1 and set errno=EWOULDBLOCK.

The client receives by some external means the IP and port of the server, and calls sendto(). It also makes an internal call bind(), so the next recvfrom() will receive the response without explicitly executing bind.

In our case, the client externally receives a string key and uses the function resolve() to get an IP address.

At this point, we begin the WebRTC handshake if the two peers are not already connected to each other. Connections to different ports of the same peer use the same WebRTC DataChannel.

We also do indirect bind()so that the server can reconnect next sendto() in case it closed for some reason.

The server is notified of the client's connection when the client writes its SDP offer under the server's port information in Firebase, and the server responds with its response there as well.

The diagram below shows an example of message flow for the socket scheme and the first message from the client to the server:

Complete diagram of the connection phase between client and server

Conclusion

If you've read this far, you might be interested in seeing the theory in action. The game can be played on teeworlds.leaningtech.com, try it!

Friendly match between colleagues

The network library code is freely available at Github. Join the conversation on our channel grid!

Source: habr.com

Porting a multiplayer game from C++ to the web with Cheerp, WebRTC and Firebase