Mikhail Salosin (hereinafter referred to as MS): – Hello everyone! My name is Mikhail. I work as a backend developer at MC2 Software, and I will talk about using Go in the backend of the mobile app 'Smotri+'.
Is anyone here a fan of hockey?
Then this app is for you. It is available for Android and iOS and serves for watching broadcasts of various sports events online and on demand. The app also has various statistics, text broadcasts, conference tables, tournament standings, and other information useful for fans.
The app also features video moments, meaning you can watch the highlights of games (goals, fights, shootouts, etc.). If you don't want to watch the entire broadcast, you can check out just the interesting bits.
What did you use in development?
Most of it was written in Go. The API that mobile clients interacted with was built on Go. A service for sending push notifications to mobiles was also developed in Go. Additionally, we had to create our own ORM, which we might discuss someday. Some minor services were written in Go as well: image resizing and uploading for editors...
For our database, we used PostgreSQL. The interface for editors was built with Ruby on Rails using the ActiveAdmin gem. The import of statistics from the statistics provider was also done in Ruby.
For system testing of the API, we utilized Python's unittest. Memcached is used for throttling API payment requests, Chef is for configuration management, Zabbix is implemented for gathering and monitoring internal statistical data of the system. Graylog2 is used for log collection, and Slate provides API documentation for clients.
Choosing a protocol
The first challenge we faced was selecting a protocol for backend interaction with mobile clients, based on the following points...
- The most important requirement: client data must be updated in real-time. This means that everyone currently watching the broadcast should receive updates almost instantly.
- To simplify, we accepted that the data synchronized with clients is not deleted but hidden using special flags.
- All sorts of rare requests (such as statistics, team compositions, team stats) are handled as regular GET requests.
- Additionally, the system needed to handle 100,000 users simultaneously without any issues.
Based on this, we had two protocol options:
- WebSockets. However, we didn't require client-to-server channels. We only needed to send updates from the server to the client, so WebSocket was an excessive option.
- Server-Sent Events (SSE) was the perfect fit! It's simple enough and essentially meets all our needs.
Server-Sent Events
A few words about how this thing works…
It operates over an HTTP connection. The client sends a request, and the server responds with Content-Type: text/event-stream, keeping the connection open to continue sending data.
Data can be sent in a format agreed upon with the clients. In our case, we sent it in such a way: the event field contained the name of the modified structure (person, player), and the data field contained JSON with the new, modified fields for the player.
Now, let's discuss how the interaction itself works.
- First, the client determines when it last synchronized with the service: it checks its local database and finds the date of the last recorded change.
- It sends a request with that date.
- In response, we send it all the updates that occurred since that date.
- After that, it establishes a connection to the live channel and keeps it open as long as it needs these updates:
We send it a list of changes: if someone scores a goal – we update the match score; if there’s an injury – that is also sent in real-time. Thus, clients instantly receive relevant data in the match events stream. Periodically, to let the client know that the server is still alive and nothing has gone wrong, we send a timestamp every 15 seconds – so it knows everything is fine and there's no need to reconnect.
How is the live connection maintained?
- First, we create a channel that will receive updates with a buffer.
- Next, we subscribe this channel to receive updates.
- We set the correct header so that the client knows everything is okay.
- We send the first ping. We simply record the current timestamp of the connection.
- After this, we read from the channel in a loop until the update channel is closed. The channel periodically receives either the current timestamp or changes that we are already recording into open connections.
The first problem we encountered was as follows: for each open connection with the client, we created a timer that ticked every 15 seconds – meaning that if we had 6,000 connections open with one machine (with one API server), we were creating 6,000 timers. This meant the machine was not able to handle the required load. The problem was not so obvious to us, but with a bit of help, we resolved it.
As a result, now our ping comes from the same channel from which the update arrives.
Accordingly, there is only one timer that ticks every 15 seconds.
Here are several helper functions – sending headers, pings, and the structure itself. That is, the table name (person, match, season) and the information about this record are passed here:
The mechanism for sending updates
Now a bit about where the changes come from. We have several people, editors, who watch the broadcast in real time. They create all the events: someone was sent off, someone got injured, there was a substitution…
Using the CMS, data enters the database. After that, the database, through the Listen/Notify mechanism, notifies the API servers about this. The API servers then distribute this information to the clients. Thus, essentially, only a few servers are connected to the database, and there is no significant load on the database because the client does not interact directly with it.
PostgreSQL: Listen/Notify
The Listen/Notify mechanism in PostgreSQL allows notifying subscribers to events that some event has changed – a record has been created in the database. For this, we wrote a simple trigger and function:
Upon inserting or modifying a record, we call the notify function on the data_updates channel, passing the table name and the identifier of the record that was modified or inserted.
For all tables that need to be synchronized with the client, we define a trigger that calls the function specified in the slide below after updating/modifying a record.
How does the API subscribe to these changes?
A Fanout mechanism is created – it distributes messages to clients. It collects all client channels and sends updates received through these channels.
Here is the standard pq library, which connects to the database and indicates that it wants to listen to the channel (data_updates), checks that the connection is open and everything is fine. I'm skipping error checking to save space (not checking can be risky).
Next, we asynchronously set a Ticker that will send a ping every 15 seconds, and we start listening to the subscribed channel. If we receive a ping, we publish that ping. If we receive a record, we publish that record to all subscribers of that Fanout.
How does Fan-out work?
In Russian, this translates to 'splitter'. We have one object that registers subscribers who want to receive updates. As soon as an update arrives for this object, it distributes that update to all its existing subscribers. It’s quite simple:
How this is implemented in Go:
There is a structure that synchronizes using Mutexes. It has a field that saves the state of the Fanout's connection to the database, i.e., it is currently listening and will receive updates, as well as a list of all existing channels – a map, where the key is the channel and the value is a struct (which isn't really used).
Two methods – Connected and Disconnected – allow the Fanout to know that we have a connection to the database, that it has appeared, and that the connection to the database has been severed. In the latter case, all clients need to be disconnected and informed that they can no longer listen and that they should reconnect since the connection with them has been closed.
There is also a Subscribe method that adds a channel to the 'listeners':
There is an Unsubscribe method that removes a channel from the listeners if the client has disconnected, as well as a Publish method that allows sending a message to all subscribers.
Question: – What is transmitted through this channel?
MS: – A model that has changed or a ping (essentially just a number, an integer).
MS: – You can send any structure, publish it – it simply gets converted to JSON, and that's it.
MS: – We receive a notification from PostgreSQL – it contains the table name and identifier. By the table name, we retrieve the corresponding record by its identifier, and then we send this structure for publication.
Infrastructure
What does it look like from an infrastructure perspective? We have 7 physical servers: one is entirely dedicated to the database, while the other six host virtual machines. There are 6 API copies: each virtual machine with API runs on a separate physical server – this is for reliability.
We have two frontends with Keepalived for improved availability, so that if necessary, one frontend can replace the other. Additionally, we have two copies of the CMS.
There is also a statistics importer. We have a DB Slave from which periodic backups are taken. There's a Pigeon Pusher – the application that sends pushes to clients, along with infrastructure components: Zabbix, Graylog2, and Chef.
In fact, this infrastructure is redundant, because we could serve 100 thousand with fewer servers. But we had the hardware – so we used it (we were told it was okay – why not?).
Advantages of Go
After we worked on this application, some obvious advantages of Go emerged.
- A great HTTP library. With it, you can create quite a lot right out of the box.
- Plus, the channels made it very easy for us to implement the mechanism for sending notifications to clients.
- The wonderful Race Detector allowed us to eliminate several critical bugs (in the staging infrastructure). Everything running on staging is launched and compiled with the Race key; and we can, accordingly, see what potential issues we have in the staging infrastructure.
- Minimalism and simplicity of the language.
We are looking for developers! If anyone is interested – please reach out.
Questions
Question from the audience (Q): – It seems to me that you missed one important point regarding Fan-out. Am I correct in understanding that when you send a response to the client, you get blocked if the client doesn't want to read?
MS: – No, we do not block. First of all, everything is behind nginx, so there are no issues with slow clients. Secondly, the client has a buffered channel – essentially, we can put up to a hundred updates there... If we can't write to the channel, it deletes it. If we see that the channel has been blocked, we simply close the channel, and that's it – the client will reconnect if there is any problem. Therefore, blocking generally does not occur here.
Q: – Couldn't you send the record to Listen/Notify directly instead of the identifier table?
MS: – Listen/Notify has a limit of 8 thousand bytes on the preload it sends. In principle, it could be sent if we were dealing with a small amount of data, but I think that the way we do it is simply more reliable. The limitations are inherent in PostgreSQL itself.
Q: – Do clients receive updates about matches they are not interested in?
MS: – Generally, yes. Typically, 2-3 matches proceed in parallel, which occurs quite rarely. If a client is watching something, they usually focus on the match that is currently happening. Then, on the client side, there is a local database where all these updates are stored, and even without an internet connection, the client can view all past matches for which they have updates. Essentially, we synchronize our server database with the client's local database so that they can work offline.
Q: – Why did you create your own ORM?
Alexey (one of the developers of "Smotri+"): – At that time (this was a year ago), there were fewer ORMs than now, when there are quite a lot. Of most existing ORMs, I dislike that most of them operate on empty interfaces. This means that the methods in these ORMs are willing to accept anything: a structure, a structure pointer, a number, or something entirely irrelevant...
Our ORM generates structures based on the data model. By itself. Therefore, all methods are specific, do not use reflection, etc. They accept structures and expect to use the structures that are provided.
Q: – How many people were involved?
MS: – Initially, two people were involved. We started somewhere in June, and by August the main part was ready (the first version). The release was in September.
Q: – In the section where you describe SSE, you do not use a timeout. Why is that?
MS: – To be honest, SSE is still an HTML5 protocol: the SSE standard is designed for communication with browsers, as far as I understand. It has additional features so that browsers can reconnect (and more), but we don’t need them because we had clients who could implement any connection logic and information retrieval. We did something more akin to SSE rather than SSE itself. It’s not the protocol.
There was no need. As far as I understand, clients implemented the connection mechanism practically from scratch. They didn’t really mind.
Q: – What additional tools did you use?
MS: – Most actively, we used govet and golint to maintain a consistent style, as well as gofmt. We didn’t use anything else.
Q: – What did you use for debugging?
MS: – Debugging was mainly done through tests. We didn’t use any debugger, we relied on GOP.
Q: – Can you pull up the slide where the Publish function is implemented? Are you not bothered by single-letter variable names?
MS: – No. They have a sufficiently ‘narrow’ scope. They are not used anywhere else, except here (besides the internals of this class), and it’s very compact – it only takes up 7 lines.
Q: – Somehow, it’s still not intuitive…
MS: – No, no, this is real code! It’s not about style. It’s just such a utilitarian, very small class – it only has 3 fields inside the class…
MS: – Basically, all the data that synchronizes with clients (season matches, players) does not change. Roughly speaking, if we will do another sport where we need to change a match, we will just take it into account in the new version of the client, and old client versions will be banned.
Q: – Are there any third-party packages for managing dependencies?
MS: – We used go dep.
Q: – The report topic mentioned something about video, but there is nothing about video in the report.
MS: – No, I don’t have anything about video in my topic. It’s called ‘Sмотри+’ – that’s the name of the application.
Q: – You mentioned that it streams to clients?..
MS: – We did not deal with streaming video. This was entirely done by ‘MegaFon’. Yes, I didn’t mention that the application is from MegaFon.
MS: – Go – for sending all data – on the account, on match events, statistics… Go is a complete backend for the application. The client needs to know where to get the link for the player so that the user can watch the match. We have links to the videos and streams that are prepared.
A little advertisement 🙂
Thank you for staying with us. Do you enjoy our articles? Want to see more interesting content? Support us by placing an order or recommending us to your friends, , a unique entry-level server alternative that we have created for you: (options available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).
Dell R730xd at half the price in the Equinix Tier IV data center in Amsterdam? Only with us in the Netherlands! Dell R420 — 2x E5-2430 2.2GHz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB — from $99! Read about how
Source: habr.com
