Video communication is the primary means of interaction between teachers and students on the Vimbox platform. We long abandoned Skype, tried several third-party solutions, and ultimately settled on the combination of WebRTC and Janus-gateway. For a while, we were satisfied, but some negative aspects continued to arise. As a result, a separate direction for video communication was created.
I asked Kirill Rogovoy, the head of the new direction, to share the evolution of video communication at Skyeng, the problems encountered, the solutions, and the workarounds we ultimately implemented. We hope this article will be useful for companies also implementing video via web applications.
A Bit of History
In the summer of 2017, Skyeng's head of development, Sergey Safonov, spoke at Backend Conf about how we 'abandoned Skype and implemented WebRTC.' Those interested can watch the recording of the presentation at (~45 min), and here I will briefly summarize its essence.
For Skyeng, video communication has always been the priority means of teacher-student interaction. Initially, we used 'Skype,' but it was categorically unsatisfactory for a whole range of reasons, primarily due to the lack of logs and the impossibility of direct integration into the web application. Therefore, we conducted various experiments.
Essentially, our requirements for video communication were approximately as follows:
— stability;
— low lesson cost;
— lesson recording;
— tracking who speaks more (it is important for us that students speak more than the teacher during lessons);
— linear scalability;
— the ability to use both UDP and TCP.
The first to try to implement Tokbox in 2013 was us. Everything was good, but it turned out to be very expensive – 113 rubles per lesson – which ate into the profits.
Then in 2015, we integrated Voximplant. Here we had the necessary function of tracking who speaks more, and the solution was significantly cheaper: with only audio recording, it cost 20 rubles per lesson. However, it only worked via UDP and did not switch to TCP. Nevertheless, about 40% of students ended up using it.
After a year, we began to have corporate clients with their specific requirements. For instance, everything must work through a browser, and only HTTP and HTTPS are allowed in the company; that is, no 'Skype' or UDP. Corporate clients equal revenue, so we returned to Tokbox, but the price issue remained unresolved.
The solution is WebRTC and Janus.
We decided to use It handles connection establishment, encoding and decoding of streams, synchronization of tracks, and quality control by managing network glitches. On our part, we must ensure stream reading from the camera and microphone, video rendering, connection management, establishing the WebRTC connection, and transmitting the streams to it, as well as sending signaling messages between clients to establish a connection (WebRTC describes only the data format, not the mechanism for transmitting it). If clients are behind NAT, WebRTC connects to STUN servers; if that doesn't help, it connects to TURN servers.
An ordinary p2p connection is not enough for us since we want to record lessons for further analysis in case of complaints. Therefore, we send the WebRTC streams through a relay. As a result, clients do not know each other's addresses, only seeing the address of the Janus server; it also serves as the signaling server. Janus has many features we need: it automatically switches to TCP if UDP is blocked on the client's side; it can record both UDP and TCP streams; it scales well; there is even a built-in plugin for echo tests. If necessary, STUN and TURN servers from Twilio connect automatically.
In the summer of 2017, we had two Janus servers plus an additional server for processing recorded raw audio and video files to avoid burdening the main processors. When connecting, the Janus servers were selected based on odd/even (connection number). At that time, this was sufficient, providing about four times the margin of safety, with an implementation rate of around 80%. At the same time, the price dropped to about 2 rubles per lesson, plus development and maintenance costs.
Returning to the topic of video communication.
We constantly monitor feedback from students and teachers to promptly identify and address problems. By the summer of 2018, the quality of the connection had firmly taken first place among complaints. On one hand, this meant we had successfully resolved other issues. On the other, we urgently needed to act: a disrupted lesson risks losing its value, sometimes along with the cost of purchasing the next package, and a disrupted introductory class could mean losing a potential client altogether.
At that time, our video connection was still in MVP mode. In simpler terms, we launched it, it worked, we scaled it once, understood how to do it – and that was great. If it works, don’t fix it. No one had specifically addressed the issue of connection quality. By August, it became clear that this couldn't continue, and we launched a separate initiative to determine what was wrong with WebRTC and Janus.
At the outset, this initiative received: an MVP solution, no metrics, no goals, no improvement processes, while 7% of teachers reported issues with connection quality (data on students was also lacking).
The new initiative is getting to work.
The team looks something like this:
- Head of the initiative, who is also the lead developer.
- QA helps test changes, seeks new ways to create unstable connection conditions, and reports problems from the front lines.
- The analyst constantly searches for various correlations in technical data, improves the analysis of user feedback, and checks the results of experiments.
- The product manager assists with overall direction and allocation of resources for experiments.
- Another developer often helps with programming and associated tasks.
Initially, we set up a relatively reliable metric to track changes in connection quality ratings (averages over days, weeks, and months). At that stage, these were ratings from teachers, which were later supplemented by student ratings. We then began to formulate hypotheses about what wasn't working, make corrections, and observe changes in dynamics. We started with low-hanging fruit: for example, we replaced the VP8 codec with VP9, and the metrics improved. We experimented with Janus settings and conducted other experiments – most of which led to no results.
At the second stage, a hypothesis emerged: WebRTC is a peer-to-peer solution, while we are using a server in between. Could the problem lie here? We started digging and found what is currently the most significant improvement.
At that time, the server was chosen from the pool using a rather simplistic algorithm: each server had its own 'weight,' depending on the channel and power, and we tried to direct users to the one with the higher 'weight,' without considering the user's geographical location. As a result, a teacher from St. Petersburg could communicate with a student from Siberia through Moscow, instead of our Janus server in St. Petersburg.
The algorithm has been revamped: now, when a user opens our platform, we collect pings from them to all servers using Ajax. When establishing a connection, we select a pair of pings (teacher-server and student-server) with the lowest total. A lower ping means a shorter network distance to the server; a shorter distance means a lower chance of packet loss; and packet loss is the biggest negative factor in video communication. The rate of negative feedback has halved over three months (to be fair, other experiments were conducted during this time, but this one likely had the most significant impact).
Recently, we discovered another subtle yet seemingly important aspect: instead of one powerful Janus server on a thick channel, it's better to have two simpler ones with lower bandwidth. This was realized after we bought powerful machines in hopes of cramming as many rooms (communication sessions) as possible at once. Servers have a bandwidth limit that we can accurately translate into the number of rooms — we know how many can be opened, for example, on 300 Mbps. Once there are too many rooms open on the server, we stop selecting it for new sessions until the load decreases. The idea was that by purchasing a powerful machine, we would push the channel to its maximum, ultimately hitting the limits of the CPU and memory, rather than the bandwidth. However, it turned out that after a certain number of open rooms (420), despite CPU, memory, and disk usage being far from their limits, negative feedback started coming in to technical support. Apparently, something becomes worse within Janus; there may be limitations as well. We began experimenting, lowering the bandwidth limit from 300 to 200 Mbps, and the problems disappeared. We have now purchased three new servers with lower limits and specifications, hoping that this will lead to a stable improvement in communication quality. We didn't investigate what the issue was, of course; quick fixes are our go-to. In our defense, we wanted to resolve the pressing problem as quickly as possible rather than elegantly; besides, Janus is a black box for us, written in C, and delving into it is very costly.
So, during this process, we:
- updated all dependencies that could be updated, both on the server and on the client (these were also experiments, and we monitored the results);
- fixed all identified bugs related to specific cases, for example, when the connection dropped and did not recover automatically;
- held numerous meetings with companies working in the field of video communication and familiar with our issues: streaming games, conducting webinars; we tried everything we found useful;
- conducted a technical review of the hardware and communication quality with the teachers who received the most complaints.
The experiments conducted and the subsequent changes have reduced the dissatisfaction with the connection among teachers from 7.1% in January 2018 to 2.5% in January 2019.
What's Next
Stabilizing our Vimbox platform is one of the main projects of the company for 2019. We have high hopes of maintaining this momentum and no longer seeing video connection issues at the top of complaints. We understand that a significant portion of these complaints is related to users' computer and internet lags, but we need to identify this portion and resolve everything else. The rest is a technical issue; it seems we should be able to handle it.
The main difficulty is that we don't know what level of quality we can realistically achieve. Determining this ceiling is the main task. Therefore, two experiments have been planned:
- comparing video through Janus with regular p2p in real conditions. This experiment has already been conducted, and no statistically significant difference has been found between our solution and p2p;
- we will implement (expensive) services from companies that earn solely on video communication solutions and compare the negativity surrounding them with what we currently have.
These two experiments will allow us to define an achievable goal and focus on it.
In addition, there are a number of tasks being addressed as part of daily work:
- we are creating a technical measurement of connection quality instead of relying on subjective feedback;
- we are generating more detailed session logs to accurately analyze failures that occur, understand when and where they happened, and what seemingly unrelated events took place at that time;
- we are preparing an automatic connection quality test before the lesson, as well as providing the client with the option to manually test the connection to reduce the amount of negativity caused by their hardware and connection;
- we will develop and conduct more load tests of video communication under poor conditions, with variable packet loss, etc.;
- we are changing server behavior in case of problems to enhance fault tolerance;
- we will warn users if their connection is not functioning properly, as Skype does, so they understand that the problem is on their side.
Starting in April, the video communication direction will become a full-fledged separate project within Skyeng, focused on its own product, not just part of Vimbox. This means we are beginning to look for people for . As always, .
. Of course, we continue to actively engage with individuals and companies working in video communication. If you want to share experiences with us — we would be happy to! Comment, reach out — we will reply to everyone.
Source: habr.com
