🥇Dailymotion's Kubernetes Adventure: Building Infrastructure in the Cloud + On-Premises

Note: translation.: Dailymotion — one of the largest video hosting services in the world and therefore a notable user of Kubernetes. In this article, system architect David Donchez shares the results of creating the company's production platform based on K8s, which began with a cloud installation in GKE and ended up as a hybrid solution that achieved better response times and saved on infrastructure costs.

When making the decision to restructure the main API Dailymotion three years ago, we wanted to develop a more efficient way to deploy applications and simplify processes in development and production. To achieve this goal, we decided to use a container orchestration platform and naturally chose Kubernetes.

Why should you create your own platform based on Kubernetes?

API production-level in the shortest possible time using Google Cloud

Summer 2016

Three years ago, shortly after Dailymotion was acquired by Vivendi, our engineering teams focused on one global goal: to create a completely new Dailymotion product.

As a result of analyzing containers, orchestration solutions, and our past experience, we concluded that Kubernetes was the right choice. Some developers already had an understanding of the basic concepts and knew how to use it, which was a huge advantage for the infrastructure transformation.

From an infrastructure perspective, a powerful and flexible system was needed to host new types of cloud-native applications. We preferred to stay in the cloud at the beginning of our journey to calmly build the most reliable local platform. We decided to deploy our applications using Google Kubernetes Engine, although we knew that sooner or later we would move to our own data centers and implement a hybrid strategy.

Why choose GKE?

We made this choice mainly for technical reasons. Additionally, there was a need to quickly provide infrastructure that met the company's business needs. We had some application deployment requirements, such as geographical distribution, scalability, and fault tolerance.

GKE Clusters at Dailymotion

Since Dailymotion is a video platform available worldwide, we really wanted to improve service quality by reducing wait times (latency). Previously our API was only available in Paris, which was not optimal. It was desirable to host applications not only in Europe but also in Asia and the USA.

This sensitivity to latency meant that significant work would be needed on the platform's network architecture. While most cloud services required creating their network in each region and then linking them via VPN or some managed service, Google Cloud allowed the creation of a fully routable single network covering all of Google's regions. This is a significant advantage in terms of operations and system efficiency.

Additionally, Google's networking services and load balancers perform excellently. They simply allow the use of arbitrary public IP addresses from each region, and the remarkable BGP protocol takes care of the rest (i.e., it redirects users to the nearest cluster). It is clear that in the event of a failure, traffic will automatically route to another region without any human intervention.

Load Balancing Monitoring in Google

Our platform also actively utilizes graphics processors. Google Cloud allows for very efficient use of them directly in Kubernetes clusters.

At that time, the infrastructure team predominantly focused on the old stack deployed on physical servers. This is why using a managed service (including Kubernetes master components) met our requirements and allowed the teams to train for working with local clusters.

As a result, we were able to start accepting production traffic on Google Cloud infrastructure just six months after work began.

However, despite a number of advantages, working with a cloud provider is associated with certain costs that can increase depending on the load. This is why we carefully analyzed each managed service utilized, planning to implement them on-premises in the future. In fact, the implementation of local clusters began in late 2016 and a hybrid strategy was initiated at that time.

Launching the Dailymotion Local Container Orchestration Platform

Fall 2016

In a situation where the entire stack was ready for production and work on the API was ongoing, there was a time to focus on regional clusters.

At that time, users were viewing over 3 billion videos each month. Of course, we had already been operating our own extensive Content Delivery Network for several years. We wanted to take advantage of this circumstance and deploy Kubernetes clusters in existing data centers.

Dailymotion's infrastructure comprised over 2,500 servers in six data centers. All of them are configured using Saltstack. We began preparing all the necessary recipes for creating master and worker nodes, as well as the etcd cluster.

Network Component

Our network is fully routable. Each server announces its IP within the network using Exabgp. We compared several network plugins, and the only one that met all our needs (due to the L3-level approach used) was Calico. It integrated perfectly into the existing network model of the infrastructure.

Since we wanted to utilize all available elements of the infrastructure, the first step was to figure out our homegrown network utility (used on all servers): to use it for announcing IP address ranges in the network with Kubernetes nodes. We allowed Calico to assign IPs to pods, but did not use it and still do not for BGP sessions on network equipment. In fact, routing is handled by Exabgp, which announces the subnets used by Calico. This allows us to reach any pod from the internal network (and specifically from load balancers).

How we manage ingress traffic

To redirect incoming requests to the correct service, we decided to use the Ingress Controller due to its integration with Kubernetes ingress resources.

Three years ago, the nginx-ingress-controller was the most mature controller: Nginx had been used for a long time and was known for its stability and performance.

In our system, we decided to deploy controllers on dedicated 10-gigabit blade servers. Each controller connected to the kube-apiserver endpoint of the respective cluster. Exabgp was also used on these servers to announce public or private IP addresses. The topology of our network allows for BGP from these controllers to route all traffic directly to the pods without using a service like NodePort. This approach helps avoid horizontal traffic between nodes and improves efficiency.

Traffic flow from the internet to the pods

Now that we have covered our hybrid platform, we can delve into the migration process.

Traffic migration from Google Cloud to Dailymotion's infrastructure

Fall 2018

After nearly two years of building, testing, and configuring, we finally have a complete Kubernetes stack ready to handle some traffic.

The current routing strategy is quite simple, yet it adequately meets our needs. In addition to public IPs (in Google Cloud and Dailymotion), we use AWS Route 53 to set policies and redirect users to the cluster of our choice.

Example of a routing policy using Route 53

With Google Cloud, this is straightforward since we utilize a single IP for all clusters, and the user is redirected to the nearest GKE cluster. For our clusters, the technology is different as their IPs vary.

During the migration, we aimed to redirect regional requests to the appropriate clusters and evaluated the benefits of this approach.

Since our GKE clusters are configured for automatic scaling using Custom Metrics, they scale up/down based on incoming traffic.

In normal operation, all regional traffic is directed to the local cluster, while GKE serves as a backup in case of issues (health checks are conducted by Route 53).

…

In the future, we aim to fully automate routing policies to achieve a self-sustaining hybrid strategy that continuously enhances user availability. As for the benefits, cloud costs have significantly decreased, and we have even managed to reduce API response times. We trust the resulting cloud platform and are ready to redirect more traffic to it if necessary.

P.S. from the translator

You might also be interested in another recent Dailymotion publication about Kubernetes. It focuses on deploying applications with Helm across multiple Kubernetes clusters and was published about a month ago.