Hey Habr!
We represent the Exness platform team. Previously, our colleagues have already written an article about . Today we want to share our experience of migrating services to Kubernetes.
To begin with, we offer you some numbers for a better understanding of what will be discussed:
- Our development department is 100+ people, including more than 10 different teams with self-sufficient QA, DevOps and Scrum processes. Development stack - Python, PHP, C++, Java and Golang.
- The size of the test and production environments is about 2000 containers each. They are running Rancher v1.6 on their virtualization and under VMware.
Motivation
As they say, nothing lasts forever under the moon, and Rancher has long announced the end of support for version 1.6. Yes, in more than three years we have learned how to prepare it and solve emerging problems, but more and more often we were faced with problems that would never be fixed. Rancher 1.6 also has a ossified rights system where you can either do almost everything or nothing.
Own virtualization, although it provided more control over data storage and its security, imposed operating costs that were difficult to tolerate with the constant growth of the company, the number of projects and requirements for them.
We wanted to follow the IaC standards and, if necessary, get capacities quickly, in any geographic location and without a vendor location, and also be able to quickly abandon them.
First steps
First of all, we wanted to build on modern technologies and solutions that would allow teams to have a faster development cycle and minimize the operating costs of interacting with a capacity-providing platform.
Of course, the first thing that came to our mind was Kubernetes, but we did not get excited and did a little research on the correctness of the choice. We evaluated only open source solutions, and Kubernetes unconditionally won in an unfair battle.
Then there was the question of choosing a tool for creating clusters. We compared the most popular solutions: kops, kubespray, kubeadm.
To start, kubeadm seemed too complicated for us, rather, a kind of inventor of the βbicycleβ, and kops lacked flexibility.
And the winner is:
We started experimenting with our own virtualization and AWS, trying to replicate our previous resource management pattern where everyone uses the same "cluster". And now we have the first cluster of 10 small virtual machines, a couple of which are located in AWS. We started trying to migrate teams there, everything seemed to be βgoodβ, and the story could be finished, but ...
First Problems
Ansible is what kubespray is built on, itβs not the tool that allows you to follow IaC: when putting nodes in / out of service, something constantly went wrong and some kind of intervention was required, and when using different OS, the playbook behaved differently. With the growing number of teams and nodes in the cluster, we began to notice that the playbook was running longer and longer, as a result, our record is 3,5 hours, and yours? π
And it seems like kubespray is just Ansible, and everything is clear at first glance, but:
At the beginning of the journey, the task was to launch capacities only in AWS and on virtualization, but then, as often happens, the requirements changed.
In light of this, it became clear that our old pattern of combining resources into one orchestration system was not suitable - in the case when the clusters are very remote and are managed by different providers.
Further more. When all teams work within the same cluster, various services with incorrectly installed NodeSelectors could fly to the βforeignβ host of another team and utilize resources there, and in the case of setting taint, there were constant messages that this or that service was not working, not distributed correctly due to the human factor. Another problem was the calculation of the cost, especially given the problems in distributing services across nodes.
A separate story was the issuance of rights to employees: each team wanted to be βat the headβ of the cluster and completely manage it, which could cause a complete collapse, since the teams are mostly independent of each other.
What to do?
Given the above and the wishes of the teams to be more independent, we made a simple conclusion: one team - one cluster.
So we got the second one:
And then the third cluster:
Then we began to think: let's say in a year our teams will have more than one cluster? In different geographical areas, for example, or under the control of different providers? And some of them will want to be able to quickly deploy a temporary cluster for any tests.
Full Kubernetes would come! This is some kind of MultiKubernetes, it turns out.
At the same time, all of us will need to somehow support all these clusters, be able to easily manage access to them, as well as create new ones and decommission old ones without manual intervention.
Since the beginning of our journey in the world of Kubernetes, some time has passed, and we decided to re-examine the available solutions. It turned out that it already exists on the market - Rancher 2.2.
At the first stage of our research, Rancher Labs had already made the first release of version 2, but although it could be raised very quickly by running a container without external dependencies with a couple of parameters or using the official HELM Chart, it seemed raw to us, and we did not know if we could rely on this decision, whether it will be developed or quickly abandoned. The cluster = clicks paradigm itself in the UI also did not suit us, and we did not want to be tied to RKE, since this is a rather narrowly focused tool.
Rancher 2.2 already had a more efficient look and, together with the previous ones, had a bunch of interesting features out of the box, such as integration with many external providers, a single point of distribution of rights and kubeconfig files, running a kubectl image with your rights in the UI, nested namespaces aka projects.
Also, a community has already formed around Rancher 2, and the HashiCorp Terraform provider was created to manage it, which helped us put everything together.
What happened
As a result, we got one small cluster in which Rancher is running, available to all other clusters, as well as many clusters associated with it, access to any of which can be issued as simply as adding a user to the ldap directory, regardless of where he is is located and the resources of which provider it uses.
With the help of gitlab-ci and Terraform, a system was created that allows you to create a cluster of any configuration in cloud providers or our own infrastructure and connect them to Rancher. All this is done in the IaC style, where each cluster is described by a repository, and its state is versioned. At the same time, most modules are connected from external repositories so that all that remains is to transfer variables or describe your custom configuration for instances, which helps to reduce the percentage of code repetition.
Of course, our journey is far from over and there are still many interesting tasks ahead, such as a single point of work with logs and metrics of any clusters, service mesh, gitops for managing loads in a multicluster, and much more. We hope you enjoy our experience!
The article was written by A. Antipov, A. Ganush, Platform Engineers.
Source: habr.com