Hi all! My name is Pavel Agaletsky. I work as a team leader in a team that develops the Lamoda delivery system. In 2018, I spoke at the HighLoad ++ conference, and today I want to present a transcript of my report.

My topic is dedicated to the experience of our company in deploying systems and services to different environments. Starting from our prehistoric times, when we deployed all systems into ordinary virtual servers, ending with a gradual transition from Nomad to Kubernetes deployment. I'll tell you why we did it and what problems we had in the process.

Deploy applications to VM

Let's start with the fact that 3 years ago all the systems and services of the company were deployed on ordinary virtual servers. Technically, it was organized in such a way that all the code of our systems lay and was assembled using automatic assembly tools, using jenkins. With the help of Ansible, it was already rolled out from our version control system to virtual servers. At the same time, each system that was in our company was deployed to at least 2 servers: one of them - on head, the second - on tail. These two systems were absolutely identical to each other in all their settings, power, configuration and so on. The only difference between them was that head received user traffic, while tail never received user traffic for itself.

Why was this done?

When we deployed new releases of our application, we wanted to be able to roll out seamlessly, that is, without noticeable consequences for users. This was achieved due to the fact that the next compiled release using Ansible was rolled out to tail. There, the people who were involved in the deployment could check and make sure that everything is fine: all metrics, sections and applications are working; the necessary scripts are launched. Only after they were convinced that everything was OK, did the traffic switch. He started going to the server that was tail before. And the one that was previously the head remained without user traffic, while with the previous version of our application available on it.

So it was seamless for users. Because the switching is instantaneous, since it is just a balancer switching. It is very easy to roll back to the previous version by simply switching the balancer back. We could also make sure that the application was ready for production even before the user traffic went to it, which was quite convenient.

What virtues do we see in all this?

1. First of all, it's enough just works. Everyone understands how such a deployment scheme works, because most people have ever deployed to ordinary virtual servers.
2. This is enough reliably, since the deployment technology is simple, tested by thousands of companies. Millions of servers are deployed this way. It's hard to break something.
3. And finally we could get atomic deployments. Deployments that occur simultaneously for users, without a noticeable stage of switching between the old version and the new one.

But we also saw several shortcomings in this all:

1. In addition to the production environment, the development environment, there are other environments. For example, qa and preproduction. At that time, we had many servers and about 60 services. For this reason it had to for each service, maintain the latest version for it virtual machine. Moreover, if you want to update libraries or install new dependencies, you need to do this in all environments. It was also necessary to synchronize the time when you are going to deploy the next new version of your application with the time when devops performs the necessary environment settings. In this case, it is easy to get into a situation where our environment will be somewhat different at once in all environments in a row. For example, in a QA environment there will be some versions of libraries, and in production - others, which will lead to problems.
2. Difficulty in updating dependencies your application. It doesn't depend on you, but on the other team. Namely, from the devops team that maintains the servers. You should give them an appropriate task and describe what you want to do.
3. At that time, we also wanted to split the large large monoliths we had into separate small services, as we understood that there would be more and more of them. At that time, we already had more than 100 of them. It was necessary to create a separate new virtual machine for each new service, which also needed to be serviced and deployed. In addition, you need not one car, but at least two. To this all is added a QA environment. This causes problems and makes it more difficult for you to build and run new systems. complex, costly and lengthy process.

Therefore, we decided that it would be more convenient to move from deploying ordinary virtual machines to deploying our applications in a docker container. If you have docker, you need a system that can run the application in a cluster, since you can’t just raise a container just like that. Usually you want to keep track of how many containers are lifted so that they are lifted automatically. For this reason, we had to choose a control system.

We thought long and hard about which one to take. The fact is that at that time this deployment stack to ordinary virtual servers was somewhat outdated, since there were not the latest versions of operating systems. At some point, even FreeBSD was there, which was not very convenient to maintain. We understood that we needed to migrate to docker as soon as possible. Our devops looked at their experience with different solutions and chose a system like Nomad.

Switching to Nomad

Nomad is a product of HashiCorp. They are also known for their other solutions:

Consul is a tool for service discovery.

"Terraform" - a system for managing servers that allows you to configure them through a configuration, the so-called infrastructure-as-a-code.

Vagrant allows you to deploy virtual machines locally or in the cloud through specific configuration files.

Nomad at that time seemed like a fairly simple solution that you can quickly switch to without changing the entire infrastructure. Plus, it's fairly easy to learn. Therefore, we chose it as the filtration system for our container.

What do you need to actually deploy your system in Nomad?

1. First of all, you need docker image your application. You need to build it and put it in the docker image store. In our case, this is artifactory - a system that allows you to push various artifacts of various types into it. It can store archives, docker images, PHP composer packages, NPM packages, and so on.
2. Also needed configuration file, which will tell Nomad what, where and how much you want to deploy.

When we talk about Nomad, it uses the HCL language as an information file format, which stands for HashiCorp Configuration Language. It is a superset of Yaml that allows you to describe your service in terms of Nomad.

It allows you to say how many containers you want to deploy, from which images to transfer various parameters to them during deployment. So you feed this file to Nomad, and it launches containers to production according to it.

In our case, we realized that just writing exactly the same, identical HCL files for each service would not be very convenient, because there are a lot of services and sometimes you want to update them. It happens that one service is deployed not in one instance, but in a variety of different ones. For example, one of the systems we have in production has over 100 instances in production. They run from the same images, but differ in configuration settings and configuration files.

Therefore, we decided that it would be convenient for us to store all our configuration files for deployment in one common repository. In this way, they became visible: they were easy to maintain and you could see what kind of systems we have. If necessary, it is also easy to update or change something. Adding a new system is also not difficult - just add a configuration file inside the new directory. Inside it are files: service.hcl, which contains a description of our service, and some env-files that allow this very service, being deployed in production, to be configured.

However, some of our systems are deployed in production not in one copy, but in several at once. Therefore, we decided that it would be convenient for us to store not configs in their pure form, but their templated form. And as the templating language we have chosen jinja 2. In this format, we store both the configs of the service itself and the env files needed for it.

In addition, we have placed in the repository a deployment script that is common for all projects, which allows you to launch and deploy your service in production, in the right environment, in the right target. In the case when we turned our HCL config into a template, then the HCL file, which before that was the usual Nomad config, in this case began to look a little different.

That is, we have replaced some config place variables with variable inserts that are taken from env files or from other sources. In addition, we got the ability to build HCL files dynamically, that is, we can use not only the usual variable inserts. Since jinja supports cycles and conditions, you can also put configuration files there, which change depending on where exactly you deploy your applications.

For example, you want to deploy your service to pre-production and production. Let's say that in pre-production you don't want to run cron scripts, but just want to see the service on a separate domain to make sure it's running. For anyone deploying a service, the process is very simple and transparent. It is enough to execute the deploy.sh file, specify which service you want to deploy and to which target. For example, you want to deploy some system to Russia, Belarus or Kazakhstan. To do this, simply change one of the parameters, and you will have the correct configuration file.

When the Nomad service is already deployed in your cluster, it looks like this.

To begin with, you need some load balancer from the outside, which will take all user traffic into itself. It will work together with Consul and learn from it where, on which node, at what IP address a particular service is located, which corresponds to a particular domain name. Services in Consul come from Nomad itself. Since these are products of the same company, they are well connected. We can say that out of the box Nomad is able to register all services launched in it inside Consul.

After your external balancer knows which service to send traffic to, it redirects it to the appropriate container or multiple containers that match your application. Naturally, it is necessary to think about safety as well. Even though all services run on the same virtual machines in containers, this usually requires that any service be denied free access to any other. We achieved this through segmentation. Each service was launched in its own virtual network, on which routing rules and rules for allowing / denying access to other systems and services were written. They could be both inside this cluster and outside it. For example, if you want to prevent a service from connecting to a particular database, this can be done by segmenting at the network level. That is, even by mistake, you cannot accidentally connect from the test environment to your production base.

How much did the transition cost us in terms of human resources?

Approximately 5-6 months took the transition of the entire company to Nomad. We moved service-by-service, but at a fairly fast pace. Each team had to create their own service containers.

We have adopted such an approach that each team is responsible for the docker images of their systems independently. Devops, on the other hand, provide the general infrastructure necessary for deployment, that is, support for the cluster itself, support for the CI system, and so on. And at that time, we had more than 60 systems moved to Nomad, which turned out to be about 2 thousand containers.

DevOps is responsible for the overall infrastructure of everything related to deployment, with servers. And each development team, in turn, is responsible for implementing containers for their specific system, since it is the team that knows what it generally needs in a particular container.

Reasons for leaving Nomad

What advantages did we get by switching to deployment using Nomad and docker as well?

1. Мы provided the same conditions for all environments. In development, QA-environment, pre-production, production, the same container images are used, with the same dependencies. Accordingly, you have practically no chance that something different from what you previously tested locally or on a test environment will turn out in production.
2. We also found that it is enough easy to add a new service. Any new systems in terms of deployment are launched very simply. It is enough to go to the repository that stores the configs, add the next config for your system there, and you are all set. You can deploy your system to production with no extra effort from devops.
3. All configuration files in one shared repository turned out to be overlooked. At the moment when we deployed our systems using virtual servers, we used Ansible, in which the configs were in the same repository. However, for most developers, this was a little more difficult to work with. Here the volume of configs and code that you need to add to deploy the service has become much smaller. Plus for devops it is very easy to fix it or change it. In the case of transitions, for example, to a new version of Nomad, they can take and massively update all the operating files lying in the same place.

But we also encountered several disadvantages:

It turned out that we failed to achieve seamless deployments in the case of Nomad. When rolling out containers from different conditions, it could turn out that it turned out to be running, and Nomad perceived it as a container ready to receive traffic. This happened even before the application inside it had time to start. For this reason, the system began to issue 500 errors for a short time, because the traffic began to go to the container, which was not yet ready to accept it.

We have encountered some bugs. The most significant bug is that Nomad doesn't handle a large cluster very well if you have many systems and containers. When you want to take one of the servers that is included in the Nomad cluster into service, there is a fairly high probability that the cluster will not feel very good and fall apart. Some containers may, for example, fall and not rise - this will cost you a lot later if all your systems in production are located in a Nomad-managed cluster.

So we decided to think about where we should go next. At that time, we became much more aware of what we want to achieve. Namely: we want reliability, a little more features than Nomad gives, and a more mature, more stable system.

In this regard, our choice fell on Kubernetes as the most popular platform for running clusters. Especially given that the size and number of our containers was large enough. For such purposes, Kubernetes seemed to be the most suitable system of those that we could look at.

Moving to Kubernetes

I’ll talk a little about what the basic concepts of Kubernetes are and how they differ from Nomad.

First of all, the most basic concept in Kubernetes is the concept of pod. Under is a group of one or more containers that always start together. And they work as if always strictly on the same virtual machine. They are available to each other by IP address 127.0.0.1 on different ports.

Let's say you have a PHP application that consists of nginx and php-fpm - the classic pattern. Most likely, you will want both nginx and php-fpm containers to always be together. Kubernetes allows you to achieve this by describing them as one common pod. This is exactly what we couldn't get with Nomad.

The second concept is deployment. The point is that the pod itself is an ephemeral thing, it starts and disappears. Whether you want to first kill all your previous containers, and then launch new versions right away, or do you want to roll them out gradually - this is precisely the concept of deployment that is responsible for this process. It describes how you deploy your pods, how many, and how to update them.

The third concept is service. Your service is actually your system that takes in some traffic and then directs it to one or more pods corresponding to your service. That is, it allows you to say that all incoming traffic to such and such a service with such and such a name must be sent to these specific pods. And at the same time, it provides you with traffic balancing. That is, you can run two pods of your application, and all incoming traffic will be evenly balanced between the pods related to this service.

And the fourth basic concept - income. This is a service that runs on a Kubernetes cluster. It acts as an external load balancer that takes over all requests. Through the Kubernetes API, Ingress can determine where these requests should be sent. And he does it very flexibly. You can say that all requests for this host and such and such a URL are sent to this service. And these requests coming to this host and to another URL are sent to another service.

The coolest thing from the point of view of someone who develops an application is that you are able to manage it all yourself. By setting the Ingress config, you can send all traffic coming to such and such an API to separate containers, registered, for example, in Go. But this traffic that comes to the same domain, but to a different URL, is sent to containers written in PHP, where there is a lot of logic, but they are not very fast.

If we compare all these concepts with Nomad, then we can say that the first three concepts are all together Service. And the last concept is absent in Nomad itself. We used an external balancer as it: it can be haproxy, nginx, nginx + and so on. In the case of a cube, you do not need to introduce this additional concept separately. However, if you look at Ingress inside, then it is either nginx, or haproxy, or traefik, but, as it were, built into Kubernetes.

All the concepts I have described are, in fact, resources that exist inside a Kubernetes cluster. To describe them in the cube, the yaml format is used, which is more readable and familiar than HCL files in the case of Nomad. But structurally, they describe the same thing in the case of, for example, a pod. They say - I want to deploy such and such pods there, with such and such images, in such and such quantity.

In addition, we realized that we didn’t want to manually create each individual resource: deployment, services, Ingress, and so on. Instead, we wanted to describe our each system in terms of Kubernetes during deployment, so that we didn’t have to manually re-create all the necessary resource dependencies in the right order. Helm was chosen as such a system that allowed us to do this.

Basic concepts in Helm

Helm is package manager for Kubernetes. It is very similar to how package managers work in programming languages. They allow you to store a service consisting of, for example, deployment nginx, deployment php-fpm, config for Ingress, configmaps (this is an entity that allows you to set env and other parameters for your system) in the form of so-called charts. At the same time, Helm runs on top of Kubernetes. That is, this is not some kind of system standing aside, but just another service running inside the cube. You interact with it via its API via a console command. Its convenience and charm is that even if helm breaks or you remove it from the cluster, your services will not disappear, since helm essentially only serves to start the system. Kubernetes itself is further responsible for the health and condition of services.

We also realized that standardization, which until then we had to do on our own through the introduction of jinja into our configs, is one of the main features of helm. All the configs you create for your systems are stored in helm as templates, a bit like jinja, but actually using the templating of the Go language in which helm is written, just like Kubernetes.

Helm adds a few more additional concepts to us.

Chart is a description of your service. In other package managers, it would be called bundle, bundle, or something similar. Here it is called chart.

Values are the variables you want to use to build your configs from templates.

Release. Every time a service that deploys with helm gets an incremental version of the release. Helm remembers what the service config was for the previous release, the year before last, and so on. Therefore, if you need to roll back, just run the helm callback command, pointing it to the previous version of the release. Even if the corresponding configuration in your repository is not available at the time of the rollback, helm will still remember what it was and roll back your system to the state it was in the previous release.

In the case when we use helm, the usual configs for Kubernetes also turn into templates in which it is possible to use variables, functions, and apply conditional statements. Thus, you can collect the config of your service depending on the environment.

In practice, we decided to do things a little differently than we did with Nomad. If Nomad stored both deployment configs and n-variables that are needed to deploy our service in one repository, then here we decided to divide them into two separate repositories. The "deploy" repository stores only the n-variables needed for deployment, while the "helm" repository stores configs or charts.

What did it give us?

Despite the fact that we do not store any really sensitive data in the configuration files themselves. For example, database passwords. They are stored as secrets in Kubernetes, but nevertheless, there are still separate things that we do not want to give access to everyone in a row. Therefore, access to the "deploy" repository is more limited, and the "helm" repository contains just a description of the service. For this reason, it can be given access safely to a larger circle of people.

Since we have not only production, but also other environments, thanks to this division, we can reuse our helm charts to deploy services not only to production, but also, for example, to a QA environment. Even to deploy them locally using minicube - this is such a thing for running Kubernetes locally.

Inside each repository, we left the division into separate directories for each service. That is, inside each directory there are templates related to the corresponding chart and describing the resources that need to be deployed to launch our system. In the "deploy" repository, we left only envy. In this case, we didn't use jinja templating because helm provides templating out of the box - that's one of its main features.

We left the deployment script - deploy.sh, which simplifies and standardizes the launch for deployment using helm. Thus, for anyone who wants to deploy, the deployment interface looks exactly the same as it was in the case of deploying through Nomad. The same deploy.sh, the name of your service, and where you want to deploy it. This causes helm to run internally. He, in turn, collects configs from templates, substitutes the necessary values ​​​​files into them, then deploys them, launching them into Kubernetes.

Conclusions

The Kubernetes service appears to be more complex than Nomad.

This is where outgoing traffic comes in Ingress. This is just the front controller, which takes over all requests and subsequently sends them to the services corresponding to the request data. It determines them based on configs that are part of the description of your application in helm and that developers set themselves. The service, on the other hand, sends requests to its pods, that is, specific containers, balancing incoming traffic between all containers that belong to this service. And, of course, we should not forget that we should not go anywhere from network-level security. Therefore, segmentation works in the Kubernetes cluster, which is based on tagging. All services have certain tags, to which the access rights of services to certain external / internal resources inside or outside the cluster are attached.

As we transitioned, we saw that Kubernetes has all the features of the Nomad we've been using so far, and adds a lot of new stuff as well. It can be extended through plugins, and in fact through custom resource types. That is, you have the opportunity not only to use something that comes with Kubernetes out of the box, but to create your own resource and service that will read your resource. This gives you more options to expand your system without having to reinstall Kubernetes and without having to make changes.

An example of such use is Prometheus, which we run inside a Kubernetes cluster. In order for it to start collecting metrics from a particular service, we need to add an additional type of resource to the service description, the so-called monitor service. Prometheus, due to the fact that it can read, being launched in Kubernetes, a custom type of resource, automatically starts collecting metrics from the new system. It's convenient enough.

The first deployment we did in Kubernetes was in March 2018. And during this time we never experienced any problems with him. It works quite stably without significant bugs. In addition, we can expand it further. To date, we have enough of the capabilities that it has, and we really like the pace of development of Kubernetes. At the moment, more than 3000 containers are in Kubernetes. The cluster occupies several Node. At the same time, it is serviceable, stable and very controllable.

Source: habr.com