It would hardly be incorrect to say that the best of people
find joy through suffering.
Ludwig van Beethoven
I am Sergey, working at Yandex.Money in the performance research team. I want to share the beginning of our story about our journey toward using orchestration — how we chose the tools and what we considered. All events in this article are happening in real time, so you, dear readers, are following the situation almost live.
Why do we need a conductor in the team?
Who is a conductor? From the French diriger — to manage, direct, lead — in the world of music, it is a person who directs the rehearsal and performance of ensemble music. In our case, this role is taken by orchestration and automation systems.
Their role is no different from that of a conductor in music — they are needed to assist the team, direct, and organize their play.
As a rule, the team possesses some set of capacities — let's call them servers, on which they implement their projects.
The approach to obtaining and operating these servers varies. Here are a few examples:
- The team makes a request, for example, to the operations group, asking for resources with specific parameters.
- The operations group provides them with the necessary amount — cloud or bare metal — and commits to maintain them in good condition according to the SLA. Configuration is also done by the operations group.
- The team receives only cloud or bare metal resources from the operations group, and they perform the configuration themselves.
- The team itself 'purchases' resources and fully maintains/configures them independently.
In our team, we use servers that need maintenance — updating the OS, installing new packages, etc.
We have identified them into two main types:
- tank group,
- service group.
The tank group consists of hosts with Yandex.Tank.
The service group includes everything related to support — these are various services for ensuring the release cycle, generating automatic reports, and so on.
At one point, managing all this manually became inconvenient, and we started thinking about automating the entire process, from filling the servers to developing, deploying, and launching our internal service.
Why is a conductor needed, even if the orchestra can play by itself?
To start, we learned Ansible and began 'filling' our bare metal servers so that we would be less dependent on system administrators — this benefits everyone, we gain new skills, and we relieve the administrators of part of the work they always have enough of without us. We strive for development beyond our specialty and for the autonomy of the team as much as possible.
In the company, working with Ansible has been established and regulated for quite some time, so we easily integrated our solution into this process.
Currently, the filling of hosts consists of three Ansible roles:
- the first role installs the OS,
- the second applies basic settings for the host, such as LDAP authorization,
- and the third installs Yandex.Tank in a Docker container along with its dependencies.
Let's move on to the services we use within the team.
For our tasks, we equally use Kotlin and Python, and a little Golang. To unify the development and deployment of our services, we decided to package them in Docker containers. This provides freedom of programming language choice while standardizing the delivery format of our applications.
A small note about ipv6 in Docker
Some of the services we interact with are only available via ipv6, so we had to figure out how to enable ipv6 for containers.
According to the official Docker documentation on ipv6, ipv6 is enabled by adding parameters to daemon.json:
{
"ipv6": true,
"fixed-cidr-v6": "2001:db8:1::/64"
}In this case, the provider must issue an ipv6 subnet, which you will write in fixed-cidr-v6.
However, we chose another option — ipv6 NAT, and here’s why:
- Currently, Docker only with ipv6.
- Having a globally routable address in each container means that all ports (even unpublished ones) become accessible to everyone unless additional filtering is implemented.
- userland proxy for publishing ports, .
ipv6 NAT is , which automatically manages the rules in ip6tables and modifies them when a new container is added.
For this solution to work correctly, several additional manipulations were necessary. It is essential to initialize ip6table_nat in the system. Having the module installed does not guarantee that it will be loaded into the kernel upon startup. We encountered this issue when we received the following error while starting a container with NAT on a fresh host:
2019/01/22 14:59:54 running [/sbin/ip6tables -t filter -N DOCKER --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory
ip6tables v1.6.2: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)The problem was resolved after adding initialization in the Ansible role using the modprobe module and loading it at OS startup with lineinfile:
- name: Add ip6table_nat module
modprobe:
name: ip6table_nat
state: present
- name: Add ip6table_nat to boot
lineinfile:
path: /etc/modules
line: 'ip6table_nat'By the way, there is a good article on Habr , which briefly and clearly describes the advantages and disadvantages of various methods for working with ipv6 in Docker.
But let's return to our initial question:
Why is a conductor needed, even if the orchestra can play by itself?
Now everyone understands how to play in our team:
- the process of ‘pouring’ servers has been created,
- the development and deployment of services have been standardized.
A reasonable question arises — how to effectively and maximally automate the deployment, updating, and monitoring of our services in Docker containers?
Even though each member of the orchestra knows their part, they can get off track and deviate from the initial intent. This is where we come to the point that without a conductor, our orchestra will not rehearse effectively and play cohesively. The conductor is responsible for all aspects of performance, ensuring everything is unified in tempo and mood.
How can we get a good conductor with minimal investment?
The topic of orchestration is quite well developed in the market. But first, let's talk about the auxiliary tools that can help the conductor.
— a system that provides two main functions:
- service discovery,
- a distributed key-value store.
In our orchestra, Consul will be responsible for service registration and storing their configurations. There are two options for registration:
- Active — this is when the service registers itself using the HTTP API;
- Passive — the service must be manually specified.
Vault is a storage solution that standardizes and unifies the secure storage and management of secrets — passwords, certificates.
Here are the benefits we will gain by using this tool:
- A single hub for creating and storing secrets, managing their lifecycle via an HTTP API.
- Transit Secrets Engine — encrypting-decrypting data without storing it. Capability to transmit data in an encrypted form over unprotected communication channels.
- Access policies that are easy to configure.
- Audit access to secrets.
- The ability to create your own CA (Certificate Authority) to manage self-signed certificates within your infrastructure.
Considering all our requirements, two options were suitable for the role of the orchestrator — Kubernetes and Nomad.
Kubernetes
So many articles and books have already been written about it (here's , for example), and many talks have been given, so let me keep it short — it’s a universal tool that can do almost anything. The price for this is not always easy setup and cluster support on Kubernetes.
Nomad
from HashiCorp, a company known for the aforementioned consul and vault.
Nomad seemed to us simpler to install and set up than Kubernetes. One binary file operates as both a server and a client. Moreover, Nomad covers the entire list of tasks we want it to solve: managing the cluster, a fast scheduler, support for multi-datacenter. Plus, by using consul and vault, we get tighter integration for orchestrating our services.
What’s currently in progress:
- servers have been prepared for deploying Consul,
- the Nomad cluster configuration will be stored in Consul, allowing Nomad to be deployed automatically,
- and simultaneously we will install vault for secret storage.
Question to the floor — should we establish an orchestrator for such tasks, or is orchestration fine without it? Tell us in the comments what you think about this.
Subscribe to our blog and stay tuned — we will soon share the final results and whether we configured the Nomad cluster as we wanted.
Join our cozy , where you can always ask for advice, help colleagues, and just chat about performance research and more.
Source: habr.com
