My name is Viktor Yagofarov, and I am responsible for the development of the Kubernetes platform at Domklik, serving as the technical lead for the Ops team (operations). I would like to share the structure of our Dev Ops processes, the specifics of operating one of the largest k8s clusters in Russia, as well as the DevOps/SRE practices that our team applies.
Ops Team
Currently, there are 15 people in the Ops team. Three of them are responsible for the office, while two work in another time zone and are available at night as well. Thus, there is always someone from Ops at the monitor, ready to respond to incidents of any complexity. We do not have night shifts, which preserves our mental health and allows everyone to rest and engage in leisure activities beyond the computer.
Everyone has different competencies: network engineers, DBAs, specialists in the ELK stack, Kubernetes admins/developers, monitoring specialists, virtualization experts, hardware specialists, etc. What unites us is that each person can partially substitute any of us: for example, introducing new nodes into the k8s cluster, updating PostgreSQL, writing CI/CD pipelines + Ansible, automating tasks using Python/Bash/Go, connecting hardware in the data center. Strong competencies in one area do not hinder the ability to shift focus and start developing in another field. For instance, I joined the company as a PostgreSQL specialist, and now my primary responsibility is Kubernetes clusters. In the team, any growth is welcomed, and there is a strong sense of support.
By the way, we are hiring. The requirements for candidates are quite standard. Personally, I think it’s important for a person to fit into the team, be non-conflicting, but also be able to stand their ground, have a desire to develop and not be afraid to try new things, and propose their own ideas. Additionally, skills in scripting languages, knowledge of the basics of Linux, and English language proficiency are mandatory. English is needed simply so that a person can quickly Google a solution to a problem in 10 seconds rather than 10 minutes in case something goes wrong. It's very hard to find specialists with deep knowledge of Linux nowadays: it's funny, but two out of three candidates cannot answer the question, 'What is Load Average? What does it consist of?' and they consider the question, 'How to gather a core dump from a C program' as something from the realm of superhumans... or dinosaurs. It’s something we have to deal with since usually, people have strong competencies in other areas, and we can teach them 'Linux.' The answer to the question, 'Why is all this important for a DevOps engineer in today's cloud world?' will have to be left outside the scope of this article, but in three words: all of this is necessary.
Tools Team
The Tools team plays a significant role in automation. Their main task is to create convenient graphical and CLI tools for developers. For example, our internal development tool Confer allows deploying an application into Kubernetes with just a few clicks, setting up resources, keys from the vault, etc. Previously, we used Jenkins + Helm 2, but we had to develop our own tool to eliminate copy-pasting and bring consistency to the software lifecycle.
The Ops team does not write pipelines for developers, but they can advise on any questions regarding their writing (some still have Helm 3).
DevOps
As for DevOps, we see it this way:
Dev teams write code, deploy it through Confer from dev -> qa/stage -> prod. The responsibility for ensuring that the code doesn’t lag and doesn’t throw errors lies with the Dev and Ops teams. During the day, the on-duty person from the Ops team should primarily respond to incidents with their application, while during the evening and night, the on-call admin (Ops) should wake up the on-call developer if they are sure that the problem does not lie within the infrastructure. All metrics and alerts in monitoring appear automatically or semi-automatically.
The Ops responsibility zone begins when the application is deployed to production, but Dev's responsibility does not end there — we are working on the same project and are in the same boat.
Developers consult with admins if help is needed in writing an admin microservice (for example, Go backend + HTML5), and admins provide advice to developers on any infrastructure issues or questions related to k8s.
By the way, we do not have a monolith at all, only microservices. Their number currently fluctuates between 900 and 1000 in the production k8s cluster, if measured by the number of deployments. The number of pods varies between 1700 and 2000. Currently, there are about 2000 pods in the production cluster.
I cannot specify the exact numbers, as we monitor unnecessary microservices and eliminate them in a semi-automated manner. Monitoring unnecessary entities in k8s is assisted by , which greatly saves resources and money.
Resource Management
Monitoring
A well-organized and informative monitoring system becomes a cornerstone in the operation of a large cluster. We have not yet found a universal solution that covers 100% of all monitoring requirements, so we periodically create different custom solutions in this environment.
- Zabbix. The good old monitoring system, which is primarily designed to track the overall state of the infrastructure. It tells us when a node fails due to CPU, memory, disks, network, and so on. Nothing extraordinary, but we also have a separate DaemonSet of agents through which we monitor the state of DNS in the cluster: we search for lagging pods of coredns and check the availability of external hosts. It may seem unnecessary, but with high traffic volumes, this component is a serious point of failure. Earlier, I already how I dealt with DNS performance in the cluster.
- Prometheus Operator. A set of various exporters provides a broad overview of all cluster components. We then visualize all this on large dashboards in Grafana, and for alerts, we use alertmanager.
Another useful tool for us has been We wrote this after encountering situations multiple times where one team's Ingress conflicts with another team's, resulting in 50x errors. Now, before deploying to production, developers check that they won't disturb anyone, and for my team, this is a useful tool for initial diagnosis of Ingress issues. Interestingly, it was initially written for admins and looked rather 'clunky', but after the tool became favored by development teams, it underwent significant transformation and no longer looked like 'an admin created a web interface for admins'. Soon we will phase out this tool, and similar situations will be validated even before the pipeline rollout.
Team Resources in Kubernetes
Before diving into examples, it's important to explain how we allocate resources for microservices.
To understand which teams and in what amounts use their resources (CPU, memory, local SSD), we assign each team its own namespace in Kubernetes and restrict its maximum capabilities in terms of CPU, memory, and disk, after discussing the teams' needs. Accordingly, one team generally will not block the entire cluster for deployment by allocating thousands of cores and terabytes of memory to itself. Access to namespaces is granted through Active Directory (we use RBAC). Namespaces and their limits are added via a pull request in the GIT repository, and then everything is automatically rolled out through the Ansible pipeline.
Example of resource allocation for a team:
namespaces:
chat-team:
pods: 23
limits:
cpu: 11
memory: 20Gi
requests:
cpu: 11
memory: 20Gi
Requests and Limits
In Kubernetes Request — this is the amount of resources guaranteed to be reserved for pod (one or more Docker containers) in the cluster. A limit is a non-guaranteed maximum. It's often visible in graphs when a certain team sets too many requests for all its applications and cannot deploy an application in Kubernetes because all requests have already been 'spent' under their namespace.
The correct way out of such a situation: monitor actual resource consumption and compare it with the requested amount (Request).
In the screenshots above, you can see that the 'requested' (Requested) CPUs align with the actual number of threads, and Limits can exceed the actual number of CPU threads =)
Now let's take a closer look at a namespace (I chose the kube-system namespace — the system namespace for components of Kubernetes) and examine the ratio of actual CPU time and memory used to what has been requested:
It is obvious that significantly more memory and CPU is reserved for system services than is actually used. In the case of kube-system, this is justified: there have been times when the nginx ingress controller or nodelocaldns peaked in CPU usage and consumed a lot of RAM, so this cushion makes sense. Moreover, we cannot rely on graphs from the last 3 hours: it is preferable to see historical metrics over a longer period.
A 'recommendations' system has been developed. For example, here you can see which resources would benefit from raising their 'limits' (the upper allowed threshold) to avoid 'throttling': the moment when CPU or memory has been exhausted within the allocated quantum of time and is waiting to be 'unthrottled':
And here are the pods that should moderate their appetites:
About throttling + resource monitoring could fill more than one article, so feel free to ask questions in the comments. In short, I can say that the task of automating such metrics is quite complex and requires a lot of time and acrobatics with window functions and CTE in Prometheus / VictoriaMetrics (these terms are put in quotes because there is almost nothing similar in PromQL, and you have to create lengthy queries that span multiple screens of text and then optimize them).
As a result, developers have tools for monitoring their namespaces in Kubernetes, and they can decide where and when to 'trim' resources for certain applications, while allowing some pods to use all CPU overnight.
Methodologies
In the company, as is currently fashionable, we adhere to DevOps and- practices. When a company has 1000 microservices, around 350 developers, and 15 admins managing the entire infrastructure, it is necessary to 'be trendy': behind all these 'buzzwords' lies an acute need for automation of everything, and admins should not be the bottleneck in the processes. SREAs Ops, we provide various metrics and dashboards for developers related to the response times of services and their errors.
We use methodologies such as:
RED , and , combining them together. We strive to minimize the number of dashboards so that at a glance, it's clear which service is currently degrading (for example, response codes per second, response time in the 99th percentile), and so on. As soon as new metrics are needed for the general dashboards, we immediately create and add them.
I haven't drawn any graphs in a month. Probably a good sign: it means that most of the 'wish list' has already been implemented. There were times I would draw some new graph at least once a day during the week.
The resulting outcome is valuable in that developers rarely come to admins with questions of 'where to look for a certain metric.'
Implementation Service Mesh is not far off and should greatly simplify life for everyone; colleagues from Tools are already close to implementing an abstract 'healthy person's Istio': the lifecycle of each HTTP(s) request will be visible in monitoring, and it will always be possible to understand 'at what stage everything broke' during inter-service (and not only) interactions. Subscribe to the company Hub news from DomClick. =)
Kubernetes Infrastructure Support
Historically, we have been using a patched version Kubespray — Ansible role for deploying, scaling, and updating Kubernetes. At some point, support for non-kubeadm installations was removed from the main branch, and a transition process to kubeadm was not proposed. As a result, Southbridge created its fork (with kubeadm support and a quick fix for critical issues).
The process of updating all k8s clusters looks like this:
- We take Kubespray from Southbridge, compare it with our branch, and merge.
- We roll out the update in Stress- 'Cube'.
- We roll out the update one node at a time (in Ansible this is 'serial: 1') in Dev- 'Cube'.
- Update Prod on Saturday evenings one node at a time.
In the future, there are plans to replace Kubespray with something faster and transition to kubeadm.
We have three 'Cubes': Stress, Dev, and Prod. We plan to launch another one (hot standby) Prod 'Cube' in the second data center. Stress and Dev live in 'virtual machines' (oVirt for Stress and VMWare cloud for Dev). Prod- 'Cube' lives on 'bare metal': these are identical nodes with 32 CPU threads, 64-128 GB of memory, and 300 GB SSD RAID 10 — a total of 50 units. Three 'thin' nodes are allocated for the 'masters' of Prod- 'Cube': 16 GB of memory, 12 CPU threads.
For production, we prefer to use 'bare metal' and avoid unnecessary layers like OpenStack: we don't need 'noisy neighbors' and CPU steal timeThe complexity of administration roughly doubles in the case of in-house OpenStack.
For CI/CD of 'Kubernetes' and other infrastructure components, we use a separate GIT server, Helm 3 (we transitioned quite painfully from Helm 2, but we are very pleased with the options, atomic), Jenkins, Ansible, and Docker. We love feature branches and deploying to different environments from a single repository.
Conclusion
This, in general terms, outlines how the DevOps process looks from the perspective of an operations engineer at DomClick. The article turned out to be less technical than I expected: stay tuned for news from DomClick on Habr, as there will be more 'hardcore' articles about Kubernetes and more.
Source: habr.com
