{"id":91199,"date":"2020-08-10T01:42:03","date_gmt":"2020-08-09T23:42:03","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov"},"modified":"2020-08-10T01:42:03","modified_gmt":"2020-08-09T23:42:03","slug":"kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kubernetes-v-domklik-kak-spat-spokojno-upravlyaya-klasterom-na-1000-mikroservisov","title":{"rendered":"Kubernetes in DomClick: how to sleep soundly while managing a cluster of 1000 microservices","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

My name is Viktor Yagofarov, and I am responsible for the development of the Kubernetes platform at Domklik, serving as the technical lead for the Ops team (operations). I would like to share the structure of our Dev Ops processes, the specifics of operating one of the largest k8s clusters in Russia, as well as the DevOps\/SRE practices that our team applies.<\/p>\n

\"Kubernetes
\n<\/a><\/noindex><\/p>\n

Ops Team<\/h4>\n

\nCurrently, there are 15 people in the Ops team. Three of them are responsible for the office, while two work in another time zone and are available at night as well. Thus, there is always someone from Ops at the monitor, ready to respond to incidents of any complexity. We do not have night shifts, which preserves our mental health and allows everyone to rest and engage in leisure activities beyond the computer.<\/p>\n

\"Kubernetes
\n
\nEveryone has different competencies: network engineers, DBAs, specialists in the ELK stack, Kubernetes admins\/developers, monitoring specialists, virtualization experts, hardware specialists, etc. What unites us is that each person can partially substitute any of us: for example, introducing new nodes into the k8s cluster, updating PostgreSQL, writing CI\/CD pipelines + Ansible, automating tasks using Python\/Bash\/Go, connecting hardware in the data center. Strong competencies in one area do not hinder the ability to shift focus and start developing in another field. For instance, I joined the company as a PostgreSQL specialist, and now my primary responsibility is Kubernetes clusters. In the team, any growth is welcomed, and there is a strong sense of support.<\/p>\n

By the way, we are hiring. The requirements for candidates are quite standard. Personally, I think it\u2019s important for a person to fit into the team, be non-conflicting, but also be able to stand their ground, have a desire to develop and not be afraid to try new things, and propose their own ideas. Additionally, skills in scripting languages, knowledge of the basics of Linux, and English language proficiency are mandatory. English is needed simply so that a person can quickly Google a solution to a problem in 10 seconds rather than 10 minutes in case something goes wrong. It's very hard to find specialists with deep knowledge of Linux nowadays: it's funny, but two out of three candidates cannot answer the question, 'What is Load Average? What does it consist of?' and they consider the question, 'How to gather a core dump from a C program' as something from the realm of superhumans... or dinosaurs. It\u2019s something we have to deal with since usually, people have strong competencies in other areas, and we can teach them 'Linux.' The answer to the question, 'Why is all this important for a DevOps engineer in today's cloud world?' will have to be left outside the scope of this article, but in three words: all of this is necessary.<\/p>\n

Tools Team<\/h4>\n

\nThe Tools team plays a significant role in automation. Their main task is to create convenient graphical and CLI tools for developers. For example, our internal development tool Confer allows deploying an application into Kubernetes with just a few clicks, setting up resources, keys from the vault, etc. Previously, we used Jenkins + Helm 2, but we had to develop our own tool to eliminate copy-pasting and bring consistency to the software lifecycle.<\/p>\n

The Ops team does not write pipelines for developers, but they can advise on any questions regarding their writing (some still have Helm 3).<\/p>\n

DevOps<\/h4>\n

\nAs for DevOps, we see it this way:<\/p>\n

Dev teams write code, deploy it through Confer from dev -> qa\/stage -> prod. The responsibility for ensuring that the code doesn\u2019t lag and doesn\u2019t throw errors lies with the Dev and Ops teams. During the day, the on-duty person from the Ops team should primarily respond to incidents with their application, while during the evening and night, the on-call admin (Ops) should wake up the on-call developer if they are sure that the problem does not lie within the infrastructure. All metrics and alerts in monitoring appear automatically or semi-automatically.<\/p>\n

The Ops responsibility zone begins when the application is deployed to production, but Dev's responsibility does not end there \u2014 we are working on the same project and are in the same boat.<\/p>\n

Developers consult with admins if help is needed in writing an admin microservice (for example, Go backend + HTML5), and admins provide advice to developers on any infrastructure issues or questions related to k8s.<\/p>\n

By the way, we do not have a monolith at all, only microservices. Their number currently fluctuates between 900 and 1000 in the production k8s cluster, if measured by the number of deployments<\/i>. The number of pods varies between 1700 and 2000. Currently, there are about 2000 pods in the production cluster.<\/p>\n

I cannot specify the exact numbers, as we monitor unnecessary microservices and eliminate them in a semi-automated manner. Monitoring unnecessary entities in k8s is assisted by useless-operator<\/a><\/noindex>, which greatly saves resources and money.<\/p>\n

Resource Management <\/h2>\n

<\/p>\n

Monitoring <\/h4>\n

\nA well-organized and informative monitoring system becomes a cornerstone in the operation of a large cluster. We have not yet found a universal solution that covers 100% of all monitoring requirements, so we periodically create different custom solutions in this environment.<\/p>\n