In anticipation of Vitaly Khabarov interviewed Dmitry Stolyarov (), the CTO and co-founder of the company "Flant." Vitaly asked Dmitry about what "Flant" does, about Kubernetes, ecosystem development, and support. They discussed why Kubernetes is needed and if it is needed at all. Also, they talked about microservices, Amazon AWS, the "I'll get lucky" approach in DevOps, the future of Kubernetes itself, why, when, and how it will take over the world, the prospects of DevOps, and what engineers should prepare for in the near future with simplification and neural networks.
in podcast form can be listened to on DevOps DeFlop â a Russian-language podcast about DevOps, and below is the text version.
Hereafter, the questions are asked by an engineer from Express42.
About "Flant"
â Dima, hello. You are the CTO of "" and also its founder. Please tell us what the company does and what your role is there?
Dmitry: From the outside, it seems like we are the kind of guys who go around setting up Kubernetes for everyone and doing something with it. But that's not true. We started as a company focused on Linux, but for a long time, our main activity has been servicing production and high-load projects turnkey. Usually, we build the entire infrastructure from scratch and then take full responsibility for it for a long time. Therefore, the main work that "Flant" performs, for which we get paid, is taking responsibility and implementing production turnkey..
As the CTO and one of the founders of the company, I work around the clock to come up with ways to increase production availability, simplify its operation, ease the lives of admins, and make life easier and more pleasant for developers.
About Kubernetes
â Lately, I've seen a lot of talks from "Flant" about Kubernetes. How did you come to it? : I've told this many times before, but I don't mind repeating it at all. I believe it's important to keep repeating this topic because confusion arises between cause and effect.
Dmitry: I have already talked about this many times, but I don't mind repeating it at all. I believe it is important to revisit this topic because there is confusion between cause and effect.
We really needed a tool. We faced a lot of problems, struggled, overcame them with various workarounds, and felt the need for a proper tool. We explored many options, built our own solutions, and gained experience. Gradually, we began using Docker almost as soon as it appearedâaround 2013. At the time of its emergence, we already had extensive experience with containers; we had even written our own equivalent of Dockerâsome workarounds in Python. With Docker's arrival, we were able to discard those workarounds and use a reliable, community-supported solution.
The story with Kubernetes is similar. By the time it started gaining tractionâspecifically version 1.2âwe already had a bunch of workarounds in both Shell and Chef that we were trying to use to orchestrate Docker. We seriously considered Rancher and various other solutions, but then Kubernetes came along, perfectly implemented the ideas we had, or even did it better. There was nothing to criticize.
Yes, there are some unfinished features here and thereâmany unfinished elements, and 1.2 is quite rough, but... Kubernetes is like a building under constructionâyou look at the project and understand that itâs going to be great. If the building currently has a foundation and two floors, you realize it's better not to move in just yet, while with software, such problems donât existâyou can start using it right away.
We never had a moment of doubt about whether to use Kubernetes or not. We were waiting for it long before it was released and tried to create our own analogs.
About Kubernetes
â Are you directly involved in the development of Kubernetes?
Dmitry: Moderately. We are more involved in the development of the ecosystem. We submit a number of pull requests: to Prometheus, various operators, Helmâinto the ecosystem. Unfortunately, I am unable to keep track of everything we do and may make mistakes, but we have not submitted a single pull request to the core.
â At the same time, do you develop many of your own tools around Kubernetes?
Dmitry: The strategy is like this: we contribute and make pull requests to everything that already exists. If pull requests are not accepted, we simply fork those projects for ourselves and continue using them until they are accepted with our builds. Then, when that reaches upstream, we transition back to the upstream version.
For example, we have the Prometheus operator, with which we switched back and forth to the upstream of our build about 5 times, I suppose. We need some feature, we sent a pull request, we need to roll it out tomorrow, and we donât want to wait for it to be released in upstream. Accordingly, we collect it for ourselves, rolling out our build with the feature that we need across all our clusters. Then, for instance, it gets brought back to upstream with the words: 'Guys, letâs make this for a more general case,' and we, or someone else, finalizes it, and over time it flows back.
We strive to develop everything that exists.Many elements that either do not yet exist, have not been conceived, or have been conceived but not implemented â we create. And not because we like the process itself or building bicycles as an industry, but simply because we need this tool. We are often asked why we made this or that thing. The answer is simple â because we needed to move forward, solve some practical problem, and we resolved it with this tool.
The path is always like this: we search very thoroughly and if we canât find any solution, like how to make a trolleybus out of a loaf of bread, then we make our own loaf and our own trolleybus.
Flant Tools
I know that Flant currently has addon operators, shell operators, and dapp/werf tools. As I understand it, these are the same tool in different incarnations. I also understand that there are many different tools within Flant. Is that correct?
DmitryWe have a lot more on GitHub. From what I can recall right now, we have statusmap â a panel for Grafana that has been widely accepted. It is mentioned in almost every second article about monitoring Kubernetes on Medium. It's impossible to briefly describe what statusmap is â it requires a separate article, but it is a very useful thing for monitoring status over time, as in Kubernetes we often need to show the status over time. We also have LogHouse â a tool based on ClickHouse and some black magic for collecting logs in Kubernetes.
Many utilities! And there will be even more, as some internal solutions will be released this year. Among the very large ones based on the addon operator, there are a bunch of addons for Kubernetes, such as how to properly set up cert-manager â a tool for managing certificates, or how to correctly set up Prometheus with a lot of extensions â a good twenty different binaries that export data and collect something, plus Prometheus offers amazing graphics and alerts. All of this is just a ton of addons for Kubernetes that are installed in the cluster, turning it from simple to sophisticated, feature-rich, and automatic, where many questions have already been resolved. Yes, we are doing a lot.
Ecosystem Development
â I believe this is a significant contribution to the development of this tool and its methods of use. Can you estimate who else might contribute similarly to the ecosystem's development?
Dmitry: In Russia, among the companies operating in our market â no one is even close.. Of course, this is a bold statement, because there are major players like Mail and Yandex â they are also doing something with Kubernetes, but even they haven't come close to the contributions made by companies globally, who are doing much more than we are. It's hard to compare 'Flant' with a team of 80 people and Red Hat, which has 300 engineers just for Kubernetes, if I'm not mistaken. It's tough to compare. We have 6 people in our RnD department, including me, who build all our tools. 6 people versus 300 engineers from Red Hat â it's a bit difficult to compare.
â Nevertheless, when even these 6 people can create something genuinely useful and transferable, when they face a practical challenge and provide a solution to the community â it's an interesting case. I understand that in large tech companies, where they have their own development and support team for Kubernetes, similar tools can indeed be developed. This is an example for them that something can be developed and given to the community, giving a boost to the entire community that uses Kubernetes.
DmitryThis is probably the integrator's trick, its characteristic. We have many projects, and we see a variety of situations. Our main way to create added value is to analyze these cases, find commonalities, and maximize cost efficiency for us. We are actively engaged in this. It's hard for me to speak about Russia and the world, but we have about 40 DevOps engineers in the company who work with Kubernetes. I don't think there are many companies in Russia with a comparable number of specialists knowledgeable about Kubernetes, if any exist at all.
I understand everything about the title of DevOps Engineer; everyone understands it and has gotten used to calling DevOps engineers as such, we aren't going to discuss that. All 40 of these wonderful DevOps engineers face problems every day and solve them; we simply analyze this experience and try to generalize it. We understand that if this knowledge stays internal, then in a year or two the tool becomes useless because somewhere in the community a ready-made solution will appear. There's no sense in hoarding this experience internallyâit's just draining energy and time into dev/null. But we absolutely don't mind. We publish everything with great pleasure and know that it needs to be published, developed, promoted, and circulated so that people can use it and add their own experiencesâthen everything grows and thrives. In two years, the tool won't end up in the trash. We don't mind continuing to invest effort because it's clear that someone is using our tool, and in two years everyone will be using it.
This is part of our larger strategy with dapp/werf.I donât remember when we started working on it, it seems like three years ago. Initially, it was done entirely in shell. It was a super proof of concept, addressing some of our specific tasksâit worked! But there are problems with shell; it's impossible to scale further, and programming in shell is quite the ordeal. We had a habit of writing in Ruby, so we transformed some aspects and kept developing it until we encountered the issue that the community, the crowd that doesnât say âwe want or donât want,â turns its nose up at Ruby, however funny that may seem. We realized we needed to write all this in Go, just to meet the first item on the checklist: The DevOps tool must be a static binary.Whether it's Go or not is not that important, but it's better to have a static binary written in Go.
We invested effort, rewrote the dapp in Go, and named it werf. The dapp is no longer supported, not being developed, and works on some last version, but there is an absolute upgrade path upward that one can follow.
Why was the dapp created
â Can you briefly explain why the dapp was created and what problems it solves?
Dmitry: The first reason is the build. Initially, we had significant problems with building when Docker wasnât capable of multi-stage builds, and we had to implement multi-stage on our own. Then, we had a ton more questions with cleaning up images. Everyone dealing with CI/CD sooner or later faces the issue of having a lot of built images, and we need to clean up whatâs unnecessary and keep what is needed.
The second reason is deployment. Yes, there's Helm, but it only addresses part of the tasks. Strangely enough, itâs stated that 'Helm is the Package Manager for Kubernetes.' Specifically, 'the.' There are also the words 'Package Manager' â what is usually expected from a Package Manager? We say, 'Package Manager â install the package!' and expect it to tell us: 'The package has been installed.'
Interestingly, we say, 'Helm, install the package,' but when it responds that it installed it, it turns out that it only started the installation â it told Kubernetes: 'Run this thing!' But whether it started or not, whether it works or not, Helm does not address this question at all.
It results in Helm being just a text preprocessor that uploads data to Kubernetes.
However, within any deployment, we want to know â did the application go live or not? Going live means that the application has been deployed there, a new version has rolled out, and at least it does not crash and responds correctly. Helm does not tackle this task at all. To solve it, a lot of effort is needed because itâs necessary to give Kubernetes the command to roll it out and monitor what happens â whether it has rolled out or not. Plus, there are many tasks related to deployment, cleaning, and building.
Plans
Later this year, we will move towards local development. We want to achieve what used to be possible with Vagrant â just typing 'vagrant up' would spin up virtual machines. We aim for a state where a project exists in Git, and we simply write 'werf up', which launches a local copy of that project, deployed in a local mini-Kub, with all directories connected for convenient development. Depending on the programming language, this is executed differently, but the goal is to facilitate smooth local development with mounted files.
The next step for us is to heavily invest in developer convenience. To quickly launch a project locally with one tool, develop it, push it to Git, and have it deployed to staging or testing based on the pipelines, and then use the same tool to go to production. This unity, unification, and reproducibility of the infrastructure from the local environment to production is a very important aspect for us. But this is not yet available in werf â it is something we are planning to implement.
However, the path to dapp/werf has always been similar to how we started with Kubernetes. We faced challenges and solved them with workarounds â coming up with various shell solutions, or anything else. Then we tried to streamline those workarounds, generalize them, and consolidate them into binaries that we simply share.
There is also another perspective on this whole story, with analogies.
Kubernetes is like a car frame with an engine. There are no doors, windows, radio, or anything â just the frame and the engine. And thereâs Helm â thatâs the steering wheel. It's great to have the steering wheel, but you also need the steering column, the steering rack, transmission, and wheels; without them, it's not functional.
In the case of werf, it's another component for Kubernetes. Right now, we have an alpha version of werf, where, for example, Helm is actually compiled into werf, because we got tired of managing it separately ourselves. There are many reasons for doing it this way, and I will explain in detail why we integrated Helm completely along with Tiller into werf .
Now werf is a more integrated component. We get a ready steering wheel, a steering pin - I'm not very knowledgeable about cars, but it's a large module that addresses a significant range of tasks. We don't need to rummage through the catalog, picking one part to match another, figuring out how to fasten them together. We receive a ready-made combine that solves a large number of tasks right away. Yet inside, it is all built from the same open-source components, still using Docker for assembly, Helm for part of the functionality, and there are a few other libraries as well. It's an integrated tool for quickly and conveniently achieving great CI/CD out of the box.
Is it difficult to maintain Kubernetes?
â You're talking about the experience of starting to use Kubernetes; for you, it's a frame, an engine, and there are many different things you can attach to it: a body, a steering wheel, pedals, seats. The question arises - how challenging is it for you to maintain Kubernetes? You have a wealth of experience; how much time and resources do you spend specifically on maintaining Kubernetes, apart from everything else?
Dmitry: This is a very complex question, and to answer it, we need to understand what maintenance is and what we want from Kubernetes. Perhaps you can elaborate?
â As far as I know and as I see, many teams want to try Kubernetes these days. Everyone is jumping in and installing it on their own. I have the feeling that people don't always grasp the complexity of this system.
Dmitry: That's true.
â How difficult is it to take Kubernetes and set it up from scratch so that it is production-ready?
Dmitry: How do you think it is to transplant a heart? I understand that's a compromising question. Using a scalpel without making mistakes isn't that difficult. If you're told where to cut and where to stitch, then the procedure itself isn't hard. The challenge is guaranteeing success each time.
Setting up Kubernetes and making it work is easy: snap! â installed, there are plenty of installation methods. But what happens when problems arise?
Questions always arise - what have we overlooked? What have we not done? Which Linux kernel parameters did we set incorrectly? My God, did we even specify them at all?! Which Kubernetes components did we install, and which did we not? Thousands of questions come up, and to answer them, you need to have spent 15-20 years immersed in this industry.
I have a fresh example on this topic that can highlight the essence of the question "Is it difficult to maintain Kubernetes?" Some time ago, we seriously considered whether we should implement Cilium as the networking layer in Kubernetes.
Let me explain what Cilium is. In Kubernetes, there are many different implementations of the networking subsystem, and one of them is really cool â it's Cilium. What's its essence? Some time ago, the ability to write hooks for the kernel was added, which intrude into the networking subsystem and various other subsystems, allowing to bypass large parts of the kernel.
Historically, the Linux kernel has ip rout, net-filter, bridges, and many different old components that are 15, 20, or even 30 years old. Overall, they work well; everything is great, but now we've piled on containers, and it looks like a tower made of 15 bricks stacked on top of each other, while you're standing on it on one leg â it feels strange. This system has historically developed with many nuances, like an appendix in the body. In some situations, there are performance problems, for example.
There is a wonderful BPF and the ability to write hooks for the kernel â the guys wrote their own hooks for the kernel. A packet comes into the Linux kernel; they take it out right at the entrance, process it as needed without bridges, without TCP, without the IP stack â in short, bypassing everything that's written in the Linux kernel, and immediately spit it out into the container.
What did we get? Very cool performance, great features â it's just fantastic! But we look at this and see that on each machine there is a program that connects to the Kubernetes API and based on the data it receives from this API, generates C code and compiles binaries that are loaded into the kernel, so that these hooks work in kernel space.
What will happen if something goes wrong? We don't know. To understand this, you need to read all this code, grasp all the logic, and thatâs incredibly complex. But, on the other hand, there are these bridges, net-filters, ip rout â I haven't read their sources, nor have the 40 engineers who work in our company. Maybe only a few understand certain parts.
Whatâs the difference? Thereâs an IP route, a Linux kernel, and a new tool â what difference does it make? We donât understand either one. But weâre afraid to use the new â why? Because if a tool has been around for 30 years, all the bugs have been found, all the pitfalls encountered; you donât need to know everything â it works like a black box, and it always works. Everyone knows where to insert their diagnostic screwdriver and when to run tcpdump. Everyone is well acquainted with diagnostic utilities and understands how this set of components operates within the Linux kernel â not how it's structured, but how to use it.
But the incredibly cool Cilium is not 30 years old; itâs still new. Kubernetes has the same issue, itâs a copy. Both Cilium and Kubernetes can be installed seamlessly, but when something goes wrong in production, can you quickly figure out what went wrong in a critical situation?
When we talk about how difficult it is to maintain Kubernetes â itâs not, itâs very simple, and yes, itâs incredibly complicated. Kubernetes works wonderfully by itself, but there are billions of nuances.
About the approach "Iâll get lucky"
â Are there companies where these nuances will almost certainly appear? Letâs suppose Yandex suddenly migrates all its services to Kubernetes; there will be quite a load there.
Dmitry: No, this conversation isnât about the load; itâs about simple things. For instance, we have Kubernetes, and weâve deployed an application there. How do we know itâs working? Thereâs no ready-made tool to confirm that the application isnât crashing. Thereâs no ready system that sends alerts â you have to set up those alerts and every graph. And we are updating Kubernetes.
Ubuntu 16.04 is available. It can be considered an old version, but we are still using it because it is LTS. It has systemd, which has the nuance that it does not clean up C groups. Kubernetes launches pods, creates C groups, then deletes the pods, and somehow it turns outâI don't remember the details, sorryâthat systemd slices remain. This leads to the fact that over time, any machine begins to lag significantly. This is not even a matter of high load. If persistent pods are launched, for example, if there is a Cron Job that continuously generates pods, then a machine with Ubuntu 16.04 will start to lag after a week. There will constantly be a high load average due to the creation of a lot of C groups. This is a problem faced by anyone who simply installs Ubuntu 16 and runs Kubernetes on top of it.
Let's say he somehow updates systemd or something else, but in the Linux kernel before 4.16, it's even funnierâwhen C groups are deleted, they leak in the kernel and actually are not removed. Therefore, after a month of work on this machine, it will be impossible to view memory statistics by pods. We pull out a file, process it, and one file takes 15 seconds to process because the kernel takes a long time to count internally through a million C groups that seem to be deleted, but they are leaking.
There are still a lot of such minor issues here and there. This is not a question that giant companies might occasionally encounter under very high loadsâno, this is a matter of everyday occurrences. People can live like this for monthsâthey installed Kubernetes, deployed the applicationâit seems to work. For many, this is normal. They won't even know that at some point this application will crash, the alert won't come, but for them, this is the norm. They used to live on virtual machines without monitoring, now they've moved to Kubernetes also without monitoringâwhat's the difference?
The question is that when we walk on ice, we never know its thickness unless we measured it beforehand. Many walk and don't worry because they walked on it before.
From my point of view, the nuance and complexity of operating any system is to ensure that the thickness of the ice is certainly enough to meet our tasks. That's what it's about.
In IT, it seems to me that there are too many approaches of âI will get lucky.â Many set up software, use libraries in the hope that they will get lucky. Overall, many do get lucky. Probably that's why it works.
â From my pessimistic assessment, it looks like this: when risks are high, and the application must work, support from âFlantâ is needed, possibly from Red Hat, or an internal team dedicated specifically to Kubernetes is required, ready to handle it.
Dmitry: Objectively, that's the case. Diving into the Kubernetes story alone for a small team involves a certain amount of risks.
Do we need containers?
â Can you tell me how widespread Kubernetes is in Russia?
Dmitry: I donât have that data, and Iâm not sure if anyone does. We keep saying: âKubernetes, Kubernetes,â but thereâs another perspective on this issue. I also donât know how widespread containers are, but I know from reports online that 70% of containers are orchestrated by Kubernetes. That was from a reliable source based on a fairly large sample globally.
Another question isâdo we need containers? My personal feeling and the overall stance of âFlantâ is that Kubernetes is the de facto standard.
Nothing but Kubernetes will exist.
This is an absolute game-changer in the field of infrastructure management. Just absolutelyâno more Ansible, Chef, virtual machines, Terraform. Iâm not even talking about the old, outdated methods. Kubernetes is an absolute changer, and from now on, it will only be this way.
It is clear that some will need a couple of years, while others may take several decades to realize this. I have no doubt that there will be nothing but Kubernetes and this new perspective: we no longer run the operating system, but use infrastructure as code, except not with code, but with ymlâdeclaratively described infrastructure. I have the feeling that it will always be this way.
â So those companies that havenât yet transitioned to Kubernetes will inevitably do so or be forgotten. Did I understand you correctly?
Dmitry: This is not entirely correct either. For instance, if we need to set up a DNS server, we can run it on FreeBSD 4.10, and it can work perfectly fine for 20 years. Just work, that's it. Perhaps, over 20 years, something might need to be updated just once. If we talk about software in the format where we launched it and it actually runs for many years without any updates or changes, then, of course, Kubernetes wonât be there. Itâs simply not needed.
Anything related to CI/CD â wherever Continuous Delivery is required, where version updates and active changes are needed, wherever resilience must be built â itâs only Kubernetes.
About microservices
â Here I have a small dissonance. To work with Kubernetes, internal or external support is required â thatâs the first point. The second â when we are just starting development, we are a small startup, we still have nothing, developing for Kubernetes or even for microservices architecture can be complex, and not always economically justified. Iâm curious about your opinion â should startups from scratch immediately start developing for Kubernetes or can they write a monolith first and then move to Kubernetes?
Dmitry: Great question. I have a talk about microservices Iâve encountered many times that people are trying to hammer nails with a microscope. The approach itself is correct, we design our internal software exactly in this way. But when doing this, you need to clearly understand what youâre doing. What I hate the most about microservices is the word "micro." Historically, this word emerged, and for some reason, people think that micro means very small, less than a millimeter, like a micrometer. That is not the case.
For example, thereâs a monolith written by 300 people, and everyone involved in the development understands there are problems, and it needs to be broken down into smaller pieces â about 10, each written by at least 30 people. This is important, necessary, and great. But when a startup comes to us, where 3 very cool and talented guys have crudely written 60 microservices, each time I'm looking for a sedative.
It seems to me that this has been discussed thousands of times â a distributed monolith has emerged in one form or another. This is economically unjustifiable and very complicated overall. I've seen this so many times that it hurts, which is why I continue to talk about it.
To the initial question, there is a conflict between the fact that, on one hand, Kubernetes is frightening to use because itâs unclear what might break or not work, and on the other hand, itâs clear that everything is moving in that direction and nothing but Kubernetes will remain. The answer is â to weigh the amount of benefit that comes against the amount of tasks you can solve.This is on one side of the scale. On the other side are the risks associated with downtime or reduced response times, levels of availability â with decreased performance metrics.
Itâs either that we move quickly, and Kubernetes allows many things to be done much faster and better, or we use reliable, time-tested solutions but move much slower. This choice each company must make. You can think of it as a path in the jungle â the first time you walk it, you might encounter a snake, a tiger, or a rabid badger, but after walking it ten times, you pave the path, clear the branches, and it becomes easier to walk. With each time, the path gets wider. Eventually, it becomes a paved road, and later a beautiful boulevard.
Kubernetes does not stand still. Again the question: Kubernetes, on one hand, consists of 4-5 binaries, and on the other hand, itâs the entire ecosystem. Itâs the operating system that runs on our machines. What is it? Ubuntu or Curios? Itâs the Linux kernel, a bunch of additional components. All these things here â they removed a poisonous snake from the road, there they put up a fence. Kubernetes is rapidly and dynamically evolving, and the volume of risks, the amount of the unknown decreases each month, and accordingly, this scale gets rebalanced.
In response to the question of what a startup should do, I would say â come to 'Flant', pay 150,000 rubles, and get a turnkey DevOps easy service. If you are a small startup with a few developers, this works. Instead of hiring your own DevOps who will need to learn how to solve your problems while you pay them a salary, you will get a turnkey solution for all issues. Yes, there are some downsides. As an outsourcer, we cannot be as involved and respond quickly to changes. But we have tons of expertise and ready practices. We guarantee that in any situation we will manage to quickly figure things out and revive any Kubernetes from the dead.
I categorically recommend outsourcing for startups and established businesses until you can allocate a team of 10 people for operations, because otherwise it makes no sense. It absolutely makes sense to outsource.
About Amazon and Google
â Can hosting solutions from Amazon or Google be considered as outsourcing?
Dmitry: Yes, of course, this addresses a number of issues. But again, there are nuances. You still need to understand how to use it. For example, there are a thousand little details in working with Amazon AWS: the Load Balancer needs to be warmed up or you have to submit a request in advance saying, 'Guys, we will get traffic, please warm up our Load Balancer!' You need to know these nuances.
When you consult people who specialize in this, you get nearly all standard issues addressed. We currently have 40 engineers, and by the end of the year there will likely be 60 â we have definitely encountered all these things. Even if we face this problem on some project again, we quickly ask each other and know how to solve it.
Perhaps the answer is that, of course, hosted solutions alleviate some part. The question is whether you are ready to trust these hosts, and will they solve your problems. Amazon and Google have established a good reputation. For all our cases â definitely. We have no additional positive experiences. All other clouds we tried to work with create a lot of problems â whether itâs Ager, all options available in Russia, or various OpenStack implementations: Headster, Overage â whatever you want. They all create problems that are undesirable to solve.
Therefore, the answer is yes, but, in fact, there aren't many mature hosted solutions.
Who needs Kubernetes?
â But seriously, who needs Kubernetes? Who should switch to Kubernetes, and who is a typical client of "Flant" that comes specifically for Kubernetes?
Dmitry: This is an interesting question because right now, many are coming to us in the wave of Kubernetes: "Guys, we know you do Kubernetes, make it for us!" We reply: "Gentlemen, we don't do Kubernetes; we do production and everything related to it." Because creating production without setting up the entire CI/CD and all that entails is simply impossible these days. Everyone has moved away from the separation of development being development and operations being operations.
Our clients expect various things, but they all hope for some kind of miracle, that they have certain problems, and nowâvoila!âKubernetes will solve them. People believe in miracles. While they understand rationally that there won't be a miracle, they still hopeâwhat if this Kubernetes solves everything for us; there's so much talk about it! What if itâs now, poof!âthe silver bullet, poof!âand we have 100% uptime, all developers can release something to production 50 times, and it won't crash. In general, a miracle!
When such people come to us, we say: "Sorry, but miracles donât happen." To be healthy, you need a balanced diet and to exercise. To have a reliable production environment, it needs to be built reliably. To have a convenient CI/CD, it needs to be set up properly. This is a lot of work that needs to be done.
To answer the question of who needs KubernetesâKubernetes is not needed by anyone.
Some people have a mistaken impression that they need Kubernetes. What people really need is a deep desire to stop thinking, worrying, or being concerned about all the infrastructure problems and the issues of launching their applications. They want their applications to just work and be deployed easily. For them, Kubernetes is the hope that they will stop hearing about stories like "we were stalled" or "we can't roll it out," or something else.
The typical person who comes to us is a tech director. They are asked two things: on one hand, they want features; on the other hand, they want stability. We propose taking that burden on ourselves and handling it. The silver bulletâmore precisely, the silver-plated oneâis that you will stop thinking about these problems and wasting time. You will have dedicated people who will handle this issue.
The statement that we or someone needs Kubernetes is incorrect.
Kubernetes is very much needed by admins because it is a really interesting tool to play around with. Let's be honest â everyone loves toys. We are all like kids at some point, and when we see something new, we want to play with it. Some may have lost that interest, for example in administration, because they have already played enough and are simply tired of it. However, no one has completely lost that interest. For instance, while I may be tired of toys in system administration and DevOps, I still love toys and continue to buy new ones. People, in one way or another, always want some toys.
Do not play with production. What I would categorically advise against, which I currently see happening widely: "Oh, a new toy!" â rush to buy it, then say: "Let's introduce it here, so we can show it to all our friends." Don't do that. I apologize; I have kids growing up and I constantly notice something in them, see it in myself, and then generalize it to others.
The final answer: you do not need Kubernetes. You need to solve your problems.
You can achieve the following:
- the production doesn't go down;
- even if it tries to crash, we know about it in advance and can mitigate the issue;
- we can change it at the speed required for our business, and do so conveniently without any issues.
There are two real needs: reliability and flexibility in deployment. Everyone working on IT projects, regardless of the industry â software for easing the world, and who understand this, need to address these needs. With the right approach, understanding, and sufficient experience, Kubernetes can help meet them.
About serverless
â Looking a little further into the future, in an effort to solve the headache of infrastructure management, deployment speed, and application changes, new solutions such as serverless are emerging. Do you see potential in this direction and, shall we say, a threat to Kubernetes and similar solutions?
DmitryHere, I need to remark again that I am not a seer who looks ahead and says - it will be this way! Although I just did the same thing. I look down and see a bunch of problems, like how transistors work in a computer. Ridiculous, right? We're facing some bugs in the CPU.
Making serverless reliable enough, cheap, efficient, and convenient requires solving all the ecosystem issues. I agree with Elon Musk on the need for a second planet to ensure humanity's resilience. Although I donât know what he means, I understand that Iâm not ready to fly to Mars myself, and that wonât happen tomorrow.
With serverless, it's clear that it's ideologically the right thing, just like having a second planet is better for humanity than one. But how do we achieve this now? Sending a single expedition is not a problem if we focus efforts on it. I think sending several expeditions to settle thousands of people there is also realistic. However, creating full resilience so that half of humanity lives there seems currently impossible and not consider it feasible.
It's the same with serverless: it's a cool concept, but it is far from solving the problems of 2019. Closer to 2030 - let's live to see that. I have no doubt we will live to see it (repeat before sleep), but right now we need to tackle other issues. It's like believing in a magical rainbow unicorn. Yes, a couple of percent of cases are solved, and they are solved excellently, but subjectively, serverless is a rainbow⊠This topic feels too far and too unclear for me. I'm not ready to speak about it. In 2019, you couldnât write any application with serverless.
How Kubernetes will develop
â While weâre heading towards this potentially wonderful distant future, what do you think about how Kubernetes and the ecosystem around it will evolve?
Dmitry: I have thought a lot about this, and I have a clear answer. First, stateful is still easier to make stateless. Kubernetes invested more in this from the start; thatâs where it all began. Stateless works practically perfectly in Kubernetesâthere's simply nothing to complain about. There are still a lot of problems, or rather nuances, with stateful. Everything works perfectly for us there, but thatâs just us. For everyone else to have it working, itâll at least take a couple more years. This is not a calculated indicator, but rather my gut feeling.
In short, stateful needs to develop significantlyâit will developâbecause all our applications store their state; there are no stateless applications. Thatâs an illusion; thereâs always a need for some kind of database and something else. Stateful is about streamlining everything possible, fixing all bugs, improving all issues we are currently facingâlet's call it adoption.
The level of the unexplored, the level of unresolved problems, the likelihood of encountering something will significantly decrease. This is an important story. And operatorsâeverything related to codifying administration logic, management logic, to achieve easy service: MySQL easy service, RabbitMQ easy service, Memcache easy serviceâabsolutely all these components we need to have things working out of the box reliably. This precisely addresses the pain points of wanting a database but not wanting to administer it, or wanting Kubernetes but not wanting to manage it.
This story regarding the development of operators in one form or another will be important in the coming couple of years.
I think the simplicity of operation will increase significantlyâthe box will become more and more of a black box, increasingly reliable, with simpler and simpler controls.
I once listened to an old interview with Isaac Asimov from the 1980s on YouTube on a show called Saturday Night Liveâlike Urgant's show but actually interesting. They asked him about the future of computers. He said that the future lies in simplicity, just like it was with the radio receiver. The radio receiver was initially a complex device. To catch a wave, you had to spend 15 minutes adjusting knobs and really understand how it all worked, grasping the physics of radio waves. In the end, the radio ended up with just one knob.
What radio is there now in 2019? The car radio receives all the waves, displaying the names of the stations. The physics of the process hasn't changed in 100 years; what has changed is the ease of use. Nowadays, and even back in 1980 during an interview with Asimov, everyone used the radio without thinking about how it works. It always worked â it was a given.
Asimov mentioned back then that it would be similar with computers â the ease of use will increase. In 1980, you needed special education just to push buttons on a computer; in the future, it wonât be like that.
I feel that with Kubernetes and infrastructure, the ease of use will also greatly increase. This seems obvious to me â itâs evident.
What will happen to engineers?
â What will happen to the engineers and system administrators who support Kubernetes?
Dmitry: What happened to accountants after the advent of 1C? Itâs pretty much the same. Before, calculations were done on paper â now they are done in a program. Productivity has increased exponentially, but labor hasnât disappeared. If ten engineers were needed to screw in a lightbulb before, now one will suffice.
The amount of software and the number of tasks seem to be growing at a rate higher than the emergence of new DevOps and the increase in efficiency. Currently, there is a specific shortage in the market, and it will last for a long time. Later, everything will reach a certain norm where the efficiency of work will increase, there will be more serverless options, AI will be integrated with Kubernetes to allocate resources just as needed, and it will handle everything on its own â humans, step aside and donât interfere.
But decisions still need to be made by someone. It's clear that the level of qualification and specialization of this person will be higher. In accounting, you donât need ten employees to keep the books just to avoid fatigue. Thatâs simply unnecessary. Many documents are automatically scanned and recognized by electronic document management systems. One smart chief accountant with significantly higher skills and a good understanding is sufficient.
Overall, this path is common in all industries. The same goes for cars: previously, a car came with a mechanic and three drivers. Now, driving a car is a simple process that we all participate in every day. No one thinks that a car is something complicated.
DevOps or systems engineering arenât going anywhereâhigh-level efficiency and productivity will continue to rise.
â Iâve also heard an interesting idea that, in fact, the amount of work will increase.
Dmitry: Of course, a hundred percent! Because the amount of software we're writing is constantly growing. The number of problems we solve with software is constantly increasing. The workload is increasing. Right now, the DevOps market is extremely overheated. This is evident in salary expectations. Ideally, without going into details, there should be juniors wanting X, mid-level wanting 1.5X, and seniors wanting 2X. But currently, if you look at the Moscow DevOps salary market, juniors are asking from X to 3X and seniors from X to 3X.
No one knows how much it costs. The salary level is measured by your confidenceâit's complete chaos, to be honest, a terribly overheated market.
Of course, this situation will change very soonâsome saturation should occur. Software development is differentâdespite the fact that developers are needed everywhere, and good developers are in high demand, the market understands what each person is worthâthe industry has stabilized. It's not the same with DevOps now.
â From what Iâve heard, I concluded that the current system administrator doesnât need to worry too much, but itâs time to enhance skills and prepare for the fact that tomorrow there will be more work, but it will be more qualified.
Dmitry: Absolutely. Overall, we are living in 2019, and the rule of life is: lifetime learningâwe learn throughout our lives. It seems to me that everyone knows and feels this now, but knowing isnât enoughâyou have to act. Every day we must change. If we donât, sooner or later weâll be left on the sidelines of our profession.
Be ready for sharp turns of 180 degrees. I don't exclude situations where something drastically changes, or something new is invented â it happens. Boom! â and now we're acting differently. It's important to be prepared for this and not to stress. It may happen that tomorrow everything I'm doing will become unnecessary â that's okay, I've learned all my life and I'm ready to learn something different. It's not a problem. There's no need to fear job security, but one must be ready to keep learning something new.
Wishes and a moment of advertising
â Do you have any wishes?
Dmitry: Yes, I have a few wishes.
The first and mercenary one â subscribe to . Dear readers, visit YouTube and subscribe to our channel. In about a month, we will start an active expansion on the video service with a ton of educational content about Kubernetes, both open and varied: from practical matters to labs, to deep fundamental theoretical aspects and how to apply Kubernetes at the level of principles and patterns.
The second mercenary wish â visit and give us stars, because we rely on them. If you don't give us stars, we will have nothing to eat. It's like manna in a computer game. We do something, we strive, some say it's terrible bicycles, others say it's all wrong, and we keep going and acting absolutely honestly. We see a problem, solve it, and share our experience. So please give us a star; it won't take away from you but will add to us, because we thrive on them.
The third, important, and already not mercenary wish â stop believing in fairy tales. You are professionals. DevOps is a very serious and responsible profession. Stop playing at work. Let it click for you, and you'll understand. Imagine that you go to the hospital, and there a doctor is experimenting on you. I understand that this might upset some, but most likely it's not about you, but about someone else. Tell others to stop too. This really ruins life for all of us â many start to treat operations, admins, and DevOps people as guys who have broken something again. This 'broken' often happens because we went to play instead of looking at things with a clear mind and understanding how this works.
This doesn't mean that we shouldn't experiment. We need to experiment, and we do it ourselves. To be honest, we also play around sometimesâit's certainly not ideal, but nothing human is alien to us. Let's declare 2019 as the year of serious, thought-out experiments, not just playing around in production. That seems reasonable.
â Thank you very much!
Dmitry: Thank you, Vitaly, for your time and for the interview. Dear readers, thank you very much if you've made it this far. I hope we've shared at least a couple of thoughts with you.
In the interview, Dmitry touched upon the topic of werf. Currently, it's a universal Swiss Army knife that solves almost all tasks. But it hasn't always been that way. At  the festival Dmitry Stolyarov will discuss this tool in detail. In his presentation, everything will be covered: the challenges and hidden nuances of Kubernetes, ways to tackle these difficulties, and the current implementation of werf in detail. Join us on May 27 and 28 as we create perfect tools.
Source: habr.com
