In anticipation Vitaly Khabarov interviewed Dmitry Stolyarov (), technical director and co-founder of Flant. Vitaly asked Dmitry about what Flant does, about Kubernetes, ecosystem development, and support. We discussed why Kubernetes is needed and whether it is needed at all. And also about microservices, Amazon AWS, the “I'm lucky” approach to DevOps, the future of Kubernetes itself, why, when and how it will take over the world, the prospects for DevOps and what engineers should prepare for in the bright and near future with simplification and neural networks.
listen as a podcast on DevOps Deflop - a Russian-language podcast about DevOps, and below - a text version.
Here and below questions are asked engineer from Express42.
About "Flant"
- Hello Dima. You are the technical directorand also its founder. Tell us, please, what does the company do and what are you in it?
Dmitriy: From the outside, it seems like we are the kind of guys who go around, install Kubernetes for everyone and do something with it. But it's not. We started as a Linux company, but for a very long time our main activity has been servicing production and turnkey highload projects. Usually we build the entire infrastructure from scratch and then we are responsible for it for a long, long time. Therefore, the main work that Flant performs, for which it receives money, is taking responsibility and implementing turnkey production.
I, as a technical director and one of the founders of the company, work around the clock to figure out how to increase the availability of production, simplify its operation, make life easier for admins, and make life more enjoyable for developers.
About Kubernetes
- Lately I have seen a lot of reports from Flant and about Kubernetes. How did you come to him?
Dmitriy: I have already talked about this many times, but I do not mind repeating it at all. I think that it is right to repeat this topic, because there is a confusion between cause and effect.
We really needed a tool. We faced a lot of problems, struggled, overcame them with different crutches and felt the need for a tool. We went through many different options, built our own bikes, accumulated experience. Gradually, we got to the point where we started using Docker almost as soon as it appeared - around 2013. At the time of its appearance, we already had a lot of experience with containers, we had already written an analogue of “Docker” - some kind of our crutches in Python. With the advent of Docker, it became possible to throw out the crutches and use a reliable and community-supported solution.
With Kubernetes, the story is similar. By the time it started to gain momentum - for us this is version 1.2 - we already had a bunch of crutches on both Shell and Chef, which we somehow tried to orchestrate Docker. We seriously looked towards Rancher and various other solutions, but then Kubernetes appeared, in which everything is implemented exactly as we would have done or even better. Nothing to complain about.
Yes, there is some kind of imperfection here, there is some kind of imperfection - a lot of imperfections, and 1.2 is generally horror, but .... Kubernetes is like a building under construction - you look at the project and understand that it will be cool. If the building now has a foundation and two floors, then you understand that it’s better not to move in yet, but there are no such problems with the software - you can already use it.
We didn't have a moment that we thought whether to use Kubernetes or not. We were waiting for him long before he appeared, and tried to make analogues ourselves.
About Kubernetes
- Are you directly involved in the development of Kubernetes itself?
Dmitriy: Mediocre. Rather, we participate in the development of the ecosystem. We send a certain number of pull requests: to Prometheus, to all sorts of operators, to Helm - to the ecosystem. Unfortunately, I am not able to follow everything we do and I can be wrong, but there is not a single pool from us to the core.
At the same time, do you develop many of your tools around Kubernetes?
Dmitriy: The strategy is this: we go and pull-request everything that is already there. If pull requests are not accepted there, we simply fork them to ourselves and live until they are accepted with our builds. Then when it reaches upstream, we switch back to the upstream version.
For example, we have a Prometheus operator, with which we switched back and forth to the upstream of our assembly already 5 times, probably. We need some kind of feature, we sent a pull request, we need to roll it out tomorrow, but we don’t want to wait until it is released to the upstream. Accordingly, we assemble for ourselves, roll our assembly with our feature, which we need for some reason, to all our clusters. Then, for example, they wrap it upstream with the words: “Guys, let's do it for a more general case”, we, or someone else, finish it, and over time it merges back again.
Everything that exists, we try to develop. Many elements that do not yet exist, have not yet been invented or invented, but did not have time to implement - we are doing it. And not because we like the process itself or cycling as an industry, but simply because we need this tool. The question is often asked, why did we do this or that thing? The answer is simple - yes, because we had to go further, solve some practical problem, and we solved it with this tool.
The way is always like this: we search very carefully and if we don’t find any solution on how to make a trolleybus out of a loaf of bread, then we make our own loaf and our own trolleybus.
Flant tools
- I know that Flant now has addon operators, shell operators, dapp / werf tools. As I understand it, this is the same instrument in different incarnations. I also understand that there are many more different instruments inside Flant. This is true?
Dmitriy: We have a lot more on GitHub. From what I remember now, we have a statusmap - a panel for Grafana, which went to everyone. It is mentioned in almost every second article about monitoring Kubernetes on Medium. It's impossible to briefly describe what a statusmap is - we need a separate article, but this is a very useful thing for monitoring status over time, since in Kubernetes we often need to show status over time. We also have LogHouse - this is a thing based on ClickHouse and black magic for collecting logs in Kubernetes.
Lots of utilities! And there will be even more, because a number of internal solutions will be released this year. Of the very large ones based on the addon operator, there are a bunch of addons to Kubernetes ala how to install sert manager correctly - a tool for managing certificates, how to properly install Prometheus with a bunch of body kit - these are about twenty different binaries that export data and collect something, to this Prometheus awesome graphics and alerts. All this is just a bunch of Kubernetes addons that are put into the cluster, and it turns from simple to cool, fancy, automatic, in which many issues have already been resolved. Yes, we do a lot.
Ecosystem development
— It seems to me that this is a very big contribution to the development of this tool and its methods of use. Can you roughly estimate who else would make the same contribution to the development of the ecosystem?
Dmitriy: In Russia, of those companies that operate in our market, no one is even close. Of course, this is a big statement, because there are big players like Mail and Yandex - they also do something with Kubernetes, but even they have not come close to the contribution of companies in the whole world that are doing much more than us. It is difficult to compare Flant with a staff of 80 people and Red Hat, in which there are 300 engineers for only one Kubernetes, if I am not mistaken. It's hard to compare. We have 6 people in the R&D department, including me, who cut all our tools. 6 people versus 300 Red Hat engineers - it's kind of hard to compare.
“Nevertheless, when even these 6 people can do something really useful and alienable, when they face a practical problem and give the solution to the community, it’s an interesting case. I understand that in large technology companies that have their own development and Kubernetes support team, in principle, the same tools can be developed. This is an example for them that can be developed and given to the community, to give impetus to the entire community that uses Kubernetes.
Dmitriy: Probably, this is the feature of the integrator, its peculiarity. We have many projects and we see many different situations. For us, the main way to create added value is to analyze these cases, find common ground and make them as cheap as possible for us. This is what we are actively doing. It’s hard for me to talk about Russia and the world, but we have about 40 DevOps engineers in the company who are engaged in Kubernetes. I do not think that there are many companies in Russia with a comparable number of specialists who understand Kubernetes, if they exist at all.
I understand everything about the title of the position DevOps engineer, everyone understands everything and is used to calling DevOps engineers DevOps engineers, we will not discuss this. All these 40 amazing DevOps engineers face problems and solve them every day, we just analyze this experience and try to generalize. We understand that if it remains inside us, then in a year or two the tool will be useless, because somewhere in the community a ready-made tool will appear. It makes no sense to accumulate this experience inside - it's just a waste of time and effort in dev/null. And we don't feel sorry for it at all. We publish everything with great pleasure and understand that it needs to be published, developed, promoted, promoted so that people can use and add their experience - then everything grows and lives. Then after two years the tool does not go to the trash. It is not a pity to continue to infuse strength, because it is clear that someone uses your instrument, and in two years everyone already uses it.
This is part of our big strategy with dapp/werf. I don’t remember when we started doing it, I think about 3 years ago. Initially, it was generally on the shell. It was a super proof of concept, we solved some of our private tasks - it worked! But there are problems with the shell, it is impossible to increase it further, programming on the shell is another occupation. We had a habit of writing in Ruby, so we redid something in Ruby, developed, developed, developed, and ran into the fact that the community, the crowd that doesn’t say “we want or don’t want”, turns up its nose at Ruby, how not funny. We realized that we had to write all this goodness in Go in order to simply comply with the first item in the checklist: DevOps tool must be a static binary. On Go or not on Go, it doesn't really matter, but a static binary written in Go is better.
We spent our energy, rewrote dapp in Go and called it werf. Dapp is no longer maintained, not developed, running on some latest version, but there is an absolute upgrade path to the top that can be followed.
Why was dapp created?
— Can you briefly tell us why dapp was created, what problems does it solve?
Dmitriy: The first reason is in the assembly. Initially, we had severe build problems when Docker did not know how to multi-stage, and we made multi-stage on our own. Then we had a bunch more questions with image cleanup. Everyone who does CI / CD, sooner rather than later, faces the problem that there are a bunch of collected images, you need to somehow clean up what is not needed, and leave what is needed.
The second reason is deployment. Yes, there is Helm, but it solves only part of the problems. Ironically, it says "Helm - the Package Manager for Kubernetes". Exactly what "the". There are also the words "Package Manager" - what is the usual expectation from the Package Manager? We say: "Package Manager - install the package!" and we expect him to tell us: "The package has been delivered."
Interestingly, we say: “Helm, install the package,” and when he answers that he installed, it turns out that he just started the installation - he indicated Kubernetes: “Run this thing!”, And whether it started or not, it works or not , Helm does not solve this issue at all.
It turns out that Helm is just a text preprocessor that loads data into Kubernetes.
But within the framework of any deployment, we want to know whether the application has rolled out to production or not? It rolled out to the prod, which means that the application went there, the new version unfolded, and it at least does not fall there and answers correctly. Helm does not solve this problem in any way. To solve it, you need to spend a lot of effort, because you need to give Kubernetes a command to roll out and monitor what is happening there - whether it turned around, whether it rolled out. And there are a lot of tasks related to deployment, cleaning, and assembly.
Plans
This year we will go into local development. We want to come to what used to be in Vagrant - we typed “vagrant up” and our virtual machines turned around. We want to come to such a state that there is a project in Git, we write “werf up” there, and it raises a local copy of this project, deployed in a local mini-Kub, with all directories convenient for development included. Depending on the development language, this is done in different ways, but, nevertheless, so that you can conveniently conduct local development under mounted files.
The next step for us is strong invest in developer friendliness. In order to quickly deploy the project locally with one tool, develop it, push it to Git, and it will also roll out to the stage or tests, depending on the pipelines, and then go to production with the same tool. This unity, unification, reproducibility of the infrastructure from the local environment to the sale is a very important point for us. But this is not yet in werf - we only plan to do it.
But the path to dapp/werf has always been the same as with Kubernetes in the beginning. We faced problems, solved them by workarounds - we came up with some solutions for ourselves on the shell, on anything. Then they tried to somehow straighten, generalize and consolidate these workarounds into binaries in this case, which we simply share.
There is another view of this whole story, with analogies.
Kubernetes is the skeleton of a car with an engine. There are no doors, no windows, no radio, no Christmas tree — nothing at all. Frame and engine only. And there is Helm - this is the steering wheel. Cool - there is a steering wheel, but you also need a steering pin, steering rack, gearbox and wheels, but you can’t do without them.
In the case of werf, this is another component to Kubernetes. Only now in our alpha version of werf, for example, Helm is compiled into werf at all, because we are tired of doing it ourselves. There are many reasons to do this, in detail about why we compiled the whole helm together with tiller inside werf, I will tell just .
Now werf is a more integrated component. We get a finished steering wheel, a steering pin - I don’t understand cars very well, but this is a large unit that already solves a fairly wide range of tasks. We do not need to climb through the catalog ourselves, select one part for another, think about how to screw them together. We get a ready-made combine that solves a large bundle of tasks at once. But inside it is all built from the same open source components, everything also uses Docker for assembly, Helm for some of the functionality, and there are several other libraries. It's an integrated tool to get cool CI/CD out of the box quickly and conveniently.
Is it difficult to maintain Kubernetes?
- You talk about the experience that you started using Kubernetes, this is a frame for you, an engine, and that you can hang a lot of different things on it: body, steering wheel, fasten pedals, seats. The question arises - how difficult is it for you to support Kubernetes? You have a wealth of experience, how much time and resources do you spend specifically supporting Kubernetes in isolation from everything else?
Dmitriy: This is a very complex question and to answer it, you need to understand what support is and what we want from Kubernetes. Maybe you can open up?
- As far as I know and as I see, now many teams want to try Kubernetes. Everyone is harnessed to it, put on their knees. I have a feeling that people don't always understand the complexity of this system.
Dmitriy: It's like that.
- How difficult is it to take and install Kubernetes from nothing, so that it is production ready?
Dmitriy: How difficult do you think it is to transplant a heart? I understand the question is compromising. Carrying a scalpel and not making a mistake is not so difficult. If you are told where to cut off and where to sew up, then the procedure itself is simple. It is difficult to guarantee from time to time that everything will work out.
Installing Kubernetes and making it work is easy: chick! - put, there are a lot of ways to install. But what happens when problems arise?
Questions always arise - what have we not taken into account yet? What haven't we done yet? What parameters of the Linux kernel were specified incorrectly? Lord, did we mention them at all?! Which Kubernetes components have we shipped and which we haven't? Thousands of questions arise, and to answer them, you need to cook in this industry for 15-20 years.
I have a recent example on this topic that may make sense of the "Is it hard to maintain Kubernetes?" problem. Some time ago, we seriously considered trying to implement Cilium as a network in Kubernetes.
Let me explain what Cilium is. There are many different implementations of the networking subsystem in Kubernetes, and one of them is very cool - this is Cilium. What is its meaning? Some time ago, the kernel introduced the ability to write hooks for the kernel, which in one way or another invade the network subsystem and various other subsystems, and allow you to bypass large chunks in the kernel.
The Linux kernel historically has an ip route, an overfilter, bridges, and many different old components that are 15, 20, 30 years old. In general, they work, everything is cool, but now they have piled up containers, and it looks like a turret of 15 bricks on top of each other, and you are standing on it on one leg - a strange feeling. This system has historically developed with many nuances, like the appendix in the body. There are performance issues in some situations, e.g.
There is a wonderful BPF and the ability to write hooks for the core - the guys wrote their own hooks for the core. The packet comes into the Linux kernel, they take it out right at the entrance, process it themselves as it should without bridges, without TCP, without an IP stack - in short, bypassing everything that is written in the Linux kernel, and immediately spit it out into the container.
What happened? Very cool performance, cool features - just cool! But we look at it and see that on each machine there is a program that connects to the Kubernetes API and, according to the data it receives from this API, generates C-code and compiles binaries that it loads into the kernel so that these hooks work in the kernel space .
What happens if something goes wrong? We do not know. To understand this, you need to read all this code, understand all the logic, and it's stunned how difficult it is. But, on the other hand, there are these bridges, net filters, ip routes - I have not read their source codes, and 40 engineers who work in our company too. Maybe some pieces understand units.
And what's the difference? It turns out that there is ip route, the Linux kernel, and there is a new tool - what's the difference, we don't understand either one or the other. But we are afraid to use the new - why? Because if the instrument is 30 years old, then in 30 years all the bugs have been found, all the rakes have been stepped on and you don’t need to know about everything - it works like a black box, and it always works. Everyone knows which diagnostic screwdriver to stick in which place, which tcpdump to run at what moment. Everyone knows diagnostic utilities well and understands how this set of components works in the Linux kernel - not how it works, but how to use it.
And the awesomely cool Cilium is not 30 years old, it has not yet matured. Same problem with Kubernetes, copy. That Cilium is great, that Kubernetes is great, but when something goes wrong in the production, are you able to quickly understand what went wrong in a critical situation?
When we say whether it is difficult to maintain Kubernetes - no, very simple, and yes, incredibly difficult. Kubernetes works great on its own, but with a billion nuances.
About the "I'm Feeling Lucky" Approach
— Are there companies where these nuances are almost guaranteed to appear? Suppose Yandex suddenly transfers all services to Kubernetes, there will be a huge load there.
Dmitriy: No, this is not a conversation about the load, but about the simplest things. For example, we have Kubernetes, we deployed an application there. How to understand that it works? There is simply no ready-made tool to understand that the application does not crash. There is no ready-made system that sends alerts, you need to set up these alerts and each chart. And we are updating Kubernetes.
There is Ubuntu 16.04. We can say that this is an old version, but we are still on it, because there is LTS. There's systemd, which has the caveat that it doesn't clean up C groups. Kubernetes starts the pods, creates C groups, then deletes the pods, and somehow it turns out - I don't remember the details, sorry - that the systemd slices remain. This leads to the fact that over time, any car starts to slow down a lot. This is not even a question about highload. If persistent pods are running, for example, if there is a Cron Job that constantly generates pods, then an Ubuntu 16.04 machine will start to slow down after a week. There will be a constantly high load average due to the fact that a bunch of C-groups are created. This is a problem that anyone who just puts Ubuntu 16 and Kubernetes on top of it has.
Let's say he somehow updates systemd or something else, but in the Linux kernel before 4.16 it's even funnier - when you delete C-groups, they leak in the kernel and are not actually deleted. Therefore, after a month of work on this machine, it will be impossible to see memory statistics for pods. We take out a file, roll it in the program, and one file rolls for 15 seconds, because the kernel takes a very long time to count millions of C-groups inside itself, which seem to have been deleted, but no - they are leaking.
There are still a lot of such little things here and there. This is not a question that giant companies can sometimes face under very heavy loads - no, it is a matter of daily things. People can live like this for months - they installed Kubernetes, deployed the application - it seems to work. Many people are fine with it. They won’t even know that someday this application will crash for some reason, the alert will not come, but for them this is the norm. Previously, they lived on virtual machines without monitoring, now they have moved to Kubernetes without monitoring too - what's the difference?
The issue is that when we walk on ice, we never know its thickness unless we measure it beforehand. Many go and do not bathe, because they used to go.
From my point of view, the nuance and complexity of operating any system is to ensure that the thickness of the ice is exactly enough to solve our problems. It's about this.
In IT, I think there are too many "I'm Feeling Lucky" approaches. Many install software, use software libraries in the hope that they will be lucky. In general, many people are lucky. That's probably why it works.
- From my pessimistic assessment, it looks like this: when the risks are high, and the application should work, then you need support from Flant, possibly from Red Hat, or you need your own internal team dedicated specifically to Kubernetes, which is ready to pull it.
Dmitriy: Objectively, it is. Getting into the story with Kubernetes on your own with a small team is a certain amount of risk.
Do we need containers?
— Can you tell us how common Kubernetes is in Russia?
Dmitriy: I don't have this data and I'm not sure if anyone has it. We say: “Kubernetes, Kubernetes”, but there is another way to look at this issue. I don't know how common containers are either, but I do know a figure from reports on the Internet that 70% of containers are orchestrated by Kubernetes. It was a reliable source for a fairly large sample around the world.
Then another question - do we need containers? I have a personal feeling and, in general, the position of the Flant company is such that Kubernetes is the de facto standard.
There will be nothing but Kubernetes.
It is an absolute game-changer in infrastructure management. Just absolute - that's it, no more Ansible, Chef, virtual machines, Terraform. I'm not talking about the old collective farm methods. Kubernetes is the absolute changer, and now it will only be so.
It is clear that someone needs a couple of years, and someone a couple of decades, to realize this. I have no doubt that there will be nothing but Kubernetes and this new look: we no longer hurt the OS, but use infrastructure as code, but not with code, but with yml - a declaratively described infrastructure. I have a feeling this will always be the case.
- That is, those companies that have not yet switched to Kubernetes will definitely switch to it or remain in oblivion. I understood you correctly?
DmitriyA: This is also not entirely true. For example, if we have a task to run a dns server, then it can be run on FreeBSD 4.10 and it can work fine for 20 years. Just work is all. Perhaps in 20 years something will need to be updated once. If we are talking about software in the format that we launched and it really works for many years without any updates, without making changes, then, of course, there will be no Kubernetes. He is not needed there.
Everything related to CI / CD - wherever you need Continuous Delivery, where you need to update versions, make active changes, wherever you need to build fault tolerance - only Kubernetes.
About microservices
- Here I have a slight dissonance. To work with Kubernetes, you need external or internal support - this is the first point. The second is when we are just starting development, we are a small startup, we don’t have anything yet, development for Kubernetes or generally for a microservice architecture can be difficult, and not always economically justified. I’m interested in your opinion - do startups need to immediately start writing for Kubernetes from scratch, or can they still write a monolith, and only then come to Kubernetes?
Dmitriy: Cool question. I have a talk about microservices Many times I have come across the fact that people try to hammer nails with a microscope. The approach itself is correct, we ourselves design the internal software in this way. But when you do this, you need to clearly understand what you are doing. What I hate the most about microservices is the word "micro". Historically, this word arose there, and for some reason people think that micro is very small, less than a millimeter, like a micrometer. This is wrong.
For example, there is a monolith that is written by 300 people, and everyone who participated in the development understands that there are problems there, and it should be broken into micro-pieces - 10 pieces, each of which is written by 30 people in the minimum version. This is important, necessary and cool. But when a startup comes to us, where 3 very cool and talented guys wrote 60 microservices on their knees, every time I look for Corvalol.
It seems to me that this has already been said thousands of times - they got a distributed monolith in one form or another. This is not economically justified, it is very difficult in general in everything. It's just that I've seen it so many times that it really hurts me, so I keep talking about it.
To the initial question, that there is a conflict between the fact that, on the one hand, Kubernetes is scary to use, because it is not clear what can break or not work there, on the other hand, it is clear that everything goes there and nothing but Kubernetes will be . Answer - weigh the amount of benefit that comes, the amount of tasks that you can solve. This is on one side of the scale. On the other hand, there are risks that are associated with downtime or with a decrease in response time, availability level - with a decrease in performance.
Here it is - either we move fast, and Kubernetes allows you to do many things much faster and better, or use reliable, time-tested solutions, but move much more slowly. This choice must be made by every company. You can consider it as a path in the jungle - when you walk for the first time, you can meet a snake, a tiger or a rabid badger, and when you went 10 times, you trod the path, removed the branches and walk easier. Each time the path gets wider. Then it is already an asphalt road, and later a beautiful boulevard.
Kubernetes does not stand still. Another question: Kubernetes, on the one hand, is 4-5 binaries, on the other hand, it is the whole ecosystem. This is the operating system that we have on the machines. What is this? Ubuntu or Curios? It's the Linux kernel, a bunch of extras. All these things here, one poisonous snake was thrown out of the way, a fence was put up there. Kubernetes is developing very quickly and dynamically, and the volume of risks, the volume of the unknown decreases every month and, accordingly, these scales are rebalanced.
Answering the question of what a startup should do, I would say - come to Flant, pay 150 thousand rubles and get DevOps easy service on a turnkey basis. If you are a small startup with several developers, this works. Instead of hiring your own DevOps, who will need to learn how to solve your problems and pay a salary at this time, you will receive a turnkey solution to all issues. Yes, there are some downsides. We, as an outsourcer, cannot be so involved and quickly respond to changes. But we have a lot of expertise, ready-made practices. We guarantee that in any situation we will definitely quickly figure it out and raise any Kubernetes from the other world.
I strongly recommend outsourcing to startups and established businesses up to a size where you can allocate a team of 10 people to operate, because otherwise there is no point. It absolutely makes sense to outsource.
About Amazon and Google
- Can a host from a solution from Amazon or Google be considered as an outsourcer?
Dmitriy: Yes, of course, it solves a number of issues. But again, nuances. You still need to understand how to use it. For example, there are a thousand little things in the work of Amazon AWS: you need to warm up the Load Balancer or write a request in advance that “guys, we will get traffic, warm up the Load Balancer for us!” You need to know these nuances.
When you contact people who specialize in this, you get almost all the typical stuff closed. We now have 40 engineers, by the end of the year there will probably be 60 of them - we have definitely come across all these things. Even if on some project we encounter this problem again, we quickly ask each other and know how to solve it.
Probably, the answer is this - of course, the hosted-story facilitates some part. The question is whether you are ready to trust these hosts and whether they will solve your problems. Amazon and Google have done well. For all our cases - for sure. We don't have any other positive experiences. All the other clouds that we tried to work with create a lot of problems - and Ager, and everything that is in Russia, and all kinds of OpenStack in different implementations: Headster, Overage - whatever you want. They all create problems that you don't want to solve.
Therefore, the answer is yes, but, in fact, there are not very many mature hosted solutions.
Who needs Kubernetes?
- And yet, who needs Kubernetes? Who should already switch to Kubernetes, who is a typical Flant client who comes specifically for Kubernetes?
Dmitriy: The question is interesting, because now, just on the wave of Kubernetes, many people come to us: “Guys, we know that you are doing Kubernetes, do it for us!”. We answer them: "Gentlemen, we do not make Kubernetes, we make prod and everything connected with it." Because to make a production without doing all the CI / CD and all this history is currently simply impossible. Everyone has moved away from the division that we have development by development, and then exploitation by exploitation.
Our clients expect different things, but everyone is waiting for some kind miracle that they have this or that problem, and now - hop! - Kubernetes will solve them. People believe in miracles. They understand with their minds that there will be no miracle, but they hope with their souls - what if this Kubernetes will decide everything for us now, they talk so much about it! Suddenly he is now - sneeze! — and a silver bullet, sneeze! - and we have 100% uptime, all developers can release 50 times anything for production, and it does not fall. In general, a miracle!
When such people come to us, we say: "Sorry, but a miracle does not happen." To be healthy, you need to eat well and exercise. To have a reliable product, it must be done reliably. To have a convenient CI / CD, it needs to be made like this. This is a lot of work that needs to be done.
Answering the question, who needs Kubernetes, no one needs Kubernetes.
Some people have the misguided feeling that they need Kubernetes. People need, they have a really deep need to stop thinking, doing, being interested in all the problems of the infrastructure and the problems of running their applications. They want applications to just work and deploy. For them, Kubernetes is the hope that they will stop hearing the story that “we were lying there”, or “we can’t roll out”, or something else.
The technical director usually comes to us. They ask him two things: on the one hand, give us features, on the other hand, stability. We suggest you take it upon yourself and do it. The silver bullet, or rather, silver-plated, is that you will stop thinking about these problems and wasting time. You will have special people who will close this issue.
The wording that we or someone needs Kubernetes is wrong.
Kubernetes is very necessary for admins, because this is a very interesting toy with which you can play, dig deeper. Let's be honest - everyone loves toys. We are all children somewhere, and when we see a new one, we want to play it. For some, this is repulsed, for example, in the administration, because they have already played enough and are already tired to the point that they simply don’t want to. But this is not repulsed from anyone completely. For example, if I'm tired of toys in the field of system administration and DevOps for a long time, then I still love toys, I still buy some new ones. All people somehow still want some kind of toys.
No need to play with production. What would I categorically not recommend doing and what I see now massively: “Ah, a new toy!” - they ran to buy it, bought it and: “Let's take it to school now, show it to all our friends.” Don't do it. I'm sorry, my children are just growing up, I constantly see something in children, I notice it in myself, and then I generalize it to others.
Final answer: You don't need Kubernetes. You need to solve your problems.
You can achieve that:
- prod does not fall;
- even if he tries to fall, we know about it in advance, and we can put something on it;
- we can change it at the speed with which we need for business, and do it conveniently, it does not cause us any problems.
There are two real needs: reliability and dynamism / rollout flexibility. Everyone who is now doing some kind of IT projects, no matter what kind of business - soft for easing the world, and who understands this, these needs need to be addressed. Kubernetes, with the right approach, with the right understanding, and with enough experience, can solve them.
About serverless
- If you look a little further into the future, then trying to solve the problem of the lack of a headache with the infrastructure, with the speed of rolling out and the speed of application changes, new solutions appear, for example, serverless. Do you feel any potential in this direction and, let's say, a danger for Kubernetes and similar solutions?
Dmitriy: Here it is necessary to make a remark again that I am not a seer who looks ahead and says - it will be like this! Even though I just did the same. I look down at my feet and see a bunch of problems there, like how transistors work in a computer. Ridiculous, right? We are facing some bugs in the CPU.
Making serverless is quite reliable, cheap, efficient and convenient, solving all ecosystem issues. Here I agree with Elon Musk that a second planet is needed to make fault tolerance for humanity. Although I don’t know what he is saying, I understand that he is not ready to fly to Mars himself and it won’t be tomorrow.
With serverless, it is clearly clear that this is an ideologically correct thing, like fault tolerance for humanity - having two planets is better than one. But how to do it now? Sending one expedition is not a problem if you concentrate your efforts on it. Sending several expeditions and settling several thousand people there, I think, is also realistic. But it seems to me now that it is impossible, not considered, to make it entirely fault-tolerant so that half of humanity lives there.
With serverless one to one: a cool thing, but it is far from the problems of 2019. Closer to 2030 - let's live to see it. I have no doubt that we will live, we will definitely live (repeat before going to bed), but now we need to solve other problems. It's like believing in the fabulous pony Rainbow. Yes, a couple of percent of cases are solved, and they are solved perfectly, but subjectively, serverless is a rainbow ... For me, this topic is too far and too incomprehensible. I'm not ready to speak. In 2019, you can’t write a single application with serverless.
How Kubernetes will evolve
- As we move towards this potentially beautiful distant future, how do you think Kubernetes and the ecosystem around it will develop?
DmitriyA: I thought about it a lot and I have a clear answer. The first - statefull - after all, stateless is easier to do. Kubernetes initially invested more in this, it all started with it. Stateless works almost perfectly in Kubernetes, there's just nothing to complain about. According to statefull, there are still a lot of problems, or rather, nuances. We already have everything working fine there, but it's us. It will take at least a couple more years for this to work for everyone. This is not a calculated figure, but my feeling from the head.
In short, statefull must - and will - develop very strongly, because all applications store their status, there are no stateless applications. This is an illusion, you always need some kind of database and something else. Statefull is about straightening everything you can, fixing all the bugs, improving all the problems that are currently being faced - let's call it adoption.
The level of the unknown, the level of unresolved problems, the level of probability of encountering something will drop dramatically. This is an important story. And operators - everything related to codifying administration logic, management logic, in order to get easy service: MySQL easy service, RabbitMQ easy service, Memcache easy service - in general, all these components that we need to get it working out of the box guaranteed. This just solves the pain that we want a database, but do not want to administer it, or we want Kubernetes, but do not want to administer it.
This story with the development of operators in one form or another will be important in the next couple of years.
I think the ease of use should increase a lot - the box will become more and more black, more and more reliable, with more and more simple twists.
I once listened to an old Isaac Asimov interview from the 80s on YouTube on Saturday Night Live - an Urgant-like show, only interesting. He was asked there about the future of computers. He said that the future is in simplicity, as it was with the radio. The radio receiver was originally a complex thing. To catch a wave, it took 15 minutes to turn the spinners, turn the spinners and generally know how everything works, understand the physics of radio wave transmission. As a result, the radio was left with one twist.
Now in 2019 which radio? In the car, the radio receiver finds all the waves, the name of the stations. The physics of the process has not changed in 100 years, the ease of use has. Now, and not only now, already in 1980, when there was an interview with Azimov, everyone used the radio and no one thought about how it works. It has always worked - it's a given.
Asimov then said that it would be similar with computers - ease of use will increase. If in 1980 it was necessary to get a special education to press buttons on a computer, then in the future this will not be the case.
I have a feeling that with Kubernetes and with the infrastructure, ease of use will also increase very much. This, in my opinion, is obvious - it lies on the surface.
What to do with engineers?
- And then what will happen to the engineers, system administrators who support Kubernetes?
Dmitriy: And what happened to the accountant after the appearance of 1C? Roughly the same. Before that, they counted on a piece of paper - now in the program. Labor productivity has increased by orders of magnitude, but labor itself has not disappeared from this. If earlier it took 10 engineers to screw in a light bulb, now one will be enough.
The number of software and the number of tasks, it seems to me, is now growing at a faster rate than new DevOps appear and efficiency increases. Now there is a specific shortage in the market and it will last for a long time. Later, everything will enter into a certain norm, at which the efficiency of work will increase, there will be more and more serverless, a neuron will be attached to Kubernetes, which will select all the resources right as it should, and in general does everything by itself, as it should - the person move away and do not interfere.
But decisions still need to be made by someone. It is clear that the level of qualification and specialization of this person is higher. Now in the accounting department, you don't need 10 employees who keep accounting books so that their arm does not get tired. It's just not necessary. Many documents are automatically scanned and recognized by the electronic document management system. One smart chief accountant is enough, already with much greater skills, with a good understanding.
In general, this way in all industries. It is the same with cars: earlier, an auto mechanic and three drivers were attached to the car. Now driving a car is the simplest process in which we all participate every day. No one thinks that a car is something complicated.
DevOps or systems engineering will not go anywhere - the high-level and efficiency of work will increase.
- I also heard an interesting idea that in fact the work will increase.
Dmitriy: Of course, one hundred percent! Because the amount of software we write is constantly growing. The number of issues that we solve with software is constantly growing. The number of jobs is growing. Now the DevOps market is terribly overheated. This can be seen in salary expectations. In a good way, without going into details, there should be juniors who want X, middles who want 1,5X, and seniors who want 2X. And now, if you look at the Moscow DevOps salary market, a junior wants from X to 3X and a senior wants from X to 3X.
Nobody knows how much it costs. The level of salary is measured by your confidence - a complete madhouse, to be honest, a terribly overheated market.
Of course, this situation will change very soon - there should be some saturation. With software development, it’s not like that - despite the fact that everyone needs developers, and everyone needs good developers, the market understands who costs how much - the industry has settled down. Not so with DevOps now.
- From what I heard, I concluded that the current system administrator should not worry much, but it's time to download skills and prepare for the fact that tomorrow there will be more work, but it will be more highly qualified.
Dmitriy: Absolutely. In general, we live in 2019 and the rule of life is this: lifetime learning - we learn all our lives. It seems to me that now everyone already knows and feels this, but knowing is not enough - you have to do it. Every day we must change. If we do not do this, then sooner or later we will be dropped off at the margins of the profession.
Be prepared for sharp 180-degree turns. I do not rule out situations when something changes dramatically, something new is invented - it happens. Hop! — and now we act differently. It is important to be prepared for this and not to worry. It may happen that tomorrow everything I do will be unnecessary - nothing, I have been studying all my life and I am ready to learn something else. It's not a problem. You should not be afraid of job security, but you need to be ready to constantly learn something new.
Wishes and a minute of advertising
- Do you have any wishes?
Dmitriy: Yes, I have a few wishes.
The first and mercantile - subscribe to . Dear readers, go to YouTube and subscribe to our channel. Somewhere in a month we will start an active expansion to the video service, we will have a bunch of educational content about Kubernetes, open and different: from practical things, up to laboratories, to deep fundamental theoretical things and how to apply Kubernetes at the level of principles and patterns.
The second mercantile wish - go to and put stars, because we eat them. If you don't give us stars, we won't have anything to eat. It's like mana in a computer game. We do something, we do, we try, someone says that these are terrible bikes, someone that everything is generally wrong, but we continue and act absolutely honestly. We see the problem, we solve it and share our experience. Therefore, give us an asterisk, it will not decrease from you, but it will come to us, because we feed on them.
The third, important, and no longer mercantile wish - stop believing in fairy tales. You are professionals. DevOps is a very serious and responsible profession. Stop playing in the workplace. Let it click you and you will understand it. Imagine that you come to the hospital, and there the doctor is experimenting on you. I understand that this may be offensive to someone, but, most likely, this is not about you, but about someone else. Tell others to stop too. This really spoils the lives of all of us - many begin to treat the operation, the admins and DevOps as dudes who have broken something again. This was “broken” most often due to the fact that we went to play, and did not look with a cold mind that it was like that here, but that’s how it is here.
This does not mean that you should not experiment. We need to experiment, we do it ourselves. To be honest, we ourselves also play sometimes - this, of course, is very bad, but nothing human is alien to us. Let's declare 2019 the year of serious thoughtful experiments, not games on the market. Probably so.
- Many thanks!
Dmitriy: Thank you, Vitaly, for the time and for the interview. Dear readers, thank you very much if you suddenly got to this point. I hope that we have brought you at least a couple of thoughts.
In an interview, Dmitry touched on the issue of werf. Now it is a universal Swiss knife that solves almost all tasks. But it was not always so. On at the festival Dmitry Stolyarov will talk about this tool in detail. in the report there will be everything: problems and hidden nuances of Kubernetes, solutions to these difficulties and the current implementation of werf in detail. Join on May 27 and 28, we will create the perfect tools.
Source: habr.com