Today I plan to tell you how to write applications and what are the requirements for your application to work well in Kubernetes. So that there is no headache with the application, so that you don’t have to invent and build some kind of “crutches” around it - and everything works as intended by Kubernetes itself.
This lecture is part of". You can view the open theoretical lectures of the Evening School . For those who prefer text rather than video, we have prepared this article.
My name is Pavel Selivanov, at the moment I am a leading DevOps engineer at Mail.ru Cloud Solutions, we make clouds, we make managed kubernetes and so on. My tasks now include helping with development, rolling out these clouds, rolling out the applications that we write, and directly developing the tools that we provide for our users.
I've been doing DevOps, I think that the last, probably, is three years. But, in principle, what DevOps is doing, I have been doing, probably, for five years already. Before that, I was rather more administrative things. I started working with Kubernetes a very long time ago - it's probably been about four years since I started working with it.
In general, I started when Kubernetes had version 1.3, probably, and maybe 1.2 - when it was still in its infancy. Now it is not in its infancy at all - and it is obvious that there is a huge demand on the market for engineers who would like to be able to use Kubernetes. And companies have a very high demand for such people. Therefore, in fact, this lecture appeared.
If we talk according to the plan of what I will talk about, it looks like this, it is written in brackets (TL;DR) - “too long; don't read. My presentation today will be endless lists.
In fact, I myself do not like such presentations when they are made, but there is such a topic that when I was preparing this presentation, I simply did not really figure out how to organize this information in a different way.
Because by and large this information is “ctrl + c, ctrl + v”, from, including our Wiki in the DevOps section, where we have written requirements for developers: “guys, so that we launch your application in Kubernetes, it it should be like this."
Therefore, the presentation turned out to be such a large list. Sorry. I will try to tell as much as possible so that it is not boring as much as possible.
What we are going to analyze now:
- these are, firstly, logs (application logs?), what to do with them in Kubernetes, how to deal with them, what they should be;
- what to do with configurations in Kubernetes, what are the best and worst ways for Kubernetes to configure an application;
- let's talk about what accessibility checks are in general, how they should look like;
- let's talk about what graceful shutdown is;
- let's talk about resources again;
- once again we will touch on the topic of data storage;
- and at the end I will tell you what the term is this mysterious cloud-native application. Cloudnativeness, as an adjective for this term.
Logs
I suggest starting with the logs - where these logs need to be shoved in Kubernetes. Here you have launched an application in Kubernetes. According to the classics, applications used to always write logs somewhere in a file. Bad applications wrote logs to a file in the home directory of the developer who ran the application. Good applications wrote logs to a file somewhere in /var/log.
Accordingly, further, good admins in the infrastructures had some things configured that these logs can rotate - the same rsyslog is elementary, which looks at these logs and when something happens to them, there are a lot of them, it creates backup copies , adds logs there, deletes old files, more than a week, half a year and some more. In theory, we should have provision so that just stupidly due to the fact that the application writes logs, the place on the production servers (combat servers?) does not end. And, accordingly, the entire production did not stop because of the logs.
When we move into the world of Kubernetes and run everything the same there, the first thing you can pay attention to is that people, as they wrote logs in a file, continue to write them.
It turns out that if we talk about Kubernetes, the right place to write logs somewhere from the docker container is just to write them from the application to the so-called Stdout / Stderr, that is, the standard output streams of the operating system, standard error output . This is the most correct, easiest and most logical way to put the logs in principle in the docker and specifically in Kubernetis. Because if your application writes logs to Stdout / Stderr, then it is the task of the docker and the Kubernetes add-on above it, what to do with these logs. Docker will by default bundle its custom files in JSON format.
Here the question already arises, what will you do next with these logs. The easiest way, of course, we have the ability to do kubectl logs and see these logs of these "pods". But, probably, this is not a very good option - something needs to be done with the logs.
For now, let's talk at the same time, since we have touched on the topic of logs, about such a thing as the logs should look like. That is, this does not apply directly to Kubernetes, but when we start to think about what to do with the logs, it would be good to think about this too.
We need some kind of tool, in a good way, which will take these logs that the docker puts in our files and send them somewhere. By and large, we usually run some agent inside Kubernetes in the form of a DaemonSet - a log collector, which is simply told where the logs that the docker puts are located. And this agent-collector simply takes them, perhaps even somehow parses them along the way, perhaps enriches them with some additional meta-information and, as a result, sends them to storage somewhere. Variations are already possible. The most common, probably, is Elasticsearch, where you can store logs, you can conveniently get them from there. Then, using a request, using Kibana, for example, build charts based on them, build alerts based on them, and so on.
The main idea, I want to repeat it again, is that inside the docker, in particular, inside Kubernetes, putting your logs into a file is a very bad idea.
Because firstly, the logs inside the container in the file are problematic to get. First you need to go into the container, do an exec there, and then look at the logs. The next point is that if you have logs in a file, then containers usually have a minimalistic environment and there are no utilities that are usually needed for normal work with logs. Warm them up, look at them, open them in a text editor. The next moment, when we have the logs in a file inside the container, if this container is deleted, you understand, the logs will die with it. Accordingly, any restart of the container - and there are no more logs. Again, bad choice.
And the last point is that inside the containers you usually have your application and that's it - it is usually the only running process. There is no question of any process that would rotate files with your logs. As soon as the logs begin to be written to a file, this means that, sorry, we will begin to lose the production server. Because, firstly, it is problematic to find them, no one monitors them, plus no one controls them - accordingly, the file grows indefinitely until the space on the server just runs out. Therefore, I say again that the logs in the docker, in particular, in Kubernetes, in a file is a bad idea.
The next moment, here I want to talk about this again - since we are touching on the topic of logs, it would be nice to talk about how the logs should look in order to make it convenient to work with them. As I said, the topic is not directly related to Kubernetes, but it is very well related to the DevOps topic. To the topic of development culture and friendship between these two different departments - Dev and Ops, so that everyone is comfortable.
So, ideally, today, logs should be written in JSON format. If you have some incomprehensible application of your own, which writes logs in incomprehensible formats, because you insert some print or something like that there, then it's time to google some framework, some wrapper that allows you to implement normal logging; enable logging parameters in JSON there, because JSON is a simple format, parsing it is simply elementary.
If your JSON does not work according to some criteria, it is not known what, then at least write the logs in a format that can be parsed. Here, rather, you should think about the fact that, for example, if you have a bunch of some containers or just processes with nginx running, and each has its own logging settings, then it probably seems that it will be very inconvenient for you to parse them. Because for each new nginx instance you need to write your own parser, because they write logs differently. Again, probably, it was worth thinking about the fact that all these nginx instances had the same logging configuration, and wrote all their logs in an absolutely uniform way. The same applies to absolutely all applications.
In the end, I still want to add fuel to the fire, that, ideally, multiline format logs should be avoided. Here's the thing, if you have ever worked with log collectors, then most likely you have seen what they promise you, that they can work with multi-line logs, they can collect them, and so on. In fact, it is normal, fully and without errors to collect multi-line logs, in my opinion, not a single collector can do it today. In a human way, so that it is convenient and without errors.
But stack trace is always multi-line logs and how to avoid them. The question here is that the log is a record of the event, and the stacktrace is actually not a log. If we collect logs and put them somewhere in Elasticsearch and then draw graphs based on them, build some reports on the work of users on your site, then when you get a stack trace, it means that you have some kind of unforeseen, an unhandled situation in your application. And stack trace makes sense to throw automatically somewhere in the system that can track them.
This is the software (the same Sentry), which is made specifically to work with the stack trace. It can create automated tasks right away, assign to someone, alert when stack traces occur, group these stack traces by the same type, and so on. In principle, it does not make much sense to talk about stack traces when we talk about logs, because these are, after all, different things with different purposes.
Configuration
Next, we have about the configuration in Kubernetes: what to do with it and how applications inside Kubernetes should be configured. In general, I usually say that docker is not about containers. Everyone knows that docker is containers, even those who have not worked with docker much. Again, Docker is not about containers.
Docker, in my opinion, is about standards. And there are standards for almost everything: standards for building your application, standards for installing your application.
And this thing - we used it before, it just became especially popular with the advent of containers - this thing is called ENV (environment) variables, that is, environment variables that are in your operating system. This is generally an ideal way to configure your application, because if you have applications in JAVA, Python, Go, Perl, God forbid, and they can all read the database host, database user, database password variables, then it is ideal. You have applications in four different languages configured in the database plan in the same way. There are no more different configs.
Everything can be configured using ENV variables. When we talk about Kubernetes, there is a great way to declare ENV variables right inside the Deployment. Accordingly, if we are talking about secret data, then secret data from ENV variables (passwords to databases, etc.), we can immediately shove into a secret, create a cluster secret and indicate in the description of ENV in Deployment that we do not directly declare the value of this variable, and the value of this database password variable will be read from the secret. This is standard Kubernetes behavior. And this is the most ideal way to configure your applications. Just at the code level, again, this applies to developers. If you are DevOps, you can ask: “Guys, please teach your application to read environment variables. And we will all be happy.”
If everyone also reads the same named environment variables in the company, then this is generally gorgeous. So that there is no such thing that some are waiting for the postgres database, others for the database name, third for the database, something else, the fourth for some dbn, so that, accordingly, there is uniformity.
The problem comes when you have so many environment variables that you just open Deployment - and there are five hundred lines of environment variables. In this case, you have simply outgrown the environment variables - and then you no longer need to torture yourself. In this case, it would make sense to start using configs. That is, train your application to use configs.
The only question is that configs are not what you think. Config.pi is not the config that is convenient to use. Or some config in your own format, alternatively gifted - that's also not the config I mean.
What I'm talking about is configuration in acceptable formats, that is, by far the most popular standard is the .yaml standard. It's clear how to read, it's human-readable, it's clear how to read from the app.
Accordingly, in addition to YAML, you can also, for example, use JSON, parsing is about as convenient as YAML in terms of reading the application configuration from there. Reading people is noticeably more inconvenient. You can try the format, a la ini. It’s quite convenient to read it directly, from a human point of view, but it can be inconvenient to process it automatically, in the sense that if you ever want to generate your configs, the ini format can already be inconvenient to generate.
But in any case, whichever format you choose, the bottom line is that from the point of view of Kubernetes, this is very convenient. You can put all your config inside Kubernetes, in ConfigMap. And then take this configmap and ask to mount it inside your pod to some specific directory where your application will read the configuration from this configmap as if it were just a file. This, in fact, is what is good to do when you have a lot of configuration options in your application. Or just some complex structure, there is nesting.
If you have a configmap, then you can very well teach your application, for example, to automatically track changes in the file where the configmap is mounted, and also automatically reload your application when changing configs. This would be ideal in general.
Again, I already spoke about this - secret information is not in the configmap, secret information is not in variables, secret information is in secrets. From there, connect this secret information to diplomas. Usually we store all descriptions of Kubernet objects, diplomas, configmaps, services in git. Accordingly, pushing the database password into git, even if it is your git, which you have internal in the company, is a bad idea. Because, at least, git remembers everything and it's not so easy to remove passwords from there.
health check
The next moment is about such a thing called Health check. In general, a Health check is simply a check that your application is working. At the same time, we most often talk about some web applications, which, accordingly, from the point of view of a helcheck (it is better not to translate here and below), this will be some special URL that they process as standard, they usually do /health.
When accessing this URL, respectively, our application says either "yes, okay, I'm doing well 200" or "no, I'm not doing well, some 500". Accordingly, if our application is not http, not a web application, we are now talking about some kind of daemon, we can figure out how to do helchecks. That is, it is not necessary, if the application is not http, then everything works without a helscheck and this cannot be done in any way. You can periodically update some information in the file, you can come up with some special command for your daemon, such as, daemon status, which will say "yes, everything is fine, the daemon is working, it is alive."
What is it for? The first most obvious, probably, why a helcheck is needed is in order to understand that the application is working. That is, just stupidly, when it is now raised, it looks like a working one, just to be sure that it works. Otherwise, it turns out that the pod with the application is running, the container is running, the instance is working, everything is fine - and there the users have already cut off all the phones from technical support and say “what are you, ... fell asleep, nothing works.”
Here is a helcheck - this is just such a way to see from the user's point of view that it works. One of the methods. Let's say so. From the point of view of Kubernetes, this is also a way to understand when the application is launched, because we understand that there is a difference between when the container started, was created and started, and when the application was launched directly in this container. Because if we take some average java application and try to run it in the dock, then forty seconds, or even a minute, or even ten, it can start perfectly. At the same time, you can at least knock on its ports, it will not answer there, that is, it is not yet ready to receive traffic.
Again, with the help of the helcheck and what we are referring to here, we can understand in Kubernetes that not just the container has risen in the application, but the application has started right up, it is already responding to the helcheck, which means you can start traffic there.
What I'm talking about now is called Readiness / Liveness tests within Kubernetes, respectively, readiness tests are responsible for the availability of the application in balancing. That is, if readiness tests are performed in the application, then everything is OK, client traffic is going to the application. If readiness probes are not performed, then the application simply does not participate, specifically this instance, does not participate in balancing, it is removed from balancing, client traffic does not go. Accordingly, Liveness samples within Kubernetes are needed in order to “restart” it if the application is “stuck”. If the liveness test does not work for an application that is declared in Kubernetes, then the application is not just removed from balancing, it is just restarted.
And here is such an important point, which I would like to say, from the point of view of practice, it is usually still more often used and more often a readiness test is needed than a liveness test. That is, just mindlessly declaring both the readiness and liveness of the probe, because Kubernetes can do that, and let's use everything that it can - not a good idea. I'll explain why. Because there is a point number two in the samples - it's that it would be nice to check the underlying service in your health checks. This means that if you have a web application that gives some information, which, in turn, it, of course, must take somewhere. In a database, for example. Well, it saves the information that enters this REST API into the same database. So, accordingly, if your helcheck responds simply like they turned to slashhelps, the application says “200, okay, everything is fine”, and at the same time the database is not available for your application, and the application on the helcheck says “200, okay, everything is fine” - this is a bad helcheck. This is not how it should work.
That is, your application, when it receives a request for /health, it doesn’t just answer, “200, ok”, it first goes, for example, to the database, tries to connect to it, does something quite elementary there, like select one, just checks that there is a connection in the database and the database can be queried. If all this was successful, then the answer is “200, ok”. If not successful, then it says that the error, the database is not available.
Therefore, in connection with this, I again return to Readiness / Liveness samples - why you most likely need a readiness sample, but a liveness sample is questionable. Because if you describe helchecks exactly as I just said, then it will turn out that we are not available in the instance partв или со всех instanceto a database, for example. When you declared a readiness test, our health checks began to fail, and, accordingly, all applications from which the database is inaccessible, they simply turn off from balancing and actually “hang” just in a running state and wait until their databases work.
If we have declared a liveness test, then imagine that our database is broken, and in Kubernetes half of everything starts restarting in Kubernetes, because the liveness test is falling. This means you need to restart. This is not at all what you want, I even had personal experience in practice. We had an application that was a chat that was written in JS and included in the Mongo database. And there was just the same problem in that it was at the dawn of my work with Kubernetes, we described the readiness, liveness of the samples on the principle that Kubernetes can do it, so we will use it. Accordingly, at some point, Mongo “blunted” a little and the sample began to fail. Accordingly, according to the rainfall test, the hearths began to “kill”.
As you understand, when they are “killed”, this is a chat, that is, a bunch of connections from clients hang to it. They are also “killed” - no, not clients, only connections - not all at the same time, and due to the fact that they are not killed at the same time, someone earlier, someone later, they do not start at the same time. Plus the standard randomness, we cannot predict the start time of the application each time with an accuracy of a millisecond, so they do it one by one instance. One info spot rises, is added to the balancing, all clients come there, it cannot withstand such a load, because it is alone, and roughly speaking, a dozen of them work there, and then falls. The next one rises, the whole load is on him, he also falls. Well, these falls just continue in a cascade. In the end, how it was decided - I just had to directly hard stop the user traffic to this application, let all instances rise and then start all the user traffic at once so that it would already be distributed to all ten instances.
If it weren't for this liveness test announced, which would force it all to restart, the application would have coped with this perfectly. But everything from balancing is turned off for us, because the databases are inaccessible and all users have “fallen off”. Then, when this database becomes available, everything is included in the balancing, but applications do not need to restart, they do not need to spend time and resources on this. They are all already here, they are ready for traffic, so traffic just opens, everything is fine - the application is in place, everything continues to work.
Therefore, the readiness and liveness of the samples are different, even more so, you can theoretically do different helchecks, one type of ready, one type of liv, for example, and check different things. On readiness tests, check your backends. And on a liveness test, for example, do not check, from the point of view that the liveness of the test is generally just an application responding, if at all it is able to respond.
Because the test of liveness, by and large, is when we are “stuck”. An endless loop has begun or something else - and no more requests are processed. Therefore, it makes sense even to separate them - and implement different logic in them.
About what you need to answer when you have a trial, when you do helchecks. It's just really a pain. Those who are familiar with this will probably laugh - but seriously, I have seen services in my life that answer “200” in response to requests in one hundred percent cases. That is, who is successful. But at the same time, in the body of the response, they write "an error such and such."
That is, the status of the answer comes to you - everything is successful. But at the same time, you must parse the body, because the body says “sorry, the request ended with an error” and this is a direct reality. I saw this in real life.
And so that it is not funny for someone, and it hurts a lot for someone, you should still follow a simple rule. In health checks, and in principle in working with web applications.
If everything went well, then answer the two-hundred answer. In principle, any two-hundredth answer will suit. If you've read ragsey very well and know that some response statuses are different from others, respond with the appropriate ones: 204, 5, 10, 15, whatever. If not very good, then just “two zero zero”. If everything went badly and the helcheck doesn't answer, then you can answer with any 502th. Again, if you understand how to respond, how do different response statuses differ from each other. If you do not understand, then XNUMX is your option to respond to helchecks if something went wrong.
Another point, I want to return a little about checking the underlying services. If you start, for example, checking all the underlying services that are behind your application, that's all. That’s what we get from the point of view of microservice architecture, we have such a thing as “low coupling” - that is, when your services are minimally dependent on each other. If one of them fails, all the others without this functionality will simply continue to work. It's just that part of the functionality doesn't work. Accordingly, if you tie all the healthchecks to each other, then you will get that one thing has fallen in the infrastructure, and due to the fact that it has fallen, all the healthchecks of all services also begin to fail - and the infrastructure in general of the entire microservice architecture is larger No. Everything crashed there.
Therefore, I want to repeat this again, that you need to check the underlying services are those without which your application in one hundred percent of cases cannot do its job. That is, it is logical that if you have a REST API through which the user saves to the database or retrieves from the database, then in the absence of a database, you cannot work with your users for sure.
But if you have users, when you get them from the database, they are additionally enriched with some other metadata, from one more backend that you enter before giving a response to the frontend - and this backend is not available, this means that you give your answer without some part of the metadata.
Next, we also have one of the sore topics when launching applications.
In fact, this is not only about Kubernetes by and large, it just so happened that the culture of some kind of development in bulk and DevOps in particular began to spread around the same time as Kubernetes. Therefore, by and large, it turns out that you need to correctly (gracefully) shut down your application without Kubernetes. And before Kubernetes, people did this, but just with the advent of Kubernetes, we started talking about it en masse.
graceful shutdown
In general, what is Graceful Shutdown and what is it for. It's about when your application terminates for some reason, you need to execute app stop - or you receive, for example, a signal from the operating system, your application must understand it and do something with it. The worst option, of course, is when your application receives a SIGTERM and is like "SIGTERM, hang on, work, do nothing." This is downright bad choice.
An almost equally bad option is when your application receives a SIGTERM and it’s like “they said the segterm, that’s all, we’re finishing, I haven’t seen it, I don’t know any user requests, I don’t know what kind of requests I have there now, they said SIGTERM, so we’re finishing ". This is also a bad choice.
What is a good option? The first point, we take into account the completion of operations. A good option is when your server still takes into account what it does if it receives a SIGTERM.
SIGTERM is a soft shutdown, it is on purpose, it can be intercepted at the code level, it can be processed, say that now, wait, we will first finish the work that we have, then we will end.
From a Kubernetes point of view, what does it look like. When we say to a pod that is running in a Kubernetes cluster, “please stop, fail,” or we have a restart, or an update when Kubernetes recreates the pods - Kubernetes sends SIGTERM to the pod message, waits for some time, and , this is the time that he is waiting for, it is also configured, there is such a special parameter in diplomas and it is called Graceful ShutdownTimeout. As you understand, it is not without reason that it is called so, and it is not without reason that we are talking about it now.
There you can specifically say how long you need to wait between when we sent SIGTERM to the application and when we realize that the application seems to be freaking out or “stuck” and is not going to end - and you need to send it SIGKILL, that is, hard complete his work. That is, accordingly, some kind of daemon works for us, it processes operations. We understand that on average, we have operations that the daemon is working on do not last more than 30 seconds at a time. Accordingly, when SIGTERM arrives, we understand that our daemon can finish the maximum after SIGTERM for 30 seconds. We write it, for example, 45 seconds just in case and say that SIGTERM. After that, we wait 45 seconds. In theory, during this time the demon had to complete its work and end itself. But if suddenly he could not, then he is most likely stuck - he no longer processes our requests normally. And it can be safely nailed down in 45 seconds.
And here, in fact, even 2 aspects can be taken into account. First, understand that if you received a request, you started working with it somehow and did not give a response to the user, but you received a SIGTERM, for example. That makes sense to finalize and give an answer to the user. This is the number one point in this plan. Point number two here is that if you write your application, generally build the architecture in such a way that you received a request for your application, then you started some work, you started downloading files from somewhere, downloading the database and something else -That. In general, your user, your request hangs for half an hour and waits for you to answer him - then, most likely, you need to work on the architecture. That is, it’s easy to take into account even common sense that if your operations are short, then it makes sense to ignore SIGTERM and refine it. If your operations are long, then it makes no sense to ignore SIGTERM in this case. It makes sense to redesign the architecture so that such long operations are avoided. So that users simply do not hang, do not wait. I don’t know, make some kind of websocket there, make reverse hooks that your server will already send to the client, anything else, but don’t force the user to hang for half an hour and just wait for the session until you answer him. Because it is unpredictable where it can break.
When your application terminates, some appropriate exit code should be returned. That is, if your application was asked to close, stop, and it was able to stop normally by itself, then you do not need to return some type of exit code 1,5,255 and so on. Anything that is not a null code, at least on Linux systems, I'm pretty sure of it, is considered unsuccessful. That is, it is considered that your application in this case ended with an error. Accordingly, in an amicable way, if your application completed without an error, you say 0 in the output. If your application crashes for some reason, you're not saying 0 in the output. And you can work with this information.
And the last option. It’s bad when a user submits a request and hangs for half an hour while you process it. But in general, I still want to say about what is generally worth on the part of the client. It doesn’t matter if you have a mobile application, front-end, and so on. It must be taken into account that in general the user session can be terminated, anything can happen. A request may be sent, for example, underprocessed and no response returned. Your front-end or your mobile application - any front-end, let's put it this way - should take this into account. If you work with websockets, this is generally the worst pain that has been in my practice.
When the developers of some regular chats do not know that, it turns out, the websocket can break. When something elementary happens on the proxy, we just change the config, and it does a reload. Naturally, all long-lived sessions are torn at the same time. Developers come running to us and say: “Guys, what are you talking about, the chat fell off for all our clients there!”. We tell them: “What are you doing? Are your clients unable to reconnect?". They say: "No, we need the sessions not to break." In short, it's actually bullshit. You need to consider the client side. Especially, I say, with long-lived sessions such as websocket, that it can break and imperceptibly for the user you need to be able to reinstall such sessions. And then everything is perfect.
Resources
In general, here I will tell you just a straight story. Again from real life. The sickest thing I've heard about resources.
Resources in this case, I mean, some kind of requests, limits that you can put on pods in your Kubernetes clusters. The funniest thing I've heard from a developer... Once, one of my fellow developers at a previous job said: "My application in the cluster does not start." I looked that it did not start, and there it either did not fit in with resources, or they set very small limits. In short, the application cannot start due to resources. I say: “Because of the resources it doesn’t start, you decide how much you need and set an adequate value.” He says: "What kind of resources?". I began to explain to him that here are Kubernetes, limits in requests and blah, blah, blah, they need to be set. The man listened for five minutes, nodded and said: “I came here to work as a developer, I don’t want to know anything about any resources. I kind of came here to write code and that’s it.” It is sad. This is a very sad concept from a developer's point of view. Especially in the modern world, so to speak, of a progressive devops.
What do you need resources for? Resources in Kubernetes are of 2 types. Some are called requests, the second - limits. By resources, we will understand, basically, there are always only two basic restrictions. That is, CPU time limits and RAM limits for a container that is running in Kubernetes.
The limit limits the upper bound on the resource usage in your application. That is, respectively, if you say 1GB OP in limits, then your application will not be able to use more than 1GB OP. And if he suddenly wants to and tries to do this, then a process called oom killer, out of memory, that is, will come and kill your application - that is, it will simply restart. Applications are not restarted on the CPU. In terms of CPU, if an application tries to use a lot, more than specified in the limits, the CPU will simply be severely selected. It does not lead to restarts. This is the limit - this is the upper limit.
And there is a request. The request is what makes Kubernetes understand how the nodes in your Kubernetes cluster are populated with applications. That is, a request is a kind of commit of your application. It says what I want to use: "I would like you to reserve this much CPU and this much memory for me." Such a simple analogy. What if we have a node that has everything available, I don't know, 8 CPUs. And a pod arrives there, which has 1 CPU in the requests, which means the node has 7 CPUs left. That is, accordingly, as soon as 8 pods arrive at this node, each of which has 1 CPU in the requests - on the node, as if from the point of view of Kubernetes, the CPU is over and more pods with requests for this node cannot be launched. If all the nodes run out of CPU, then Kubernetes will start saying that there are no suitable nodes in the cluster in order to run your pods, because the CPU has run out.
What are requests for and why without requests, I think that nothing needs to be launched in Kubernetes? Let's imagine a hypothetical situation. You run your application without requests, Kubernetes does not know how much you have, which nodes you can push. Well, shoves, shoves, shoves on the nodes. At some point, traffic will start coming to your application. And one of the applications suddenly starts using resources up to the limits that it has according to the limits. It turns out that there is another application nearby and it also needs resources. The node starts to actually physically run out of resources, for example, OP. Resources, for example, random access memory (RAM), begin to physically run out on the node. When a node runs out of OP, you will stop responding first of all to the docker, then the kulet, then the OS. They will simply go into unconsciousness and EVERYTHING will stop working for you for sure. That is, this will lead to the fact that your node will get stuck, and it will need to be restarted. In short, the situation is not very good.
And when you have requests, the limits are still not very different, at least not many times more than the limits or requests, then you can have such a normal rational filling of applications by Kubernetes cluster nodes. At the same time, Kubernetes is approximately aware of how much it put where, how much is used where. I mean, it's just that moment. It is important to understand it. And it is important to control that it is affixed.
Data Storage
The next thing we have is about data storage. What to do with them and in general, how to deal with persistence in Kubernetes?
I think, again, we have, within , there was a topic about the database in Kubernetes. And it seems to me that I even roughly know what colleagues told you to the question: “Is it possible to run a database in Kubernetes?”. For some reason, it seems to me that colleagues should have told you that if you ask the question whether it is possible to run a database in Kubernetes, then it is impossible.
Here the logic is simple. Just in case, I’ll explain once again, if you’re a straight cool dude who can build a fairly fault-tolerant distributed network storage system, understand how to pull up a database for this business, how cloud native in containers should work in a database in general. Most likely, you do not have a question how to run it. If you have such a question, and you want to make it all turn around and stand right to death in production and never fall, then this does not happen. You're guaranteed to shoot yourself in the foot with this approach. So it's better not to.
What to do with the data that our application would like to store, some pictures that users upload, some things that our application generates during operation, at startup, for example? What to do with them in Kubernetes?
In general, ideally, yes, of course, Kubernetes is very well provided for and in general was originally conceived for stateless applications. That is, for those applications that do not store information at all. This is the ideal option.
But, of course, the ideal option, as it were, does not always exist. So what? The first and easiest point is to take some S3, but not self-made, which is also not clear how it works, but from some provider. A good normal provider - and teach your application to use S3. That is, when your user wants to upload a file, say "here, please, upload it to S3." When he wants to get it, say: "Here's a link to S3 backing and take it from here." This is the ideal option.
If suddenly, for some reason, such an ideal option does not fit, you have an application that you do not write, you do not develop, or it is some kind of terrible legacy, it cannot use the S3 protocol, but should work with local directories in local folders . Take something more or less simple, deploy this Kubernetes business. That is, to immediately fence Ceph for some minimal tasks, it seems to me that the idea is bad. Because Ceph is, of course, good and trendy. But if you don’t really understand what you are doing, then by adding something to Ceph, you can very easily and simply never get it from there again. Because, as you know, Ceph stores data in its cluster in binary form, and not in the form of simple files. Therefore, if the Ceph cluster suddenly breaks, then there is a complete and high probability that you will never get your data from there again.
We will have a course on Ceph, you can .
Because it's better to have something simple type on an NFS server. Kubernetes knows how to work with them, you can mount a directory under the NFS server - your application is just like a local directory. At the same time, of course, you need to understand that, again, you need to do something with your NFS, you need to understand that sometimes it may become inaccessible and provide for the question of what you will do in this case. Perhaps it should be backed up somewhere on a separate machine.
The next point I talked about was what to do if your application generates some files in the process. For example, when it starts, it generates some kind of static file, which is based on some information that the application receives only at the time of launch. Here's a moment. If there is not much such data, then you can not bother at all, just put this application under yourself and work. Only here the question is what, look. Very often legacy all sorts of systems, such as WordPress and so on, especially with modified ingenious plugins of some kind, ingenious php developers, they often know how to make it so that they generate some kind of file for themselves. Accordingly, one generates one file, the second generates the second file. They are different. Balancing happens in a cluster of Kubernetes clients just by chance. Respectively, together it turns out in instance they are not able to work. One gives one information, the other gives the user another information. This is what you should avoid. That is, in Kubernetes, everything that you run must be guaranteed to be able to work in several instances. Because Kubernetes is a moving thing. Accordingly, he can move anything, at any time, without asking anyone in general. Therefore, it must be counted on. Everything launched in one instance will fall sooner or later. The more reservations you have, the better. But again, I say, if you have few such files, then you can put them right under you, they weigh a small amount. If there are a little more of them, they probably should not be shoved inside the container.
I would advise, there is such a beautiful thing in Kubernetes, you can use volume. In particular, there is a volume of type empty dir. That is, it's just that Kubernetes will automatically create a directory in its service directories on the server where you started. And give it to you to use. There is only one important point here. That is, your data will not be stored inside the container, but, as it were, on the host on which you are running. Kubernetes, moreover, can control such empty dirs under normal configuration and can control their maximum size and not allow it to be exceeded. The only point is that what you have written in the empty dir is not lost when you restart the pod. That is, if your pod falls down by mistake and rises again, the information in the empty dir will not go anywhere. Under it again with a new start can use - and that's good. If your pod leaves somewhere, then naturally, it will leave without data. That is, as soon as the pod from the node where it was started with empty dir disappears, the empty dir is deleted.
What else is empty dir good for? For example, it can be used as a cache. Let's imagine that we have an application that generates something on the fly, gives it to users, and does it for a long time. Therefore, the application, for example, generates and gives it to users, and at the same time puts it somewhere for itself, so that the next time the user comes for the same thing, give it immediately generated faster. You can ask Kubernetes to create an empty dir in memory. And thus, your caches in general can simply work at lightning speed - in terms of disk access speed. That is, you have an empty dir in memory, it is stored in memory in the OS, and at the same time, for you, for the user inside the pod, it looks like just a local directory. You do not need the application to specifically teach some magic. You just directly take and put your file in the directory, but, in fact, in memory on the OS. This is also a very handy feature in terms of Kubernetes.
What problems does Minio have? The main problem with Minio is that in order for this thing to work, it needs to be running somewhere, and there must be some kind of file system, that is, storage, for it. And here we meet with the same problems that Ceph has. That is, Minio must store its files somewhere. It's just an HTTP interface to your files. Moreover, the functionality is clearly poorer than that of the Amazon S3. Previously, it was not able to properly authorize the user. Now, as far as I know, it already knows how to make buckets with different authorizations, but again, it seems to me that the main problem is, so to speak, the underlying storage system to a minimum.
Empty dir in memory how does it affect the limits? Doesn't affect limits in any way. It lies in the memory, it turns out, of the host, and not in the memory of your container. That is, your container does not see the empty dir in memory as part of its occupied memory. The host sees it. Accordingly, yes, from the point of view of kubernetes, when you start using this, it would be good to understand that you give part of your memory to the empty dir. And accordingly, understand that memory can run out not only because of applications, but also because someone writes to these empty dirs.
cloud nativeness
And the final subtopic, what is Cloudnative. Why is it needed. Cloudnativeness and so on.
That is, those applications that can and are written in order to work in a modern cloud infrastructure. But, in fact, Cloudnative has another such aspect. That this is not only an application that takes into account all the requirements of modern cloud infrastructure, but also knows how to work with this modern cloud infrastructure, take advantage of the fact that it works in these clouds. Do not just overdo yourself and work in the clouds, but directly use the advantages of the fact that you work in the cloud.
Let's just take Kubernetes as an example. You have an application running in Kubernetes. You can always have your application, or rather admins for your application, can always make a service account. That is, an account for authorization in Kubernetes itself in its server. Screw there any rights that we need. And you can access Kubernetes from within your application. What can be done in this way? For example, from the application to receive data about where your other applications, other similar instances are located, and together somehow cluster on top of Kubernetes, if there is such a need.
Again, recently we literally had a case. We have one controller who monitors the queue. And when some new tasks appear in this queue, it goes to Kubernetes - and inside Kubernetes creates a new pod. Gives this pod some new task and within this pod, the pod executes the task, sends a response to the controller itself, and the controller then does something with this information. The database, for example, adds up. That is, again, this is a plus that our application works in Kubernetes. We can use the built-in Kubernetes functionality itself in order to somehow expand and make the functionality of our application more convenient. That is, not to fence some kind of magic on how to launch an application, how to launch a worker. In Kubernetes, just throw a request in the app if the application is written in python.
The same applies if we go beyond Kubernetes. We have our Kubernetes running somewhere - it’s good if it’s in some kind of cloud. Again, we can use, and even should, I believe, use the capabilities of the cloud itself, where we are launched. From the elementary that the cloud provides us. Balancing, that is, we can create cloud balancers and use them. This is a direct advantage of what we can use. Because cloud balancing, firstly, just stupidly removes responsibility from us for how it works, how it is configured. Plus, it’s very convenient, because the usual Kubernetes with clouds can integrate.
The same goes for scaling. Regular Kubernetes can integrate with cloud providers. He knows how to understand that if the cluster runs out of nodes, that is, the place of the node is over, then you need to add - Kubernetes itself will add new nodes to your cluster and start running pods on them. That is, when your load comes, your hearths begin to increase in number. When the nodes in the cluster run out for these pods, Kubernetes launches new nodes and, accordingly, the number of pods can still increase. And it's very convenient. This is a direct opportunity to scale the cluster on the fly. Not very fast, in the sense that it's not a second, it's more like a minute to add new nodes.
But from my experience, again, the coolest thing I've ever seen. When the Cloudnative cluster scaled based on the time of day. It was a kind of backend service that people in the back office used. That is, they come to work at 9 am, start logging in, respectively, the Cloudnative cluster, where it's all running, starts to swell, launch new pods, so that everyone who comes to work can work with the application. When they leave work at 8pm or 6pm, the Kubernetes clusters notice that no one else is using the application and start shrinking. Save up to 30% guaranteed. It then worked in Amazon, in Russia at that time there was no one at all who could do it so cool.
I'm just saying, saving 30 percent simply because we use Kubernetes, we use the capabilities of the cloud. Now this can be done in Russia. I will not advertise anyone, of course, but let's just say that there are providers who can do this, provide it right from the box at the click of a button.
The last point, which I also want to pay attention to. In order for your application, your infrastructure to be Cloudnative, it makes sense to finally start adapting the approach called Infrastructure as a Code. That is, it means that your application, or rather your infrastructure, needs exactly the same as the code describe your application, your business logic in the form of code. And work with it as with code, that is, test it, roll it out, store it in git, apply CICD to it.
And this is exactly what allows you, firstly, to always have control over your infrastructure, always understand what state it is in. Second, avoid manual operations that cause errors. Thirdly, to avoid just what is called turnover, when you constantly need to perform the same manual tasks. Fourth, it allows you to recover much faster in the event of a failure. Here in Russia, every time I talk about this, there are necessarily a huge number of people who say: “Yeah, it’s clear, yes, you have approaches, in short, you don’t need to fix anything.” But it's true. If something breaks in your infrastructure, then from the point of view of the Cloudnative approach and from the point of view of Infrastructure as a Code, than to fix it, go to the server, figure out what is broken and fix it, it’s easier to delete the server and create it again. And I will get it all back.
All of these issues are discussed in more detail at . Click the link to view the program and conditions. The convenient thing is that you can master Kubernetes by working from home or from work for 1-2 hours a day.
Source: habr.com