An important point in the operation of distributed systems is the handling of failures. Kubernetes helps with this by using controllers that monitor the state of your system and restart services that have stopped working. However, Kubernetes can force stop your applications to ensure overall system health. In this series, we'll look at how you can help Kubernetes do its job more efficiently and reduce application downtime.
Prior to containers, most applications ran on virtual or physical machines. If the application crashed or hung, it took a long time to cancel the running task and reload the program. In the worst case, someone had to solve this problem manually at night, at the most inopportune time. If only 1-2 working machines were performing an important task, such a failure in work was completely unacceptable.
Therefore, instead of manually restarting, they began to use monitoring at the process level to automatically restart the application in case of an abnormal termination. If the program fails, the monitoring process captures the exit code and restarts the server. With the advent of systems such as Kubernetes, this kind of response to system failures was simply integrated into the infrastructure.
Kubernetes uses a watch-diff-take-action event loop to make sure resources stay healthy on their way from containers to the nodes themselves.
This means that you no longer need to manually start monitoring processes. If a resource fails the Health Check, Kubernetes will simply automatically provide a replacement for it. That being said, Kubernetes does much more than just keep your applications crashing. It can create more copies of the application to run on multiple machines, update the application, or run multiple versions of your application at the same time.
Therefore, there are many reasons why Kubernetes can abort a perfectly healthy container. For example, if you upgrade your deployment, Kubernetes will slowly stop old pods while starting new ones. If you shut down a node, Kubernetes will terminate all pods in that node. Finally, if a node runs out of resources, Kubernetes will shut down all pods to free those resources.
Therefore, it is very important that your application terminates with minimal impact on the end user and minimal recovery time. This means that before shutting down, it must save any data that needs to be saved, close all network connections, complete any remaining work, and have time to complete other urgent tasks.
In practice, this means that your application must be able to process the SIGTERM message, the process termination signal that is the default signal for the kill utility in Unix operating systems. Upon receiving this message, the application should terminate.
Once Kubernetes decides to end a pod, a number of things happen. Let's take a look at each step that Kubernetes takes when a container or pod is terminated.
Let's say we want to terminate one of the pods. At this point, it will stop receiving new traffic - the containers running in the pod will not be affected, but all new traffic will be blocked.
Let's take a look at the preStop hook, which is a special command or HTTP request that is sent to containers in a Pod. If your application doesn't shut down correctly when it receives a SIGTERM, you can use preStop to shut down properly.
Most programs exit gracefully when they receive a SIGTERM signal, but if you're using third-party code or some system you don't have full control over, the preStop hook is a great way to force a graceful shutdown without changing the application.
After executing this hook, Kubernetes will send a SIGTERM signal to the containers in the pod, which will let them know that they will be shut down soon. Upon receiving this signal, your code will jump into the shutdown process. This process may include stopping any long-lived connections such as a database connection or a WebSocket thread, saving the current state, and the like.
Even if you use the preStop hook, it's important to check what exactly happens to your application when you send it a SIGTERM signal, and how it behaves, so that events or system changes caused by a pod shutdown don't come as a surprise to you.
At this point, before taking further action, Kubernetes will wait for a specified amount of time, called terminationGracePeriodSecond, or the graceful shutdown period upon receiving a SIGTERM signal.
By default, this period is 30 seconds. It is important to note that it runs in parallel with the preStop hook and the SIGTERM signal. Kubernetes will not wait for the preStop hook and SIGTERM to finish - if your application terminates before the end of the TerminationGracePeriod, Kubernetes will immediately move on to the next step. Therefore, make sure that the value of this period in seconds is not less than the time required for the correct shutdown of the pod, and if it is more than 30s, increase the period to the desired value in YAML. In the example given, it is 60s.
And finally, the last step - if the containers are still running after the terminationGracePeriod, they will send a SIGKILL signal and be forcefully deleted. At this point, Kubernetes will also clean up all other objects in the pod.
Kubernetes terminates pods for many reasons, so make sure your application is terminated gracefully anyway to keep the service running smoothly.
Some ads π
Thank you for staying with us. Do you like our articles? Want to see more interesting content? Support us by placing an order or recommending to friends, , a unique analogue of entry-level servers, which was invented by us for you: (available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).
Dell R730xd 2 times cheaper in Equinix Tier IV data center in Amsterdam? Only here in the Netherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - from $99! Read about
Source: habr.com