{"id":70831,"date":"2020-02-21T14:42:15","date_gmt":"2020-02-21T11:42:15","guid":{"rendered":"https:\/\/prohoster.info\/blog\/obnovlenie-kubernetes-klastera-bez-prostoya"},"modified":"2020-03-03T16:14:45","modified_gmt":"2020-03-03T13:14:45","slug":"obnovlenie-kubernetes-klastera-bez-prostoya","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/obnovlenie-kubernetes-klastera-bez-prostoya","title":{"rendered":"Updating a Kubernetes Cluster Without Downtime","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Updating a Kubernetes Cluster Without Downtime\" src=\"\/wp-content\/uploads\/2020\/02\/53cdba726a58c16679c6443191f28321.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>The update process for your Kubernetes cluster<\/em><\/p>\n<p><\/p>\n<p>At some point while using a Kubernetes cluster, there is a need to upgrade the running nodes. This can involve package updates, kernel upgrades, or deploying new virtual machine images. In Kubernetes terminology, this is referred to as <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/workloads\/pods\/disruptions\/#voluntary-and-involuntary-disruptions\">\"Voluntary Disruption\"<\/a><\/noindex>.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>This post is part of a series of 4 posts:<\/p>\n<p><\/p>\n<ol>\n<li>This post.<\/li>\n<li>Graceful termination of pods in a Kubernetes cluster<\/li>\n<li>Delayed termination of the pod upon its deletion<\/li>\n<li>How to avoid downtime in your Kubernetes cluster using PodDisruptionBudgets<\/li>\n<\/ol>\n<p><\/p>\n<p><em>(Translator's note: translations of the other articles in the series will be available soon)<\/em><\/p>\n<p><\/p>\n<p>In this article, we will describe all the tools Kubernetes provides to achieve zero downtime for the nodes running in your cluster.<\/p>\n<p><\/p>\n<h2 id=\"opredelenie-problemy\">Defining the problem<\/h2>\n<p><\/p>\n<p>Initially, we will take a naive approach to identify problems and assess the potential risks of this approach, accumulating knowledge to address each of the problems we encounter throughout the series. As a result, we will achieve a configuration that utilizes lifecycle hooks, readiness probes, and Pod disruption budgets to achieve our zero downtime goal. <\/p>\n<p><\/p>\n<p>To start our journey, let's take a concrete example. Suppose we have a Kubernetes cluster with two nodes, running an application with two pods, located behind <code>Service<\/code>:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Updating a Kubernetes Cluster Without Downtime\" src=\"\/wp-content\/uploads\/2020\/02\/f1eb999a0a088d7ed5b54c71e9b9ed1e.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><em>Let's start with two pods with Nginx and the Service running on our two Kubernetes cluster nodes.<\/em><\/p>\n<p><\/p>\n<p>We want to upgrade the kernel version of the two working nodes in our cluster. How do we do this? A simple solution would be to boot new nodes with the updated configuration and then shut down the old nodes while launching the new ones. While this will work, there will be several issues with this approach:<\/p>\n<p><\/p>\n<ul>\n<li>When you shut down the old nodes, the pods running on them will also be shut down. What if the pods need to be cleaned up for proper shutdown? The virtualization system you are using may not wait for the cleanup process to finish.<\/li>\n<li>What if you turn off all nodes at the same time? You will experience considerable downtime while the pods migrate to the new nodes.<\/li>\n<\/ul>\n<p><\/p>\n<p>We need a way to correctly migrate pods from old nodes while ensuring that none of our workflows are running while we make changes to the node. Or when we perform a complete cluster replacement, as in the example (i.e., replacing VM images), we want to move running applications from the old nodes to the new ones. In both cases, we want to prevent new pods from being scheduled on the old nodes, and then evict all running pods from them. To achieve these goals, we can use the command <code>kubectl drain<\/code>.<\/p>\n<p><\/p>\n<h2 id=\"pereraspredelenie-vseh-podov-s-nody\">Evicting all pods from the node<\/h2>\n<p><\/p>\n<p>The drain operation allows you to evict all pods from a node. During the execution of the drain, the node is marked as unschedulable (flag <code>NoSchedule<\/code>). This prevents new pods from being placed on it. Then, the drain starts evicting pods from the node, terminating containers that are currently running on the node by sending a signal <code>TERM<\/code> to the containers in the pod.<\/p>\n<p><\/p>\n<p>Although <code>kubectl drain<\/code> it will do a great job of evicting pods, there are two more factors that can cause the drain operation to fail:<\/p>\n<p><\/p>\n<ul>\n<li>Your application must be able to terminate correctly when it receives <code>TERM<\/code> the signal. When pods are evicted, Kubernetes sends the signal <code>TERM<\/code> to the containers and waits for them to stop for a specified amount of time, after which, if they have not stopped, it forcibly terminates them. In any case, if your container does not handle the signal correctly, you may still shut down the pods improperly if they happen to be active at that specific moment (for example, if a transaction is occurring in the database).<\/li>\n<li>You lose all pods that contain your application. It may be unavailable when new containers are launched on the new nodes, or, if your pods are deployed without controllers, they may not restart at all. <\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"izbegaem-prostoya\">Minimizing downtime<\/h2>\n<p><\/p>\n<p>To minimize downtime from voluntary disruptions, such as during the drain operation for a node, Kubernetes provides the following failure handling options:<\/p>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/workloads\/pods\/pod\/#termination-of-pods\">Graceful termination<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/containers\/container-lifecycle-hooks\/\">Lifecycle hooks<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/workloads\/pods\/disruptions\/#how-disruption-budgets-work\">PodDisruptionBudgets<\/a><\/noindex><\/li>\n<\/ul>\n<p><\/p>\n<p>In the other parts of the cycle, we will use these Kubernetes features to mitigate the impact of migrating pods. To better trace the main idea, we'll use our example above with the following resource configuration:<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">---\napiVersion: apps\/v1\nkind: Deployment\nmetadata:\n name: nginx-deployment\n labels:\n   app: nginx\nspec:\n replicas: 2\n selector:\n   matchLabels:\n     app: nginx\n template:\n   metadata:\n     labels:\n       app: nginx\n   spec:\n     containers:\n     - name: nginx\n       image: nginx:1.15\n       ports:\n       - containerPort: 80\n---\nkind: Service\napiVersion: v1\nmetadata:\n name: nginx-service\nspec:\n selector:\n   app: nginx\n ports:\n - protocol: TCP\n   targetPort: 80\n   port: 80<\/code><\/pre>\n<p><\/p>\n<p>This configuration is a minimal example <code>Deployment<\/code>, which manages nginx pods in the cluster. Additionally, the configuration describes a resource <code>Service<\/code>, which can be used to access the nginx pods in the cluster.<\/p>\n<p><\/p>\n<p>Throughout the lifecycle, we will iteratively expand this configuration so that it ultimately includes all features provided by Kubernetes to minimize downtime.<\/p>\n<p><\/p>\n<p><em>To get a fully integrated and tested version of Kubernetes cluster updates for zero downtime on AWS and other resources, visit <noindex><a rel=\"nofollow\" href=\"https:\/\/gruntwork.io\/\">Gruntwork.io<\/a><\/noindex>.<\/em><\/p>\n<p><\/p>\n<h2 id=\"takzhe-chitayte-drugie-stati-v-nashem-bloge\">Also, read other articles in our blog:<\/h2>\n<p><\/p>\n<ul>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/nixys\/blog\/481932\/\">Zero Downtime Deployment and databases<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/nixys\/blog\/480072\/\">Kubernetes: why it is crucial to set up system resource management?<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/nixys\/blog\/481992\/\">Tekton Pipeline \u2014 Kubernetes-native pipelines<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/nixys\/blog\/473578\/\">Building dynamic modules for Nginx<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/nixys\/blog\/473014\/\">Introduction to Hashicorp Consul\u2019s Kubernetes Authorization<\/a><\/noindex><\/li>\n<li><noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/nixys\/blog\/468779\/\">What the migration from ClickHouse without authorization to ClickHouse with authorization led to<\/a><\/noindex><\/li>\n<\/ul>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/nixys\/blog\/489164\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u043e\u0446\u0435\u0441\u0441 \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u044f \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0433\u043e Kubernetes-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0412 \u043a\u0430\u043a\u043e\u0439-\u0442\u043e \u043c\u043e\u043c\u0435\u043d\u0442 \u043f\u0440\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0438 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 Kubernetes \u0432\u043e\u0437\u043d\u0438\u043a\u0430\u0435\u0442 \u043f\u043e\u0442\u0440\u0435\u0431\u043d\u043e\u0441\u0442\u044c \u0432 \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u0438 \u0440\u0430\u0431\u043e\u0442\u0430\u044e\u0449\u0438\u0445 \u043d\u043e\u0434. \u041e\u043d\u043e \u043c\u043e\u0436\u0435\u0442 \u0432\u043a\u043b\u044e\u0447\u0430\u0442\u044c \u0432 \u0441\u0435\u0431\u044f \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u044f \u043f\u0430\u043a\u0435\u0442\u043e\u0432, \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u0435 \u044f\u0434\u0440\u0430 \u0438\u043b\u0438 \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 \u043d\u043e\u0432\u044b\u0445 \u043e\u0431\u0440\u0430\u0437\u043e\u0432 \u0432\u0438\u0440\u0442\u0443\u0430\u043b\u044c\u043d\u044b\u0445 \u043c\u0430\u0448\u0438\u043d. \u0412 \u0442\u0435\u0440\u043c\u0438\u043d\u043e\u043b\u043e\u0433\u0438\u0438 Kubernetes \u044d\u0442\u043e \u043d\u0430\u0437\u044b\u0432\u0430\u0435\u0442\u0441\u044f &quot;Voluntary Disruption&quot;. \u042d\u0442\u043e\u0442 \u043f\u043e\u0441\u0442 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0447\u0430\u0441\u0442\u044c\u044e \u0446\u0438\u043a\u043b\u0430 \u0438\u0437 4 \u043f\u043e\u0441\u0442\u043e\u0432: \u042d\u0442\u043e\u0442 \u043f\u043e\u0441\u0442. \u041a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u043e\u0435 \u0437\u0430\u0432\u0435\u0440\u0448\u0435\u043d\u0438\u0435 \u0440\u0430\u0431\u043e\u0442\u044b pod\u2019\u043e\u0432 \u0432 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":70832,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-70831","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u043e\u0446\u0435\u0441\u0441 \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u044f \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0433\u043e.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/obnovlenie-kubernetes-klastera-bez-prostoya\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u0435 Kubernetes-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0431\u0435\u0437 \u043f\u0440\u043e\u0441\u0442\u043e\u044f | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u043e\u0446\u0435\u0441\u0441 \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u044f \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0433\u043e.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/obnovlenie-kubernetes-klastera-bez-prostoya\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-02-21T11:42:15+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-03T13:14:45+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kubernetes cluster update without downtime | ProHoster","description":"The update process for your.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/obnovlenie-kubernetes-klastera-bez-prostoya","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u0435 Kubernetes-\u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0431\u0435\u0437 \u043f\u0440\u043e\u0441\u0442\u043e\u044f | ProHoster","og:description":"\u041f\u0440\u043e\u0446\u0435\u0441\u0441 \u043e\u0431\u043d\u043e\u0432\u043b\u0435\u043d\u0438\u044f \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0433\u043e.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/obnovlenie-kubernetes-klastera-bez-prostoya","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-02-21T11:42:15+00:00","article:modified_time":"2020-03-03T13:14:45+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"70831","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:13:22","updated":"2022-09-29 13:06:50","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/70831","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=70831"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/70831\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/70832"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=70831"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=70831"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=70831"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}