{"id":33773,"date":"2019-10-31T21:54:36","date_gmt":"2019-10-31T18:54:36","guid":{"rendered":"https:\/\/prohoster.info\/blog\/rezervirovanie-v-kubernetes-ono-sushhestvuet\/"},"modified":"2019-10-31T21:54:36","modified_gmt":"2019-10-31T18:54:36","slug":"rezervirovanie-v-kubernetes-ono-sushhestvuet","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rezervirovanie-v-kubernetes-ono-sushhestvuet","title":{"rendered":"Kubernetes Reservation: It Exists","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

My name is Sergey, I am from the company ITSumma, and I want to tell you how we approach redundancy in Kubernetes. Recently, I have been heavily involved in consulting on the implementation of various DevOps solutions for different teams, specifically working on projects utilizing K8s. At the Uptime Day 4 conference, which focused on redundancy in complex architectures, I presented a report on the redundancy of a \"cube,\" and here is a loose summary of it. Just to clarify beforehand, it is not a direct guide to action but rather a generalization of thoughts on the given topic.<\/p>\n

\"Kubernetes
\n
\nIn principle, monitoring and redundancy are the two main tools for increasing the resilience of any project. But, you might say, in K8s everything balances itself, everything scales up automatically, and if something happens, it will recover by itself... So, at first glance, when researching this topic, the question of how to approach redundancy in K8s received the internet's response of \"why bother?\" Many believe that K8s is some sort of magical solution that eliminates all infrastructure problems and ensures that a project will never go down. But... the world is not what it seems.
\n<\/a><\/noindex>
\nHow did we approach the redundancy process before? We had identical environments for hosting \u2014 either they were virtual machines or physical servers servers<\/a>, to which we applied three basic practices: <\/p>\n

    \n
  1. synchronization of code and static files<\/li>\n
  2. synchronization of configurations<\/li>\n
  3. database replication<\/li>\n<\/ol>\n

    \nAnd voila: at any moment we switch to the backup site, everyone is happy, we stand up and disperse. <\/p>\n

    \"Kubernetes
    \n
    \nWhat do we have to increase the constant availability of our Kubernetes application? The first thing the unofficial documentation suggests is to set up multiple machines, create multiple masters\u2014the number of which must satisfy quorum requirements within the cluster, and ensure that etcd, api, MC, scheduler, etc. are running on each master. It all seems great: when several worker nodes or masters fail, our cluster will rebalance, and the application will continue to run. It looks like magic! But often our cluster is located within a single data center, which can raise certain questions. What if a backhoe comes and digs up a cable, or lightning strikes, or a universal flood occurs? Everything goes down, and our cluster is no more. How should we approach redundancy considering this side of the problem? <\/p>\n

    First of all, you need another hot standby cluster, meaning a cluster you can switch to at any moment. From the Kubernetes perspective, the infrastructure must be fully identical. So if there are any non-standard plugins for filesystem interaction, custom ingress solutions, they should be completely identical across your two (or three, or ten, as many as your budget and admin resources allow) clusters. It is necessary to clearly define two sets of applications (deployments, statefulsets, daemonsets, cronjobs, etc.): which of them can operate on standby constantly, and which are better not to start until an actual switch. <\/p>\n

    Should our standby cluster be completely identical to our production cluster? No. In the past, while dealing with monolithic projects and hardware infrastructure, we maintained almost completely identical environments; however, within Kubernetes, I believe this should not be the case. Let's explore why.<\/p>\n

    For example, let's start with the basic entities of Kubernetes \u2014 deployments \u2014 they should be identical. Applications must be running that can take over traffic processing at any moment and allow our project to continue operating. When it comes to configuration files, we need to consider whether they should be identical or not. This means that if we, smart individuals, do not use any prohibited substances and do not store the database in K8s, then our configmaps must contain access settings for the production database (the backup process is built separately). Accordingly, to ensure access to the backup instance of the database, we must have a separate configuration file (configmap). We work in exactly the same way with secrets: passwords for database access, API keys; at any point in time, either the production secret or the backup can be operational. So we already have two Kubernetes entities, whose backup versions must not be identical to the production ones. The next entity to focus on is the cronjob. The cronjobs in the backup must not, under any circumstances, be identical to the set of cronjobs in the production cluster! If we bring up a backup cluster and fully activate it with all included cronjobs \u2014 then, for instance, people could receive two emails at the same time instead of one. Or some data synchronization with external sources could happen twice, consequently leading us to despair, weep, scream, and argue. <\/p>\n

    \"Kubernetes
    \n
    \nSo how do people on the internet suggest we organize a backup cluster? The second most popular answer after \"why bother?\" is the use of Kubernetes Federation. <\/p>\n

    What is it? Let's say it's a large meta-cluster. If we imagine the architecture of Kubernetes\u2014where we have a master and several nodes\u2014then from the federation perspective, we also have a master and several nodes, but each node is a separate cluster. So, we are working with the same entities and primitives as with a single Kubernetes instance, but instead of manipulating our physical machines, we are working with entire clusters. Within the federation, we have full synchronization of federated resources from parents to children. For example, if we deploy something through the federation, it will get deployed on each of our child clusters. If we take any configmap or secret and roll it out through the federation, it will propagate to all our child clusters; at the same time, the federation allows us to customize our resources on the children. So, we took a configmap, deployed it through the federation, and then, if we need to make any adjustments on specific clusters, we go to edit it on the individual cluster, and that change will not synchronize anywhere else. <\/p>\n

    Kubernetes Federation is a relatively new tool that does not support the full range of resources offered by K8s. At the time of the documentation's first release, only config maps, deployment under replica set, and ingress were supported. Secrets were not supported, nor was work with volumes. This is a very limited set, especially if we enjoy customizing things\u2014such as passing our own resources to Kubernetes through custom resource definitions\u2014we cannot shove them into the federation. So, it\u2019s a solution that seems true but makes us shoot ourselves in the foot periodically. On the other hand, the federation allows flexible management of our replica set. For instance, if we want to run 10 replicas of our application, the federation will proportionally distribute this number across the clusters by default. This can also be configured! You can specify that the production cluster needs to hold 6 replicas of our application, and for resource-saving or personal amusement\u2014only 4 replicas on the backup cluster. That\u2019s pretty convenient, too. However, with the federation, we have to use some new solutions, deploy additional components on the fly, and challenge ourselves to think a bit more. <\/p>\n

    Is there a simpler way to approach the process of backing up Kubernetes? What tools do we actually have?<\/p>\n

    First of all, we always have some CI\/CD system, meaning we don't do things manually, or write create\/apply commands on the servers. The system generates YAML files for our containers. <\/p>\n

    Secondly, we have several clusters; we have either one or multiple (if we are smart) registries that we have also reserved. And there is a wonderful utility called kubectl that can work with multiple clusters simultaneously. <\/p>\n

    \"Kubernetes
    \n
    \nSo, in my opinion, the simplest and most accurate solution for building a backup cluster is a straightforward parallel deployment. There is some pipeline in the CI\/CD system; first, we build our containers, test them, and deploy applications via kubectl to multiple independent clusters. We can make simultaneous deployments across several clusters. Accordingly, we also address the delivery of configurations at this stage. We can pre-define a set of configurations for our production cluster and a set for the backup cluster and deploy the production environment to the production cluster and the backup environment to the backup cluster at the CI\/CD system level. Compared to federation, we do not need to go to each child cluster after defining the federated resource and redefine anything. We did this in advance. Aren't we clever?<\/p>\n

    But\u2026 there's\u2026 I was about to write there\u2019s a \"root of all evil,\" but there are actually two. First, the file system. There\u2019s some PV, or we use external storage. If we store files within the cluster, we need to adhere to the old practices that date back to the times of physical infrastructure: for example, synchronize using lsync. Or any other tool you personally prefer. We deploy everything on other machines and carry on.<\/p>\n

    Secondly, and actually the even more important sticking point \u2014 the database. If we are smart and do not keep the database in Kubernetes, then the data backup process follows the same old scheme \u2014 master-slave replication, then switching over, catching up the replica, and we'll live well. But if we keep our DB inside the cluster, there are actually many ready-made solutions for organizing a master-slave replica and many solutions for deploying a DB inside Kubernetes.
    \n A billion reports have already been written about database backup, a billion articles have been published, nothing new really needs to be said here. In general, follow your dreams, live as you wish, invent some complex workarounds for yourself too, but make sure to think about how you will back everything up.<\/p>\n

    Now let's discuss how we will generally switch to a backup site in case of a fire. First, we deploy stateless applications in parallel. They don't affect the business logic of our applications or our project; we can continuously run two sets of applications, and they can start receiving traffic. It's crucial to check whether we need to redefine configurations during the switch to the backup site. For example, we have a production Kubernetes cluster, a backup Kubernetes cluster, an external master database, and a backup master database. We have four options for how these applications in production can interact with each other. The database might switch, and we need to redirect traffic in the production cluster to the new database, or the cluster might fail \u2014 and we switched to the backup but continue to work with the production database. The third option is when both fail and we switch both applications, redefining our configuration so that the new applications work with the new database. <\/p>\n

    So, what conclusions can we draw from all this? <\/p>\n

    \"Kubernetes
    \n
    \nConclusion one: living with a backup is good. But it's expensive. Ideally, you shouldn't just have one backup. Ideally, you should have multiple backups. First, the backup should be at least not in one data center, and secondly, it should ideally be with a different host. I have unfortunately seen this happen in my experience. I can't name the projects, but there was a fire in a data center... I was like: let's switch to the backup! But the backup servers were in the same rack...<\/p>\n

    Or imagine if Amazon was banned in Russia (which has happened). And that's it: what's the point of having our backup in another Amazon? It's also inaccessible. So, I repeat: we keep a backup, at a minimum, in another data center, and preferably with another host. <\/p>\n

    The second takeaway is: if your application in Kubernetes interacts with some external sources (this can be both a database and some external API), ensure you define it as a service with an external endpoint so that when switching, you don't have to redeploy 15 applications that connect to the same database. Define your database as a separate service, access it as if it\u2019s inside your cluster: if your database goes down, you just change the IP in one place and continue to live happily. <\/p>\n

    And finally: I love \"the cube\", as well as experimenting with it. I also enjoy sharing the results of these experiments and my personal experiences. That\u2019s why I recorded a series of webinars about K8s, welcome to our YouTube channel<\/a><\/noindex> for more details.
    \n
    Source:     habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

    \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0421\u0435\u0440\u0433\u0435\u0439, \u044f \u0438\u0437 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 ITSumma, \u0438 \u044f \u0445\u043e\u0447\u0443 \u0432\u0430\u043c \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0434\u0445\u043e\u0434\u0438\u043c \u043a \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044e \u0432 Kubernetes. \u0412 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0435\u0435 \u0432\u0440\u0435\u043c\u044f \u044f \u043c\u043d\u043e\u0433\u043e \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u043a\u043e\u043d\u0441\u0443\u043b\u044c\u0442\u0430\u0442\u0438\u0432\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u043e\u0439 \u043f\u043e \u0432\u043d\u0435\u0434\u0440\u0435\u043d\u0438\u044e \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u043d\u044b\u0445 devops-\u0440\u0435\u0448\u0435\u043d\u0438\u0439 \u0434\u043b\u044f \u0440\u0430\u0437\u043b\u0438\u0447\u043d\u044b\u0445 \u043a\u043e\u043c\u0430\u043d\u0434, \u0438, \u0432 \u0447\u0430\u0441\u0442\u043d\u043e\u0441\u0442\u0438, \u043f\u043b\u043e\u0442\u043d\u043e \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u043f\u043e \u043f\u0440\u043e\u0435\u043a\u0442\u0430\u043c \u0441 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435\u043c K8s. \u041d\u0430 \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u0438 Uptime day 4, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0431\u044b\u043b\u0430 \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u0430 \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044e \u0432 \u0441\u043b\u043e\u0436\u043d\u044b\u0445 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":25449,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-33773","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Backup in Kubernetes: it exists | ProHoster","description":"My name is Sergey, I am from ITSumma, and I want to tell you how we approach backup in Kubernetes.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rezervirovanie-v-kubernetes-ono-sushhestvuet","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0420\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0432 Kubernetes: \u043e\u043d\u043e \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u0435\u0442 | ProHoster","og:description":"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0421\u0435\u0440\u0433\u0435\u0439, \u044f \u0438\u0437 \u043a\u043e\u043c\u043f\u0430\u043d\u0438\u0438 ITSumma, \u0438 \u044f \u0445\u043e\u0447\u0443 \u0432\u0430\u043c \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c, \u043a\u0430\u043a \u043c\u044b \u043f\u043e\u0434\u0445\u043e\u0434\u0438\u043c \u043a \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044e \u0432 Kubernetes.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/rezervirovanie-v-kubernetes-ono-sushhestvuet","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:54:36+00:00","article:modified_time":"2019-10-31T18:54:36+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"33773","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-02-09 17:03:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:33:25","updated":"2026-02-09 17:03:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/33773","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=33773"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/33773\/revisions"}],"predecessor-version":[{"id":159074,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/33773\/revisions\/159074"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/25449"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=33773"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=33773"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=33773"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}