{"id":35955,"date":"2019-10-31T22:08:26","date_gmt":"2019-10-31T19:08:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/razvertyvanie-prilozhenij-na-neskolkih-klasterah-kubernetes-s-helm\/"},"modified":"2019-10-31T22:08:26","modified_gmt":"2019-10-31T19:08:26","slug":"razvertyvanie-prilozhenij-na-neskolkih-klasterah-kubernetes-s-helm","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razvertyvanie-prilozhenij-na-neskolkih-klasterah-kubernetes-s-helm","title":{"rendered":"Deploying Applications Across Multiple Kubernetes Clusters with Helm","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<\/p>\n<p><\/p>\n<h2 id=\"kak-dailymotion-ispolzuet-kubernetes-razvertyvanie-prilozheniy\">How Dailymotion Uses Kubernetes: Deploying Applications<\/h2>\n<p><\/p>\n<p>At Dailymotion, we started using Kubernetes in production three years ago. However, deploying applications across multiple clusters has been quite a challenge, so over the past few years, we have been working to improve our tools and workflows.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h3 id=\"s-chego-nachalos\">How It Started<\/h3>\n<p><\/p>\n<p>Here, we will explain how we deploy our applications across multiple Kubernetes clusters worldwide.<\/p>\n<p><\/p>\n<p>To deploy multiple Kubernetes objects at once, we use <noindex><a rel=\"nofollow\" href=\"https:\/\/helm.sh\/\">Helm<\/a><\/noindex>, and all our charts are stored in a single git repository. To deploy a full stack of applications consisting of multiple services, we utilize a so-called umbrella chart. Essentially, this is a chart that declares dependencies and allows for the initialization of the API and its services with a single command.<\/p>\n<p><\/p>\n<p>Additionally, we wrote a small Python script on top of Helm to handle checks, create charts, add secrets, and deploy applications. All of these tasks are executed on a central CI platform using a Docker image.<\/p>\n<p><\/p>\n<p>Let's get to the point.<\/p>\n<p><\/p>\n<blockquote><p>Note: By the time you read this, the first release candidate for Helm 3 has already been announced. The main version includes a whole set of enhancements aimed at resolving some issues we faced in the past.<\/p><\/blockquote>\n<p><\/p>\n<h3 id=\"rabochiy-process-razrabotki-chartov\">Chart Development Workflow<\/h3>\n<p><\/p>\n<p>For applications, we use branching, and we decided to apply the same approach to charts.<\/p>\n<p><\/p>\n<ul>\n<li>The branch <strong>dev<\/strong> is used to create charts that will be tested on development clusters.<\/li>\n<li>When a pull request is submitted to <strong>master<\/strong>, they are validated in staging.<\/li>\n<li>Finally, we create a pull request to merge changes into the branch <strong>prod<\/strong> and apply them in production.<\/li>\n<\/ul>\n<p><\/p>\n<p>Each environment has its own private repository that stores our charts, and we use <noindex><a rel=\"nofollow\" href=\"https:\/\/chartmuseum.com\/\">Chartmuseum<\/a><\/noindex> with very useful APIs. This ensures strict isolation between environments and the validation of charts in real conditions before using them in production.<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/rz\/vt\/uk\/rzvtuk6q7b_rkeaxnyujzhmfjog.png\"><\/a><\/noindex><\/p>\n<p><\/p>\n<p><em>Chart Repositories in Different Environments<\/em><\/p>\n<p><\/p>\n<p>It is worth noting that when developers push to the dev branch, their chart version is automatically sent to the dev Chartmuseum. This way, all developers use a single dev repository, and it is crucial to specify your chart version carefully to avoid unintentionally using someone else's changes.<\/p>\n<p><\/p>\n<p>Moreover, our small Python script checks Kubernetes objects against the Kubernetes OpenAPI specifications using <noindex><a rel=\"nofollow\" href=\"https:\/\/kubeval.instrumenta.dev\/\">Kubeval<\/a><\/noindex>, before publishing them to Chartmuseum.<\/p>\n<p><\/p>\n<h3 id=\"obschee-opisanie-rabochego-processa-razrabotki-charta\">Overview of the chart development workflow<\/h3>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/yd\/cw\/qw\/ydcwqw-vqfitu5bj8ky4nh7xxea.png\"><\/a><\/noindex><\/p>\n<p><\/p>\n<ol>\n<li>Configuring pipeline tasks according to specifications <noindex><a rel=\"nofollow\" href=\"https:\/\/gazr.io\/\">gazr.io<\/a><\/noindex> for quality control (lint, unit-test).<\/li>\n<li>Sending the Docker image with Python tools that deploy our applications.<\/li>\n<li>Setting up the environment by branch name.<\/li>\n<li>Validating Kubernetes yaml files with Kubeval.<\/li>\n<li>Automatically incrementing the version of the chart and its parent charts (charts that depend on the changing chart).<\/li>\n<li>Sending the chart to Chartmuseum corresponding to its environment<\/li>\n<\/ol>\n<p><\/p>\n<h2 id=\"upravlenie-razlichiyami-v-klasterah\">Managing differences in clusters<\/h2>\n<p><\/p>\n<h3 id=\"federaciya-klasterov\">Cluster federation<\/h3>\n<p><\/p>\n<p>There was a time when we used <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/cluster-administration\/federation\/\">Kubernetes cluster federation<\/a><\/noindex>, where Kubernetes objects could be declared from a single API endpoint. But issues arose. For example, some Kubernetes objects could not be created at the federation endpoint, making it difficult to manage unioned objects and others for individual clusters.<\/p>\n<p><\/p>\n<p>To solve the issue, we began managing clusters independently, significantly simplifying the process (we used the first version of federation; something might have changed in the second).<\/p>\n<p><\/p>\n<h3 id=\"georaspredelennaya-platforma\">Geo-distributed platform<\/h3>\n<p><\/p>\n<p>Currently, our platform is distributed across 6 regions \u2014 3 locally and 3 in the cloud.<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/jq\/c1\/mr\/jqc1mru36g5byams4wddzducauw.png\"><\/a><\/noindex><br \/>\n<em>Distributed deployment<\/em><\/p>\n<p><\/p>\n<h3 id=\"globalnye-znacheniya-helm\">Global Helm values<\/h3>\n<p><\/p>\n<p>4 global Helm values allow defining differences between clusters. For all our charts, there are minimum default values.<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">global:\n  cloud: True\n  env: staging\n  region: us-central1\n  clusterName: staging-us-central1<\/code><\/pre>\n<p><\/p>\n<p><em>Global values<\/em><\/p>\n<p><\/p>\n<p>These values help define the context for our applications and are used for different tasks: monitoring, tracing, logging, making external calls, scaling, etc.<\/p>\n<p><\/p>\n<ul>\n<li>\"cloud\": we have a hybrid Kubernetes platform. For example, our API is deployed in GCP zones and in our data centers.<\/li>\n<li>\"env\": some values may change for non-production environments. For instance, resource definitions and autoscaling configurations.<\/li>\n<li>\"region\": this information helps determine the location of the cluster and can be used to identify the nearest endpoints for external services.<\/li>\n<li>\"clusterName\": if and when we want to define a value for an individual cluster.<\/li>\n<\/ul>\n<p><\/p>\n<p>Here is a specific example:<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">{{\n\/* Returns Horizontal Pod Autoscaler replicas for GraphQL*\/}}\n{{- define \"graphql.hpaReplicas\" -}}\n{{- if eq .Values.global.env \"prod\" }}\n{{- if eq .Values.global.region \"europe-west1\" }}\nminReplicas: 40\n{{- else }}\nminReplicas: 150\n{{- end }}\nmaxReplicas: 1400\n{{- else }}\nminReplicas: 4\nmaxReplicas: 20\n{{- end }}\n{{- end -}}<\/code><\/pre>\n<p><\/p>\n<p><em>Example of a Helm template<\/em><\/p>\n<p><\/p>\n<p>This logic is defined in a helper template to avoid cluttering Kubernetes YAML.<\/p>\n<p><\/p>\n<h3 id=\"obyavlenie-prilozheniya\">Application declaration<\/h3>\n<p><\/p>\n<p>Our deployment tools are based on several YAML files. Below is an example of how we declare a service and its scaling topology (number of replicas) in the cluster.<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">releases:\n  - foo.world\n\nfoo.world:                # Release name\n  services:               # List of dailymotion's apps\/projects\n    foobar:\n      chart_name: foo-foobar\n      repo: git@github.com:dailymotion\/foobar\n      contexts:\n        prod-europe-west1:\n          deployments:\n            - name: foo-bar-baz\n              replicas: 18\n            - name: another-deployment\n              replicas: 3<\/code><\/pre>\n<p><\/p>\n<p><em>Service definition<\/em><\/p>\n<p><\/p>\n<p>This is a diagram of all the steps that define our deployment workflow. The last step deploys the application simultaneously across multiple working clusters.<\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/habrastorage.org\/webt\/iu\/qh\/xk\/iuqhxki4nfqoi0mus258j0feylu.png\"><\/a><\/noindex><br \/>\n<em>Deployment steps in Jenkins<\/em><\/p>\n<p><\/p>\n<h3 id=\"a-sekrety\">And what about secrets?<\/h3>\n<p><\/p>\n<p>Regarding security, we track all secrets from various sources and store them in a unique repository <noindex><a rel=\"nofollow\" href=\"https:\/\/www.vaultproject.io\/\">Vault<\/a><\/noindex> in Paris.<\/p>\n<p><\/p>\n<p>Our deployment tools extract secret values from Vault, and when it\u2019s time to deploy, they insert them into Helm.<\/p>\n<p><\/p>\n<p>For this, we defined a mapping between secrets in Vault and the secrets required by our applications:<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">secrets:                                                                                                                                                                                                        \n     - secret_id: \"stack1-app1-password\"                                                                                                                                                                                  \n       contexts:                                                                                                                                                                                                   \n         - name: \"default\"                                                                                                                                                                                         \n           vaultPath: \"\\\/kv\\\/dev\\\/stack1\\\/app1\\\/test\"                                                                                                                                                               \n           vaultKey: \"password\"                                                                                                                                                                                    \n         - name: \"cluster1\"                                                                                                                                                                           \n           vaultPath: \"\\\/kv\\\/dev\\\/stack1\\\/app1\\\/test\"                                                                                                                                                               \n           vaultKey: \"password\"<\/code><\/pre>\n<p><\/p>\n<ul>\n<li>We have established general rules that must be followed when recording secrets in the Vault.<\/li>\n<li>If the secret pertains <strong>to a specific context or cluster<\/strong>, a specific entry needs to be added. (In this case, the context cluster1 has a distinct value for the secret stack-app1-password).<\/li>\n<li>Otherwise, the <strong>default<\/strong>.<\/li>\n<li>value is used. <strong>For each item in this list, a<\/strong> Kubernetes secret<\/li>\n<\/ul>\n<p><\/p>\n<pre><code class=\"plaintext\">is inserted as a key-value pair. Therefore, the secret template in our charts is very straightforward.<\/code><\/pre>\n<p><\/p>\n<h2 id=\"problemy-i-ogranicheniya\">Issues and Limitations<\/h2>\n<p><\/p>\n<h3 id=\"rabota-s-neskolkimi-repozitoriyami\">Working with Multiple Repositories<\/h3>\n<p><\/p>\n<p>Currently, we separate the development of charts and applications. This means that developers have to work in two git repositories: one for the application and another for defining its deployment in Kubernetes. Two git repositories mean two workflows, and it can be easy for newcomers to get confused.<\/p>\n<p><\/p>\n<h3 id=\"upravlyat-obobschennymi-chartami-hlopotno\">Managing generic charts is cumbersome.<\/h3>\n<p><\/p>\n<p>As mentioned, generic charts are very convenient for defining dependencies and quickly deploying multiple applications. However, we use <code>--reuse-values<\/code>, to avoid passing all values each time we deploy an application that is part of this generic chart.<\/p>\n<p><\/p>\n<p>In the continuous delivery workflow, we have only two values that change regularly: the number of replicas and the image tag (version). Other, more stable values are changed manually, which is quite complex. Moreover, a single error in deploying a generalized chart can lead to serious failures, as we have seen from our own experience.<\/p>\n<p><\/p>\n<h3 id=\"obnovlenie-neskolkih-faylov-konfiguracii\">Updating several configuration files<\/h3>\n<p><\/p>\n<p>When a developer adds a new application, they have to modify several files: the application declaration, the secrets list, and add the application to dependencies if it is included in the generalized chart.<\/p>\n<p><\/p>\n<h3 id=\"razresheniya-jenkins-slishkom-rasshireny-v-vault\">Jenkins permissions are overly broad in Vault<\/h3>\n<p><\/p>\n<p>Currently, we have one <noindex><a rel=\"nofollow\" href=\"https:\/\/www.vaultproject.io\/docs\/auth\/approle.html\">AppRole<\/a><\/noindex>, which reads all secrets from Vault.<\/p>\n<p><\/p>\n<h3 id=\"process-otkata-ne-avtomatizirovan\">The rollback process is not automated<\/h3>\n<p><\/p>\n<p>To roll back, a command needs to be executed on several clusters, which is prone to errors. We perform this operation manually to ensure the correct version identifier is specified.<\/p>\n<p><\/p>\n<h2 id=\"my-dvizhemsya-v-storonu-gitops\">We are moving towards GitOps<\/h2>\n<p><\/p>\n<h3 id=\"nasha-cel\">Our goal<\/h3>\n<p><\/p>\n<p>We want to return the chart to the repository of the application it deploys.<\/p>\n<p><\/p>\n<p>The workflow will be the same as for development. For example, when a branch is pushed to master, deployment will start automatically. The main difference between this approach and the current workflow will be that <strong>everything will be managed in git<\/strong> (the application itself and the method of its deployment in Kubernetes).<\/p>\n<p><\/p>\n<p>There are several advantages:<\/p>\n<p><\/p>\n<ul>\n<li>Much <strong>clearer<\/strong> for the developer. It's easier to learn how to apply changes in the local chart.<\/li>\n<li>The service deployment definition can be specified <strong>right where the code<\/strong> of the service is.<\/li>\n<li><strong>Managing the deletion of generalized charts<\/strong>. The service will have its own Helm release. This will allow for managing the application lifecycle (rollback, upgrade) at the finest level, without affecting other services.<\/li>\n<li><strong>The advantages of git<\/strong> for managing charts: undo changes, audit log, etc. If you need to revert a chart change, this can be done using git. The deployment starts automatically.<\/li>\n<li>One can think about enhancing the development workflow using tools like <strong>Skaffold<\/strong>, which allows developers to test changes in an environment similar to production.<\/li>\n<\/ul>\n<p><\/p>\n<h3 id=\"dvuhetapnaya-migraciya\">Two-stage migration<\/h3>\n<p><\/p>\n<p>Our developers have been using this workflow for 2 years, so we need a seamless migration. That\u2019s why we decided to add an intermediate step on the way to our goal.<br \/>\nThe first step is simple:<\/p>\n<p><\/p>\n<ul>\n<li>We maintain a similar structure for application deployment configuration, but in a single object named DailymotionRelease.<\/li>\n<\/ul>\n<p><\/p>\n<pre><code class=\"plaintext\">apiVersion: \"v1\"\nkind: \"DailymotionRelease\"\nmetadata:\n  name: \"app1.ns1\"\n  environment: \"dev\"\n  branch: \"mybranch\"\nspec:\n  slack_channel: \"#admin\"\n  chart_name: \"app1\"\n  scaling:\n    - context: \"dev-us-central1-0\"\n      replicas:\n        - name: \"hermes\"\n          count: 2\n    - context: \"dev-europe-west1-0\"\n      replicas:\n        - name: \"app1-deploy\"\n          count: 2\n  secrets:\n    - secret_id: \"app1\"\n      contexts:\n        - name: \"default\"\n          vaultPath: \"\\\/kv\\\/dev\\\/ns1\\\/app1\\\/test\"\n          vaultKey: \"password\"\n        - name: \"dev-europe-west1-0\"\n          vaultPath: \"\\\/kv\\\/dev\\\/ns1\\\/app1\\\/test\"\n          vaultKey: \"password\"<\/code><\/pre>\n<p><\/p>\n<ul>\n<li>1 release per application (no generic charts).<\/li>\n<li>Charts in the application's git repository.<\/li>\n<\/ul>\n<p><\/p>\n<p>We spoke with all developers, so the migration process has already begun. The first stage is still monitored using the CI platform. Soon I will write another post about the second stage: how we transitioned to the GitOps workflow with <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/weaveworks\/flux\">Flux<\/a><\/noindex>. I will share how we set everything up and what challenges we faced (multiple repositories, secrets, etc.). Stay tuned for updates.<\/p>\n<p><\/p>\n<p>Here we tried to outline our progress in the application deployment workflow over the last few years, which has led to thoughts about the GitOps approach. We have not yet reached our goal and will report on the results, but we are now convinced that we made the right decision to simplify everything and bring it closer to developers' habits.<\/p>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/southbridge\/blog\/458934\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041a\u0430\u043a Dailymotion \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 Kubernetes: \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u041c\u044b \u0432 Dailymotion \u043d\u0430\u0447\u0430\u043b\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c Kubernetes \u0432 \u043f\u0440\u043e\u0434\u0430\u043a\u0448\u0435\u043d\u0435 3 \u0433\u043e\u0434\u0430 \u043d\u0430\u0437\u0430\u0434. \u041d\u043e \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u0442\u044c \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f \u043d\u0430 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430\u0445 \u0442\u043e \u0435\u0449\u0435 \u0443\u0434\u043e\u0432\u043e\u043b\u044c\u0441\u0442\u0432\u0438\u0435, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u0432 \u043f\u043e\u0441\u043b\u0435\u0434\u043d\u0438\u0435 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u043b\u0435\u0442 \u043c\u044b \u0441\u0442\u0430\u0440\u0430\u043b\u0438\u0441\u044c \u0443\u043b\u0443\u0447\u0448\u0438\u0442\u044c \u043d\u0430\u0448\u0438 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u044b \u0438 \u0440\u0430\u0431\u043e\u0447\u0438\u0435 \u043f\u0440\u043e\u0446\u0435\u0441\u0441\u044b. \u0421 \u0447\u0435\u0433\u043e \u043d\u0430\u0447\u0430\u043b\u043e\u0441\u044c \u0417\u0434\u0435\u0441\u044c \u043c\u044b \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0435\u043c, \u043a\u0430\u043a \u043c\u044b \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u0435\u043c \u043d\u0430\u0448\u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f \u043d\u0430 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430\u0445 Kubernetes \u043f\u043e [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-35955","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.1.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041a\u0430\u043a Dailymotion \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 Kubernetes: \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u041c\u044b \u0432 Dailymotion \u043d\u0430\u0447\u0430\u043b\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c Kubernetes \u0432.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razvertyvanie-prilozhenij-na-neskolkih-klasterah-kubernetes-s-helm\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.1.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u0420\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u043d\u0430 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430\u0445 Kubernetes \u0441 Helm | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041a\u0430\u043a Dailymotion \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 Kubernetes: \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u041c\u044b \u0432 Dailymotion \u043d\u0430\u0447\u0430\u043b\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c Kubernetes \u0432.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razvertyvanie-prilozhenij-na-neskolkih-klasterah-kubernetes-s-helm\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:08:26+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:08:26+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Deploying Applications Across Multiple Kubernetes Clusters with Helm | ProHoster","description":"How Dailymotion Uses Kubernetes: Application Deployment We at Dailymotion started using Kubernetes in.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razvertyvanie-prilozhenij-na-neskolkih-klasterah-kubernetes-s-helm","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0420\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u043d\u0430 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430\u0445 Kubernetes \u0441 Helm | ProHoster","og:description":"\u041a\u0430\u043a Dailymotion \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 Kubernetes: \u0440\u0430\u0437\u0432\u0435\u0440\u0442\u044b\u0432\u0430\u043d\u0438\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u041c\u044b \u0432 Dailymotion \u043d\u0430\u0447\u0430\u043b\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c Kubernetes \u0432.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/razvertyvanie-prilozhenij-na-neskolkih-klasterah-kubernetes-s-helm","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:08:26+00:00","article:modified_time":"2019-10-31T19:08:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"35955","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 01:25:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:53:31","updated":"2026-01-22 01:25:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35955","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=35955"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/35955\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=35955"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=35955"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=35955"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}