{"id":53120,"date":"2019-11-24T00:00:00","date_gmt":"2019-11-23T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/3-way-merge-v-werf-deploj-v-kubernetes-s-helm-na-steroidah"},"modified":"2020-02-18T14:00:59","modified_gmt":"2020-02-18T11:00:59","slug":"3-way-merge-v-werf-deploj-v-kubernetes-s-helm-na-steroidah","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/3-way-merge-v-werf-deploj-v-kubernetes-s-helm-na-steroidah","title":{"rendered":"3-way merge in werf: deploying in Kubernetes with Helm \"on steroids\"","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>What we (and not just us) have been waiting for has happened: <noindex><a rel=\"nofollow\" href=\"https:\/\/werf.io\/\">werf<\/a><\/noindex>, our Open Source utility for building applications and deploying them to Kubernetes now supports applying changes using 3-way merge patches! In addition, there is now the ability to adopt existing K8s resources into Helm releases without recreating these resources.<\/p>\n<p><img decoding=\"async\" alt=\"3-way merge in werf: deploying in Kubernetes with Helm &quot;on steroids&quot;\" src=\"\/wp-content\/uploads\/2019\/11\/e5be3544c201f7b2c97fba216a49506d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIn short, we set <code>WERF_THREE_WAY_MERGE=enabled<\/code> \u2014 and get a deployment 'as in <code>kubectl apply<\/code>', compatible with existing Helm 2 installations and even a bit more.<\/p>\n<p>But let's start with the theory: what exactly are 3-way merge patches, how did people come to the approach of generating them, and why are they important in CI\/CD processes with Kubernetes-based infrastructure? After that, we will see what 3-way merge looks like in werf, what modes are used by default, and how to manage them.<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>What is a 3-way merge patch?<\/h2>\n<p>\nSo, let's start with the task of deploying resources described in YAML manifests to Kubernetes.<\/p>\n<p>To work with resources, Kubernetes API offers the following basic operations: create, patch, replace, and delete. The expectation is that these operations should be used to construct a convenient continuous deployment of resources into the cluster. How?<\/p>\n<h3>Imperative kubectl commands<\/h3>\n<p>\nThe first approach to managing objects in Kubernetes is the use of imperative kubectl commands to create, modify, and delete these objects. Simply put:<\/p>\n<ul>\n<li> with the command <code>kubectl run<\/code> can launch a Deployment or Job:\n<pre><code class=\"bash\">kubectl run --generator=deployment\/apps.v1 DEPLOYMENT_NAME --image=IMAGE<\/code><\/pre>\n<\/li>\n<li> with the command <code>kubectl scale<\/code> \u2014 change the number of replicas:\n<pre><code class=\"bash\">kubectl scale --replicas=3 deployment\/mysql<\/code><\/pre>\n<\/li>\n<li>etc.<\/li>\n<\/ul>\n<p>\nThis approach may seem convenient at first glance. However, there are problems: <\/p>\n<ol>\n<li> It is difficult to <b>automate<\/b>.<\/li>\n<li> How <b>reflect the configuration<\/b> in Git? How to review changes occurring with the cluster?<\/li>\n<li> How to ensure <b>reproducibility<\/b> of the configuration upon restart?<\/li>\n<li>\u2026<\/li>\n<\/ol>\n<p>\nIt is clear that this approach does not work well with storing alongside the application code and infrastructure as code (IaC; or even <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/458878\/\">GitOps<\/a><\/noindex> as a more modern alternative gaining popularity in the Kubernetes ecosystem). Therefore, these commands in kubectl did not receive further development.<\/p>\n<h3>Operations create, get, replace, and delete<\/h3>\n<p>\nWith initial <b>creation<\/b> it's straightforward: we send the manifest to the <code>create<\/code> kube API operation and the resource is created. The YAML representation of the manifest can be stored in Git, and to create, we use the command <code>kubectl create -f manifest.yaml<\/code>.<\/p>\n<p>C <b>deletion<\/b> is also simple: we plug the same <code>manifest.yaml<\/code> from Git into the command <code>kubectl delete -f manifest.yaml<\/code>.<\/p>\n<p>The operation <b><code>replace<\/code><\/b> allows you to completely replace the resource configuration with a new one without recreating the resource. This means that before making a change to the resource, it makes sense to request the current version with an operation <code>get<\/code>, modify it, and then update it with an operation <code>replace<\/code>. The kube apiserver has built-in <noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Optimistic_concurrency_control\">optimistic locking<\/a><\/noindex> and if the object has changed after the operation <code>get<\/code> , the operation <code>replace<\/code> will not succeed.<\/p>\n<p>To store the configuration in Git and update using replace, you need to perform an operation <code>get<\/code>, merge the config from Git with what we received, and execute <code>replace<\/code>. By default, kubectl only allows the use of the command <code>kubectl replace -f manifest.yaml<\/code>, where <code>manifest.yaml<\/code> \u2014 a fully prepared (in our case \u2014 merged) manifest that needs to be deployed. Thus, the user needs to implement the merging of manifests, which is a non-trivial task\u2026<\/p>\n<p>It should also be noted that although <code>manifest.yaml<\/code> it is stored in Git, we cannot know in advance whether to create the object or update it \u2014 this must be done by user software.<\/p>\n<p>Total: <b>Can we build a continuous rollout<\/b> only using create, replace, and delete, ensuring the storage of infrastructure configuration in Git along with the code and convenient CI\/CD?<\/p>\n<p>In principle, we can\u2026 For this, <b>it will be necessary to implement the merge operation<\/b> of manifests and some wrapper that:<\/p>\n<ul>\n<li> checks for the existence of the object in the cluster,<\/li>\n<li> performs the initial creation of the resource,<\/li>\n<li> updates or deletes it.<\/li>\n<\/ul>\n<p>\nWhen updating, it should be taken into account that <i>the resource may have changed<\/i> since the last <code>get<\/code> and automatically handle the optimistic locking case \u2014 making retries for the update.<\/p>\n<p>However, why reinvent the wheel when kube-apiserver offers another way to update resources: the operation <code>patch<\/code>, which relieves the user of some described problems?<\/p>\n<h3>Patch<\/h3>\n<p>\nNow we\u2019ve come to patches.<\/p>\n<p>Patches are the primary way to apply changes to existing objects in Kubernetes. The operation <code>patch<\/code> works in such a way that:<\/p>\n<ul>\n<li> the kube-apiserver user needs to send a patch in JSON format and specify the object,<\/li>\n<li> and the apiserver will figure out the current state of the object and adjust it to the required state.<\/li>\n<\/ul>\n<p>\nOptimistic locking is not required in this case. This operation is more declarative compared to replace, although it may initially seem otherwise.<\/p>\n<p>Thus:<\/p>\n<ul>\n<li> using the operation <code>create<\/code> we create the object from the manifest in Git,<\/li>\n<li> using <code>delete<\/code> \u2014 delete it if the object is no longer needed,<\/li>\n<li> using <code>patch<\/code> \u2014 modify the object, bringing it to the state described in Git.<\/li>\n<\/ul>\n<p>\nHowever, to do this, it is necessary to create <i>the correct patch<\/i>!<\/p>\n<h3>How patches work in Helm 2: 2-way merge<\/h3>\n<p>\nUpon the initial installation of the Helm release, an operation is executed <code>create<\/code> for the chart resources.<\/p>\n<p>When updating the Helm release for each resource:<\/p>\n<ul>\n<li> it calculates the patch between the resource version from the previous chart and the current version of the chart,<\/li>\n<li> and applies this patch.<\/li>\n<\/ul>\n<p>\nThis patch will be called <b>2-way merge patch<\/b>, because two manifests are involved in its creation:<\/p>\n<ul>\n<li> the resource manifest from the previous release,<\/li>\n<li> the resource manifest from the current resource.<\/li>\n<\/ul>\n<p>\nDuring deletion, the operation <code>delete<\/code> is invoked in kube apiserver for resources that were declared in the previous release but not declared in the current one.<\/p>\n<p>The 2-way merge patch approach has a problem: it leads to <b>a desynchronization between the actual resource state in the cluster and the manifest in Git.<\/b>.<\/p>\n<h3>Illustration of the problem with an example<\/h3>\n<p><\/p>\n<ul>\n<li> In Git, the chart contains a manifest in which the field <code>image<\/code> in the Deployment has the value <code>ubuntu:18.04<\/code>.<\/li>\n<li> The user has changed the value of this field to <code>kubectl edit<\/code> ubuntu:19.04 <code>Upon redeploying the Helm chart,<\/code>.<\/li>\n<li> no patch is generated <i>, because the field<\/i>in the previous release version and the current chart is the same. <code>image<\/code> After the redeployment,<\/li>\n<li> it remains <code>image<\/code> , even though the chart specifies <code>Upon redeploying the Helm chart,<\/code>We have encountered desynchronization and lost declarative control. <code>ubuntu:18.04<\/code>.<\/li>\n<\/ul>\n<p>\nWhat is a synchronized resource?<\/p>\n<h3>It is impossible to achieve complete<\/h3>\n<p>\nGenerally speaking, <i>correspondence between the resource manifest in the running cluster and the manifest from Git. This is because the real manifest may have operational annotations\/labels, additional containers, and other data that dynamic controllers add and remove from resources. We cannot and do not want to keep this data in Git. However, we want the fields we explicitly specified in Git to take the corresponding values upon deployment.<\/i> Thus, a general<\/p>\n<p>rule for a synchronized resource <b>: at deployment, one can only change or remove the fields that are explicitly mentioned in the Git manifest (or were mentioned in the previous version and are now removed).<\/b>3-way merge patch<\/p>\n<h3>: generates a patch between the last applied version of the manifest from Git and the target version of the manifest from Git, taking into account the current version of the manifest from the running cluster. The resulting patch must correspond to the rule of a synchronized resource:<\/h3>\n<p>\nThe main idea <noindex><a rel=\"nofollow\" href=\"https:\/\/en.wikipedia.org\/wiki\/Merge_(version_control)#Three-way_merge\">: generates a patch between the last applied version of the manifest from Git and the target version of the manifest from Git, taking into account the current version of the manifest from the running cluster. The resulting patch must correspond to the rule of a synchronized resource:<\/a><\/noindex>new fields added to the target version are added via the patch;<\/p>\n<ul>\n<li> New fields added to the target version are included through a patch;<\/li>\n<li> Fields that existed in the last applied version and do not exist in the target will be reset using a patch;<\/li>\n<li> Fields in the current version of the object that differ from the target version of the manifest will be updated using a patch.<\/li>\n<\/ul>\n<p>\nThis is exactly how patches are generated. <code>kubectl apply<\/code>:<\/p>\n<ul>\n<li> The last applied version of the manifest is saved in the annotation of the object itself, <\/li>\n<li> the target is taken from the specified YAML file,<\/li>\n<li> the current is from the running cluster.<\/li>\n<\/ul>\n<p>\nNow that we've covered the theory, it's time to explain what we've done in werf.<\/p>\n<h2>Applying changes in werf<\/h2>\n<p>\nPreviously, werf, like Helm 2, used 2-way-merge patches.<\/p>\n<h3>Repair patch<\/h3>\n<p>\nTo transition to a new type of patches \u2014 3-way-merge \u2014 the first step we introduced is the so-called <b>repair patches.<\/b>.<\/p>\n<p>During deployment, a standard 2-way-merge patch is used, but werf also generates a patch that synchronizes the actual state of the resource with what is written in Git (this patch is created using the same rules for the synchronized resource described above).<\/p>\n<p>In case of desynchronization, at the end of the deployment, the user receives a WARNING with a corresponding message and a patch that needs to be applied to bring the resource to a synchronized state. This patch is also recorded in a special annotation <code>werf.io\/repair-patch<\/code>. It is assumed that the user will manually <b>apply this patch themselves: werf will not apply it in principle.<\/b> The generation of repair patches is a temporary measure that allows testing the creation of patches based on the 3-way-merge principle, but does not apply these patches automatically. Currently, this mode of operation is enabled by default.<\/p>\n<p>3-way-merge patch only for new releases<\/p>\n<h3>Starting from December 1, 2019, beta and alpha versions of werf will begin<\/h3>\n<p>\nto use full 3-way-merge patches for applying changes only for new Helm releases being rolled out through werf. Existing releases will continue to use the approach with 2-way-merge + repair patches. <b>default<\/b> This mode of operation can be explicitly enabled by setting<\/p>\n<p>WERF_THREE_WAY_MERGE_MODE=onlyNewReleases <code>: this feature has been appearing in werf over several releases: in the alpha channel, it became stable starting from version<\/code> already downloaded now.<\/p>\n<p><i><b>Note<\/b>v1.0.5-alpha.19 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/werf\/releases\/tag\/v1.0.5-alpha.19\">, and in the beta channel \u2014 from<\/a><\/noindex>3-way-merge patch for all releases. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/werf\/releases\/tag\/v1.0.4-beta.20\">v1.0.4-beta.20<\/a><\/noindex>.<\/i><\/p>\n<h3>3-way-merge patch for all releases<\/h3>\n<p>\nStarting December 15, 2019, beta and alpha versions of werf will by default use full 3-way-merge patches to apply changes for all releases.<\/p>\n<p>WERF_THREE_WAY_MERGE_MODE=onlyNewReleases <code>WERF_THREE_WAY_MERGE_MODE=enabled<\/code> already downloaded now.<\/p>\n<h3>How to handle resource autoscaling?<\/h3>\n<p>\nIn Kubernetes, there are 2 types of autoscaling: HPA (Horizontal Pod Autoscaler) and VPA (Vertical Pod Autoscaler).<\/p>\n<p>Horizontal automatically selects the number of replicas, while vertical sets the amount of resources. Both the number of replicas and resource requirements are specified in the resource manifest (see <code>spec.replicas<\/code> or <code>spec.containers[].resources.limits.cpu<\/code>, <code>spec.containers[].resources.limits.memory<\/code> and <noindex><a rel=\"nofollow\" href=\"https:\/\/kubernetes.io\/docs\/concepts\/configuration\/manage-compute-resources-container\/#resource-requests-and-limits-of-pod-and-container\">others<\/a><\/noindex>).<\/p>\n<p>Problem: if the user configures a resource in the chart with specific values for resources or replicas, and if auto-scalers are enabled for that resource, then on each deployment, werf will reset these values to what is recorded in the chart manifest.<\/p>\n<p>There are two solutions to the problem. First, it is best to refrain from explicitly specifying autoscalable values in the chart manifest. However, if this option is not suitable for some reason (for example, because it's convenient to set initial limits for resources and the number of replicas in the chart), werf offers the following annotations:<\/p>\n<ul>\n<li> <code>werf.io\/set-replicas-only-on-creation=true<\/code><\/li>\n<li> <code>werf.io\/set-resources-only-on-creation=true<\/code><\/li>\n<\/ul>\n<p>\nWith such an annotation, werf will not reset the corresponding values on each deployment but will only set them at the initial creation of the resource.<\/p>\n<p>For more details \u2014 see the project documentation on <noindex><a rel=\"nofollow\" href=\"https:\/\/werf.io\/documentation\/reference\/deploy_process\/resources_update_methods_and_adoption.html#deal-with-hpa\">HPA<\/a><\/noindex> and <noindex><a rel=\"nofollow\" href=\"https:\/\/werf.io\/documentation\/reference\/deploy_process\/resources_update_methods_and_adoption.html#deal-with-vpa\">VPA<\/a><\/noindex>.<\/p>\n<h3>Disable 3-way-merge patch usage<\/h3>\n<p>\nUsers can temporarily disable the use of new patches in werf using the environment variable <code>WERF_THREE_WAY_MERGE_MODE=disabled<\/code>. However, starting <b>from March 1, 2020, this restriction will stop working<\/b> and only the use of 3-way-merge patches will be possible.<\/p>\n<h2>Resource adoption in werf<\/h2>\n<p>\nMastering the application of changes via 3-way-merge patches allowed us to immediately implement a feature like adopting existing resources in the cluster to a Helm release.<\/p>\n<p>Helm 2 has a problem: you cannot add a resource to the chart manifests that already exists in the cluster without recreating that resource from scratch (see <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/helm\/helm\/issues\/6031#issuecomment-531579500\">#6031<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/helm\/helm\/issues\/3275\">#3275<\/a><\/noindex>). We taught werf to accept existing resources into the release. To do this, you need to annotate the current version of the resource from the running cluster (for example, using <code>kubectl edit<\/code>):<\/p>\n<pre><code class=\"plaintext\">\"werf.io\/allow-adoption-by-release\": RELEASE_NAME<\/code><\/pre>\n<p>\nNow the resource needs to be described in the chart, and during the next deployment with werf of the release with the corresponding name, the existing resource will be included in this release and remain under its management. Moreover, during the resource's adoption into the release, werf will bring the current state of the resource from the working cluster to the state described in the chart, using the same 3-way-merge patches and the synchronized resource rule.<\/p>\n<p><i><b>Note<\/b>: configuration <code>WERF_THREE_WAY_MERGE_MODE<\/code> has no effect on resource adoption \u2014 in the case of adoption, a 3-way-merge patch is always used.<\/i><\/p>\n<p>Details can be found in <noindex><a rel=\"nofollow\" href=\"https:\/\/werf.io\/documentation\/reference\/deploy_process\/resources_update_methods_and_adoption.html#resources-adoption\">the documentation<\/a><\/noindex>.<\/p>\n<h2>Conclusions and further plans<\/h2>\n<p>\nI hope that after reading this article, it's clearer what 3-way-merge patches are and why we arrived at them. From a practical perspective, the implementation of them in the werf project is another step towards improving Helm-like deployments. Now we can forget about configuration synchronization issues that often arose when using Helm 2. At the same time, a new useful feature for adopting already pulled Kubernetes resources into Helm releases has been added.<\/p>\n<p>In Helm-like deployments, there are still some problems and difficulties, such as the use of Go templates, and we will continue to address them.<\/p>\n<p>Information about resource update methods and adoption can also be found on <noindex><a rel=\"nofollow\" href=\"https:\/\/werf.io\/documentation\/reference\/deploy_process\/resources_update_methods_and_adoption.html\">this documentation page<\/a><\/noindex>.<\/p>\n<h3>Helm 3<\/h3>\n<p>\nDeserving separate mention is the <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/news\/t\/475722\/\">recently released<\/a><\/noindex> literally just days ago new major version of Helm \u2014 v3, \u2014 which also uses 3-way-merge patches and eliminates Tiller. The new version of Helm requires <noindex><a rel=\"nofollow\" href=\"https:\/\/helm.sh\/docs\/topics\/v2_v3_migration\/\">migration<\/a><\/noindex> existing installations to convert them to the new release storage format.<\/p>\n<p>Werf has already eliminated the use of Tiller, switched to 3-way-merge, and added <noindex><a rel=\"nofollow\" href=\"https:\/\/werf.io\/documentation\/reference\/deploy_process\/differences_with_helm.html\">and much more<\/a><\/noindex>, while remaining compatible with existing Helm 2 installations (no migration scripts need to be executed). Therefore, as long as werf is not switched to Helm 3, werf users do not lose the main advantages of Helm 3 over Helm 2 (they are also present in werf).<\/p>\n<p>However, werf's switch to the Helm 3 codebase is inevitable and will occur in the near future. It is expected to be in werf 1.1 or werf 1.2 (currently, the main version of werf is 1.0; for more information about the versioning system of werf, see <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/werf#backward-compatibility-promise\">here<\/a><\/noindex>). By that time, Helm 3 will have stabilized.<\/p>\n<h2>P.S.<\/h2>\n<p>\nAlso read in our blog:<\/p>\n<ul>\n<li> Cycle of release notes for new features in werf:\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/468049\/\">Using werf for deploying complex Helm charts<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/465131\/\">Support for monorepo and multirepo in werf and its connection to Docker Registry<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/463613\/\">You can now build Docker images in werf using a regular Dockerfile<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/460351\/\">werf is our tool for CI\/CD in Kubernetes (overview and video report)<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/469541\/\">Building and deploying identical microservices with werf and GitLab CI<\/a><\/noindex>\u00bb;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/453734\/\">Introducing Helm 3<\/a><\/noindex>\u00bb.<\/li>\n<\/ul>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/476646\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c \u0442\u043e, \u0447\u0435\u0433\u043e \u043c\u044b (\u0438 \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u043c\u044b) \u0434\u043e\u043b\u0433\u043e \u0436\u0434\u0430\u043b\u0438: werf, \u043d\u0430\u0448\u0430 Open Source-\u0443\u0442\u0438\u043b\u0438\u0442\u0430 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u0438 \u0438\u0445 \u0434\u043e\u0441\u0442\u0430\u0432\u043a\u0438 \u0432 Kubernetes, \u0442\u0435\u043f\u0435\u0440\u044c \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u0442 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0435 \u0438\u0437\u043c\u0435\u043d\u0435\u043d\u0438\u0439 \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e 3-way-merge-\u043f\u0430\u0442\u0447\u0435\u0439! \u0412 \u0434\u043e\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u0435 \u043a \u044d\u0442\u043e\u043c\u0443, \u043f\u043e\u044f\u0432\u0438\u043b\u0430\u0441\u044c \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u044c adoption\u2019\u0430 \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 K8s-\u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432 \u0432 Helm-\u0440\u0435\u043b\u0438\u0437\u044b \u0431\u0435\u0437 \u043f\u0435\u0440\u0435\u0441\u043e\u0437\u0434\u0430\u043d\u0438\u044f \u044d\u0442\u0438\u0445 \u0440\u0435\u0441\u0443\u0440\u0441\u043e\u0432. \u0415\u0441\u043b\u0438 \u0441\u043e\u0432\u0441\u0435\u043c \u043a\u043e\u0440\u043e\u0442\u043a\u043e, \u0442\u043e \u0441\u0442\u0430\u0432\u0438\u043c WERF_THREE_WAY_MERGE=enabled \u2014 \u043f\u043e\u043b\u0443\u0447\u0430\u0435\u043c \u0434\u0435\u043f\u043b\u043e\u0439 \u00ab\u043a\u0430\u043a \u0432 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":53121,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-53120","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0421\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c \u0442\u043e, \u0447\u0435\u0433\u043e \u043c\u044b (\u0438 \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u043c\u044b) \u0434\u043e\u043b\u0433\u043e \u0436\u0434\u0430\u043b\u0438: werf, \u043d\u0430\u0448\u0430 Open Source-\u0443\u0442\u0438\u043b\u0438\u0442\u0430 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u0438 \u0438\u0445 \u0434\u043e\u0441\u0442\u0430\u0432\u043a\u0438 \u0432 Kubernetes, \u0442\u0435\u043f\u0435\u0440\u044c \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u0442.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/3-way-merge-v-werf-deploj-v-kubernetes-s-helm-na-steroidah\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd473-way merge \u0432 werf: \u0434\u0435\u043f\u043b\u043e\u0439 \u0432 Kubernetes \u0441 Helm \u00ab\u043d\u0430 \u0441\u0442\u0435\u0440\u043e\u0438\u0434\u0430\u0445\u00bb | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0421\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c \u0442\u043e, \u0447\u0435\u0433\u043e \u043c\u044b (\u0438 \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u043c\u044b) \u0434\u043e\u043b\u0433\u043e \u0436\u0434\u0430\u043b\u0438: werf, \u043d\u0430\u0448\u0430 Open Source-\u0443\u0442\u0438\u043b\u0438\u0442\u0430 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u0438 \u0438\u0445 \u0434\u043e\u0441\u0442\u0430\u0432\u043a\u0438 \u0432 Kubernetes, \u0442\u0435\u043f\u0435\u0440\u044c \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u0442.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/3-way-merge-v-werf-deploj-v-kubernetes-s-helm-na-steroidah\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-11-23T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T11:00:59+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd473-way merge in werf: deploying to Kubernetes with Helm \"on steroids\" | ProHoster","description":"The moment we've all been waiting for has finally arrived: werf, our Open Source tool for building applications and delivering them to Kubernetes, now supports.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/3-way-merge-v-werf-deploj-v-kubernetes-s-helm-na-steroidah","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd473-way merge \u0432 werf: \u0434\u0435\u043f\u043b\u043e\u0439 \u0432 Kubernetes \u0441 Helm \u00ab\u043d\u0430 \u0441\u0442\u0435\u0440\u043e\u0438\u0434\u0430\u0445\u00bb | ProHoster","og:description":"\u0421\u043b\u0443\u0447\u0438\u043b\u043e\u0441\u044c \u0442\u043e, \u0447\u0435\u0433\u043e \u043c\u044b (\u0438 \u043d\u0435 \u0442\u043e\u043b\u044c\u043a\u043e \u043c\u044b) \u0434\u043e\u043b\u0433\u043e \u0436\u0434\u0430\u043b\u0438: werf, \u043d\u0430\u0448\u0430 Open Source-\u0443\u0442\u0438\u043b\u0438\u0442\u0430 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439 \u0438 \u0438\u0445 \u0434\u043e\u0441\u0442\u0430\u0432\u043a\u0438 \u0432 Kubernetes, \u0442\u0435\u043f\u0435\u0440\u044c \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0435\u0442.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/3-way-merge-v-werf-deploj-v-kubernetes-s-helm-na-steroidah","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-11-23T21:00:00+00:00","article:modified_time":"2020-02-18T11:00:59+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"53120","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 06:11:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 20:31:28","updated":"2026-01-24 06:11:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/53120","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=53120"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/53120\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/53121"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=53120"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=53120"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=53120"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}