🥇3-way merge in werf: deploying to Kubernetes with Helm "on steroids"

What we (and not just us) have been waiting for has happened: werf, our Open Source utility for building applications and deploying them to Kubernetes now supports applying changes using 3-way merge patches! In addition, there is now the ability to adopt existing K8s resources into Helm releases without recreating these resources.

In short, we set WERF_THREE_WAY_MERGE=enabled — and get a deployment 'as in kubectl apply', compatible with existing Helm 2 installations and even a bit more.

But let's start with the theory: what exactly are 3-way merge patches, how did people come to the approach of generating them, and why are they important in CI/CD processes with Kubernetes-based infrastructure? After that, we will see what 3-way merge looks like in werf, what modes are used by default, and how to manage them.

What is a 3-way merge patch?

So, let's start with the task of deploying resources described in YAML manifests to Kubernetes.

To work with resources, Kubernetes API offers the following basic operations: create, patch, replace, and delete. The expectation is that these operations should be used to construct a convenient continuous deployment of resources into the cluster. How?

Imperative kubectl commands

The first approach to managing objects in Kubernetes is the use of imperative kubectl commands to create, modify, and delete these objects. Simply put:

with the command kubectl run can launch a Deployment or Job:

kubectl run --generator=deployment/apps.v1 DEPLOYMENT_NAME --image=IMAGE

with the command kubectl scale — change the number of replicas:
```
kubectl scale --replicas=3 deployment/mysql
```
etc.

This approach may seem convenient at first glance. However, there are problems:

It is difficult to automate.
How reflect the configuration in Git? How to review changes occurring with the cluster?
How to ensure reproducibility of the configuration upon restart?
…

It is clear that this approach does not work well with storing alongside the application code and infrastructure as code (IaC; or even GitOps as a more modern alternative gaining popularity in the Kubernetes ecosystem). Therefore, these commands in kubectl did not receive further development.

Operations create, get, replace, and delete

With initial creation it's straightforward: we send the manifest to the create kube API operation and the resource is created. The YAML representation of the manifest can be stored in Git, and to create, we use the command kubectl create -f manifest.yaml.

C deletion is also simple: we plug the same manifest.yaml from Git into the command kubectl delete -f manifest.yaml.

The operation replace allows you to completely replace the resource configuration with a new one without recreating the resource. This means that before making a change to the resource, it makes sense to request the current version with an operation get, modify it, and then update it with an operation replace. The kube apiserver has built-in optimistic locking and if the object has changed after the operation get , the operation replace will not succeed.

To store the configuration in Git and update using replace, you need to perform an operation get, merge the config from Git with what we received, and execute replace. By default, kubectl only allows the use of the command kubectl replace -f manifest.yaml, where manifest.yaml — a fully prepared (in our case — merged) manifest that needs to be deployed. Thus, the user needs to implement the merging of manifests, which is a non-trivial task…

It should also be noted that although manifest.yaml it is stored in Git, we cannot know in advance whether to create the object or update it — this must be done by user software.

Total: Can we build a continuous rollout only using create, replace, and delete, ensuring the storage of infrastructure configuration in Git along with the code and convenient CI/CD?

In principle, we can… For this, it will be necessary to implement the merge operation of manifests and some wrapper that:

checks for the existence of the object in the cluster,
performs the initial creation of the resource,
updates or deletes it.

When updating, it should be taken into account that the resource may have changed since the last get and automatically handle the optimistic locking case — making retries for the update.

However, why reinvent the wheel when kube-apiserver offers another way to update resources: the operation patch, which relieves the user of some described problems?

Patch

Now we’ve come to patches.

Patches are the primary way to apply changes to existing objects in Kubernetes. The operation patch works in such a way that:

the kube-apiserver user needs to send a patch in JSON format and specify the object,
and the apiserver will figure out the current state of the object and adjust it to the required state.

Optimistic locking is not required in this case. This operation is more declarative compared to replace, although it may initially seem otherwise.

Thus:

using the operation create we create the object from the manifest in Git,
using delete — delete it if the object is no longer needed,
using patch — modify the object, bringing it to the state described in Git.

However, to do this, it is necessary to create the correct patch!

How patches work in Helm 2: 2-way merge

Upon the initial installation of the Helm release, an operation is executed create for the chart resources.

When updating the Helm release for each resource:

it calculates the patch between the resource version from the previous chart and the current version of the chart,
and applies this patch.

This patch will be called 2-way merge patch, because two manifests are involved in its creation:

the resource manifest from the previous release,
the resource manifest from the current resource.

During deletion, the operation delete is invoked in kube apiserver for resources that were declared in the previous release but not declared in the current one.

The 2-way merge patch approach has a problem: it leads to a desynchronization between the actual resource state in the cluster and the manifest in Git..

Illustration of the problem with an example

In Git, the chart contains a manifest in which the field image in the Deployment has the value ubuntu:18.04.
The user has changed the value of this field to kubectl edit ubuntu:19.04 Upon redeploying the Helm chart,.
no patch is generated , because the fieldin the previous release version and the current chart is the same. image After the redeployment,
it remains image , even though the chart specifies Upon redeploying the Helm chart,We have encountered desynchronization and lost declarative control. ubuntu:18.04.

What is a synchronized resource?

It is impossible to achieve complete

Generally speaking, correspondence between the resource manifest in the running cluster and the manifest from Git. This is because the real manifest may have operational annotations/labels, additional containers, and other data that dynamic controllers add and remove from resources. We cannot and do not want to keep this data in Git. However, we want the fields we explicitly specified in Git to take the corresponding values upon deployment. Thus, a general

rule for a synchronized resource : at deployment, one can only change or remove the fields that are explicitly mentioned in the Git manifest (or were mentioned in the previous version and are now removed).3-way merge patch

: generates a patch between the last applied version of the manifest from Git and the target version of the manifest from Git, taking into account the current version of the manifest from the running cluster. The resulting patch must correspond to the rule of a synchronized resource:

The main idea : generates a patch between the last applied version of the manifest from Git and the target version of the manifest from Git, taking into account the current version of the manifest from the running cluster. The resulting patch must correspond to the rule of a synchronized resource:new fields added to the target version are added via the patch;

New fields added to the target version are included through a patch;
Fields that existed in the last applied version and do not exist in the target will be reset using a patch;
Fields in the current version of the object that differ from the target version of the manifest will be updated using a patch.

This is exactly how patches are generated. kubectl apply:

The last applied version of the manifest is saved in the annotation of the object itself,
the target is taken from the specified YAML file,
the current is from the running cluster.

Now that we've covered the theory, it's time to explain what we've done in werf.

Applying changes in werf

Previously, werf, like Helm 2, used 2-way-merge patches.

Repair patch

To transition to a new type of patches — 3-way-merge — the first step we introduced is the so-called repair patches..

During deployment, a standard 2-way-merge patch is used, but werf also generates a patch that synchronizes the actual state of the resource with what is written in Git (this patch is created using the same rules for the synchronized resource described above).

In case of desynchronization, at the end of the deployment, the user receives a WARNING with a corresponding message and a patch that needs to be applied to bring the resource to a synchronized state. This patch is also recorded in a special annotation werf.io/repair-patch. It is assumed that the user will manually apply this patch themselves: werf will not apply it in principle. The generation of repair patches is a temporary measure that allows testing the creation of patches based on the 3-way-merge principle, but does not apply these patches automatically. Currently, this mode of operation is enabled by default.

3-way-merge patch only for new releases

Starting from December 1, 2019, beta and alpha versions of werf will begin

to use full 3-way-merge patches for applying changes only for new Helm releases being rolled out through werf. Existing releases will continue to use the approach with 2-way-merge + repair patches. default This mode of operation can be explicitly enabled by setting

WERF_THREE_WAY_MERGE_MODE=onlyNewReleases : this feature has been appearing in werf over several releases: in the alpha channel, it became stable starting from version already downloaded now.

Notev1.0.5-alpha.19 , and in the beta channel — from3-way-merge patch for all releases. v1.0.4-beta.20.

3-way-merge patch for all releases

Starting December 15, 2019, beta and alpha versions of werf will by default use full 3-way-merge patches to apply changes for all releases.

WERF_THREE_WAY_MERGE_MODE=onlyNewReleases WERF_THREE_WAY_MERGE_MODE=enabled already downloaded now.

How to handle resource autoscaling?

In Kubernetes, there are 2 types of autoscaling: HPA (Horizontal Pod Autoscaler) and VPA (Vertical Pod Autoscaler).

Horizontal automatically selects the number of replicas, while vertical sets the amount of resources. Both the number of replicas and resource requirements are specified in the resource manifest (see spec.replicas or spec.containers[].resources.limits.cpu, spec.containers[].resources.limits.memory and others).

Problem: if the user configures a resource in the chart with specific values for resources or replicas, and if auto-scalers are enabled for that resource, then on each deployment, werf will reset these values to what is recorded in the chart manifest.

There are two solutions to the problem. First, it is best to refrain from explicitly specifying autoscalable values in the chart manifest. However, if this option is not suitable for some reason (for example, because it's convenient to set initial limits for resources and the number of replicas in the chart), werf offers the following annotations:

werf.io/set-replicas-only-on-creation=true
werf.io/set-resources-only-on-creation=true

With such an annotation, werf will not reset the corresponding values on each deployment but will only set them at the initial creation of the resource.

For more details — see the project documentation on HPA and VPA.

Disable 3-way-merge patch usage

Users can temporarily disable the use of new patches in werf using the environment variable WERF_THREE_WAY_MERGE_MODE=disabled. However, starting from March 1, 2020, this restriction will stop working and only the use of 3-way-merge patches will be possible.

Resource adoption in werf

Mastering the application of changes via 3-way-merge patches allowed us to immediately implement a feature like adopting existing resources in the cluster to a Helm release.

Helm 2 has a problem: you cannot add a resource to the chart manifests that already exists in the cluster without recreating that resource from scratch (see #6031, #3275). We taught werf to accept existing resources into the release. To do this, you need to annotate the current version of the resource from the running cluster (for example, using kubectl edit):

"werf.io/allow-adoption-by-release": RELEASE_NAME

Now the resource needs to be described in the chart, and during the next deployment with werf of the release with the corresponding name, the existing resource will be included in this release and remain under its management. Moreover, during the resource's adoption into the release, werf will bring the current state of the resource from the working cluster to the state described in the chart, using the same 3-way-merge patches and the synchronized resource rule.

Note: configuration WERF_THREE_WAY_MERGE_MODE has no effect on resource adoption — in the case of adoption, a 3-way-merge patch is always used.

Details can be found in the documentation.

Conclusions and further plans

I hope that after reading this article, it's clearer what 3-way-merge patches are and why we arrived at them. From a practical perspective, the implementation of them in the werf project is another step towards improving Helm-like deployments. Now we can forget about configuration synchronization issues that often arose when using Helm 2. At the same time, a new useful feature for adopting already pulled Kubernetes resources into Helm releases has been added.

In Helm-like deployments, there are still some problems and difficulties, such as the use of Go templates, and we will continue to address them.

Information about resource update methods and adoption can also be found on this documentation page.

Helm 3

Deserving separate mention is the recently released literally just days ago new major version of Helm — v3, — which also uses 3-way-merge patches and eliminates Tiller. The new version of Helm requires migration existing installations to convert them to the new release storage format.

Werf has already eliminated the use of Tiller, switched to 3-way-merge, and added and much more, while remaining compatible with existing Helm 2 installations (no migration scripts need to be executed). Therefore, as long as werf is not switched to Helm 3, werf users do not lose the main advantages of Helm 3 over Helm 2 (they are also present in werf).

However, werf's switch to the Helm 3 codebase is inevitable and will occur in the near future. It is expected to be in werf 1.1 or werf 1.2 (currently, the main version of werf is 1.0; for more information about the versioning system of werf, see here). By that time, Helm 3 will have stabilized.