Proper Comparison of Kubernetes Apply, Replace and Patch

For Kubernetes, there are several options for updating resources: apply, edit, patch, and replace. There is confusion about what each of them does and when to apply them. Let's figure it out.

Proper Comparison of Kubernetes Apply, Replace and Patch

If search on google the phrase "kubernetes apply vs replace" is Answer on StackOverflow, which is not correct. When searching "kubernetes apply vs patch" the very first link is documentation on kubectl patch, which does not include comparison apply и patch. This article will look at the various options, as well as the proper use of each.

During the life cycle of a Kubernetes resource (service, deployment, ingress, etc.), sometimes you need to change, add or remove some properties of this resource. For example, add a note, increase or decrease the number of replicas.

Kubernetes CLI

If you are already working with Kubernetes clusters through the CLI, then you are already familiar with apply и edit. Team apply reads the resource specification from the file and does an "upsert" to the Kubernetes cluster, i.e. creates the resource if it doesn't exist and updates it if it exists. Team edit reads the resource via the API, then writes the resource specification to a local file, which is then opened in a text editor. After you edit and save the file, kubectl will send the changes made back through the API, which will carefully apply those changes to the resource.

Not everyone knows commands patch и replace. Team patch allows you to change part of the resource specification by providing only the changed part on the command line. Team replace works the same as edit, but everything needs to be done manually: you need to download the current version of the resource specification, for example, using kubectl get -o yaml, edit it, then use replace to update a resource with a modified specification. Team replace will not work if there were any changes between reading and replacing the resource.

Kubernetes API

You are probably familiar with the methods CoreV1().Pods().Update(), replaceNamespacedService or patch_namespaced_deployment, if you work with clusters via client library for the Kubernetes API using some programming language. The library handles these methods with HTTP requests using the methods PUT и PATCH. In this case, update и replace use PUT, patch, no matter how trite it may be, uses PATCH.

It is worth noting that kubectl also works with clusters via API. In other words, kubectlis a wrapper on top of the client library for the Go language, providing much of the ability to provide subcommands in a more compact and readable form in addition to the regular API features. For example, as you may have noticed, the method apply was not mentioned in the previous paragraph. Currently (May 2020, approx. translator) all logic kubectl apply, i.e. creating non-existent resources and updating existing ones, works entirely on the code side kubectl. Efforts are being made on logic transfer apply to the API side, but it's still in beta. I'll write more below.

patch by default

Best to apply patchif you wish to update the resource. This is how both client libraries work on top of the Kubernetes API, and kubectl (not surprising, because it is a wrapper for the client library, approx. translator).

Work strategically

All teams kubectl apply, edit и patch use the method PATCH in HTTP requests to update an existing resource. If you delve into the implementation of commands in more detail, then all use the approach strategic merge patching to update resources even though the command patch may use other approaches (more on this below). The strategic-merge patching approach attempts to "get it right" when merging a supplied specification with an existing specification. More specifically, it tries to merge both objects and arrays, which means that the changes are usually additive. For example, running the command patch with a new environment variable in the pod container specification causes that environment variable to be added to existing environment variables rather than overwriting them. To delete using this approach, you must force the value of the parameter to be null in the provided specification. Which of the teams kubectl is the best way to update?

If you create and manage your resources with kubectl apply, when updating it is better to always use kubectl applyTo kubectl could manage configuration and properly track requested changes from application to application. Advantage of always using apply is that it keeps track of the previously applied BOM, allowing it to know when BOM properties and array elements are explicitly removed. This allows you to use apply to remove properties and array elements, while normal strategic merge won't work. Teams edit и patch do not update notes that kubectl apply uses to track its changes, so any changes that are tracked and made through the Kubernetes API but made through commands edit и patch, invisible to subsequent commands applyThat is, apply does not remove them, even if they do not appear in the input specification for apply (The documentation says that edit и patch make updates to the notes used applybut not in practice).

If you are not using the command apply, can be used as editAnd patch, choosing the command that best suits the change being made. When adding and changing BOM properties, both approaches are roughly the same. When deleting specification properties or array elements edit behaves like a one-time run apply, including keeping track of what the specification was before and after it was edited, so you can explicitly remove properties and array elements from the resource. You need to explicitly set the property value to null in the specification for patchto remove it from the resource. Removing an array element using strategic-merge patching is more complex because it requires the use of merge directives. See other upgrade approaches below for more acceptable alternatives.

To implement update methods in the client library that behave similar to the above commands kubectl, should be set in requests content-type в application/strategic-merge-patch+json. If you want to remove properties in the specification, you need to explicitly set their values ​​to null in a similar way kubectl patch. If you need to delete array elements, you should include merge directives in the update specification, or use a different approach to updates.

Other Update Approaches

Kubernetes supports two other approaches to updates: JSON merge patch и JSON patch. The JSON merge patch approach takes a partial Kubernetes specification as input and supports object merging similar to the strategic-merge patching approach. The difference between the two is that it only supports array replacement, including the array of containers in the pod spec. This means that when using JSON merge patch, you need to provide full specifications for all containers in case any property of any container changes. So this approach is useful for removing elements from an array in a BOM. On the command line, you can select JSON merge patch using kubectl patch --type=merge. When working with the Kubernetes API, you should use the request method PATCH and installation content-type в application/merge-patch+json.

The JSON patch approach, instead of providing a partial resource specification, uses providing the changes you want to make to the resource as an array, where each array element is a description of the change being made to the resource. This approach is a more flexible and powerful way of expressing changes to be made, but at the cost of having the list of changes to be made in a separate, non-Kubernetes format instead of submitting a partial resource specification. IN kubectl you can select JSON patch using kubectl patch --type=json. When using the Kubernetes API, this approach works using the request method PATCH and installation content-type в application/json-patch+json.

Confidence is needed - use replace

In some cases, you need to be sure that no changes will be made to the resource between the time the resource is read and when it is updated. In other words, it is worth making sure that all changes will be atomic. In this case, to update the resources, you should use replace. For example, if you have a ConfigMap with a counter that is updated by multiple sources, be sure that two sources do not update the counter at the same time, causing the update to be lost. To demonstrate, imagine a sequence of events using the approach patch:

  • A and B get the current state of the resource from the API
  • Each one updates the specification locally by incrementing the counter by one and also adding "A" or "B" respectively to the "updated-by" annotation
  • And refreshes the resource a little faster
  • B updates resource

As a result, update A is lost. Last operation patch wins, the counter is incremented by one instead of two, and the "updated-by" annotation value ends in "B" and does not contain "A". Let's compare the above with what happens when updates are done using the approach replace:

  • A and B get the current state of the resource from the API
  • Each one updates the specification locally by incrementing the counter by one and also adding "A" or "B" respectively to the "updated-by" annotation
  • And refreshes the resource a little faster
  • B tries to update the resource, but the update is rejected by the API because the version of the resource is in the specification replace does not match the current version of the resource in Kubernetes because the version of the resource was incremented by A's replace operation.

In the above case, B will have to re-fetch the resource, make changes to the new state, and try again to do replace. As a result, the counter will be incremented by two and the "updated-by" annotation will have "AB" at the end.

The above example assumes that when executing replace a complete replacement of the entire resource is performed. Specification used for replace, must not be partial, or parts as in apply, but complete, including the addition resourceVersion to specification metadata. If you have not included resourceVersion or the version you provided is not the current one, the replacement will be rejected. So the best approach to use replace – read the resource, update it and replace it immediately. Using kubectl, it might look like this:

$ kubectl get deployment my-deployment -o json 
    | jq '.spec.template.spec.containers[0].env[1].value = "new value"' 
    | kubectl replace -f -

It is worth noting that the following two commands, executed in sequence, will be executed successfully, because deployment.yaml does not contain a property .metadata.resourceVersion

$ kubectl create -f deployment.yaml
$ kubectl replace -f deployment.yaml

It would seem that this contradicts what was said above, i.e. "adding resourceVersion to the metadata of the specification." Is it wrong to say so? No, it is not, because if kubectl notices that you didn't specify resourceVersion, it will read it from the resource and add it to the specification you specified, and only then execute replace. Since this is potentially dangerous if you rely on atomicity, the magic works completely on the side kubectl, you should not rely on it when using client libraries that work with the API. In this case, you will have to read the current resource specification, update it, and then execute PUT request.

You can’t do a patch - do a replace

Sometimes you need to make some changes that cannot be handled by the API. In these cases, you can force replacement of the resource by deleting and re-creating it. This is done using kubectl replace --force. Running the command immediately removes the resources and then recreates them with the provided spec. There is no "force replace" handler in the API, and to do so via the API requires two steps. First you need to delete the resource by setting it to gracePeriodSeconds to zero (0) and propagationPolicy in "Background" and then re-create that resource with the desired specification.

Warning: this approach is potentially dangerous, may lead to an undefined state

Apply on the server side

As mentioned above, Kubernetes developers are working on implementing the logic apply of kubectl in the Kubernetes API. Logics apply available in Kubernetes 1.18 via kubectl apply --server-side or via the API using the method PATCH с content-type application/apply-patch+YAML.

Note: JSON is also valid YAML, so it's okay to send the spec as JSON even if content-type will application/apply-patch+yaml.

Besides that logic kubectl becomes available to everyone through the API, apply on the server side, keeps track of who is responsible for the fields in the specification, thus allowing secure multiple access for conflict-free editing of the specification. In other words, if apply on the server side will become more widespread, there will be a universal secure resource management interface for different clients, for example, kubectl, Pulumi or Terraform, GitOps, as well as self-written scripts using client libraries.

Results

I hope this short overview of the different ways to update resources in clusters was helpful to you. It's good to know that adversaries aren't just apply versus replace, it's possible to update a resource with apply , edit , patch , or replace . After all, in principle, each approach has its own scope. For atomic changes, replace is preferable, otherwise you should use a strategic-merge patch via apply. As a last resort, I'm counting on you to understand that you can't trust Google or StackOerflow when searching for "kubernetes apply vs replace". At least until this article replaces the current answer.

Proper Comparison of Kubernetes Apply, Replace and Patch

Source: habr.com

Add a comment