My name is Dmitry Sugrobov, I'm a developer at Leroy Merlin. In the article I will tell you why Helm is needed, how it simplifies working with Kubernetes, what has changed in the third version, and how to use it to update applications in production without downtime.
This is a summary based on a speech at a conference by If you don't want to read, watch the video.
Why we use Kubernetes in production
Leroy Merlin is the leader in the DIY retail market in Russia and Europe. Our company has more than a hundred developers, 33 internal employees and a huge number of people visiting hypermarkets and the site. In order to make them all happy, we decided to stick to the industry standard. Develop new applications using microservice architecture; use containers to isolate environments and deliver correctly; and for orchestration use Kubernetes. The price of using orchestrators is rapidly becoming cheaper: the number of engineers who own the technology is growing on the market, and providers appear offering Kubernetes as a service.
Everything that Kubernetes does, of course, can be done in other ways, for example, by smearing some Jenkins and docker-compose with scripts, but why complicate life if there is a ready-made and reliable solution? Therefore, we came to Kubernetes and have been using it in production for a year now. We now have twenty-four Kubernetes clusters, the oldest of which is over a year old with about two hundred pods.
The curse of a large number of YAML files in Kubernetes
To run a microservice in Kubernetes, we will create at least five YAML files: for Deployment, Service, Ingress, ConfigMap, Secrets - and send it to the cluster. For the next application, we will write the same package of yamliki, with the third - another one, and so on. Multiplying the number of documents by the number of environments, we already get hundreds of files, and this is not even taking into account dynamic environments.
Adam Reese, Helm's core maintainer, introduced the concept of "", which looks like this:
- Copy YAML - copy a YAML file.
- Paste YAML - paste it.
- Fix Indents - fix indents.
- Repeat - repeat again.
The option is working, but you have to copy YAML files many times. To change this cycle, they came up with Helm.
What is helm
First, Helm package manager, which helps you find and install the programs you need. To install, for example, MongoDB, you do not need to go to the official website and download binaries, just run the command helm install stable/mongodb.
Secondly, Helm - template engine, helps to parameterize files. Let's return to the situation with YAML files in Kubernetes. It's easier to write the same YAML file, add some placeholders to it, in which Helm will substitute values. That is, instead of a large set of yamliks, there will be a set of templates (templates) into which the necessary values ββwill be substituted at the right time.
Thirdly, Helm - deployment wizard. With it, you can install, rollback and update applications. Let's see how to do it.
How to use Helm to deploy your own applications
Install the Helm client on the computer, following the official . Then we will create a set of YAML files. Instead of specifying specific values, let's leave placeholders that Helm will fill in with information in the future. A set of such files is called a Helm chart. It can be sent to the Helm console client in three ways:
- specify a folder with templates;
- pack into a .tar archive and point to it;
- put the template in a remote repository and add a link to the repository in the Helm client.
You also need a file with values ββ\uXNUMXb\uXNUMXb- values.yaml. The data from there will be substituted into the template. Let's create it too.
The second version of Helm has an additional server application called Tiller. It hangs outside Kubernetes and waits for requests from the Helm client, and when called, it substitutes the necessary values ββinto the template and sends it to Kubernetes.
Helm 3 is simpler: instead of processing templates on the server, the information is now processed entirely on the side of the Helm client and sent directly to the Kubernetes API. This simplification improves the security of the cluster and facilitates the rollout scheme.
How it all works
Run the command helm install. Specify the name of the application release, give the path to values.yaml. At the end, we will specify the repository that contains the chart and the name of the chart. In the example, these are "lmru" and "bestchart" respectively.
helm install --name bestapp --values values.yaml lmru/bestchart
Execution of the command is possible only once, when re-executing instead of install need to use upgrade. For simplicity, instead of two commands, you can run the command upgrade with additional key --install. At the first execution, Helm will send a command to install the release, and will update it in the future.
helm upgrade --install bestapp --values values.yaml lmru/bestchart
Pitfalls of deploying new application versions with Helm
At this point in the story, I'm playing Who Wants to Be a Millionaire with the audience and we're figuring out how to get Helm to update the app version. .
When I studied Helm's work, I was surprised by the strange behavior when trying to update versions of running applications. I updated the application code, uploaded a new image to the docker registry, sent the command to deploy - and nothing happened. Below are some not-so-good ways to update apps. By studying each of them in more detail, you begin to understand the internal structure of the instrument and the reasons for this non-obvious behavior.
Method 1. Do not change information since the last run
As the saying goes Helm, "Kubernetes charts can be big and complex, so Helm tries to keep things as simple as possible." Therefore, if you update the latest version of the application image in the docker registry and run the command helm upgrade, then nothing will happen. Helm will think that nothing has changed and there is no need to send a command to Kubernetes to update the application.
Hereinafter, the latest tag is shown solely as an example. When this tag is specified, Kubernetes will download the image from the docker registry every time, regardless of the imagePullPolicy parameter. Using latest in production is undesirable and causes side effects.
Method 2. Update LABEL in image
As written in the same , "Helm will only update an app if it has changed since the last release." The logical option for this would be to update the LABEL in the docker image itself. However, Helm does not look into the application images and is not aware of any changes to them. Accordingly, when updating labels in the image, Helm will not know about them, and the application update command will not be received in Kubernetes.
Method 3. Use the key --force
Let's turn to the manuals and look for the right key. The key that makes the most sense --force. Despite the telling name, the behavior differs from what is expected. Instead of a forced update of the application, its real purpose is to restore a release that is in the FAILED status. If you do not use this key, then you need to sequentially execute commands helm delete && helm install --replace. Instead, it is suggested to use the key --force, which automates the sequential execution of these commands. More info in this . In order to tell Helm to update the version of the application, unfortunately, this key will not work.
Method 4. Change labels directly in Kubernetes
Updating label directly in cluster using command kubectl edit - bad idea. This action will lead to information inconsistency between the running application and the one that was originally sent for deployment. The behavior of Helm during deployment in this case differs from its version: Helm 2 will not do anything, and Helm 3 will deploy a new version of the application. To understand the reason, you need to understand how Helm works.
How Helm works
To determine if an application has changed since the last release, Helm can use:
- a running application in Kubernetes;
- new values.yaml and current chart;
- Helm's internal release information.
For the most inquisitive: where does Helm store internal release information?By executing the command helm history, we will get all the information about the versions installed with Helm.
There is also detailed information about sent patterns and values. We can request it:
In the second version of Helm, this information is in the same namespace where Tiller is running (by default, kube-system), in the ConfigMap, marked with the label βOWNER=TILLERβ:
At the time of the appearance of the third version of Helm, the information moved to secrets, moreover, to the same namespace where the application is running. Thanks to this, it became possible to run several applications simultaneously in different namespaces with the same release name. In the second version, it was a severe headache when namespaces are isolated, but can influence each other.
The second Helm, when trying to figure out if an update is needed, uses only two sources of information: what was provided to him now, and internal information about releases, which lies in the ConfigMap.
The third Helm uses a three-way merge strategy: in addition to that information, it also takes into account the application that is running right now in Kubernetes.
For this reason, the old version of Helm will not do anything, since it does not take into account the information of the application in the cluster, but Helm 3 will receive the changes and send the new application for deployment.
Method 5: Use --recreate-pods key
With a key --recreate-pods you can achieve what was originally planned to be obtained using the key --force. The containers will restart and, according to the imagePullPolicy: Always policy for the latest tag (more on this in the footnote above), Kubernetes will download and run the new version of the image. This will not be done in the best way: without taking into account the StrategyType of the deployment, it will abruptly turn off all the old application instances and go to launch new ones. During the restart, the system will not work, users will suffer.
In Kubernetes itself, a similar problem has also existed for a long time. And now, 4 years after the opening , the problem has been fixed, and starting with version 1.15 of Kubernetes, the possibility of rolling-restart pods appears.
Helm simply turns off all applications and launches new containers nearby. In production, you canβt do this, so as not to cause a simple application. This is only needed for development needs, it can only be done in stage environments.
How to update app version using Helm?
We will change the values ββsent to Helm. Typically, these are values ββthat are substituted for the image tag. In the case of latest, often used for non-productive environments, the annotation acts as mutable information, which is useless for Kubernetes itself, and for Helm it will act as a signal to update the application. Annotation value filling options:
- random value using the standard function
{{ randAlphaNum 6 }}.
There is a caveat: after each deployment using a chart with such a variable, the annotation value will be unique, and Helm will assume that there are changes. It turns out that we will always restart the application, even if we have not changed its version. This is not critical, since there will be no downtime, but still unpleasant. - Insert current date and time β
{{ .Release.Date }}.
A variant is like a random value with a permanently unique variable. - A better way is to use checksums. This is the SHA of the image or the SHA of the last commit in the git -
{{ .Values.sha }}.
They will need to be counted and sent to the Helm client on the calling side, for example in Jenkins. If the application has changed, then the checksum will also change. Therefore, Helm will only update the application when needed.
Let's summarize our efforts
- Helm makes changes in the least invasive way, so any change at the application image level in the Docker Registry will not result in an update: nothing will happen after the command is executed.
- Key
--forceused to repair problematic releases and is not associated with a forced upgrade. - Key
--recreate-podswill forcibly update applications, but will do it in a vandal way: it will abruptly turn off all containers. Users will suffer from this, itβs not worth doing this on the sale. - Directly make changes to the Kubernetes cluster using the command
kubectl editdon't: we'll break the consistency, and the behavior will differ depending on the version of Helm. - With the release of the new version of Helm, many nuances have appeared. Issues in the Helm repository are described in clear language, they will help you understand the details.
- Adding a mutable annotation to a chart will make it more flexible. This will allow the application to roll out correctly, without downtime.
Thought from the category of "peace in the world", working in all areas of life: read the instructions before use, not after. Only having complete information will it be possible to build reliable systems and make users happy.
Other related links:
This report was first presented at by Mail.ru Cloud Solutions. See other performances and subscribe to announcements of events in Telegram .
Source: habr.com