🥇Content-based tagging in the werf builder: why and how it works?

werf — our open-source GitOps CLI utility for building and delivering applications to Kubernetes. In release v1.1 a new feature was introduced in the image builder: content-based tagging. Until now, the typical tagging scheme in werf assumed tagging Docker images by Git tag, Git branch, or Git commit. However, all these schemes have disadvantages that are completely resolved by the new tagging strategy. Details about it and what makes it so good are below.

Deployment of microservices from a single Git repository

It's often the case that an application is divided into many more or less independent services. Releases of these services can occur independently: one or several services can be released at once, while others should continue to operate without any changes. However, from the perspective of code storage and project management, it is convenient to keep such application services in a single repository.

There are situations where services are truly independent and not related to a single application. In this case, they will be located in separate projects and their release will be carried out through separate CI/CD processes in each of the projects.

However, in reality, developers often break a single application into several microservices, but creating a separate repository and project for each one is clearly overkill. This discussion will focus on this situation: several such microservices reside in a single project repository, and releases occur through a unified process in CI/CD.

Tagging by Git branch and Git tag

Let's assume the most common tagging strategy is tag-or-branch. For Git branches, images are tagged by the branch name; at any one time, there can be only one published image for that branch name. For Git tags, images are tagged by the tag name respectively.

When a new Git tag is created — for example, when a new version is released — a new Docker tag will be created for all project images in Docker Registry:

myregistry.org/myproject/frontend:v1.1.10
myregistry.org/myproject/myservice1:v1.1.10
myregistry.org/myproject/myservice2:v1.1.10
myregistry.org/myproject/myservice3:v1.1.10
myregistry.org/myproject/myservice4:v1.1.10
myregistry.org/myproject/myservice5:v1.1.10
myregistry.org/myproject/database:v1.1.10

These new image names are integrated into the Kubernetes configuration via Helm templates. When the deployment command is executed, werf deploy the field is updated image in the Kubernetes resource manifests, and the corresponding resources are restarted due to the changed image name.

The Problem: in cases where the content of the image hasn't actually changed from the previous release (Git tag), but only its Docker tag has, an unnecessary restart of this application occurs, potentially leading to some downtime, even though there was no real reason to initiate that restart.

As a result, with the current tagging scheme, multiple separate Git repositories need to be maintained, raising the issue of how to organize the deployment of these multiple repositories. Overall, this scheme becomes overloaded and complex. It’s better to consolidate many services into a single repository and create Docker tags to avoid unnecessary restarts.

Tagging by Git commit

Werf also offers a tagging strategy related to Git commits.

A Git commit serves as the identifier for the content of the Git repository and depends on the history of file changes in the Git repository, so it seems logical to use it for tagging images in the Docker Registry.

However, tagging by Git commit has the same drawbacks as tagging by Git branches or Git tags:

An empty commit could have been created that does not change any files, but the Docker image tag would still be altered.
A merge commit could have been created that does not change files, but the Docker image tag would still be altered.
A commit could alter files in Git that are not included in the image, and again the Docker image tag would be changed.

Tagging by Git branch name does not reflect the version of the image.

There is yet another issue related to the tagging strategy by Git branches.

Tagging by branch name works as long as the commits in that branch are collected sequentially in chronological order.

If a user rebuilds an old commit related to some branch in the current scheme, werf will overwrite the image with a newly built version corresponding to that Docker tag for the old commit. Deployments using that tag risk pulling a different version of the image during pod restarts, causing our application to lose its connection with the CI system and become desynchronized.

Moreover, during consecutive pushes to one branch with a short time interval between them, an older commit may be processed later than a newer one: the old image version can overwrite the new one by the Git branch tag. Such problems can be addressed by a CI/CD system (for example, in GitLab CI, a pipeline for a series of commits is triggered for the last one). However, not all systems support this, and there should be a more reliable way to prevent such a fundamental issue.

What is content-based tagging?

So, what is content-based tagging — tagging images based on content.

For creating Docker tags, checksums are used rather than Git primitives (Git branch, Git tag...), which are linked to:

the image content. The image tag identifier reflects its content. When building a new version, this identifier will remain unchanged if no files in the image were modified;
the history of the image's creation in Git. Images associated with different Git branches and different build histories via werf will have different tag identifiers.

As such a tag identifier, the so-called stage signature.

is used. from, Each image consists of a set of stages:, before-install, install, git-archive, imports-after-install,… before-setup git-latest-patch etc. Each stage has an identifier reflecting its content — stage signature.

(stage signature) The final image, consisting of these stages, is tagged by the so-called signature of the set of these stages —stages signature

, which generalizes all the stages of the image. werf.yaml Each image from the configuration

will generally have its own such signature and, accordingly, Docker tag.

The stage signature resolves all the mentioned issues:
It is resilient to empty Git commits.
It is resilient to Git commits that change files irrelevant to the image.

It avoids the problem of overwriting the current image version when restarting builds for older Git commits of the branch.

This is now the recommended tagging strategy and is used by default in werf for all CI systems.

How to enable and use in werf werf publish: The relevant option appeared in the command

--tag-by-stages-signature=true|false In the CI system, the tagging strategy is set by the commandwerf ci-env. Previously, a parameter was defined for itwerf ci-env --tagging-strategy=tag-or-branch. Now, if specified, it is changed to if you do not specify this option, werf will default to using the tagging strategy stages-signature. The command In the CI system, the tagging strategy is set by the command will automatically set the necessary flags for the command werf build-and-publish (or werf publish), so no additional options need to be specified for these commands.

For example, the command:

werf publish --stages-storage :local --images-repo registry.hello.com/web/core/system --tag-by-stages-signature

... can create the following images:

registry.hello.com/web/core/system/backend:4ef339f84ca22247f01fb335bb19f46c4434014d8daa3d5d6f0e386d
registry.hello.com/web/core/system/frontend:f44206457e0a4c8a54655543f749799d10a9fe945896dab1c16996c6

Here 4ef339f84ca22247f01fb335bb19f46c4434014d8daa3d5d6f0e386d — this is the stage signature of the image backend, and f44206457e0a4c8a54655543f749799d10a9fe945896dab1c16996c6 — this is the stage signature of the image frontend.

When using special functions werf_container_image and werf_container_env no changes are required in Helm templates: these functions will automatically generate correct image names.

Example configuration in the CI system:

type multiwerf && source <(multiwerf use 1.1 beta)
type werf && source <(werf ci-env gitlab)
werf build-and-publish|deploy

More information on configuration is available in the documentation:

Total

The new option werf publish --tag-by-stages-signature=true|false.
The new value of the option werf ci-env --tagging-strategy=stages-signature|tag-or-branch (if not specified, it will default to stages-signature).
If previously tagging options were used based on Git commits (WERF_TAG_GIT_COMMIT or the option werf publish --tag-git-commit COMMIT), it is necessary to switch to the tagging strategy stages-signature.
New projects should be immediately switched to the new tagging scheme.
Old projects transitioning to werf 1.1 should ideally be switched to the new tagging scheme; however, the old one tag-or-branch is still supported.

Content-based tagging solves all the issues discussed in the article:

Resilience of Docker tag names to empty Git commits.
Resilience of Docker tag names to Git commits that change irrelevant files for the image.
Does not lead to issues with overwriting the current image version when re-running builds for old Git commits on Git branches.

Use it! And don't forget to visit us at GitHub, to create an issue or find an existing one, upvote, create a PR, or just follow the project's development.