🥇The challenge of "smart" container image cleanup and its solution in werf

This article addresses the issue of cleaning up images that accumulate in container registries (Docker Registry and its analogs) in the context of modern CI/CD pipelines for cloud-native applications deployed in Kubernetes. It outlines the main criteria for the relevance of images and the resulting challenges in automating cleanup, saving space, and meeting team needs. Finally, using a specific Open Source project as an example, we will discuss how these challenges can be overcome.

Introduction

The number of images in a container registry can rapidly grow, taking up more storage space and consequently significantly increasing costs. To control, limit, or maintain acceptable growth of the space occupied in the registry, it is common to:

use a fixed number of tags for images;
somehow clean up images.

The first limitation is sometimes acceptable for small teams. If developers can make do with permanent tags (latest, main, test, boris , etc.), the registry will not bloat in size, and there may be little need to think about cleanup for a long time. After all, all outdated images are overwritten, and there’s simply no work left for cleanup (everything is handled by the built-in garbage collector).

However, this approach severely limits development and is rarely applicable to the CI/CD of modern projects. An integral part of development has become automation, which allows for much faster testing, deploying, and delivering new functionality to users. For example, in all our projects, a CI pipeline is automatically created with each commit. In it, an image is built, tested, deployed to various Kubernetes environments for debugging and remaining checks, and if everything goes well, the changes reach the end user. This has long ceased to be rocket science; for many, it’s a routine matter—likely for you as well, since you’re reading this article.

As bug fixes and new functionality development occur simultaneously, and releases can happen several times a day, it is clear that the development process is accompanied by a significant number of commits, which means— a large number of images in the registry.As a result, the question of organizing effective registry cleanup, i.e., removing outdated images, becomes pressing.

But how can one even determine if an image is relevant?

Relevance Criteria for Images

In the vast majority of cases, the main criteria will be as follows:

1. The first (most obvious and critical of all) — these are the images that are currently in use in Kubernetes.Removing these images can lead to significant costs due to production downtime (for example, images may be required for replication) or negate the efforts of the team working on debugging in any of the environments. (For this reason, we even created a special Prometheus exporter, which monitors the absence of such images in any Kubernetes cluster.)

2. The second (less obvious, but also very important and again related to operation) — images that are needed for rollback in case serious issues are detected in the current version. For example, with Helm, these are images used in saved release versions. (By the way, by default, Helm has a limit of 256 revisions, but it's unlikely that anyone actually needs to keep that many versions..) After all, we store versions specifically so that they can be used later, i.e., to "rollback" to them when necessary.

3. The third — developer needs:all images related to their current work. For example, if we are considering a PR, it makes sense to keep the image that corresponds to the latest commit and, say, the previous commit: this way, the developer can quickly return to any task and work with the latest changes.

4. The fourth — images that correspond to versions of our application, i.e., are the final product: v1.0.0, 20.04.01, sierra, etc.

NB: The criteria outlined here were formulated based on experience working with dozens of development teams from various companies. However, of course, depending on the specifics of development processes and the infrastructure used (for example, if Kubernetes is not used), these criteria may differ.

Criteria Compliance and Existing Solutions

Popular container registry services typically offer their image cleanup policies: you can specify the conditions under which a tag is removed from the registry. However, the possibilities of these conditions are limited to parameters such as names, creation time, and the number of tags*.

* Depends on the specific implementations of the container registry. We considered the options for the following solutions: Azure CR, Docker Hub, ECR, GCR, GitHub Packages, GitLab Container Registry, Harbor Registry, JFrog Artifactory, Quay.io — as of September 2020.

This set of parameters is quite sufficient to meet the fourth criterion — that is, to filter images that correspond to versions. However, for all other criteria, a compromise solution must be chosen (a stricter or, on the contrary, a more lenient policy) depending on expectations and financial capabilities.

For instance, the third criterion — related to developers' needs — can be solved by organizing processes within teams: specific naming of images, maintaining special allow lists, and internal agreements. However, in the end, it still needs to be automated. And if the capabilities of ready-made solutions are insufficient, something custom must be created.

The situation is similar for the first two criteria: they cannot be satisfied without obtaining data from an external system — that very system where applications are deployed (in our case, it's Kubernetes).

Workflow illustration in Git

Suppose you work in Git approximately like this:

The icon with a head on the diagram marks the container images currently deployed in Kubernetes for any users (end users, testers, managers, etc.) or used by developers for debugging and similar purposes.

What will happen if the cleanup policies allow keeping (not deleting) images only by specified tag names?

Clearly, such a scenario will please no one.

What will change if the policies allow not deleting images by a specified time interval / number of latest commits?

The result became significantly better, yet still far from ideal. After all, we still have developers who need images in the registry (or even deployed in K8s) for debugging bugs…

To summarize the market situation: the functions available in container registries do not provide sufficient flexibility in cleanup, and the main reason is — there is no way to interact with the outside world.It turns out that teams requiring such flexibility have to implement image deletion "externally" using the Docker Registry API (or the native API of the respective implementation).

However, we were looking for a universal solution that would automate image cleanup for different teams using different registries…

Our Path to Universal Image Cleanup

Where does this need come from? The fact is that we are not just a separate group of developers, but a team that serves multiple such groups, helping to comprehensively address CI/CD issues. And the main technical tool for this is an Open Source utility. werfIts feature is that it does not perform a single function but supports continuous delivery processes at all stages: from building to deployment.

Publishing images in the registry* (immediately after they are built) is an obvious function of such a utility. And since images are stored there, if your storage is not unlimited, you also need to take responsibility for their subsequent cleanup. We will discuss how we succeeded in this while meeting all specified criteria.

* Although the registries themselves can vary (Docker Registry, GitLab Container Registry, Harbor, etc.), their users face the same problems. In our case, the universal solution is independent of the registry implementation, as it operates outside the registries themselves and offers the same behavior for all.

Although we use werf as an example of implementation, we hope that the approaches used will be useful to other teams facing similar challenges.

So, we set about external implementation of a mechanism for image cleanup — instead of the capabilities already built into the registries for containers. The first step was to use the Docker Registry API to create the same basic policies regarding the number of tags and their creation time (mentioned above). An allow list based on images used in the deployed infrastructure was added,, i.e. Kubernetes. For the latter, it was sufficient to iterate through all deployed resources via the Kubernetes API and obtain a list of values. image.

Such a trivial solution addressed the most critical problem (criterion #1), but it was only the beginning of our journey to improve the cleanup mechanism. The next — and much more interesting — step was the solution to link published images with Git history.

Tagging Schemes

Initially, we chose an approach where the final image had to store the necessary information for cleanup, and we built the process based on tagging schemes. When publishing an image, the user selected a specific tagging option (git-branch, git-commit or git-tag) and used the corresponding value. In CI systems, setting these values was done automatically based on environment variables. Essentially the final image was linked to a specific Git primitive, storing the necessary data for cleanup in labels.

Within this approach, a set of policies emerged that allowed us to use Git as the single source of truth:

When a branch/tag was deleted in Git, any associated images in the registry were automatically removed.
The number of images linked to Git tags and commits could be regulated by the number of tags used in the chosen scheme and the creation time of the related commit.

Overall, the resulting implementation met our needs, but soon we faced a new challenge. During our use of tagging schemes based on Git primitives, we encountered a number of drawbacks. (As their description goes beyond the scope of this article, anyone interested can find more details here.) Thus, having decided to move to a more efficient approach to tagging (content-based tagging), we had to reconsider the implementation of image cleanup.

The new algorithm

Why? In content-based tagging, each tag can satisfy multiple commits in Git. With image cleanup, we could no longer rely only on the commit during which the new tag was added to the registry.

For the new cleanup algorithm, it was decided to move away from tagging schemes and build the process on meta-images, each of which stores a link of:

the commit during which publishing took place (regardless of whether the image changed, was added, or remained the same in the container registry);
and our internal identifier corresponding to the built image.

In other words, a connection was established between the published tags and commits in Git.

The final configuration and general algorithm

Users configuring the cleanup now have access to policies that determine the selection of relevant images. Each such policy is defined by:

a set of references, i.e., Git tags or Git branches that are used during scanning;
and a limit on the number of images searched for each reference from the set.

For illustration — here's how the default policy configuration looks:

cleanup:
  keepPolicies:
  - references:
      tag: /.*//
      limit:
        last: 10
  - references:
      branch: /.*//
      limit:
        last: 10
        in: 168h
        operator: And
    imagesPerReference:
      last: 2
      in: 168h
      operator: And
  - references:  
      branch: /^(main|staging|production)$/
    imagesPerReference:
      last: 10

This configuration contains three policies that adhere to the following rules:

Keep an image for the 10 most recent Git tags (based on the tag creation date).
Keep at most 2 images published in the last week for no more than 10 branches active in the last week.
Keep 10 images for branches main, staging and production.

The resulting algorithm boils down to the following steps:

Fetching manifests from the container registry.
Excluding images used in Kubernetes, as we have already preselected them by querying the K8s API.
Scanning the Git history and excluding images according to specified policies.
Removing the remaining images.

Returning to our illustration, here's what it looks like with werf:

However, even if you're not using werf, a similar approach to advanced image cleanup — in one implementation or another (according to your preferred image tagging approach) — can also be applied in other systems/utilities. It suffices to keep in mind the issues that arise and find the opportunities in your stack that allow you to integrate their solution most smoothly. We hope that the path we have taken will help you look at your specific case with new details and insights.

Conclusion

Sooner or later, most teams face the problem of registry overflow.
When searching for solutions, it is essential first to define the criteria for the relevance of the image.
Tools offered by popular container registry services enable very simple cleanup processes that do not account for the "external world": images used in Kubernetes and the peculiarities of team workflows.
A flexible and efficient algorithm should understand CI/CD processes and operate not only on Docker image data.