{"id":96065,"date":"2020-10-07T13:42:09","date_gmt":"2020-10-07T11:42:09","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/problema-umnoj-ochistki-obrazov-kontejnerov-i-eyo-reshenie-v-werf"},"modified":"2020-10-07T13:42:09","modified_gmt":"2020-10-07T11:42:09","slug":"problema-umnoj-ochistki-obrazov-kontejnerov-i-eyo-reshenie-v-werf","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/problema-umnoj-ochistki-obrazov-kontejnerov-i-eyo-reshenie-v-werf","title":{"rendered":"The problem of 'intelligent' container image cleanup and its solution in werf","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"The
\n
\nThis article addresses the issue of cleaning up images that accumulate in container registries (Docker Registry and its analogs) in the context of modern CI\/CD pipelines for cloud-native applications deployed in Kubernetes. It outlines the main criteria for the relevance of images and the resulting challenges in automating cleanup, saving space, and meeting team needs. Finally, using a specific Open Source project as an example, we will discuss how these challenges can be overcome.<\/p>\n

Introduction<\/h2>\n

\nThe number of images in a container registry can rapidly grow, taking up more storage space and consequently significantly increasing costs. To control, limit, or maintain acceptable growth of the space occupied in the registry, it is common to:<\/p>\n

    \n
  1. use a fixed number of tags for images;<\/li>\n
  2. somehow clean up images.<\/li>\n<\/ol>\n

    <\/a><\/noindex>
    \nThe first limitation is sometimes acceptable for small teams. If developers can make do with permanent tags (latest<\/code>, main<\/code>, test<\/code>, boris<\/code> , etc.), the registry will not bloat in size, and there may be little need to think about cleanup for a long time. After all, all outdated images are overwritten, and there\u2019s simply no work left for cleanup (everything is handled by the built-in garbage collector).<\/p>\n

    However, this approach severely limits development and is rarely applicable to the CI\/CD of modern projects. An integral part of development has become automation<\/strong>, which allows for much faster testing, deploying, and delivering new functionality to users. For example, in all our projects, a CI pipeline is automatically created with each commit. In it, an image is built, tested, deployed to various Kubernetes environments for debugging and remaining checks, and if everything goes well, the changes reach the end user. This has long ceased to be rocket science; for many, it\u2019s a routine matter\u2014likely for you as well, since you\u2019re reading this article.<\/p>\n

    As bug fixes and new functionality development occur simultaneously, and releases can happen several times a day, it is clear that the development process is accompanied by a significant number of commits, which means\u2014 a large number of images in the registry.<\/strong>As a result, the question of organizing effective registry cleanup, i.e., removing outdated images, becomes pressing.<\/p>\n

    But how can one even determine if an image is relevant?<\/p>\n

    Relevance Criteria for Images<\/h2>\n

    \nIn the vast majority of cases, the main criteria will be as follows:<\/p>\n

    1. The first (most obvious and critical of all) \u2014 these are the images that are currently in use in Kubernetes.<\/strong>Removing these images can lead to significant costs due to production downtime (for example, images may be required for replication) or negate the efforts of the team working on debugging in any of the environments. (For this reason, we even created a special <\/i>Prometheus exporter<\/i><\/a><\/noindex>, which monitors the absence of such images in any Kubernetes cluster.)<\/i><\/p>\n

    2. The second (less obvious, but also very important and again related to operation) \u2014 images that are needed for rollback in case serious issues are detected<\/strong> in the current version. For example, with Helm, these are images used in saved release versions. (By the way, by default, Helm has a limit of 256 revisions, but it's unlikely that anyone actually needs to keep that<\/i> many versions..) After all, we store versions specifically so that they can be used later, i.e., to \"rollback\" to them when necessary.<\/p>\n

    3. The third \u2014 developer needs:<\/strong>all images related to their current work. For example, if we are considering a PR, it makes sense to keep the image that corresponds to the latest commit and, say, the previous commit: this way, the developer can quickly return to any task and work with the latest changes. <\/p>\n

    4. The fourth \u2014 images that correspond to versions of our application<\/strong>, i.e., are the final product: v1.0.0, 20.04.01, sierra, etc.<\/p>\n

    NB: The criteria outlined here were formulated based on experience working with dozens of development teams from various companies. However, of course, depending on the specifics of development processes and the infrastructure used (for example, if Kubernetes is not used), these criteria may differ. <\/p>\n

    Criteria Compliance and Existing Solutions<\/h2>\n

    \nPopular container registry services typically offer their image cleanup policies: you can specify the conditions under which a tag is removed from the registry. However, the possibilities of these conditions are limited to parameters such as names, creation time, and the number of tags*.<\/p>\n

    * It depends on the specific implementations of the container registry. We considered the capabilities of the following solutions: Azure CR, Docker Hub, ECR, GCR, GitHub Packages, GitLab Container Registry, Harbor Registry, JFrog Artifactory, Quay.io \u2014 as of September 2020.<\/i><\/p>\n

    This set of parameters is quite sufficient to meet the fourth criterion \u2014 that is, to filter images that correspond to versions. However, for all other criteria, a compromise solution must be chosen (a stricter or, on the contrary, a more lenient policy) depending on expectations and financial capabilities.<\/p>\n

    For instance, the third criterion \u2014 related to developers' needs \u2014 can be solved by organizing processes within teams: specific naming of images, maintaining special allow lists, and internal agreements. However, in the end, it still needs to be automated. And if the capabilities of ready-made solutions are insufficient, something custom must be created.<\/p>\n

    The situation is similar for the first two criteria: they cannot be satisfied without obtaining data from an external system \u2014 that very system where applications are deployed (in our case, it's Kubernetes).<\/p>\n

    Workflow illustration in Git<\/h3>\n

    \nSuppose you work in Git approximately like this:<\/p>\n

    \"The
    \n
    \nThe icon with a head on the diagram marks the container images currently deployed in Kubernetes for any users (end users, testers, managers, etc.) or used by developers for debugging and similar purposes.<\/i><\/p>\n

    What will happen if the cleanup policies allow keeping (not deleting) images only by specified tag names<\/b>?<\/p>\n

    \"The
    \n
    \nClearly, such a scenario will please no one.<\/p>\n

    What will change if the policies allow not deleting images by a specified time interval \/ number of latest commits<\/b>?<\/p>\n

    \"The
    \n
    \nThe result became significantly better, yet still far from ideal. After all, we still have developers who need images in the registry (or even deployed in K8s) for debugging bugs\u2026<\/p>\n

    To summarize the market situation: the functions available in container registries do not provide sufficient flexibility in cleanup, and the main reason is \u2014 there is no way to interact with the outside world.<\/strong>It turns out that teams requiring such flexibility have to implement image deletion \"externally\" using the Docker Registry API (or the native API of the respective implementation).<\/p>\n

    However, we were looking for a universal solution that would automate image cleanup for different teams using different registries\u2026<\/p>\n

    Our Path to Universal Image Cleanup<\/h2>\n

    \nWhere does this need come from? The fact is that we are not just a separate group of developers, but a team that serves multiple such groups, helping to comprehensively address CI\/CD issues. And the main technical tool for this is an Open Source utility. werf<\/a><\/noindex>Its feature is that it does not perform a single function but supports continuous delivery processes at all stages: from building to deployment.<\/p>\n

    Publishing images in the registry* (immediately after they are built) is an obvious function of such a utility. And since images are stored there, if your storage is not unlimited, you also need to take responsibility for their subsequent cleanup. We will discuss how we succeeded in this while meeting all specified criteria.<\/p>\n

    * Although the registries themselves can vary (Docker Registry, GitLab Container Registry, Harbor, etc.), their users face the same problems. In our case, the universal solution is independent of the registry implementation, as it operates outside the registries themselves and offers the same behavior for all.<\/i><\/p>\n

    Although we use werf as an example of implementation, we hope that the approaches used will be useful to other teams facing similar challenges.<\/p>\n

    So, we set about external<\/i> implementation of a mechanism for image cleanup \u2014 instead of the capabilities already built into the registries for containers. The first step was to use the Docker Registry API to create the same basic policies regarding the number of tags and their creation time (mentioned above). An allow list based on images used in the deployed infrastructure was added,<\/strong>, i.e. Kubernetes. For the latter, it was sufficient to iterate through all deployed resources via the Kubernetes API and obtain a list of values. image<\/code>.<\/p>\n

    Such a trivial solution addressed the most critical problem (criterion #1), but it was only the beginning of our journey to improve the cleanup mechanism. The next \u2014 and much more interesting \u2014 step was the solution to link published images with Git history<\/strong>.<\/p>\n

    Tagging Schemes<\/h3>\n

    \nInitially, we chose an approach where the final image had to store the necessary information for cleanup, and we built the process based on tagging schemes. When publishing an image, the user selected a specific tagging option (git-branch<\/code>, git-commit<\/code> or git-tag<\/code>) and used the corresponding value. In CI systems, setting these values was done automatically based on environment variables. Essentially the final image was linked to a specific Git primitive<\/strong>, storing the necessary data for cleanup in labels.<\/p>\n

    Within this approach, a set of policies emerged that allowed us to use Git as the single source of truth:<\/p>\n