May 27 in the main hall of the DevOpsConf 2019 conference, held as part of the festival , within the framework of the section "Continuous Delivery", a report was made "werf - our tool for CI / CD in Kubernetes". It talks about those problems and challenges that everyone faces when deploying to Kubernetes, as well as about the nuances that may not be immediately noticeable. Analyzing possible solutions, we show how this is implemented in the Open Source tool .
Since the presentation, our utility (formerly known as dapp) has crossed a historical milestone in 1000 stars on GitHub - We hope that the growing community of its users will make life easier for many DevOps engineers.
So, we present (~47 minutes, much more informative than the article) and the main extract from it in text form. Go!
Delivering Code to Kubernetes
The speech in the report will no longer be about werf, but about CI / CD in Kubernetes, implying that our software is packaged in Docker containers (I talked about this in ), and K8s will be used to run it in production (more on this in ).
What does delivery look like in Kubernetes?
- There is a Git repository with code and instructions for building it. The application is built into a Docker image and published to the Docker Registry.
- The same repository has instructions on how to deploy and run the application. At the deployment stage, these instructions are sent to Kubernetes, which receives the desired image from the registry and runs it.
- Plus, there are usually tests. Some of these can be performed when publishing an image. It is also possible (following the same instructions) to deploy a copy of the application (in a separate K8s namespace or a separate cluster) and run tests there.
- Finally, we need a CI system that receives events from Git (or button presses) and calls all the designated stages: build, publish, deploy, test.
There are a few important notes here:
- Since we have an immutable infrastructure (immutable infrastructure), application image that is used at all stages (staging, production, etc.), should be alone. I talked about this in more detail and with examples. .
- Since we are following infrastructure as code approach (IAC), application code, instructions for its assembly and launch should be exactly in one repository. For more on this, see .
- Delivery chain (delivery) we usually see it like this: the application was assembled, tested, released (release stage) and that's it - delivery happened. But in reality, the user gets what you rolled out, not when you delivered it to production, and when he was able to go there and this production worked. So I believe the delivery chain is ending only during operation (run), or more precisely, even at the moment when the code was removed from production (replacing it with a new one).
Let's return to the Kubernetes delivery scheme indicated above: it was not only invented by us, but literally everyone who dealt with this problem. In fact, this pattern is now called GitOps (For more on the term and the ideas behind it, see ). Let's look at the stages of the scheme.
Build stage
It would seem that in 2019, what can be said about building Docker images, when everyone knows how to write Dockerfiles and run docker build?.. Here are the nuances that I would like to pay attention to:
- Image weight matters, so use to leave in the image only what is really needed for the application to work.
- Number of layers must be minimized by combining chains from
RUN- commands by meaning. - However, this adds to the problem debugging, because when the assembly falls, you have to find the right command from the chain that caused the problem.
- Assembly speed is important because we want to quickly roll out changes and look at the result. For example, you don't want to rebuild dependencies in language libraries every time you build an application.
- Often, from one Git repository, you need many images, which can be solved by a set of Dockerfiles (or named stages in one file) and a Bash script with their sequential assembly.
This was just the tip of the iceberg that everyone faces. But there are other problems, and in particular:
- Often at the assembly stage we need something mount (for example, cache the result of a command like apt'a in a third-party directory).
- We want Ansible instead of writing in shell.
- We want build without Docker (why do we need an additional virtual machine in which we need to configure everything for this when there is already a Kubernetes cluster in which containers can be run?).
- Parallel Assembly, which can be understood in different ways: different commands from the Dockerfile (if multi-stage is used), several commits of the same repository, several Dockerfiles.
- Distributed assembly: we want to collect something in pods that are "ephemeral" because they lose the cache, which means that it must be stored somewhere separately.
- Finally, I named the pinnacle of desires automagic: it would be ideal to go to the repository, type some command and get a ready-made image, assembled with an understanding of how and what to do correctly. However, personally I'm not sure that all the nuances can be foreseen in this way.
And here are the projects:
- - a collector from Docker Inc (already integrated into current versions of Docker), which is trying to solve all these problems;
- - a Google builder that allows you to build without Docker;
- - an attempt by CNCF to make automagic and, in particular, an interesting solution with rebase for layers;
- and a bunch of other utilities like , ...
β¦ and see how many stars they have on GitHub. That is, on the one hand, docker build is and can do something, but in reality something the issue is not fully resolved - proof of this is the parallel development of alternative assemblers, each of which solves some part of the problems.
Assembly in werf
So we got to (earlier like dapp) β Open Source-utility company "Flant", which we have been doing for many years. It all started 5 years ago with Bash scripts that optimize the assembly of Dockerfiles, and for the last 3 years full-fledged development has been carried out within the same project with its own Git repository (first in Ruby, and then on Go, and at the same time renamed). What build issues are resolved in werf?
The issues highlighted in blue have already been implemented, the parallel build has been done within the same host, and the issues highlighted in yellow are planned to be completed by the end of the summer.
Publishing stage in the registry (publish)
scored docker push... - what can be difficult about uploading an image to the registry? And then the question arises: βWhat tag to put on the image?β It arises for the reason that we have gitflow (or other Git strategy) and Kubernetes, and the industry wants to ensure that what happens in Kubernetes follows what happens in Git. After all, Git is our only source of truth.
What's so hard about that? Guarantee reproducibility: from a commit in Git that is inherently immutable (immutable), to the Docker image, which should be kept the same.
It is also important for us determine the origin, because we want to understand what commit the application running in Kubernetes was built from (then we can do diffs and things like that).
Tagging strategies
The first one is simple git day. We have a registry with an image tagged as 1.0. Kubernetes has stage and production, where this image is downloaded. In Git we make commits and at some point put a tag 2.0. We collect it according to the instructions from the repository and place it in the registry with the tag 2.0. We roll out to the stage and, if everything is fine, then to production.
The problem with this approach is that we first put the tag, and only then tested and rolled it out. Why? Firstly, it's simply illogical: we give out a version to software that we haven't even checked yet (we can't do otherwise, because in order to check, you need to put a tag). Secondly, this path is not compatible with Gitflow.
The second option - git commit + tag. The master branch has a tag 1.0; for it in the registry - an image deployed to production. In addition, the Kubernetes cluster has preview and staging loops. Next we follow Gitflow: upstream for development (develop) we make new features, as a result of which a commit with an identifier appears #c1. We collect it and publish it in the registry using this identifier (#c1). With the same identifier, we roll out to preview. Do the same with commits #c2 ΠΈ #c3.
When we realized that there are enough features, we begin to stabilize everything. Create a branch in Git release_1.1 (on the base #c3 of develop). You won't need to compile this release, because. this was done in the previous step. Therefore, we can simply roll it out to staging. Fixing bugs in #c4 and similarly roll out to staging. At the same time, the development of develop, where changes are periodically taken from release_1.1. At some point, we get a compiled and rolled out commit for staging, which we are satisfied with (#c25).
Then we merge (with fast-forward) the release branch (release_1.1) in master. We put a tag with the new version on this commit (1.1). But this image is already built in the registry, so in order not to build it again, we just add a second tag to the existing image (now it has tags in the registry #c25 ΠΈ 1.1). After that, we roll it out to production.
There is a drawback that one image is rolled out for staging (#c25), and on production - as it were different (1.1), but we know that "physically" this is the same image from the registry.
The real disadvantage is that there is no support for merge commits, you need to do fast-forward.
You can go ahead and do the trick... Consider a simple Dockerfile example:
FROM ruby:2.3 as assets
RUN mkdir -p /app
WORKDIR /app
COPY . ./
RUN gem install bundler && bundle install
RUN bundle exec rake assets:precompile
CMD bundle exec puma -C config/puma.rb
FROM nginx:alpine
COPY --from=assets /app/public /usr/share/nginx/www/publicLet's build a file from it according to the principle that we take:
- SHA256 from the identifiers of the used images (
ruby:2.3ΠΈnginx:alpine), which are checksums of their contents; - all commands (
RUN,CMDetc.); - SHA256 from files that were added.
... and take the checksum (again SHA256) from such a file. This signature anything that defines the contents of the Docker image.
Let's go back to the diagram and instead of commits, we will use such signatures, i.e. tag images with signatures.
Now, when we need to 'merge' changes from a release to master, for example, we can do a real merge commit: it will have a different ID but the same signature. With the same identifier, we will also roll out the image to production.
The disadvantage is that now it will not be possible to determine what kind of commit was rolled out to production - checksums work only in one direction. This problem is solved by an additional layer with metadata - I'll tell you more later.
Tagging in werf
In werf, we have gone even further and are preparing to make a distributed build with a cache that is not stored on one machine ... So, we are building two types of Docker images, we call them training ΠΈ image.
The werf Git repository stores specific build instructions that describe the different build steps (beforeInstall, install, beforeSetup, setup). We build the first stage image with a signature defined as the checksum of the first steps. Then we add the source code, for the new stage image we calculate its checksum ... These operations are repeated for all stages, as a result of which we get a set of stage images. Then we make the final image-image, which also contains metadata about its origin. And we already tag this image in different ways (details later).
Suppose after that a new commit appears, in which only the application code has been changed. What will happen? For code changes, a patch will be created, a new stage image will be prepared. Its signature will be defined as the checksum of the old stage image and the new patch. A new final image-image will be formed from this image. Similar behavior will occur when changes are made in other stages.
Thus, stage images are a cache that can be stored distributed, and image images already created from it are loaded into the Docker Registry.
Registry cleaning
This is not about removing layers that remained hanging after the removed tags - this is a standard feature of the Docker Registry itself. We are talking about a situation where a lot of Docker tags accumulate and we understand that we no longer need some of them, but they take up space (and / or we pay for it).
What are the cleaning strategies?
- Maybe just nothing don't clean. Sometimes it's really easier to pay a little for extra space than to unravel a huge tangle of tags. But this only works up to a certain point.
- Full reset. If you delete all images and rebuild only those that are relevant in the CI system, then a problem may arise. If the container is restarted in production, a new image will be loaded for it - one that has not yet been tested by anyone. This kills the idea of ββan immutable infrastructure.
- blue-green. One registry began to overflow - we load images into another. The same problem as in the previous method: at what point can the registry that started to overflow be cleared?
- By time. Delete all images older than 1 month? But there will definitely be a service that has not been updated for a whole month ...
- Manually determine what can be removed.
There are two really viable options: no cleaning, or a combination of blue-green + manually. In the latter case, we are talking about the following: when you understand that it is time to clean the registry, create a new one and add all new images to it for, for example, a month. And after a month, see which pods in Kubernetes are still using the old registry, and transfer them to the new registry too.
What have we come to yard? We collect:
- Git head: all tags, all branches, - assuming that everything that is tagged in Git, we need in images (and if not, then we need to delete it in Git itself);
- all pods that are currently rolled out to Kubernetes;
- old ReplicaSets (what was recently downloaded), and we also plan to scan Helm releases and select the latest images there.
... and make a whitelist from this set - a list of images that we will not delete. We clean everything else, after which we find the orphan stage images and delete them too.
Deploy stage
Reliable declarativeness
The first point that I would like to pay attention to in the deployment is the rollout of the updated resource configuration, declared declaratively. The original YAML document describing Kubernetes resources is always very different from the result that actually works in the cluster. Because Kubernetes adds to the configuration:
- identifiers;
- official information;
- set of default values;
- section with current status;
- changes made as part of the admission webhook;
- the result of the work of various controllers (and the scheduler).
So when a new resource configuration appears (new), we cannot just take and overwrite the current, βliveβ configuration with it (live). To do this, we have to compare new with the last applied configuration (last applied) and roll onto live received patch.
This approach is called 2 way merge. It is used, for example, in Helm.
There is also 3 way merge, which is different in that:
- comparing last applied ΠΈ new, we look at what has been removed;
- comparing new ΠΈ live, we look at what has been added or changed;
- the summed patch is applied to live.
We deploy 1000+ applications with Helm, so we actually live with 2-way merge. However, it has a number of problems that we have solved with our patches that help Helm to work normally.
Real rollout status
After the next event, our CI system has generated a new configuration for Kubernetes, it passes it on for use (apply) to the cluster - using Helm or kubectl apply. Next, the already described N-way merge takes place, to which the Kubernetes API responds approvingly to the CI system, and that system to its user.
However, there is a huge problem: successful application does not mean successful rollout. If Kubernetes understands what changes need to be applied, applies it - we still don't know what the result will be. For example, updating and restarting pods in the frontend may succeed, but not in the backend, and we will get different versions of the running application images.
In order to do everything right, this scheme suggests an additional link - a special tracker that will receive status information from the Kubernetes API and transmit it for further analysis of the real state of affairs. We have created an Open Source library in Go - (see her announcement ), which solves this problem and is built into werf.
The behavior of this tracker at the werf level is configured using annotations that are placed on Deployments or StatefulSets. Main annotation β fail-mode understands the following meanings:
-
IgnoreAndContinueDeployProcess- we ignore the problems of rolling out this component and continue the deployment; -
FailWholeDeployProcessImmediately- an error in this component stops the deployment process; -
HopeUntilEndOfDeployProcess- we hope that this component will work by the end of the deployment.
For example, such a combination of resources and annotation values fail-mode:
When we deploy for the first time, the database (MongoDB) may not be ready yet - Deployments will fall. But you can wait for the moment for it to start, and the deployment will still pass.
There are two more annotations for kubedog in werf:
-
failures-allowed-per-replica- the number of allowed falls for each replica; -
show-logs-until- regulates the moment until which werf shows (in stdout) logs from all rolled out pods. The default isPodIsReady(to ignore messages, which we don't really want when the pod starts getting traffic), but values ββare also validControllerIsReadyΠΈEndOfDeploy.
What else do we want from deployment?
In addition to the two points already described, we would like to:
- see logs - and only the necessary ones, and not all in a row;
- track progress, because if the job "silently" hangs for several minutes, it is important to understand what is happening there;
- have automatic rollback in case something went wrong (and therefore it is critical to know the real status of the deployment). The rollout must be atomic: either it runs to the end, or everything returns to its previous state.
Results
We, as a company, need a CI system and a utility to implement all the described nuances at different stages of delivery (build, publish, deploy). .
Instead of a conclusion:
With the help of werf, we have made good progress in solving a lot of problems for DevOps engineers, and we would be glad if the wider community at least tried this tool in action. It will be easier to achieve a good result together.
Videos and slides
Video from the performance (~47 minutes):
Report presentation:
PS
Other reports about Kubernetes in our blog:
- «» (Dmitry Stolyarov; April 27, 2019 at Stachka);
- «» (Andrey Polovov; April 8, 2019 on Saint HighLoad++);
- «» (Dmitry Stolyarov; November 8, 2018 on HighLoad++);
- «» (Dmitry Stolyarov; May 28, 2018 at RootConf);
- «» (Dmitry Stolyarov; November 7, 2017 on HighLoad++);
- «» (Dmitry Stolyarov; June 6, 2017 at RootConf).
Source: habr.com