{"id":36754,"date":"2019-10-31T22:13:37","date_gmt":"2019-10-31T19:13:37","guid":{"rendered":"https:\/\/prohoster.info\/blog\/werf-nash-instrument-dlya-ci-cd-v-kubernetes-obzor-i-video-doklada\/"},"modified":"2019-10-31T22:13:37","modified_gmt":"2019-10-31T19:13:37","slug":"werf-nash-instrument-dlya-ci-cd-v-kubernetes-obzor-i-video-doklada","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/werf-nash-instrument-dlya-ci-cd-v-kubernetes-obzor-i-video-doklada","title":{"rendered":"werf is our tool for CI\/CD in Kubernetes (overview and video report)","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>On May 27, in the main hall of the DevOpsConf 2019 conference, part of the festival <noindex><a rel=\"nofollow\" href=\"http:\/\/ritfest.ru\/2019\/\">RIT++ 2019<\/a><\/noindex>, during the \"Continuous Delivery\" section, a presentation titled \"werf \u2014 our tool for CI\/CD in Kubernetes\" was delivered. It discusses the <b>problems and challenges that everyone faces when deploying in Kubernetes<\/b>, as well as nuances that may not be immediately apparent. By examining potential solutions, we demonstrate how this is implemented in the Open Source tool <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/werf\">werf<\/a><\/noindex>.<\/p>\n<p>Since the presentation, our utility (formerly known as dapp) has surpassed a historic milestone of <b>1000 stars on GitHub<\/b> \u2014 we hope that the growing community of its users will simplify life for many DevOps engineers.<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/c2d1ad5133c0de944b60ae37e3dbe598.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSo, let\u2019s present <noindex><a rel=\"nofollow\" href=\"https:\/\/www.youtube.com\/watch?v=cK3ackGUTLw\"><b>the video of the presentation<\/b><\/a><\/noindex> (~47 minutes, much more informative than the article) and the main takeaway from it in text form. Let\u2019s go!<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2>Code Delivery in Kubernetes<\/h2>\n<p>\nThe talk will no longer focus primarily on werf but on CI\/CD in Kubernetes, assuming that our software is packaged in Docker containers <i>(I discussed this in <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/322686\/\">the 2016 presentation<\/a><\/noindex>)<\/i>, and K8s will be used for its launch in production <i>(discussed in <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/331188\/\">2017)<\/a><\/noindex>)<\/i>.<\/p>\n<p>How does delivery look in Kubernetes?<\/p>\n<ul>\n<li> There is a Git repository with the code and instructions for building it. The application is built into a Docker image and published to the Docker Registry.<\/li>\n<li> In the same repository, there are instructions for deploying and running the application. At the deployment stage, these instructions are sent to Kubernetes, which retrieves the necessary image from the registry and launches it.<\/li>\n<li> Additionally, there are usually tests. Some of them can be run when publishing the image. It's also possible (using the same instructions) to deploy a copy of the application (in a separate K8s namespace or separate cluster) and run tests there.<\/li>\n<li> Finally, you need a CI system that receives events from Git (or button clicks) and triggers all defined stages: build, publish, deploy, test.<\/li>\n<\/ul>\n<p>\n<img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/ec0d1fd1bf1a68cca92badf7cf1affe1.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThere are a few important notes here:<\/p>\n<ol>\n<li> Since we have immutable infrastructure <i>(immutable infrastructure)<\/i>, the application image used at all stages (staging, production, etc.) <b>must be the same<\/b>. <i>I discussed this in more detail with examples <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/324274\/\">here<\/a><\/noindex>.<\/i><\/li>\n<li> Since we follow the infrastructure as code <i>(IaC)<\/i>, the application's code, instructions for building and running it must reside <b>in one repository<\/b>. <i>For more details, see the <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/324274\/\">same presentation<\/a><\/noindex>.<\/i><\/li>\n<li> The delivery chain <i>(delivery)<\/i> We usually see it this way: the application is built, tested, and released. <i>(release stage)<\/i> And that's it \u2014 the delivery has occurred. But in reality, the user gets what you've deployed only when you delivered it to production, and when they were able to access it and the production was running. Therefore, I believe that the delivery chain ends <b>do not<\/b> only at the operational stage <b>(run)<\/b> <i>, and to be more precise, even at the moment when the code was removed from production (replacing it with a new one).<\/i>Let's return to the delivery scheme outlined above in Kubernetes: it was invented not just by us, but literally by everyone who dealt with this problem. Essentially, this pattern is now called GitOps.<\/li>\n<\/ol>\n<p>\n(you can read more about the term and the ideas behind it <i>. Let's look at the stages of the scheme. <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/458878\/\">here<\/a><\/noindex>)<\/i>Build stage<\/p>\n<h2>One might think that in 2019 there's not much to say about building Docker images when everyone knows how to write Dockerfiles and run them.<\/h2>\n<p>\nIt may seem that there\u2019s nothing new to say in 2019 about building Docker images when everyone knows how to write Dockerfiles and run <code>docker build<\/code>?.. \u0412\u043e\u0442 \u043d\u044e\u0430\u043d\u0441\u044b, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0445\u043e\u0442\u0435\u043b\u043e\u0441\u044c \u0431\u044b \u043e\u0431\u0440\u0430\u0442\u0438\u0442\u044c \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435:<\/p>\n<ol>\n<li> <b>matters, so use<\/b> multi-stage <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.docker.com\/develop\/develop-images\/multistage-build\/\">to keep only what is truly necessary for the application to work in the image.<\/a><\/noindex>The number of layers<\/li>\n<li> <b>should be minimized by combining commands that are logically connected.<\/b> However, this adds problems <code>RUN<\/code>to debugging,<\/li>\n<li> because when a build fails, you have to find the specific command in the chain that caused the issue. <b>Build speed<\/b>is important because we want to quickly roll out changes and see the results. For instance, we don't want to rebuild dependencies in language libraries with every application build.<\/li>\n<li> <b>Often, one Git repository requires<\/b> many images,<\/li>\n<li> which can be solved with a set of Dockerfiles (or named stages in one file) and a Bash script to build them in sequence. <b>This was just the tip of the iceberg that everyone encounters. But there are other issues, particularly:<\/b>, which can be resolved with a set of Dockerfiles (or named stages in a single file) and a Bash script for their sequential assembly.<\/li>\n<\/ol>\n<p>\nmount something<\/p>\n<ol>\n<li> (for example, cache the result of an apt-type command in an external directory). <b>We want<\/b> (for example, caching the output of commands like apt in a separate directory).<\/li>\n<li> to build without Docker <b>Ansible<\/b> instead of writing it in shell.<\/li>\n<li> to build without Docker <b>Parallel build.<\/b> (why do we need an additional virtual machine to configure everything for this when we already have a Kubernetes cluster where we can run containers?).<\/li>\n<li> <b>Parallel build<\/b>, which can be interpreted in various ways: different commands from a Dockerfile (if using multi-stage), several commits from the same repository, multiple Dockerfiles.<\/li>\n<li> <b>Distributed Build<\/b>: we want to build something in pods that are 'ephemeral', as they lose their cache, which means it needs to be stored somewhere separately.<\/li>\n<li> Finally, I called the pinnacle of wishes <b>automagic<\/b>: it would be ideal to enter the repository, type some command, and receive a ready-made image, built with an understanding of how and what should be done correctly. However, personally, I am not sure that all nuances can be accounted for in that way.<\/li>\n<\/ol>\n<p>\nAnd there are projects:<\/p>\n<ul>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/moby\/buildkit\">moby\/buildkit<\/a><\/noindex> \u2014 a builder from Docker Inc (already integrated into the current versions of Docker), which is trying to solve all these issues;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/GoogleContainerTools\/kaniko\">kaniko<\/a><\/noindex> \u2014 a builder from Google, allowing builds without Docker;<\/li>\n<li> <noindex><a rel=\"nofollow\" href=\"https:\/\/buildpacks.io\/\">Buildpacks.io<\/a><\/noindex> \u2014 an attempt by CNCF to create automagic and, in particular, an interesting rebasing solution for layers;<\/li>\n<li> and a whole bunch of other utilities, such as <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/containers\/buildah\">buildah<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/genuinetools\/img\">genuinetools\/img<\/a><\/noindex>\u2026<\/li>\n<\/ul>\n<p>\n\u2026 and take a look at how many stars they have on GitHub. So, on one hand, <code>docker build<\/code> there is and can do something, but in reality <b>the issue is not completely resolved<\/b> \u2014 evidence of this is the parallel development of alternative builders, each addressing some part of the problems.<\/p>\n<h2>Building in werf<\/h2>\n<p>\nThis is how we got to <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/werf\">werf<\/a><\/noindex> <i>(previously <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/333682\/\">famous<\/a><\/noindex> like dapp)<\/i> \u2014 an open source utility from the company 'Flant' that we've been developing for many years. It all started about 5 years ago with Bash scripts optimizing the assembly of Dockerfiles, and for the last 3 years, there has been a full-fledged development within a single project with its own Git repository <i>(initially in Ruby, then <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/437044\/\">rewritten<\/a><\/noindex> in Go, and at the same time renamed)<\/i>. What build issues are addressed in werf?<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/ab6aa8831b49977a419fd4cc3543eb83.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe problems marked in blue have already been implemented, parallel builds have been made within one host, and the issues highlighted in yellow we plan to complete by the end of summer.<\/p>\n<h2>Publishing stage in the registry (publish)<\/h2>\n<p>\nWe gathered <code>docker push<\/code>\u2026 \u2014 what could be difficult about uploading an image to the registry? And here comes the question: \"What tag should be assigned to the image?\" It arises because we have <b>Gitflow<\/b> (or another Git strategy) and Kubernetes, as the industry strives for what happens in Kubernetes to follow what is done in Git. After all, Git is our single source of truth.<\/p>\n<p>What's difficult about this? <b>Guaranteeing reproducibility<\/b>: from a commit in Git, which is inherently immutable <i>(immutable)<\/i>, to a Docker image that must remain the same.<\/p>\n<p>It is also important for us <b>to determine the origin<\/b>, because we want to know from which commit the application running in Kubernetes was built (so we can do diffs and similar tasks).<\/p>\n<h3>Tagging strategies<\/h3>\n<p>\nThe first is a simple <b>git tag<\/b>. We have a registry with an image tagged as <code>1.0<\/code>. In Kubernetes, there is a stage and production where this image is pushed. In Git, we make commits and at some point place a tag <code>2.0<\/code>. We build it according to the instructions from the repository and store it in the registry with the tag <code>2.0<\/code>. We roll it out to the stage and, if everything is fine, then to production.<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/8545b84bfa63a91773f0d7dc1c2bd49f.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe problem with this approach is that we first tagged it and only then tested and rolled it out. Why? Firstly, it is simply illogical: we are issuing a version of the software that we haven't even tested (we can't do it another way since to test, we need to place a tag). Secondly, this path does not fit with Gitflow.<\/p>\n<p>The second option is <b>git commit + tag<\/b>. In the master branch, there is a tag <code>1.0<\/code>; for it in the registry \u2014 an image deployed on production. Additionally, in the Kubernetes cluster, there are preview and staging environments. Next, we follow Gitflow: in the main development branch (<code>develop<\/code>) we create new features, resulting in a commit with the identifier <code>#c1<\/code>. We build and publish it in the registry using this identifier (<code>#c1<\/code>). With the same identifier, we roll it out to preview. We do the same with commits <code>#c2<\/code> and <code>#c3<\/code>.<\/p>\n<p>When we realize that the features are sufficient, we begin to stabilize everything. In Git, we create a branch <code>release_1.1<\/code> (based on <code>#c3<\/code> from <code>develop<\/code>). We won\u2019t need to build this release since it was done in the previous step. Therefore, we can simply push it to staging. We fix bugs in <code>#c4<\/code> and likewise roll it out to staging. At the same time, development continues in <code>develop<\/code>, where changes are periodically pulled from <code>release_1.1<\/code>. At some point, we end up with a built and pushed commit on staging that we are satisfied with (<code>#c25<\/code>).<\/p>\n<p>Then we perform a merge (with a fast-forward) of the release branch (<code>release_1.1<\/code>) into master. We place a tag with the new version on this commit (<code>1.1<\/code>). But this image is already built in the registry, so to avoid building it again, we simply add a second tag to the existing image (now it has tags in the registry <code>#c25<\/code> and <code>1.1<\/code>). After that, we roll it out to production.<\/p>\n<p>There is a downside that on staging a single image is pushed (<code>#c25<\/code>), while on production \u2014 a sort of other one (<code>1.1<\/code>), but we know that 'physically' it's the same image from the registry.<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/b3fa2c34442aa401d1e7b30eb593be9a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe real downside is that there is no support for merge commits; fast-forward only is required.<\/p>\n<p>We can go further and do a trick\u2026 Let's consider an example of a simple Dockerfile:<\/p>\n<pre><code class=\"plaintext\">FROM ruby:2.3 as assets\nRUN mkdir -p \/app\nWORKDIR \/app\nCOPY . .\/\nRUN gem install bundler &amp;&amp; bundle install\nRUN bundle exec rake assets:precompile\nCMD bundle exec puma -C config\/puma.rb\n\nFROM nginx:alpine\nCOPY --from=assets \/app\/public \/usr\/share\/nginx\/www\/public<\/code><\/pre>\n<p>\nWe'll build a file based on the principle of taking:<\/p>\n<ul>\n<li> SHA256 of the identifiers of used images (<code>ruby:2.3<\/code> and <code>nginx:alpine<\/code>), which are checksums of their contents;<\/li>\n<li> all commands (<code>RUN<\/code>, <code>CMD<\/code> etc.);<\/li>\n<li> SHA256 of the files that were added.<\/li>\n<\/ul>\n<p>\n... and we'll take the checksum (again SHA256) of such a file. This is <b>the signature<\/b> of everything that defines the contents of a Docker image.<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/b2af0b7952eefe26ddbb145955e778e8.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLet's return to the scheme and <b>instead of commits, we will use such signatures<\/b>, i.e., tag images with signatures.<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/115f290bd5951614c041b3d510fae38e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNow, when we need to, for example, merge changes from the release back into master, we can perform a true merge commit: it will have a different identifier but the same signature. With that same identifier, we\u2019ll release the image to production.<\/p>\n<p>The downside is that now it will be impossible to determine which commit was deployed to production \u2014 checksums only work one way. This problem is solved with an additional layer of metadata \u2014 I will explain this further.<\/p>\n<h3>Tagging in werf<\/h3>\n<p>\nIn werf, we have gone even further and are preparing to implement distributed builds with a cache that is not stored on a single machine\u2026 So, we assemble Docker images of two types, which we call <i>stage<\/i> and <i>image<\/i>.<\/p>\n<p>In the Git repository, werf stores specific instructions for building, describing different stages of the build (<i>beforeInstall<\/i>, <i>install<\/i>, <i>beforeSetup<\/i>, <i>setup<\/i>). We build the first stage image with a signature defined as the checksum of the initial steps. Then we add the source code; for the new stage image, we calculate its checksum\u2026 These operations are repeated for all stages, resulting in a set of stage images. We then create the final image that also contains metadata about its origin. And this image we tag in various ways (details later).<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/22df8b6347b45be19ecb889c102b92b5.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLet a new commit be generated afterwards, which only modifies the application code. What will happen? A patch will be created for the code changes, a new stage image will be prepared. Its signature will be defined as the checksum of the old stage image and the new patch. From this image, a new final image will be formed. The same behavior will occur with changes at other stages.<\/p>\n<p>Thus, stage images are a cache that can be stored distributedly, while the images created from them are uploaded to the Docker Registry.<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/35805eca81605bd64c6910b7965b567e.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Cleaning the registry<\/h3>\n<p>\nThis is not about deleting layers that remain dangling after deleted tags \u2014 this is a standard capability of Docker Registry itself. It\u2019s about the situation where many Docker tags accumulate, and we understand that some of them are no longer needed, yet they occupy space (and\/or we pay for it).<\/p>\n<p>What cleaning strategies are there?<\/p>\n<ol>\n<li> One can simply do nothing <b>not clean<\/b>. Sometimes it is indeed easier to pay a bit for extra space than to untangle a huge mess of tags. But this only works up to a certain point.<\/li>\n<li> <b>Complete reset<\/b>. If all images are deleted and only the current ones are rebuilt in the CI system, a problem may arise. If a container on production restarts, it will pull a new image \u2014 one that has not been tested by anyone. This undermines the idea of immutable infrastructure.<\/li>\n<li> <b>Blue-green<\/b>. One registry started to fill up \u2014 we upload images to another. The same problem as in the previous method: at what point can we clean the registry that started to overflow?<\/li>\n<li> <b>By time<\/b>. Remove all images older than 1 month? But there will definitely be a service that has not been updated for a whole month...<\/li>\n<li> <b>Manually<\/b> determine what can already be deleted.<\/li>\n<\/ol>\n<p>\nThere are truly viable options: either do not clean or a combination of blue-green + manual. In the latter case, when you realize it\u2019s time to clean the registry, you create a new one and add all new images to it over the course of, say, a month. After a month, you check which pods in Kubernetes are still using the old registry and move them to the new registry as well.<\/p>\n<p>Where we ended up in <b>werf<\/b>? \u041c\u044b \u0441\u043e\u0431\u0438\u0440\u0430\u0435\u043c:<\/p>\n<ol>\n<li> Git head: all tags, all branches \u2014 assuming that everything tagged in Git is needed in the images (and if not, it should be removed in Git itself);<\/li>\n<li> all pods that are currently being pulled in Kubernetes;<\/li>\n<li> old ReplicaSets (what was recently pulled), as well as we plan to scan Helm releases and select the latest images there.<\/li>\n<\/ol>\n<p>\n\u2026 and we create a whitelist from this set \u2014 a list of images that we will not delete. Everything else is cleaned up, after which we find orphaned stage images and remove them as well.<\/p>\n<h2>Deployment stage<\/h2>\n<p><\/p>\n<h3>Reliable declarativity<\/h3>\n<p>\nThe first point to draw attention to in deployment is the rollout of an updated resource configuration, declared declaratively. The original YAML document describing Kubernetes resources often differs significantly from the result that actually works in the cluster. This is because Kubernetes adds to the configuration:<\/p>\n<ol>\n<li> identifiers;<\/li>\n<li> metadata;<\/li>\n<li> numerous default values;<\/li>\n<li> a section with the current status;<\/li>\n<li> changes made during the admission webhook operation;<\/li>\n<li> the results of various controllers' (and scheduler's) work.<\/li>\n<\/ol>\n<p>\nTherefore, when a new resource configuration (<i>new<\/i>) appears, we cannot simply overwrite the current, 'live' configuration (<i>live<\/i>). For this, we need to compare <i>new<\/i> with the previously applied configuration (<i>\"last-applied\"<\/i>) and apply the obtained patch. <i>live<\/i> This approach is called<\/p>\n<p>2-way merge. <b>It is used, for example, in Helm.<\/b>There is also a<\/p>\n<p>3-way merge, <b>which differs in that:<\/b>when comparing,<\/p>\n<ul>\n<li> we look at what has been removed; <i>\"last-applied\"<\/i> and <i>new<\/i>we look at what has been added or changed;<\/li>\n<li> we look at what has been removed; <i>new<\/i> and <i>live<\/i>the summarized patch is applied to<\/li>\n<li> We deploy over 1000 applications with Helm, so we effectively live with 2-way merge. However, it has a number of issues that we resolved with our patches, helping Helm function properly. <i>live<\/i>.<\/li>\n<\/ul>\n<p>\nWe deploy over 1000 applications with Helm, so we effectively operate with a 2-way merge. However, it has a number of issues that we have resolved with our patches, helping Helm to work properly.<\/p>\n<h3>After our CI system generates a new configuration for Kubernetes upon a recurring event, it passes it for application<\/h3>\n<p>\n(apply) <i>to the cluster \u2014 using Helm or<\/i> . The described N-way merge then occurs, to which the Kubernetes API responds positively to the CI system, and it to its user. <code>kubectl apply<\/code>However, there is a huge problem: the<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/fcc8251525f32c913f30fbcdc4296198.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nsuccessful application does not mean a successful rollout. <b>If Kubernetes understands what changes need to be applied and applies them \u2014 we still do not know what the outcome will be. For example, updating and restarting pods in the frontend may succeed, while in the backend it may not, and we end up with different versions of the application images running.<\/b>. If Kubernetes understands what changes need to be applied, it applies them \u2014 we still don't know what the result will be. For example, updating and restarting pods in the frontend may succeed, while in the backend it may not, leading to different versions of the running application images.<\/p>\n<p>To do everything correctly, an additional link is suggested in this scheme \u2014 a special tracker that will receive status information from the Kubernetes API and pass it on for further analysis of the actual situation. We created an Open Source library in Go \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/kubedog\"><b>kubedog<\/b><\/a><\/noindex> <i>(see its announcement <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/434160\/\">here<\/a><\/noindex>)<\/i>, \u2014 which solves this problem and is integrated into werf.<\/p>\n<p>The behavior of this tracker at the werf level is configured using annotations applied to Deployments or StatefulSets. The main annotation is <code>fail-mode<\/code> \u2014 which understands the following values:<\/p>\n<ul>\n<li> <code>IgnoreAndContinueDeployProcess<\/code> \u2014 ignore rollout issues of this component and continue the deployment;<\/li>\n<li> <code>FailWholeDeployProcessImmediately<\/code> \u2014 an error in this component stops the deployment process;<\/li>\n<li> <code>HopeUntilEndOfDeployProcess<\/code> \u2014 hope that this component will function by the end of the deployment.<\/li>\n<\/ul>\n<p>\nFor example, such a combination of resources and annotation values <code>fail-mode<\/code>:<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/40e161710e8a535cd95f1eb66ca8407b.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nWhen we deploy for the first time, the database (MongoDB) may not be ready yet \u2014 the Deployments may fail. But we can wait for it to start, and the deployment will still succeed.<\/p>\n<p>There are also two more annotations for kubedog in werf:<\/p>\n<ul>\n<li> <code>failures-allowed-per-replica<\/code> \u2014 the number of allowed failures per replica;<\/li>\n<li> <code>show-logs-until<\/code> \u2014 regulates the moment until which werf shows (in stdout) logs from all the rolling out pods. By default, this is <code>PodIsReady<\/code> (to ignore messages that are unlikely to be needed when traffic starts coming to the pod), however, values <code>ControllerIsReady<\/code> and <code>EndOfDeploy<\/code>.<\/li>\n<\/ul>\n<p><\/p>\n<h3>What else do we want from the deployment?<\/h3>\n<p>\nIn addition to the two points already described, we would like to:<\/p>\n<ul>\n<li> see <b>logs<\/b> \u2014 specifically the relevant ones, not all logs;<\/li>\n<li> track <b>progress<\/b>, because if a job is \"silently\" hanging for several minutes, it is important to understand what is happening;<\/li>\n<li> have <b>an automatic rollback<\/b> in case something goes wrong (and therefore it\u2019s critical to know the real status of the deployment). The rollout should be atomic: either it completes successfully or everything reverts to its previous state.<\/li>\n<\/ul>\n<p><\/p>\n<h2>Summary<\/h2>\n<p>\nAs a company, to implement all the nuances described at different stages of delivery (build, publish, deploy), we only need a CI system and the utility <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/flant\/werf\">werf<\/a><\/noindex>.<\/p>\n<p>In conclusion:<\/p>\n<p><img decoding=\"async\" alt=\"werf is our tool for CI\/CD in Kubernetes (overview and video report)\" src=\"\/wp-content\/uploads\/2019\/08\/bafba54f2df8740a1a12c93b3476e49a.jpeg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nWith werf, we have made significant progress in solving a large number of issues faced by DevOps engineers, and we would be happy if a broader community at least tries this utility in action. Achieving good results together will be easier.<\/p>\n<h2>Videos and slides<\/h2>\n<p>\nVideo of the presentation (~47 minutes):<\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"cK3ackGUTLw\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/cK3ackGUTLw\/hqdefault.jpg\" alt=\"Play video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<p>Presentation of the report:<\/p>\n<p><center><iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"\/\/speakerdeck.com\/player\/2033277984c04900b18940588edf1161\" frameborder=\"0\" allowfullscreen><\/iframe><\/center><\/p>\n<h2>P.S.<\/h2>\n<p>\nOther reports about Kubernetes on our blog:<\/p>\n<ul>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/459326\/\">Auto-scaling and Resource Management in Kubernetes<\/a><\/noindex>\u00bb <i>(Dmitry Stolyarov; April 27, 2019 at \"Stachka\")<\/i>;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/449096\/\">Expanding and Supplementing Kubernetes<\/a><\/noindex>\u00bb <i>(Andrey Polovov; April 8, 2019 at Saint HighLoad++)<\/i>;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/431500\/\">Databases and Kubernetes<\/a><\/noindex>\u00bb <i>(Dmitry Stolyarov; November 8, 2018 at HighLoad++)<\/i>;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/412901\/\">Monitoring and Kubernetes<\/a><\/noindex>\u00bb <i>(Dmitry Stolyarov; May 28, 2018, at RootConf)<\/i>;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/345116\/\">Best Practices for CI\/CD with Kubernetes and GitLab<\/a><\/noindex>\u00bb <i>(Dmitry Stolyarov; November 7, 2017, at HighLoad++)<\/i>;<\/li>\n<li> \u00ab<noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/331188\/\">Our experience with Kubernetes in small projects<\/a><\/noindex>\u00bb <i>(Dmitry Stolyarov; June 6, 2017, at RootConf)<\/i>.<\/li>\n<\/ul>\n<p>Source: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/460351\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>27 \u043c\u0430\u044f \u0432 \u0433\u043b\u0430\u0432\u043d\u043e\u043c \u0437\u0430\u043b\u0435 \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u0438 DevOpsConf 2019, \u043f\u0440\u043e\u0445\u043e\u0434\u044f\u0449\u0435\u0439 \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0444\u0435\u0441\u0442\u0438\u0432\u0430\u043b\u044f \u0420\u0418\u0422++ 2019, \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0441\u0435\u043a\u0446\u0438\u0438 \u00ab\u041d\u0435\u043f\u0440\u0435\u0440\u044b\u0432\u043d\u0430\u044f \u043f\u043e\u0441\u0442\u0430\u0432\u043a\u0430\u00bb, \u043f\u0440\u043e\u0437\u0432\u0443\u0447\u0430\u043b \u0434\u043e\u043a\u043b\u0430\u0434 \u00abwerf \u2014 \u043d\u0430\u0448 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0434\u043b\u044f CI\/CD \u0432 Kubernetes\u00bb. \u0412 \u043d\u0451\u043c \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442\u0441\u044f \u043e \u0442\u0435\u0445 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0445 \u0438 \u0432\u044b\u0437\u043e\u0432\u0430\u0445, \u0441 \u043a\u043e\u0442\u043e\u0440\u044b\u043c\u0438 \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u0435\u0442\u0441\u044f \u043a\u0430\u0436\u0434\u044b\u0439 \u043f\u0440\u0438 \u0434\u0435\u043f\u043b\u043e\u0435 \u0432 Kubernetes, \u0430 \u0442\u0430\u043a\u0436\u0435 \u043e \u043d\u044e\u0430\u043d\u0441\u0430\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u043e\u0433\u0443\u0442 \u0431\u044b\u0442\u044c \u0437\u0430\u043c\u0435\u0442\u043d\u044b \u043d\u0435 \u0441\u0440\u0430\u0437\u0443. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":27531,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-36754","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"27 \u043c\u0430\u044f \u0432 \u0433\u043b\u0430\u0432\u043d\u043e\u043c \u0437\u0430\u043b\u0435 \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u0438 DevOpsConf 2019, \u043f\u0440\u043e\u0445\u043e\u0434\u044f\u0449\u0435\u0439 \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0444\u0435\u0441\u0442\u0438\u0432\u0430\u043b\u044f \u0420\u0418\u0422++ 2019, \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0441\u0435\u043a\u0446\u0438\u0438 \u00ab\u041d\u0435\u043f\u0440\u0435\u0440\u044b\u0432\u043d\u0430\u044f \u043f\u043e\u0441\u0442\u0430\u0432\u043a\u0430\u00bb, \u043f\u0440\u043e\u0437\u0432\u0443\u0447\u0430\u043b.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/werf-nash-instrument-dlya-ci-cd-v-kubernetes-obzor-i-video-doklada\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"en_US\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47werf \u2014 \u043d\u0430\u0448 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0434\u043b\u044f CI\/CD \u0432 Kubernetes (\u043e\u0431\u0437\u043e\u0440 \u0438 \u0432\u0438\u0434\u0435\u043e \u0434\u043e\u043a\u043b\u0430\u0434\u0430) | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"27 \u043c\u0430\u044f \u0432 \u0433\u043b\u0430\u0432\u043d\u043e\u043c \u0437\u0430\u043b\u0435 \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u0438 DevOpsConf 2019, \u043f\u0440\u043e\u0445\u043e\u0434\u044f\u0449\u0435\u0439 \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0444\u0435\u0441\u0442\u0438\u0432\u0430\u043b\u044f \u0420\u0418\u0422++ 2019, \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0441\u0435\u043a\u0446\u0438\u0438 \u00ab\u041d\u0435\u043f\u0440\u0435\u0440\u044b\u0432\u043d\u0430\u044f \u043f\u043e\u0441\u0442\u0430\u0432\u043a\u0430\u00bb, \u043f\u0440\u043e\u0437\u0432\u0443\u0447\u0430\u043b.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/werf-nash-instrument-dlya-ci-cd-v-kubernetes-obzor-i-video-doklada\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T19:13:37+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T19:13:37+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47werf \u2014 our tool for CI\/CD in Kubernetes (overview and presentation video) | ProHoster","description":"On May 27, in the main hall of the DevOpsConf 2019 conference, which took place as part of the RIT++ 2019 festival, during the \"Continuous Delivery\" section, it was announced.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/werf-nash-instrument-dlya-ci-cd-v-kubernetes-obzor-i-video-doklada","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47werf \u2014 \u043d\u0430\u0448 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442 \u0434\u043b\u044f CI\/CD \u0432 Kubernetes (\u043e\u0431\u0437\u043e\u0440 \u0438 \u0432\u0438\u0434\u0435\u043e \u0434\u043e\u043a\u043b\u0430\u0434\u0430) | ProHoster","og:description":"27 \u043c\u0430\u044f \u0432 \u0433\u043b\u0430\u0432\u043d\u043e\u043c \u0437\u0430\u043b\u0435 \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u0438 DevOpsConf 2019, \u043f\u0440\u043e\u0445\u043e\u0434\u044f\u0449\u0435\u0439 \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0444\u0435\u0441\u0442\u0438\u0432\u0430\u043b\u044f \u0420\u0418\u0422++ 2019, \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0441\u0435\u043a\u0446\u0438\u0438 \u00ab\u041d\u0435\u043f\u0440\u0435\u0440\u044b\u0432\u043d\u0430\u044f \u043f\u043e\u0441\u0442\u0430\u0432\u043a\u0430\u00bb, \u043f\u0440\u043e\u0437\u0432\u0443\u0447\u0430\u043b.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/werf-nash-instrument-dlya-ci-cd-v-kubernetes-obzor-i-video-doklada","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:13:37+00:00","article:modified_time":"2019-10-31T19:13:37+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"36754","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 04:44:19","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:39:23","updated":"2026-01-22 04:44:19","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/36754","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=36754"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/36754\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media\/27531"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=36754"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=36754"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=36754"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}