🥇Continuous Delivery practices with Docker (review and video)

We will start our blog with publications created based on the latest speeches of our technical director distol (Dmitry Stolyarov). All of them took place in 2016 at various professional events and were dedicated to the topic of DevOps and Docker. One video, from the Docker Moscow meeting at the Badoo office, we are already published Online. New ones will be accompanied by articles that convey the essence of the reports. So…

May 31 at the conference RootConf 2016, which was held as part of the Russian Internet Technologies festival (RIT++ 2016), the section "Continuous deployment and deployment" opened with the report "Best practices of Continuous Delivery with Docker". It summarized and systematized the best practices for building a Continuous Delivery (CD) process using Docker and other Open Source products. We work with these solutions in production, which allows us to rely on practical experience.

If you have the opportunity to spend an hour on video with the reportWe encourage you to watch it in its entirety. Otherwise, the main squeeze is presented below in text form.

Continuous Delivery with Docker

Under Continuous Delivery we understand the chain of events, as a result of which the application code from the Git repository first comes to production, and then gets into the archive. It looks like this: Git → Build (assembly) → Test (testing) → Release (release) → Operate (later maintenance).

Most of the report is devoted to the build stage (application assembly), and the release and operate topics are reviewed. We will talk about problems and patterns that allow them to be solved, and specific implementations of these patterns may be different.

Why is Docker needed here at all? It’s not for nothing that we decided to talk about Continuous Delivery practices in the context of this Open Source tool. Although the entire report is devoted to its use, many reasons are revealed already when considering the main pattern of rolling out application code.

Main rollout pattern

So, when rolling out new versions of the application, we will certainly encounter downtime problem, which is formed during the switchover of the production server. Traffic from the old version of the application to the new one cannot be switched instantly: first, we must make sure that the new version is not only successfully downloaded, but also “warmed up” (that is, completely ready to serve requests).

Thus, for some time, both versions of the application (old and new) will work simultaneously. which automatically leads to shared resource conflict: network, file system, IPC, etc. With Docker, this problem is easily solved by running different versions of the application in separate containers, for which resource isolation is guaranteed within the same host (server/virtual machine). Of course, you can get by with some tricks without isolation at all, but if there is a ready-made and convenient tool, then there is a reverse reason - not to neglect it.

Containerization provides many other benefits for deploying. Every application depends on certain version (or version range) interpreter, the presence of modules / extensions, etc., as well as their versions. And this applies not only to the immediate executable environment, but to the entire environment, including system software and its version (up to the Linux distribution used). Due to the fact that containers contain not only application code, but also pre-installed system and application software of the required versions, you can forget about problems with dependencies.

Generalized main rollout pattern new versions, taking into account the following factors:

First, the old version of the application runs in the first container.
Then the new version is rolled out and "warmed up" in the second container. It is noteworthy that this new version itself can carry not only the updated application code, but also any of its dependencies, as well as system components (for example, a new version of OpenSSL or the entire distribution).
When the new version is fully ready to serve requests, traffic switches from the first container to the second.
Now the old version can be stopped.

This approach with deploying different versions of the application in separate containers provides another convenience - fast rollback to the old version (after all, it is enough to switch traffic to the desired container).

The final first recommendation sounds so that even the Captain cannot find fault: “[when organizing Continuous Delivery with Docker] Use Docker [and understand what it gives]". Remember that this is not a “silver bullet” that solves any problems, but a tool that provides a wonderful foundation.

Reproducibility

By "reproducibility" we mean a generalized set of problems encountered in the operation of applications. We are talking about such cases:

Scenarios checked by the quality department for staging must be accurately reproduced in production.
Applications are published on servers that can receive packages from different mirror repositories (over time, they are updated, and with them, the versions of installed applications).
“Everything works for me locally!” (... and developers are not allowed into production.)
It is required to check something in the old (archived) version.
...

Their general essence boils down to the fact that full compliance of the environments used (as well as the absence of the human factor) is necessary. How to guarantee reproducibility? Make Docker images based on code from Git, and then use them for any tasks: on test sites, in production, on local machines of programmers ... At the same time, it is important to minimize the actions that are performed after assembling an image: the simpler, the less likely there are errors.

Infrastructure is code

If the infrastructure requirements (availability of server software, its version, etc.) are not formalized and “programmed”, then the rollout of any application update can end in sad consequences. For example, you have already switched to PHP 7.0 for staging and rewritten the code accordingly - then its appearance in production with some old PHP (5.5) will certainly surprise someone. You may not forget about a major change in the version of the interpreter, but “the devil is in the details”: the surprise can be in a minor update of any dependency.

The approach to solve this problem is known as IaC (Infrastructure as Code, "infrastructure as code") and involves storing the infrastructure requirements along with the application code. Using it, developers and DevOps specialists can work with the same Git repository of the application, but on different parts of it. From this code, a Docker image is created in Git, in which the application is deployed taking into account all the specifics of the infrastructure. Simply put, the scripts (rules) for building images should be in the same repository with the source code and be merged together.

In the case of a layered application architecture - for example, there is nginx that comes in front of an application already running inside a Docker container - Docker images must be created from the code in Git for each layer. Then in the first image there will be an application with an interpreter and other "closest" dependencies, and in the second - the parent nginx.

Docker images, communication with Git

We divide all Docker images built from Git into two categories: temporary and release. Temporary images are tagged by the name of the branch in Git, can be overwritten by the next commit, and are only rolled out for preview (not for production). This is their key difference from release ones: you never know what specific commit is in them.

It makes sense to collect in temporary images: the master branch (you can automatically roll it out to a separate site in order to constantly see the current version of master), branches with releases, branches of specific innovations.

After the preview of temporary images comes to the need for translation in production, the developers put a certain tag. By tag automatically collected release image (its tag corresponds to a tag from Git) and rolls out to staging. If it is successfully checked by the quality department, it goes to production.

dapp

Everything described (rolling out, building images, subsequent maintenance) can be implemented independently using Bash scripts and other "improvised" tools. But if you do this, then at some point the implementation will lead to great complexity and poor manageability. Understanding this, we came to create our own specialized Workflow utility for building CI / CD - dapp.

Its source code is written in Ruby, open and published on GitHub. Unfortunately, the documentation is currently the weakest point of the tool, but we are working on it. And more than once we will write and talk about dapp, because. we sincerely can't wait to share its features with the entire interested community, but in the meantime, send your issues and pull requests and / or follow the development of the project on GitHub.

Updated August 13, 2019: currently a project dapp renamed to yard, its code has been completely rewritten in Go, and its documentation has been greatly improved.

Kubernetes

Another off-the-shelf Open Source tool that has already received significant recognition in the professional environment is Kubernetes, a cluster to manage Docker. The topic of its use in the operation of projects built on Docker is beyond the scope of the report, so the presentation is limited to an overview of some interesting features.

For rolling out Kubernetes offers:

readiness probe - checking the readiness of a new version of the application (to switch traffic to it);
rolling update - sequential image update in a cluster of containers (shutdown, update, preparation for launch, traffic switching);
synchronous update - updating the image in the cluster with a different approach: first on half of the containers, then on the rest;
canary releases - launching a new image on a limited (small) number of containers for monitoring anomalies.

Since Continuous Delivery is not only a release of a new version, Kubernetes has a number of features for the subsequent maintenance of the infrastructure: built-in monitoring and logging across all containers, automatic scaling, etc. All this is already working and is just waiting for proper implementation into your processes.

Final Recommendations

Use docker.
Make application Docker images for all needs.
Follow the principle "Infrastructure is code".
Link Git with Docker.
Regulate the order of rollout.
Use a ready platform (Kubernetes or other).

Videos and slides

Video from the performance (about an hour) published on YouTube (the report itself starts from the 5th minute - follow the link to play from this moment).

Report presentation:

Continuous Delivery practices with Docker (review and video)