Docker-in-Docker is a virtualized Docker daemon environment running within a container to build container images. The primary aim of creating Docker-in-Docker was to assist in the development of Docker itself. Many people use it to run Jenkins CI. At first, this seems fine, but problems arise that can be avoided by installing Docker in the Jenkins CI container. This article explains how to do that. If you are interested in the final solution without details, just read the last section of the article titled 'Problem Solution.'
Docker-in-Docker: 'Good'
More than two years ago, I inserted into Docker āprivileged and wrote . The goal was to assist the core team in developing Docker faster. Before Docker-in-Docker, the typical development cycle was like this:
- hackity hack;
- build;
- stop the running Docker daemon;
- start a new Docker daemon;
- test;
- loop again.
If you wanted to create a neat, reproducible build (that is, in a container), it became more complex:
- hackity hack;
- ensure that a working version of Docker was running;
- build new Docker with old Docker;
- stop the Docker daemon;
- start a new Docker daemon;
- test;
- stop the new Docker daemon;
- repeat.
With the advent of Docker-in-Docker, the process became simpler:
- hackity hack;
- build + run in one step;
- loop again.
Isn't it much better this way?
Docker-in-Docker: 'Bad'
However, contrary to popular belief, Docker-in-Docker is not made up of 100% stars, ponies, and unicorns. I mean, there are several issues that developers need to be aware of.
One of them concerns LSM (Linux Security Modules) like AppArmor and SELinux: when running a container, the 'inner Docker' may try to apply security profiles that may conflict or confuse the 'outer Docker.' This is the most complex issue that needed to be addressed when attempting to integrate the original implementation of the --privileged flag. My changes worked, and all tests would have passed on my Debian machine and the test virtual machines on Ubuntu, but they would collapse and burn on Michael Crosby's machine (as far as I remember, he had Fedora). I can't recall the exact reason for the problem, but it might have been because Mike is a wise man who works with SELINUX=enforce (I used AppArmor), and my changes did not take SELinux profiles into account.
Docker-in-Docker: 'Evil'
The second issue is related to the storage drivers of Docker. When you run Docker-in-Docker, the outer Docker operates over the regular filesystem (EXT4, BTRFS, or whatever you have available), while the inner Docker runs over a copy-on-write filesystem (AUFS, BTRFS, Device Mapper, etc., depending on what the outer Docker is configured to use). This creates many combinations that wonāt work. For example, you cannot run AUFS on top of AUFS.
If you run BTRFS on top of BTRFS, it should initially work, but once nested subvolumes appear, you wonāt be able to delete the parent subvolume. The Device Mapper module lacks namespaces, so if multiple Docker instances use it on the same machine, they will all be able to see (and influence) each otherās images and the containerās backup devices. This is bad.
There are workarounds for many of these problems. For example, if you want to use AUFS in the inner Docker, just turn the folder /var/lib/docker into that, and everything will be fine. Docker has added some basic namespaces to target Device Mapper names, so if multiple Docker calls are executed on one machine, they wonāt 'step' on each other.
However, such a setup is not at all simple, as can be seen from these in the dind repository on GitHub.
Docker-in-Docker: it gets even worse
What about the build cache? This can also be quite complicated. People often ask me, 'If I run Docker-in-Docker, how can I use images located on my host instead of pulling everything again in my inner Docker?'
Some enterprising individuals have tried to bind /var/lib/docker from the host to the Docker-in-Docker container. Sometimes they share /var/lib/docker among multiple containers.
Want to corrupt your data? Because that's exactly what will damage your data!
The Docker daemon was clearly designed to have exclusive access to /var/lib/docker. Nothing else should 'touch, poke, or probe' any Docker files in this folder.
Why is that? Because it's the result of one of the hardest lessons learned while developing dotCloud. The dotCloud container engine worked with multiple processes simultaneously accessing /var/lib/dotcloud. Clever tricks, like atomic file replacements (instead of in-place edits), 'peppering' code with recommended and required locks, and other experiments with safe systems like SQLite and BDB, didnāt always work. When we redesigned our container engine, which eventually became Docker, one of the main design decisions was to centralize all container operations under a single daemon to put an end to all this nonsense of concurrent access.
Donāt get me wrong: it is entirely possible to do something good, reliable, and fast that involves multiple processes and modern concurrent management. But we think it's simpler and easier to write and maintain code using Docker as a single player.
This means that if you share the /var/lib/docker directory between multiple Docker instances, you will run into issues. Of course, it might work, especially in the early stages of testing. 'Hey, Mom, I can run Ubuntu using Docker!' But try doing something more complex, like pulling the same image from two different instances, and you'll see the world burn.
This means that if your CI system runs builds and rebuilds, every time you restart a Docker-in-Docker container, you risk dropping a nuclear bomb in its cache. That's not cool at all!
Troubleshooting
Let's take a step back. Do you really need Docker-in-Docker, or do you just want to be able to run Docker, namely build and run containers and images from your CI system, while that CI system itself is running in a container?
I bet most people want the latter option, meaning they want a CI system like Jenkins to be able to run containers. And the easiest way to do this is to simply mount the Docker socket into your CI container using the -v flag.
In simpler terms, when you run your CI container (Jenkins or another), instead of hacking something together with Docker-in-Docker, start it with the line:
docker run -v /var/run/docker.sock:/var/run/docker.sock ...Now this container will have access to the Docker socket and, therefore, will be able to run containers. Except that instead of launching 'child' containers, it will launch 'sibling' containers.
Try this using the official Docker image (which contains the Docker binary):
docker run -v /var/run/docker.sock:/var/run/docker.sock
-ti dockerThis looks and works like Docker-in-Docker, but it's not Docker-in-Docker: when this container creates additional containers, they will be created in the upper-level Docker. You won't experience the nesting side effects, and the build cache will be shared across multiple calls.
Note: previous versions of this article recommended binding the Docker binary from the host to the container. This has now become unreliable, as the Docker mechanism no longer extends to static or nearly static libraries.
Thus, if you want to use Docker from Jenkins CI, you have 2 options:
install the Docker CLI using the base image packaging system (i.e., if your image is based on Debian, use .deb packages), or use the Docker API.
A little advertisement š
Thank you for staying with us. Do you enjoy our articles? Want to see more interesting content? Support us by placing an order or recommending us to your friends, , a unique entry-level server alternative that we have created for you: (options available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).
Dell R730xd at half the price in the Equinix Tier IV data center in Amsterdam? Only with us in the Netherlands! Dell R420 ā 2x E5-2430 2.2GHz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB ā from $99! Read about how
Source: habr.com
