Docker-in-Docker is a virtualized Docker daemon running on the container itself to build container images. The main purpose of creating Docker-in-Docker was to help develop Docker itself. Many people use it to run Jenkins CI. This seems normal at first, but then problems arise that can be avoided by installing Docker in the Jenkins CI container. This article explains how to do it. If you are interested in the final solution without details, just read the last section of the Problem Solving article.
Docker-in-Docker: Good
Over two years ago I put in Docker βprivileged and wrote . The goal was to help the core team develop Docker faster. Before Docker-in-Docker, the typical development cycle was:
- hackity hack;
- assembly (build);
- stop running Docker daemon;
- starting a new Docker daemon;
- testing;
- cycle repeat.
If you wanted to make a beautiful, reproducible assembly (that is, in a container), then it became more intricate:
- hackity hack;
- make sure a working version of Docker is running;
- build new Docker with old Docker;
- stop the docker daemon;
- start a new Docker daemon;
- test;
- stop the new Docker daemon;
- repeat.
With the advent of Docker-in-Docker, the process has been simplified:
- hackity hack;
- assembly + launch in one step;
- cycle repeat.
Isn't that so much better?
Docker-in-Docker: "Bad"
However, contrary to popular belief, Docker-in-Docker is not 100% stars, ponies, and unicorns. I mean there are a few issues that a developer needs to be aware of.
One concerns LSMs (Linux Security Modules) such as AppArmor and SELinux: when running a container, the "internal Docker" may try to apply security profiles that will conflict or obfuscate the "external Docker". This is the hardest problem to solve when trying to combine the original implementation of the --privileged flag. My changes worked and all tests would also pass on my Debian machine and test Ubuntu VMs, but they would crash and burn on Michael Crosby's machine (he had Fedora as far as I remember). I can't remember the exact cause of the problem, but it may have been because Mike is a wise man who works with SELINUX=enforce (I used AppArmor) and my changes didn't respect SELinux profiles.
Docker-in-Docker: "Evil"
The second problem is related to Docker storage drivers. When you run Docker-in-Docker, the external Docker runs on top of a regular file system (EXT4, BTRFS, or whatever you have), while the internal Docker runs on top of a copy-on-write system (AUFS, BTRFS, Device Mapper, etc.). , depending on what is configured to use external Docker). In this case, there are many combinations that will not work. For example, you won't be able to run AUFS on top of AUFS.
If you're running BTRFS on top of BTRFS, this should work at first, but once there are nested subvolumes, the parent subvolume can't be deleted. The Device Mapper module is namespaceless, so if multiple Docker instances use it on the same machine, they will all be able to see (and influence) images of each other and container backup devices. This is bad.
There are workarounds to solve many of these problems. For example, if you want to use AUFS in internal Docker, just turn the /var/lib/docker folder into a volume and you'll be fine. Docker has added some base namespaces to the Device Mapper target names so that if multiple Docker calls are made on the same machine, they won't "step on" each other.
However, this setup is not at all easy, as you can see from these in the dind repository on GitHub.
Docker-in-Docker: Getting Worse
What about the build cache? This can be quite tricky too. People often ask me βif I'm running Docker-in-Docker, how can I use the images hosted on my host instead of pulling everything in my internal Docker againβ?
Some enterprising people have tried to link /var/lib/docker from the host to a Docker-in-Docker container. Sometimes they share /var/lib/docker with multiple containers.
Want to corrupt data? Because that's exactly what will corrupt your data!
The docker daemon was clearly designed to have exclusive access to /var/lib/docker. Nothing else should "touch, poke, or feel" any Docker files that are in this folder.
Why is it so? Because it is the result of one of the hardest lessons learned in the development of dotCloud. The dotCloud container engine worked by having multiple processes accessing /var/lib/dotcloud at the same time. Cunning tricks such as atomic file replacement (instead of editing in place), peppering code with advisory and mandatory locks, and other experiments with secure systems like SQLite and BDB didn't always work. When we were redesigning our container engine, which eventually became Docker, one of the main design decisions was to collect all container operations under a single daemon to do away with all this concurrency nonsense.
Don't get me wrong: it's entirely possible to make something nice, reliable, and fast that includes multiple processes and modern parallel control. But we think it's easier and easier to write and maintain code with Docker as the only player.
This means that if you share the /var/lib/docker directory across multiple Docker instances, you will be in trouble. Of course, this can work, especially in the early stages of testing. βListen, Ma, I can run ubuntu with docker!β But try something more complex, like pulling the same image from two different instances, and you'll see the world burn.
This means that if your CI system is doing builds and rebuilds, then every time you restart your Docker-in-Docker container, you run the risk of dropping a nuclear bomb into its cache. It's not cool at all!
The solution
Let's take a step back. Do you really need Docker-in-Docker, or do you just want to be able to run Docker, namely build and run containers and images from your CI system, while that CI system itself is in a container?
I bet most people want the latter option, i.e. they want a CI system like Jenkins to be able to run containers. And the easiest way to do this is to simply insert a Docker socket into your CI container by associating it with the -v flag.
Simply put, when you start your CI container (Jenkins or otherwise), instead of hacking along with Docker-in-Docker, start it with the line:
docker run -v /var/run/docker.sock:/var/run/docker.sock ...This container will now have access to the Docker socket and therefore be able to run containers. Except that instead of launching "child" containers, it will launch "sibling" containers.
Try this using the official docker image (which contains the docker binary):
docker run -v /var/run/docker.sock:/var/run/docker.sock
-ti dockerIt looks and works like Docker-in-Docker, but it's not Docker-in-Docker: when this container creates additional containers, they will be created in top-level Docker. You won't experience the side effects of nesting, and the assembly cache will be shared across multiple calls.
Note: Previous versions of this article advised linking the Docker binary from the host to the container. This has now become unreliable as the Docker engine no longer extends to static or near-static libraries.
So if you want to use Docker from Jenkins CI you have 2 options:
installing the Docker CLI using the basic image packaging system (i.e. if your image is based on Debian, use the .deb packages), using the Docker API.
Some ads π
Thank you for staying with us. Do you like our articles? Want to see more interesting content? Support us by placing an order or recommending to friends, , a unique analogue of entry-level servers, which was invented by us for you: (available with RAID1 and RAID10, up to 24 cores and up to 40GB DDR4).
Dell R730xd 2 times cheaper in Equinix Tier IV data center in Amsterdam? Only here in the Netherlands! Dell R420 - 2x E5-2430 2.2Ghz 6C 128GB DDR3 2x960GB SSD 1Gbps 100TB - from $99! Read about
Source: habr.com