Historien om problemet med å flytte docker-lagring (docker root)

For just a couple of days ago, it was decided to move the docker storage (the directory where docker keeps all container files and images) to a separate partition on one of the servers, which
had a larger capacity. The task seemed trivial and didn’t promise any trouble...

Let's get started:

1. Stop and kill all the containers of our application:

docker-compose down

if there are many containers, and they are in different compose files, you can do it like this:

docker rm -f $(docker ps -q)

2. Stop the docker daemon:

systemctl stop docker

3. Move the directory to the desired location:

cp -r /var/lib/docker /docker/data/storage

4. Inform the docker daemon to look in the new directory. There are several options: either specify the new path through the -g flag, or use the systemd configs that we used. Or a symlink. I won’t elaborate on this much, as the internet is full of manuals on moving the docker root to a new location.

5. Start the docker daemon and observe that it looks where it should:

systemctl status docker

In one of the lines of the output, we should see:

├─19493 /usr/bin/dockerd --data-root=/docker/data/storage

We’ve confirmed that we passed the option to the daemon, now let’s check if it applied it (thanks inkvizitor68sl)!

docker info | awk '/Root Dir/ {print $NF}' 

6. Start our application:

docker-compose up -d

7. Check

And here’s where things start to get interesting, the DB, MQ, all good! The database is intact, everything works... except nginx. We have our own build of nginx with Kerberos and courtesans. And looking at the container logs indicated that it couldn’t write to /var/tmp — Permission denied. I’m rubbing my temples and trying to analyze the situation... How could this happen? The docker image didn’t change. We just moved the directory. It always worked, and now this... Just for experiment, I entered the container myself and changed the permissions on that directory, it was root, root 755, gave root, root 777. And everything got started... A thought echoed in my mind — this is nonsense... I thought, maybe I overlooked something...

I decided that we messed up the file permissions during the transfer. We stopped the application, the docker daemon, deleted the new directory and copied the directory /var/lib/docker using rsync -a.

I thought, now everything should be fine, let’s start docker and the application.

Hmm… the problem remains… My eye is twitching. I rushed to the console of my virtual machine, where I run various tests. I had this nginx image, and I went inside the container, and here in the directory /var/tmp the permissions are set to root, root 777. That is, the same ones I had to set manually. But the images are identical!

The filesystem xfs was used everywhere.

I compared using the command

docker inspect my-nginx:12345

All hashes are identical, exactly the same. On the server as well as on my virtual machine. I deleted the local nginx image and pulled it again from the registry, which for various reasons is on the same machine. And the problem persists… Now my second eye is twitching.

I no longer remember what thoughts were in my head, aside from screams of ‘AAAAAA’ and the like. It’s 4 AM outside, and I started looking at the Docker sources to understand the layer image hashing principles. Opened the third can of energy drink. And ultimately, it dawned on me that hashing considers only the file, its content, but ACCESS RIGHTS! In some mysterious way, our permissions got mixed up, even though selinux is disabled, acl is not used, and there is no sticky bit.

I deleted the local image, also removed the image from the Docker registry, and pushed it again. And everything worked. It turns out that during the transfer the permissions got mixed up, both inside the local image and inside the image stored in the registry. As I mentioned, for various reasons it was located on this same machine. And consequently, in the same directory /var/lib/docker.

Anticipating the question, did we try to revert Docker's gaze to the old directory — no, we did not try, unfortunately, the circumstances did not allow. And I really wanted to figure it out.

After writing this article, the solution to the problem seems obvious, but at the time of troubleshooting it did not seem so. I honestly Googled and did not find similar situations.

In conclusion: I solved the problem, but still don't understand the reason =(

If anyone has any ideas or possible insights about the causes of this issue — I would be extremely grateful for your comments!

Kilde: habr.com