History of the problem with transferring docker storage (docker root)

A couple of days ago, it was decided to move the docker storage (the directory where docker keeps all container files and images) to a separate partition on one of the servers that
had greater capacity. The task seemed trivial and did not forecast any trouble…

Let's get started:

1. Stop and kill all the containers of our application:

docker-compose down

if there are many containers and they are in different compose files, you can do it this way:

docker rm -f $(docker ps -q)

2. Stop the docker daemon:

systemctl stop docker

3. Move the directory to the desired location:

cp -r /var/lib/docker /docker/data/storage

4. Inform the docker daemon to look at the new directory. There are several options: either specify the new path to the daemon with the -g flag, or use the systemd configs that we used. Or create a symlink. I won't go into great detail on this, as there are plenty of manuals online about moving docker root to a new location.

5. Start the docker daemon and check that it points to the correct location:

systemctl status docker

In one of the output lines, we should see:

├─19493 /usr/bin/dockerd --data-root=/docker/data/storage

We confirmed that the option was passed to the daemon; now let's check if it was applied (thanks to inkvizitor68sl)!

docker info | awk '/Root Dir/ {print $NF}' 

6. Start our application:

docker-compose up -d

7. Check

And this is where things get interesting: the DB, MQ, everything is fine! The database is intact, everything works… except for nginx. We have our own build of nginx with Kerberos and other features. And reviewing the container logs indicated that it could not write to /var/tmp — Permission denied. I am rubbing my temples and trying to analyze the situation… How could this happen? The docker image hasn’t changed. We just moved the directory. It always worked and then suddenly… As an experiment, I entered the container and changed the permissions on this directory from root, root 755to root, root 777.And everything started working… A thought crossed my mind — this doesn't make sense… I thought, maybe I missed something…

I decided that we mishandled file permissions during the transfer. We stopped the application, the docker daemon, deleted the new directory, and copied the /var/lib/docker directory again using rsync -a.

I think now everything should be fine; we’ll bring up docker and the application.

And the problem remains... My eye has started twitching. I rushed to the console of my virtual machine, where I run various tests. I had this nginx image, and I accessed the container, and there in the /var/tmp directory the permissions are set to root, root 777. That is, the same permissions I had to set manually. But the images are identical!

The xfs file system was used everywhere.

I compared using the command

docker inspect my-nginx:12345

All the hashes are identical, exactly the same. Both on the server and on my virtual machine. I deleted the local nginx image and pulled it again from the registry, which for various reasons is on the same machine. And the problem persists… Now my other eye has started twitching.

I no longer remember what thoughts were in my head, besides screams of 'AAAAAA!' and other sounds. It's 4 AM outside, and I've gone through the Docker sources to understand the hashing principle of the image layers. I've opened the third energy drink. And in the end, it dawned on me that hashing only considers the file and its content, but NOT PERMISSIONS!! So, for some mysterious reason, our permissions got messed up, and selinux is disabled, acl is not used, and there is no sticky bit.

I deleted the local image, also deleted the image from the docker registry and pushed it again. And everything worked. So, it turns out that when transferring, permissions got corrupted, both inside the local image and in the image lying in the registry. As I already said, for various reasons it was located on the same machine. And consequently in the same directory /var/lib/docker.

Anticipating the question of whether we tried to revert Docker's gaze to the old directory — no, we didn't try, sadly, circumstances didn't allow. And I really wanted to figure it out.

After writing this article, the solution to the problem seems obvious, but at the time it didn't seem so. I honestly googled and found no similar situations.

In conclusion: I solved the problem, but I still don't understand the cause =(

If anyone has any ideas or possible causes for this problem — I would be very grateful for your comments!

Source: habr.com