Prije nekoliko dana odlučeno je da se na jednom od servera Docker storage (direktorij gdje Docker pohranjuje sve datoteke kontejnera i slika) premjesti na zaseban dio s većim kapacitetom. Zadatak se činilo trivijalnim i nije predodređivao nevolje...
Započinjemo:
1. Zaustavljamo i ubijamo sve kontejnere naše aplikacije:
Ako ima puno kontejnera i nalaze se u različitim compose datotekama, to možemo učiniti ovako:
docker-compose downdocker rm -f $(docker ps -q)
2. Zaustavljamo Docker demon:systemctl stop docker
3. Premještamo direktorij na željeno mjesto:cp -r /var/lib/docker /docker/data/storage
4. Obavještavamo Docker demona da gleda u novi direktorij. Ima nekoliko opcija: ili putem zastavice -g naznačimo demonu novi put, ili koristimo systemd konfiguracije, što smo mi i učinili. Ili simbolički link. Neću previše objašnjavati, na internetuima priručnika o premještanju docker root-a na novo mjesto.
5. Pokrećemo Docker demona i promatramo da gleda kamo treba:
systemctl status dockerU jednom od redova izlaza trebali bismo vidjeti:
├─19493 /usr/bin/dockerd --data-root=/docker/data/storageProvjerili smo da je demonu proslijeđena opcija, sada ćemo provjeriti je li je primijenio (hvala )!
docker info | awk '/Root Dir/ {print $NF}' 6. Pokrećemo našu aplikaciju:
docker-compose up -d7. Provjeravamo
I ovdje počinje najzanimljiviji dio, SGBD, MQ, sve je u redu! Baza je netaknuta, sve radi... osim nginx-a. Imamo vlastitu verziju nginx-a s Kerberosom i kurtižankama. Pregled logova kontejnera pokazao je da ne može pisati u /var/tmp — Permission denied. Zategnem prste i pokušavam analizirati situaciju… Kako to? Docker slika se nije mijenjala. Samo smo premjestili direktorij. Oduvijek je radilo, i sada… Iz eksperimenta sam ručno ušao u kontejner i promijenio dozvole za taj direktorij, bile su root, root 755, postavio sam root, root 777. I sve je proradilo… U glavi se javila misao — kakva glupost… Mislio sam, možda nisam nešto uzeo u obzir…
Odlučio sam da smo izgubili dozvole na datotekama prilikom premještanja. Zaustavili smo aplikaciju, Docker demona, izbrisali novu direktorij i ponovno kopirali direktorij /var/lib/docker koristeći rsync -a.
Mislim, sada je sigurno sve u redu, pokrećemo Docker, aplikaciju.
I… the problem remains… My eye twitched. I rushed to the console of my virtual machine, where I run various tests, I had this nginx image, and I accessed the container, and here in the directory /var/tmp, the permissions are root, root 777. That is, the same ones I had to set manually. But the images are identical!
The xfs filesystem was used everywhere.
I compared it using the command
docker inspect my-nginx:12345All hashes are identical, all exactly the same. On the server and on my virtual machine. I deleted the local nginx image and pulled it again from the registry, which for various reasons is on the same machine. And the problem remains… Now my second eye twitched.
I no longer remember what thoughts were in my head, apart from screams of “AAAAAAA” and the like. It’s 4 AM outside, and the docker sources were needed to understand the principles of image layer hashing. I opened the third can of energy drink. And in the end, it dawned on me that hashing only considers the file, its contents, but NO ACCESS RIGHTS! That is, for some mysterious reason, our permissions were messed up, although selinux is disabled, acl is not used, and there is no sticky bit.
I deleted the local image, also removed the image from the docker registry and pushed it again. And everything worked. It turns out that during the transfer, the permissions were messed up, both inside the local image and inside the image stored in the registry. As I mentioned, for various reasons it was located on the same machine. And consequently in the same directory /var/lib/docker.
And anticipating the question, whether we tried to return docker's gaze to the old directory — no, we didn’t, unfortunately, circumstances did not allow it. And I really wanted to figure it out.
After writing this article, the solution to the problem seems obvious to me, but at the time of troubleshooting, it did not seem so. Honestly googled, and did not find similar situations.
Result: problem solved, the reason still not understood =(
If someone knows or suspects any possible reasons for this problem — I would be extremely grateful in the comments!
Izvor: habr.com
