{"id":36736,"date":"2019-10-31T22:13:26","date_gmt":"2019-10-31T19:13:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/istoriya-problemy-perenosa-docker-storage-docker-root\/"},"modified":"2019-10-31T22:13:26","modified_gmt":"2019-10-31T19:13:26","slug":"istoriya-problemy-perenosa-docker-storage-docker-root","status":"publish","type":"post","link":"https:\/\/prohoster.info\/no\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","title":{"rendered":"Historien om problemet med \u00e5 flytte docker-lagring (docker root)","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

For just a couple of days ago, it was decided to move the docker storage (the directory where docker keeps all container files and images) to a separate partition on one of the servers, which
\nhad a larger capacity. The task seemed trivial and didn\u2019t promise any trouble...
\n<\/a><\/noindex>
\nLet's get started:<\/p>\n

1. Stop and kill all the containers of our application: <\/p>\n

docker-compose down<\/code><\/pre>\n
\nif there are many containers, and they are in different compose files, you can do it like this: <\/p>\n
docker rm -f $(docker ps -q)<\/code><\/pre>\n
\n2. Stop the docker daemon: <\/p>\n
systemctl stop docker<\/code><\/pre>\n
\n3. Move the directory to the desired location: <\/p>\n
cp -r \/var\/lib\/docker \/docker\/data\/storage<\/code><\/pre>\n
\n4. Inform the docker daemon to look in the new directory. There are several options: either specify the new path through the -g flag, or use the systemd configs that we used. Or a symlink. I won\u2019t elaborate on this much, as the internet is full of<\/a><\/noindex> manuals on moving the docker root to a new location.<\/p>\n
5. Start the docker daemon and observe that it looks where it should: <\/p>\n
systemctl status docker<\/code><\/pre>\n
\nIn one of the lines of the output, we should see:<\/p>\n
\u251c\u250019493 \/usr\/bin\/dockerd --data-root=\/docker\/data\/storage<\/code><\/pre>\n
\nWe\u2019ve confirmed that we passed the option to the daemon, now let\u2019s check if it applied it (thanks inkvizitor68sl<\/a><\/noindex>)!<\/p>\n
docker info | awk '\/Root Dir\/ {print $NF}' <\/code><\/pre>\n
\n6. Start our application: <\/p>\n
docker-compose up -d<\/code><\/pre>\n
\n7. Check<\/p>\n
And here\u2019s where things start to get interesting, the DB, MQ, all good! The database is intact, everything works... except nginx. We have our own build of nginx with Kerberos and courtesans. And looking at the container logs indicated that it couldn\u2019t write to \/var\/tmp \u2014 Permission denied. I\u2019m rubbing my temples and trying to analyze the situation... How could this happen? The docker image didn\u2019t change. We just moved the directory. It always worked, and now this... Just for experiment, I entered the container myself and changed the permissions on that directory, it was root, root 755<\/u>, gave root, root 777<\/u>. And everything got started... A thought echoed in my mind \u2014 this is nonsense... I thought, maybe I overlooked something... <\/p>\n
I decided that we messed up the file permissions during the transfer. We stopped the application, the docker daemon, deleted the new directory and copied the directory \/var\/lib\/docker using rsync -a<\/code>. <\/p>\n
I thought, now everything should be fine, let\u2019s start docker and the application.<\/p>\n
Hmm\u2026 the problem remains\u2026 My eye is twitching. I rushed to the console of my virtual machine, where I run various tests. I had this nginx image, and I went inside the container, and here in the directory \/var\/tmp the permissions are set to root, root 777. That is, the same ones I had to set manually. But the images are identical! <\/p>\n
The filesystem xfs was used everywhere.<\/u><\/p>\n
I compared using the command <\/p>\n
docker inspect my-nginx:12345<\/code><\/pre>\n
\nAll hashes are identical, exactly the same. On the server as well as on my virtual machine. I deleted the local nginx image and pulled it again from the registry, which for various reasons is on the same machine. And the problem persists\u2026 Now my second eye is twitching.<\/p>\n
I no longer remember what thoughts were in my head, aside from screams of \u2018AAAAAA\u2019 and the like. It\u2019s 4 AM outside, and I started looking at the Docker sources to understand the layer image hashing principles. Opened the third can of energy drink. And ultimately, it dawned on me that hashing considers only the file, its content, but ACCESS RIGHTS<\/b>! In some mysterious way, our permissions got mixed up, even though selinux is disabled, acl is not used, and there is no sticky bit. <\/p>\n
I deleted the local image, also removed the image from the Docker registry, and pushed it again. And everything worked. It turns out that during the transfer the permissions got mixed up, both inside the local image and inside the image stored in the registry. As I mentioned, for various reasons it was located on this same machine. And consequently, in the same directory \/var\/lib\/docker.<\/p>\n
Anticipating the question, did we try to revert Docker's gaze to the old directory \u2014 no, we did not try, unfortunately, the circumstances did not allow. And I really wanted to figure it out.<\/p>\n
After writing this article, the solution to the problem seems obvious, but at the time of troubleshooting it did not seem so. I honestly Googled and did not find similar situations.<\/p>\n
In conclusion: I solved the problem, but still don't understand the reason =(<\/p>\n
If anyone has any ideas or possible insights about the causes of this issue \u2014 I would be extremely grateful for your comments!
\n
Kilde: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041d\u0435 \u0434\u0430\u043b\u0435\u0435, \u0447\u0435\u043c \u043f\u0430\u0440\u0443 \u0434\u043d\u0435\u0439 \u043d\u0430\u0437\u0430\u0434 \u0431\u044b\u043b\u043e \u0440\u0435\u0448\u0435\u043d\u043e \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0438\u0437 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0432\u044b\u043d\u0435\u0441\u0442\u0438 docker storage (\u043a\u0430\u0442\u0430\u043b\u043e\u0433, \u0433\u0434\u0435 \u0434\u043e\u043a\u0435\u0440 \u0445\u0440\u0430\u043d\u0438\u0442 \u0432\u0441\u0435 \u0444\u0430\u0439\u043b\u044b \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u043e\u0431\u0440\u0430\u0437\u043e\u0432) \u043d\u0430 \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u0434\u0435\u043b, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043e\u0431\u043b\u0430\u0434\u0430\u043b \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0435\u043c\u043a\u043e\u0441\u0442\u044c\u044e. \u0417\u0430\u0434\u0430\u0447\u0430, \u043a\u0430\u0437\u0430\u043b\u043e\u0441\u044c \u0431\u044b, \u0442\u0440\u0438\u0432\u0438\u0430\u043b\u044c\u043d\u0430\u044f \u0438 \u043d\u0435 \u043f\u0440\u0435\u0434\u0432\u0435\u0449\u0430\u043b\u0430 \u0431\u0435\u0434\u044b\u2026 \u041f\u0440\u0438\u0441\u0442\u0443\u043f\u0430\u0435\u043c: 1. \u041e\u0441\u0442\u0430\u043d\u0430\u0432\u043b\u0438\u0432\u0430\u0435\u043c \u0438 \u0443\u0431\u0438\u0432\u0430\u0435\u043c \u0432\u0441\u0435 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u043d\u0430\u0448\u0435\u0433\u043e \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f: docker-compose down \u0435\u0441\u043b\u0438 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 \u043c\u043d\u043e\u0433\u043e, \u0438 \u043e\u043d\u0438 \u0432 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-36736","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47The story of the Docker storage transfer problem (Docker root) | ProHoster","description":"Just a couple of days ago, it was decided to move the Docker storage (the directory where Docker stores all container and image files) to a separate partition that had a larger capacity.","canonical_url":"https:\/\/prohoster.info\/no\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"nn_NO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0418\u0441\u0442\u043e\u0440\u0438\u044f \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u044b \u043f\u0435\u0440\u0435\u043d\u043e\u0441\u0430 docker storage (docker root) | ProHoster","og:description":"\u041d\u0435 \u0434\u0430\u043b\u0435\u0435, \u0447\u0435\u043c \u043f\u0430\u0440\u0443 \u0434\u043d\u0435\u0439 \u043d\u0430\u0437\u0430\u0434 \u0431\u044b\u043b\u043e \u0440\u0435\u0448\u0435\u043d\u043e \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0438\u0437 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0432\u044b\u043d\u0435\u0441\u0442\u0438 docker storage (\u043a\u0430\u0442\u0430\u043b\u043e\u0433, \u0433\u0434\u0435 \u0434\u043e\u043a\u0435\u0440 \u0445\u0440\u0430\u043d\u0438\u0442 \u0432\u0441\u0435 \u0444\u0430\u0439\u043b\u044b \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u043e\u0431\u0440\u0430\u0437\u043e\u0432) \u043d\u0430 \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u0434\u0435\u043b, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043e\u0431\u043b\u0430\u0434\u0430\u043b \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0435\u043c\u043a\u043e\u0441\u0442\u044c\u044e.","og:url":"https:\/\/prohoster.info\/no\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:13:26+00:00","article:modified_time":"2019-10-31T19:13:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"36736","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 04:38:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:39:24","updated":"2026-01-22 04:38:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts\/36736","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/comments?post=36736"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts\/36736\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/media?parent=36736"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/categories?post=36736"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/tags?post=36736"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}