{"id":36736,"date":"2019-10-31T22:13:26","date_gmt":"2019-10-31T19:13:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/istoriya-problemy-perenosa-docker-storage-docker-root\/"},"modified":"2019-10-31T22:13:26","modified_gmt":"2019-10-31T19:13:26","slug":"istoriya-problemy-perenosa-docker-storage-docker-root","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","title":{"rendered":"History of the problem with transferring docker storage (docker root)","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

A couple of days ago, it was decided to move the docker storage (the directory where docker keeps all container files and images) to a separate partition on one of the servers that
\nhad greater capacity. The task seemed trivial and did not forecast any trouble\u2026
\n<\/a><\/noindex>
\nLet's get started:<\/p>\n

1. Stop and kill all the containers of our application: <\/p>\n

docker-compose down<\/code><\/pre>\n
\nif there are many containers and they are in different compose files, you can do it this way: <\/p>\n
docker rm -f $(docker ps -q)<\/code><\/pre>\n
\n2. Stop the docker daemon: <\/p>\n
systemctl stop docker<\/code><\/pre>\n
\n3. Move the directory to the desired location: <\/p>\n
cp -r \/var\/lib\/docker \/docker\/data\/storage<\/code><\/pre>\n
\n4. Inform the docker daemon to look at the new directory. There are several options: either specify the new path to the daemon with the -g flag, or use the systemd configs that we used. Or create a symlink. I won't go into great detail on this, as there are plenty<\/a><\/noindex> of manuals online about moving docker root to a new location.<\/p>\n
5. Start the docker daemon and check that it points to the correct location: <\/p>\n
systemctl status docker<\/code><\/pre>\n
\nIn one of the output lines, we should see:<\/p>\n
\u251c\u250019493 \/usr\/bin\/dockerd --data-root=\/docker\/data\/storage<\/code><\/pre>\n
\nWe confirmed that the option was passed to the daemon; now let's check if it was applied (thanks to inkvizitor68sl<\/a><\/noindex>)!<\/p>\n
docker info | awk '\/Root Dir\/ {print $NF}' <\/code><\/pre>\n
\n6. Start our application: <\/p>\n
docker-compose up -d<\/code><\/pre>\n
\n7. Check<\/p>\n
And this is where things get interesting: the DB, MQ, everything is fine! The database is intact, everything works\u2026 except for nginx. We have our own build of nginx with Kerberos and other features. And reviewing the container logs indicated that it could not write to \/var\/tmp \u2014 Permission denied. I am rubbing my temples and trying to analyze the situation\u2026 How could this happen? The docker image hasn\u2019t changed. We just moved the directory. It always worked and then suddenly\u2026 As an experiment, I entered the container and changed the permissions on this directory from root, root 755<\/u>to root, root 777.<\/u>And everything started working\u2026 A thought crossed my mind \u2014 this doesn't make sense\u2026 I thought, maybe I missed something\u2026 <\/p>\n
I decided that we mishandled file permissions during the transfer. We stopped the application, the docker daemon, deleted the new directory, and copied the \/var\/lib\/docker directory again using rsync -a<\/code>. <\/p>\n
I think now everything should be fine; we\u2019ll bring up docker and the application.<\/p>\n
And the problem remains... My eye has started twitching. I rushed to the console of my virtual machine, where I run various tests. I had this nginx image, and I accessed the container, and there in the \/var\/tmp directory the permissions are set to root, root 777. That is, the same permissions I had to set manually. But the images are identical! <\/p>\n
The xfs file system was used everywhere.<\/u><\/p>\n
I compared using the command <\/p>\n
docker inspect my-nginx:12345<\/code><\/pre>\n
\nAll the hashes are identical, exactly the same. Both on the server and on my virtual machine. I deleted the local nginx image and pulled it again from the registry, which for various reasons is on the same machine. And the problem persists\u2026 Now my other eye has started twitching.<\/p>\n
I no longer remember what thoughts were in my head, besides screams of 'AAAAAA!' and other sounds. It's 4 AM outside, and I've gone through the Docker sources to understand the hashing principle of the image layers. I've opened the third energy drink. And in the end, it dawned on me that hashing only considers the file and its content, but NOT PERMISSIONS!<\/b>! So, for some mysterious reason, our permissions got messed up, and selinux is disabled, acl is not used, and there is no sticky bit. <\/p>\n
I deleted the local image, also deleted the image from the docker registry and pushed it again. And everything worked. So, it turns out that when transferring, permissions got corrupted, both inside the local image and in the image lying in the registry. As I already said, for various reasons it was located on the same machine. And consequently in the same directory \/var\/lib\/docker.<\/p>\n
Anticipating the question of whether we tried to revert Docker's gaze to the old directory \u2014 no, we didn't try, sadly, circumstances didn't allow. And I really wanted to figure it out.<\/p>\n
After writing this article, the solution to the problem seems obvious, but at the time it didn't seem so. I honestly googled and found no similar situations.<\/p>\n
In conclusion: I solved the problem, but I still don't understand the cause =(<\/p>\n
If anyone has any ideas or possible causes for this problem \u2014 I would be very grateful for your comments!
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041d\u0435 \u0434\u0430\u043b\u0435\u0435, \u0447\u0435\u043c \u043f\u0430\u0440\u0443 \u0434\u043d\u0435\u0439 \u043d\u0430\u0437\u0430\u0434 \u0431\u044b\u043b\u043e \u0440\u0435\u0448\u0435\u043d\u043e \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0438\u0437 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0432\u044b\u043d\u0435\u0441\u0442\u0438 docker storage (\u043a\u0430\u0442\u0430\u043b\u043e\u0433, \u0433\u0434\u0435 \u0434\u043e\u043a\u0435\u0440 \u0445\u0440\u0430\u043d\u0438\u0442 \u0432\u0441\u0435 \u0444\u0430\u0439\u043b\u044b \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u043e\u0431\u0440\u0430\u0437\u043e\u0432) \u043d\u0430 \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u0434\u0435\u043b, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043e\u0431\u043b\u0430\u0434\u0430\u043b \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0435\u043c\u043a\u043e\u0441\u0442\u044c\u044e. \u0417\u0430\u0434\u0430\u0447\u0430, \u043a\u0430\u0437\u0430\u043b\u043e\u0441\u044c \u0431\u044b, \u0442\u0440\u0438\u0432\u0438\u0430\u043b\u044c\u043d\u0430\u044f \u0438 \u043d\u0435 \u043f\u0440\u0435\u0434\u0432\u0435\u0449\u0430\u043b\u0430 \u0431\u0435\u0434\u044b\u2026 \u041f\u0440\u0438\u0441\u0442\u0443\u043f\u0430\u0435\u043c: 1. \u041e\u0441\u0442\u0430\u043d\u0430\u0432\u043b\u0438\u0432\u0430\u0435\u043c \u0438 \u0443\u0431\u0438\u0432\u0430\u0435\u043c \u0432\u0441\u0435 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u043d\u0430\u0448\u0435\u0433\u043e \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f: docker-compose down \u0435\u0441\u043b\u0438 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 \u043c\u043d\u043e\u0433\u043e, \u0438 \u043e\u043d\u0438 \u0432 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-36736","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47The story of the docker storage move issue (docker root) | ProHoster","description":"Just a couple of days ago, it was decided to move the Docker storage (the directory where Docker keeps all container files and images) to a separate partition with greater capacity.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0418\u0441\u0442\u043e\u0440\u0438\u044f \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u044b \u043f\u0435\u0440\u0435\u043d\u043e\u0441\u0430 docker storage (docker root) | ProHoster","og:description":"\u041d\u0435 \u0434\u0430\u043b\u0435\u0435, \u0447\u0435\u043c \u043f\u0430\u0440\u0443 \u0434\u043d\u0435\u0439 \u043d\u0430\u0437\u0430\u0434 \u0431\u044b\u043b\u043e \u0440\u0435\u0448\u0435\u043d\u043e \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0438\u0437 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0432\u044b\u043d\u0435\u0441\u0442\u0438 docker storage (\u043a\u0430\u0442\u0430\u043b\u043e\u0433, \u0433\u0434\u0435 \u0434\u043e\u043a\u0435\u0440 \u0445\u0440\u0430\u043d\u0438\u0442 \u0432\u0441\u0435 \u0444\u0430\u0439\u043b\u044b \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u043e\u0431\u0440\u0430\u0437\u043e\u0432) \u043d\u0430 \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u0434\u0435\u043b, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043e\u0431\u043b\u0430\u0434\u0430\u043b \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0435\u043c\u043a\u043e\u0441\u0442\u044c\u044e.","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:13:26+00:00","article:modified_time":"2019-10-31T19:13:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"36736","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 04:38:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:39:24","updated":"2026-01-22 04:38:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/36736","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=36736"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/36736\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=36736"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=36736"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=36736"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}