{"id":36736,"date":"2019-10-31T22:13:26","date_gmt":"2019-10-31T19:13:26","guid":{"rendered":"https:\/\/prohoster.info\/blog\/istoriya-problemy-perenosa-docker-storage-docker-root\/"},"modified":"2019-10-31T22:13:26","modified_gmt":"2019-10-31T19:13:26","slug":"istoriya-problemy-perenosa-docker-storage-docker-root","status":"publish","type":"post","link":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","title":{"rendered":"Historien om flytningen af docker storage (docker root)","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

For just a couple of days ago, it was decided to move the docker storage (the directory where Docker keeps all container files and images) to a separate partition on one of the servers, which
\nhad greater capacity. The task seemed trivial and didn\u2019t foresee any troubles...
\n<\/a><\/noindex>
\nWe begin:<\/p>\n

1. Stop and kill all containers of our application: <\/p>\n

docker-compose down<\/code><\/pre>\n
\nif there are many containers and they are in different composes, you can do it like this: <\/p>\n
docker rm -f $(docker ps -q)<\/code><\/pre>\n
\n2. Stop the Docker daemon: <\/p>\n
systemctl stop docker<\/code><\/pre>\n
\n3. Move the directory to the desired location: <\/p>\n
cp -r \/var\/lib\/docker \/docker\/data\/storage<\/code><\/pre>\n
\n4. Inform the Docker daemon to look in the new directory. There are several options: either specify the new path to the daemon with the -g flag, or use systemd configs, which is what we used. Or a symlink. I won\u2019t go into detail here; there are plenty of manuals on the internet about moving the Docker root to a new location.<\/a><\/noindex> 5. Start the Docker daemon and check to see if it\u2019s pointing where it should:<\/p>\n
systemctl status docker <\/p>\n
In one of the output lines, we should see:<\/code><\/pre>\n
\n\u251c\u250019493 \/usr\/bin\/dockerd --data-root=\/docker\/data\/storage<\/p>\n
We confirmed that the option was passed to the daemon, now let's check if it was applied (thanks to<\/code><\/pre>\n
\ninkvizitor68sl docker info | awk '\/Root Dir\/ {print $NF}'<\/a><\/noindex>)!<\/p>\n
6. Start our application: <\/code><\/pre>\n
\n7. Check <\/p>\n
docker-compose up -d<\/code><\/pre>\n
\nAnd here is where the most interesting part begins, the DB, MQ, everything is fine! The database is intact, everything works\u2026 except for nginx. We have our own nginx build with Kerberos and courtesans. And looking at the container logs indicated that it cannot write to \/var\/tmp \u2014 Permission denied. I\u2019m rubbing my temples and trying to analyze the situation\u2026 How could this happen? The Docker image hasn\u2019t changed. We simply moved the directory. It always worked, and now this\u2026 As an experiment, I manually entered the container and changed the permissions of this directory. They were<\/p>\n
root, root 755 , I gave it<\/u>root, root 777 . And everything started working\u2026 A thought echoed in my head \u2014 this is ridiculous\u2026 I thought, maybe I overlooked something\u2026<\/u>I decided that we lost access rights to the files during the transfer. We stopped the application, the Docker daemon, deleted the new directory, and copied the directory \/var\/lib\/docker using <\/p>\n
rsync -a I think now everything is definitely fine, let's start Docker and the application.<\/code>. <\/p>\n
Jeg tror, at alt nu er i orden, vi starter docker, applikationen.<\/p>\n
Og... problemet er stadig der... Mit \u00f8je er begyndt at tr\u00e6kke. Jeg sprang hen til konsollen p\u00e5 min virtuelle maskine, hvor jeg k\u00f8rer forskellige tests. Jeg havde dette nginx-billede, og jeg gik ind i containeren, hvor kataloget \/var\/tmp har rettighederne root, root 777. Alts\u00e5 de samme, som jeg var n\u00f8dt til at s\u00e6tte manuelt. Men billederne er identiske! <\/p>\n
XFS-filsystemet blev brugt overalt.<\/u><\/p>\n
Jeg sammenlignede ved hj\u00e6lp af kommandoen <\/p>\n
docker inspect my-nginx:12345<\/code><\/pre>\n
\nAlle hashes er identiske, alle er helt ens. B\u00e5de p\u00e5 serveren og p\u00e5 min virtuelle maskine. Jeg slettede det lokale nginx-billede og hentede det igen fra registry, som af en r\u00e6kke grunde ligger p\u00e5 den samme maskine. Og problemet er det samme\u2026 Nu begynder mit andet \u00f8je ogs\u00e5 at tr\u00e6kke.<\/p>\n
Jeg kan ikke huske, hvilke tanker der var i mit hoved, ud over skrig som 'AAAAAAA' og lignende. Klokken er 4 om natten, og jeg kiggede p\u00e5 Docker-kildeteksterne i h\u00e5bet om at forst\u00e5 princippet for lagene i billedhastigheden. Jeg \u00e5bnede den tredje energidrik. Og til sidst gik det op for mig, at hashing kun tager h\u00f8jde for filen, dens indhold, men IKKE RETTIGHEDER<\/b>! S\u00e5 p\u00e5 en eller anden mystisk m\u00e5de er vores rettigheder blevet korrupte, sk\u00f8nt selinux er sl\u00e5et fra, acl ikke bruges, og sticky bit ikke er tilg\u00e6ngeligt. <\/p>\n
Jeg slettede ogs\u00e5 det lokale billede, samt billedet fra Docker-registry og pushede det igen. Og s\u00e5 virkede det. Det viser sig, at rettighederne blev korrupte ved overf\u00f8rslen, b\u00e5de inde i det lokale billede og inde i billedet, der l\u00e5 i registry. Som jeg n\u00e6vnte, l\u00e5 det af flere grunde p\u00e5 den samme maskine. Og som et resultat i det samme katalog \/var\/lib\/docker.<\/p>\n
Og forudse sp\u00f8rgsm\u00e5let om vi havde fors\u00f8gt at vende Docker tilbage til det gamle katalog \u2014 nej, vi har ikke pr\u00f8vet, desv\u00e6rre tillod omst\u00e6ndighederne ikke det. Men jeg \u00f8nskede virkelig at forst\u00e5.<\/p>\n
Efter skrivning af denne artikel virker l\u00f8sningen p\u00e5 problemet \u00e5benlys, men p\u00e5 tidspunktet for analysen virkede den ikke s\u00e5dan. \u00c6rligt, jeg googlede, og fandt ikke lignende situationer.<\/p>\n
Resultat: Problemet blev l\u00f8st, men jeg forstod ikke \u00e5rsagen =(<\/p>\n
Hvis nogen har nogle ideer om mulige \u00e5rsager til dette problem \u2014 ville jeg v\u00e6re meget glad for jeres kommentarer!
\n
Kilde: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u041d\u0435 \u0434\u0430\u043b\u0435\u0435, \u0447\u0435\u043c \u043f\u0430\u0440\u0443 \u0434\u043d\u0435\u0439 \u043d\u0430\u0437\u0430\u0434 \u0431\u044b\u043b\u043e \u0440\u0435\u0448\u0435\u043d\u043e \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0438\u0437 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0432\u044b\u043d\u0435\u0441\u0442\u0438 docker storage (\u043a\u0430\u0442\u0430\u043b\u043e\u0433, \u0433\u0434\u0435 \u0434\u043e\u043a\u0435\u0440 \u0445\u0440\u0430\u043d\u0438\u0442 \u0432\u0441\u0435 \u0444\u0430\u0439\u043b\u044b \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u043e\u0431\u0440\u0430\u0437\u043e\u0432) \u043d\u0430 \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u0434\u0435\u043b, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043e\u0431\u043b\u0430\u0434\u0430\u043b \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0435\u043c\u043a\u043e\u0441\u0442\u044c\u044e. \u0417\u0430\u0434\u0430\u0447\u0430, \u043a\u0430\u0437\u0430\u043b\u043e\u0441\u044c \u0431\u044b, \u0442\u0440\u0438\u0432\u0438\u0430\u043b\u044c\u043d\u0430\u044f \u0438 \u043d\u0435 \u043f\u0440\u0435\u0434\u0432\u0435\u0449\u0430\u043b\u0430 \u0431\u0435\u0434\u044b\u2026 \u041f\u0440\u0438\u0441\u0442\u0443\u043f\u0430\u0435\u043c: 1. \u041e\u0441\u0442\u0430\u043d\u0430\u0432\u043b\u0438\u0432\u0430\u0435\u043c \u0438 \u0443\u0431\u0438\u0432\u0430\u0435\u043c \u0432\u0441\u0435 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u044b \u043d\u0430\u0448\u0435\u0433\u043e \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f: docker-compose down \u0435\u0441\u043b\u0438 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432 \u043c\u043d\u043e\u0433\u043e, \u0438 \u043e\u043d\u0438 \u0432 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-36736","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Historien om problemet med at flytte Docker-lageret (Docker root) | ProHoster","description":"For bare et par dage siden blev det besluttet at flytte Docker-lageret (kataloget, hvor Docker gemmer alle container- og billedfiler) til en separat partition, som havde en st\u00f8rre kapacitet.","canonical_url":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"da_DK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u0418\u0441\u0442\u043e\u0440\u0438\u044f \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u044b \u043f\u0435\u0440\u0435\u043d\u043e\u0441\u0430 docker storage (docker root) | ProHoster","og:description":"\u041d\u0435 \u0434\u0430\u043b\u0435\u0435, \u0447\u0435\u043c \u043f\u0430\u0440\u0443 \u0434\u043d\u0435\u0439 \u043d\u0430\u0437\u0430\u0434 \u0431\u044b\u043b\u043e \u0440\u0435\u0448\u0435\u043d\u043e \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0438\u0437 \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u0432 \u0432\u044b\u043d\u0435\u0441\u0442\u0438 docker storage (\u043a\u0430\u0442\u0430\u043b\u043e\u0433, \u0433\u0434\u0435 \u0434\u043e\u043a\u0435\u0440 \u0445\u0440\u0430\u043d\u0438\u0442 \u0432\u0441\u0435 \u0444\u0430\u0439\u043b\u044b \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u043e\u0432, \u043e\u0431\u0440\u0430\u0437\u043e\u0432) \u043d\u0430 \u043e\u0442\u0434\u0435\u043b\u044c\u043d\u044b\u0439 \u0440\u0430\u0437\u0434\u0435\u043b, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043e\u0431\u043b\u0430\u0434\u0430\u043b \u0431\u043e\u043b\u044c\u0448\u0435\u0439 \u0435\u043c\u043a\u043e\u0441\u0442\u044c\u044e.","og:url":"https:\/\/prohoster.info\/da\/blog\/administrirovanie\/istoriya-problemy-perenosa-docker-storage-docker-root","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T19:13:26+00:00","article:modified_time":"2019-10-31T19:13:26+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"36736","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-22 04:38:20","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 01:39:24","updated":"2026-01-22 04:38:20","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/36736","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/comments?post=36736"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/36736\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/media?parent=36736"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/categories?post=36736"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/tags?post=36736"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}