{"id":33803,"date":"2019-10-31T21:54:45","date_gmt":"2019-10-31T18:54:45","guid":{"rendered":"https:\/\/prohoster.info\/blog\/on-vam-ne-drook\/"},"modified":"2019-10-31T21:54:45","modified_gmt":"2019-10-31T18:54:45","slug":"on-vam-ne-drook","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/on-vam-ne-drook","title":{"rendered":"He's not your Rook","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

With the growing popularity of Rook, it's worth discussing its pitfalls and the issues that await you on this journey.<\/p>\n

About me: Experience in ceph administration since the hammer version, founder of the community t.me\/ceph_ru<\/a><\/noindex> on Telegram.<\/p>\n

To back up my claims, I will refer to accepted Hub posts (based on ratings) about issues with ceph. I've encountered most of the problems discussed in these posts as well. Links to the sources will be at the end of the post.<\/p>\n

In the post about Rook, we mention ceph not just for fun \u2014 Rook is essentially ceph wrapped in Kubernetes, which means it inherits all its problems. Let's start with the issues of ceph.
\n<\/a><\/noindex><\/p>\n

Simplification of cluster management<\/h3>\n

\nOne of the advantages of Rook is the convenience of managing ceph through Kubernetes.<\/p>\n

However, ceph has over 1000 tuning parameters, while through Rook, we can only edit a smaller portion of them.<\/p>\n

Example on Luminous
\n> ceph daemon mon.a config show | wc -l
\n1401<\/p><\/blockquote>\n

Rook is positioned as a convenient way to install and upgrade ceph.<\/b>
\nInstalling ceph without Rook poses no issues \u2014 an ansible playbook can be written in 30 minutes, but there are many problems with upgrading. <\/p>\n

Quote from Krok's post<\/i><\/p>\n

Example: incorrect operation of crush tunables after upgrading from hummer to jewel<\/p>\n

> ceph osd crush show-tunables
\n{
\n\u2026
\n \"straw_calc_version\": 1,
\n \"allowed_bucket_algs\": 22,
\n \"profile\": \"unknown\",
\n \"optimal_tunables\": 0,
\n\u2026
\n}<\/p><\/blockquote>\n

But even within minor versions, problems can arise.<\/p>\n

Example: Updating 12.2.6 brings the cluster to a health err state and a conditionally broken PG.
\nceph.com\/releases\/v12-2-8-released<\/a><\/noindex><\/p>\n

Not to update, wait and test? But we're using Rook for the convenience of upgrades as well.<\/p>\n

The complexity of disaster recovery for a cluster in Rook <\/h3>\n

\nExample: OSD crashes with errors falling at its feet. You suspect the problem lies within one of the parameters in the config, want to change the config for a specific daemon, but can't because you have Kubernetes and DaemonSet.<\/p>\n

There is no alternative. ceph tell osd.Num injectargs doesn't work \u2014 the OSD is down.<\/p>\n

The complexity of debugging<\/h3>\n

\nFor certain settings and performance tests, you need to connect directly to the socket of the osd daemon. In the case of Rook, you first need to find the right container, then enter it, discover the debug tooling is missing, and feel very disappointed.<\/p>\n

The complexity of sequentially bringing up OSD<\/h3>\n

\nExample: OSD crashes due to OOM, rebalance begins, after which subsequent ones crash.<\/p>\n

Solution: Bring up OSDs one by one, waiting for each to fully join the cluster before bringing up the next. (See the report Ceph: Anatomy of a Disaster for more details).<\/p>\n

In the case of baremetal installations, this is simply done manually; for Rook and one OSD per node, there are no significant issues, but problems with sequential startups will arise if there are more than one OSD per node. <\/p>\n

Of course, these issues can be resolved, but we are implementing Rook for simplification, and instead, we are encountering complications.<\/p>\n

The complexity of selecting limits for Ceph daemons.<\/h3>\n

\nFor baremetal installations of Ceph, it's fairly straightforward to calculate the necessary resources for the cluster\u2014formulas and studies are available. However, if you're using weak CPUs, you'll still need to conduct a series of performance tests and understand what NUMA is, but this is still simpler than in Rook.<\/p>\n

In the case of Rook, besides memory limits that can be calculated, there's the question of setting CPU limits.<\/p>\n

And here you\u2019ll have to work hard with performance tests. If you underestimate the limits, you'll get a slow cluster; if you set it to unlimit, you'll have high CPU usage during rebalancing, which will negatively affect your applications in Kubernetes.<\/p>\n

Network interaction issues v1<\/h3>\n

\nFor Ceph, it is recommended to use a 2x10Gb network. One for client traffic and the other for Ceph's internal needs (rebalancing). If you're using Ceph on baremetal, this separation is easy to configure, but if you're using Rook, network separation will pose problems, as not every cluster configuration allows two different networks to be assigned to a pod.<\/p>\n

Network interaction issues v2<\/h3>\n

\nIf you choose not to separate the networks, during rebalancing, Ceph traffic will saturate the entire bandwidth and your applications in Kubernetes will slow down or crash. You can reduce the speed of Ceph rebalancing, but this comes with the risk of the second node dropping out of the cluster due to disk failures or OOM, leading to a guaranteed read-only state for the cluster.<\/p>\n

Long rebalancing means long delays for applications.<\/h3>\n

\nQuote from the post Ceph: Anatomy of a Disaster.<\/i><\/p>\n

Performance of the test cluster:<\/p>\n

A 4KB write operation takes 1 ms, with a performance of 1000 operations\/second in a single thread.<\/p>\n

A 4MB operation (object size) takes 22 ms, with a performance of 45 operations\/second.<\/p>\n

Therefore, when one domain out of three fails, the cluster remains in a degraded state for some time, and half of the hot objects are spread across different versions, half of the write operations will begin with forced recovery.<\/p>\n

The time for forced recovery is estimated approximately \u2014 write operations to a degraded object.<\/p>\n

First, we read 4 MB in 22 ms, write for 22 ms, and then for 1 ms we write 4 KB of actual data. In total, it takes 45 ms for one write operation to a degraded object on SSD, whereas the normal performance was 1 ms \u2014 a performance drop of 45 times.<\/p>\n

The higher the percentage of degraded objects, the worse it becomes.<\/p><\/blockquote>\n

It turns out that the speed of rebalancing is critically important for the proper functioning of the cluster.<\/p>\n

Specific server settings for Ceph<\/h3>\n

\nCeph often requires specific host tuning. <\/p>\n

For example: sysctl settings and JumboFrame, some of these settings may negatively affect your payload.<\/p>\n

The actual necessity of Rook remains questionable.<\/h3>\n

\nIf you are in the cloud, you have storage from your cloud provider, which is much more convenient. <\/p>\n

If you are on your own servers, managing Ceph will be easier without Kubernetes. <\/p>\n

Are you renting servers from some low-cost hosting? Then you can expect a lot of fun with the network, its latency, and bandwidth, which clearly negatively affects Ceph.<\/p>\n

Total:<\/b> Implementing Kubernetes and implementing storage are different tasks with different inputs and different solution options \u2014 mixing them means making possibly dangerous trade-offs in favor of one or the other. Combining these solutions will be very difficult even at the design stage, and there is also an operation period.<\/p>\n

List of references:<\/p>\n

Post #1<\/a><\/noindex> But you say Ceph\u2026 is it really that good?
\nPost #2<\/a><\/noindex> Ceph. Anatomy of a disaster
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"

\u0412 \u0441\u0432\u044f\u0437\u0438 \u0441 \u043d\u0430\u0431\u0438\u0440\u0430\u044e\u0449\u0435\u0439 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u043e\u0441\u0442\u044c\u044e Rook \u0445\u043e\u0447\u0435\u0442\u0441\u044f \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u0442\u044c \u043e \u0435\u0433\u043e \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0445 \u043a\u0430\u043c\u043d\u044f\u0445 \u0438 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0436\u0434\u0443\u0442 \u0432\u0430\u0441 \u043d\u0430 \u043f\u0443\u0442\u0438. \u041e \u0441\u0435\u0431\u0435: \u041e\u043f\u044b\u0442 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f ceph \u0441 \u0432\u0435\u0440\u0441\u0438\u0438 hammer, \u043e\u0441\u043d\u043e\u0432\u0430\u0442\u0435\u043b\u044c \u043a\u043e\u043c\u044c\u044e\u043d\u0438\u0442\u0438 t.me\/ceph_ru \u0432 \u0442\u0435\u043b\u0435\u0433\u0440\u0430\u043c. \u0414\u0430\u0431\u044b \u043d\u0435 \u0431\u044b\u0442\u044c \u0433\u043e\u043b\u043e\u0441\u043b\u043e\u0432\u043d\u044b\u043c \u044f \u0431\u0443\u0434\u0443 \u0441\u0441\u044b\u043b\u0430\u0442\u044c\u0441\u044f \u043d\u0430 \u043f\u0440\u0438\u043d\u044f\u0442\u044b\u0435 \u0445\u0430\u0431\u0440\u043e\u043c (\u0441\u0443\u0434\u044f \u043f\u043e \u0440\u0435\u0439\u0442\u0438\u043d\u0433\u0443) \u043f\u043e\u0441\u0442\u044b \u043e \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0445 \u0441 ceph. \u0421 \u0431\u041e\u043b\u044c\u0448\u0435\u0439 \u0447\u0430\u0441\u0442\u044c\u044e \u043f\u0440\u043e\u0431\u043b\u0435\u043c \u0432 […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-33803","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47He\u2019s not your dRook | ProHoster","description":"With the rising popularity of Rook, I want to talk about its pitfalls and the problems that await you along the way. About myself: Experience in managing Ceph since the Hammer version, founder of the community.","canonical_url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/on-vam-ne-drook","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043d \u0432\u0430\u043c \u043d\u0435 \u0434Rook | ProHoster","og:description":"\u0412 \u0441\u0432\u044f\u0437\u0438 \u0441 \u043d\u0430\u0431\u0438\u0440\u0430\u044e\u0449\u0435\u0439 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u043e\u0441\u0442\u044c\u044e Rook \u0445\u043e\u0447\u0435\u0442\u0441\u044f \u043f\u043e\u0433\u043e\u0432\u043e\u0440\u0438\u0442\u044c \u043e \u0435\u0433\u043e \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0445 \u043a\u0430\u043c\u043d\u044f\u0445 \u0438 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0445, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0436\u0434\u0443\u0442 \u0432\u0430\u0441 \u043d\u0430 \u043f\u0443\u0442\u0438. \u041e \u0441\u0435\u0431\u0435: \u041e\u043f\u044b\u0442 \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f ceph \u0441 \u0432\u0435\u0440\u0441\u0438\u0438 hammer, \u043e\u0441\u043d\u043e\u0432\u0430\u0442\u0435\u043b\u044c \u043a\u043e\u043c\u044c\u044e\u043d\u0438\u0442\u0438","og:url":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/on-vam-ne-drook","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:54:45+00:00","article:modified_time":"2019-10-31T18:54:45+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"33803","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 16:46:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 02:33:24","updated":"2026-01-21 16:46:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/33803","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=33803"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/33803\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=33803"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=33803"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=33803"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}