With the growing popularity of Rook, it's worth discussing its pitfalls and the issues that await you on this journey.
About me: Experience in ceph administration since the hammer version, founder of the community on Telegram.
To back up my claims, I will refer to accepted Hub posts (based on ratings) about issues with ceph. I've encountered most of the problems discussed in these posts as well. Links to the sources will be at the end of the post.
In the post about Rook, we mention ceph not just for fun — Rook is essentially ceph wrapped in Kubernetes, which means it inherits all its problems. Let's start with the issues of ceph.
Simplification of cluster management
One of the advantages of Rook is the convenience of managing ceph through Kubernetes.
However, ceph has over 1000 tuning parameters, while through Rook, we can only edit a smaller portion of them.
Example on Luminous
> ceph daemon mon.a config show | wc -l
1401
Rook is positioned as a convenient way to install and upgrade ceph.
Installing ceph without Rook poses no issues — an ansible playbook can be written in 30 minutes, but there are many problems with upgrading.
Quote from Krok's post
Example: incorrect operation of crush tunables after upgrading from hummer to jewel
> ceph osd crush show-tunables
{
…
"straw_calc_version": 1,
"allowed_bucket_algs": 22,
"profile": "unknown",
"optimal_tunables": 0,
…
}
But even within minor versions, problems can arise.
Example: Updating 12.2.6 brings the cluster to a health err state and a conditionally broken PG.
Not to update, wait and test? But we're using Rook for the convenience of upgrades as well.
The complexity of disaster recovery for a cluster in Rook
Example: OSD crashes with errors falling at its feet. You suspect the problem lies within one of the parameters in the config, want to change the config for a specific daemon, but can't because you have Kubernetes and DaemonSet.
There is no alternative. ceph tell osd.Num injectargs doesn't work — the OSD is down.
The complexity of debugging
For certain settings and performance tests, you need to connect directly to the socket of the osd daemon. In the case of Rook, you first need to find the right container, then enter it, discover the debug tooling is missing, and feel very disappointed.
The complexity of sequentially bringing up OSD
Example: OSD crashes due to OOM, rebalance begins, after which subsequent ones crash.
Solution: Bring up OSDs one by one, waiting for each to fully join the cluster before bringing up the next. (See the report Ceph: Anatomy of a Disaster for more details).
In the case of baremetal installations, this is simply done manually; for Rook and one OSD per node, there are no significant issues, but problems with sequential startups will arise if there are more than one OSD per node.
Of course, these issues can be resolved, but we are implementing Rook for simplification, and instead, we are encountering complications.
The complexity of selecting limits for Ceph daemons.
For baremetal installations of Ceph, it's fairly straightforward to calculate the necessary resources for the cluster—formulas and studies are available. However, if you're using weak CPUs, you'll still need to conduct a series of performance tests and understand what NUMA is, but this is still simpler than in Rook.
In the case of Rook, besides memory limits that can be calculated, there's the question of setting CPU limits.
And here you’ll have to work hard with performance tests. If you underestimate the limits, you'll get a slow cluster; if you set it to unlimit, you'll have high CPU usage during rebalancing, which will negatively affect your applications in Kubernetes.
Network interaction issues v1
For Ceph, it is recommended to use a 2x10Gb network. One for client traffic and the other for Ceph's internal needs (rebalancing). If you're using Ceph on baremetal, this separation is easy to configure, but if you're using Rook, network separation will pose problems, as not every cluster configuration allows two different networks to be assigned to a pod.
Network interaction issues v2
If you choose not to separate the networks, during rebalancing, Ceph traffic will saturate the entire bandwidth and your applications in Kubernetes will slow down or crash. You can reduce the speed of Ceph rebalancing, but this comes with the risk of the second node dropping out of the cluster due to disk failures or OOM, leading to a guaranteed read-only state for the cluster.
Long rebalancing means long delays for applications.
Quote from the post Ceph: Anatomy of a Disaster.
Performance of the test cluster:
A 4KB write operation takes 1 ms, with a performance of 1000 operations/second in a single thread.
A 4MB operation (object size) takes 22 ms, with a performance of 45 operations/second.
Therefore, when one domain out of three fails, the cluster remains in a degraded state for some time, and half of the hot objects are spread across different versions, half of the write operations will begin with forced recovery.
The time for forced recovery is estimated approximately — write operations to a degraded object.
First, we read 4 MB in 22 ms, write for 22 ms, and then for 1 ms we write 4 KB of actual data. In total, it takes 45 ms for one write operation to a degraded object on SSD, whereas the normal performance was 1 ms — a performance drop of 45 times.
The higher the percentage of degraded objects, the worse it becomes.
It turns out that the speed of rebalancing is critically important for the proper functioning of the cluster.
Specific server settings for Ceph
Ceph often requires specific host tuning.
For example: sysctl settings and JumboFrame, some of these settings may negatively affect your payload.
The actual necessity of Rook remains questionable.
If you are in the cloud, you have storage from your cloud provider, which is much more convenient.
If you are on your own servers, managing Ceph will be easier without Kubernetes.
Are you renting servers from some low-cost hosting? Then you can expect a lot of fun with the network, its latency, and bandwidth, which clearly negatively affects Ceph.
Total: Implementing Kubernetes and implementing storage are different tasks with different inputs and different solution options — mixing them means making possibly dangerous trade-offs in favor of one or the other. Combining these solutions will be very difficult even at the design stage, and there is also an operation period.
List of references:
But you say Ceph… is it really that good?
Ceph. Anatomy of a disaster
Source: habr.com
