🥇Our hands are not for idleness: restoring a Rook cluster in K8s

We have already discussed, how/why we like Rook: it notably simplifies working with storage in Kubernetes clusters. However, with this simplicity come certain complexities. We hope the new material will help you better understand these complexities before they manifest.

To make reading more interesting, let's start with the consequences of a hypothetical problem in the cluster.

Everything is lost!

Imagine that one day you set up and launched Rook in your K8s cluster, it was performing well, but at some 'wonderful' moment, the following happens:

New pods cannot mount RBD images from Ceph.
Commands like lsblk and df do not work on the Kubernetes nodes. This automatically means: 'something is wrong' with the mounted RBD images on the nodes. They cannot be read, which indicates that the monitors are inaccessible...
Yes, there are no working monitors in the cluster. Moreover, there are not even any pods with OSDs, or the MGR pod.

When the pod was launched rook-ceph-operator? Не так давно, как его деплоили. Почему? Rook-operator решил сделать новый кластер… Как же нам теперь восстановить работу кластера и данные в нём?

To begin with, let's take a longer, more interesting route by conducting a thoughtful investigation into the 'guts' of Rook and a step-by-step recovery of its components. Of course, there is a shorter correct way: using backups. As we know, admins can be divided into two types: those who do not make backups and those who already do... But more on that after the investigation.

A bit of practice, or The long way

Let's inspect and restore the monitors

So, let's look at the ConfigMaps list: there are necessary ones for backup, rook-ceph-config and rook-config-override. They appear after a successful cluster deployment.

NB: In newer versions, after accepting this PR, ConfigMaps are no longer an indicator of successful cluster deployment.

To proceed with further actions, we need a hard reboot of all servers that have mounted RBD images (ls /dev/rbd*). It should be done via sysrq (or 'by foot' in the data center). This requirement arises from the need to unmount the mounted RBDs, for which a regular reboot won't work (it will be futile to attempt to unmount them normally).

The theater begins with a coat rack, and a Ceph cluster begins with monitors. Let's take a look at them.

Rook mounts the following entities into the monitor pod:

Volumes:
 rook-ceph-config:
   Type:      ConfigMap (a volume populated by a ConfigMap)
   Name:      rook-ceph-config
 rook-ceph-mons-keyring:
   Type:        Secret (a volume populated by a Secret)
   SecretName:  rook-ceph-mons-keyring
 rook-ceph-log:
   Type:          HostPath (bare host directory volume)
   Path:          /var/lib/rook/kube-rook/log
 ceph-daemon-data:
   Type:          HostPath (bare host directory volume)
   Path:          /var/lib/rook/mon-a/data
Mounts:
  /etc/ceph from rook-ceph-config (ro)
  /etc/ceph/keyring-store/ from rook-ceph-mons-keyring (ro)
  /var/lib/ceph/mon/ceph-a from ceph-daemon-data (rw)
  /var/log/ceph from rook-ceph-log (rw)

Let's see what's in the secret rook-ceph-mons-keyring:

kind: Secret
data:
 keyring: LongBase64EncodedString=

Decoding will yield a standard keyring with permissions for the admin and monitors:

[mon.]
       key = AQAhT19dlUz0LhBBINv5M5G4YyBswyU43RsLxA==
       caps mon = "allow *"
[client.admin]
       key = AQAhT19d9MMEMRGG+wxIwDqWO1aZiZGcGlSMKp==
       caps mds = "allow *"
       caps mon = "allow *"
       caps osd = "allow *"
       caps mgr = "allow *"

Let's remember this. Now let's take a look at the keyring in the secret rook-ceph-admin-keyring:

kind: Secret
data:
 keyring: anotherBase64EncodedString=

What's in it?

[client.admin]
       key = AQAhT19d9MMEMRGG+wxIwDqWO1aZiZGcGlSMKp==
       caps mds = "allow *"
       caps mon = "allow *"
       caps osd = "allow *"
       caps mgr = "allow *"

The same one. Let's check further… Here's, for example, a secret rook-ceph-mgr-a-keyring:

[mgr.a]
       key = AQBZR19dbVeaIhBBXFYyxGyusGf8x1bNQunuew==
       caps mon = "allow *"
       caps mds = "allow *"
       caps osd = "allow *"

Ultimately, we find a few more secrets in the ConfigMap rook-ceph-mon:

kind: Secret
data:
 admin-secret: AQAhT19d9MMEMRGG+wxIwDqWO1aZiZGcGlSMKp==
 cluster-name: a3ViZS1yb29r
 fsid: ZmZiYjliZDMtODRkOS00ZDk1LTczNTItYWY4MzZhOGJkNDJhCg==
 mon-secret: AQAhT19dlUz0LhBBINv5M5G4YyBswyU43RsLxA==

And this is the original list of keyrings, from which all the above-mentioned secrets are derived.

As it's known (see dataDirHostPath downward API support (simultaneously with this in the documentation), Rook stores such data in two places. So let's head over to the nodes to check out the keyrings located in the directories mounted in the pods with monitors and OSDs. For this, we will find on the nodes /var/lib/rook/mon-a/data/keyring and see:

# cat /var/lib/rook/mon-a/data/keyring
[mon.]
       key = AXAbS19d8NNUXOBB+XyYwXqXI1asIzGcGlzMGg==
       caps mon = "allow *"

Suddenly the secret turned out to be different — not like in the ConfigMaps.

What about the admin keyring? We have that too:

# cat /var/lib/rook/kube-rook/client.admin.keyring
[client.admin]
       key = AXAbR19d8GGSMUBN+FyYwEqGI1aZizGcJlHMLgx= 
       caps mds = "allow *"
       caps mon = "allow *"
       caps osd = "allow *"
       caps mgr = "allow *"

Here's the problem. An error occurred: the cluster was recreated... but in reality, it didn't.

It becomes clear that the secrets contain newly generated keyrings, and they do not are from our old cluster. Therefore:

we take the keyring from the monitor from the file /var/lib/rook/mon-a/data/keyring (or from a backup);
modify the keyring in the secret rook-ceph-mons-keyring;
specify the keyring from the admin and monitor in the ConfigMap rook-ceph-mon;
delete the controllers of the pods with monitors.

A miracle will not take long: the monitors will appear and start. Hooray, the beginning is laid!

We will restore the OSD

Enter the pod rook-operator: calling ceph mon dump shows that all monitors are in place, and ceph -s — regarding their quorum. However, if we look at the OSD tree (ceph osd tree), we see something strange: OSD instances have started appearing, but they are empty. It turns out they also need to be restored somehow. But how?

Meanwhile, the ConfigMaps that we need have appeared, rook-ceph-config and rook-config-override, as well as many other ConfigMaps with names like rook-ceph-osd-$nodename-config. Let's take a look at them:

kind: ConfigMap
data:
 osd-dirs: '{"/mnt/osd1":16,"/mnt/osd2":18}'

Everything is wrong, everything is mixed up!

Let's scale the operator pod down to zero, delete the generated pods' Deployments with OSD, and fix these ConfigMaps. But where to get the correct OSD mapping by nodes?

Let's try digging again through the directories /mnt/osd[1-2] on the nodes — hoping that we can find something to latch onto.
In the directory /mnt/osd1 There are 2 subdirectories: osd0 and osd16. The latter is indeed the ID specified in the ConfigMap (16)?
We'll check the sizes and see that osd0 much bigger osd16.

We conclude that osd0 — this is the needed OSD that was specified as /mnt/osd1 in the ConfigMap (after all, we are using directory based osd.)

Step by step, we check all nodes and correct the ConfigMaps. After all the adjustments, we can launch the Rook operator pod and read its logs. And in them, everything is wonderful:

I am the cluster operator;
I found disks on the nodes;
I found monitors;
the monitors have connected, i.e., formed a quorum;
I am launching OSD deployments…

Let's go back into the Rook operator pod and check the cluster's health… yes, we were a bit mistaken in our conclusions about the OSD names on some nodes! No worries: we corrected the ConfigMaps again, deleted the extra directories from the new OSDs, and reached the long-awaited state HEALTH_OK!

Let's check the images in the pool:

# rbd ls -p kube
pvc-9cfa2a98-b878-437e-8d57-acb26c7118fb
pvc-9fcc4308-0343-434c-a65f-9fd181ab103e
pvc-a6466fea-bded-4ac7-8935-7c347cff0d43
pvc-b284d098-f0fc-420c-8ef1-7d60e330af67
pvc-b6d02124-143d-4ce3-810f-3326cfa180ae
pvc-c0800871-0749-40ab-8545-b900b83eeee9
pvc-c274dbe9-1566-4a33-bada-aabeb4c76c32
…

Everything is in place — the cluster is saved!

I am lazy about backups, or The Quick Way

If backups for Rook were made, then the restoration procedure becomes significantly simpler and boils down to the following:

Scale down the Rook operator deployment to zero;
Delete all deployments except for the Rook operator;
Restore all secrets and ConfigMaps from the backup;
Restore the contents of the directories /var/lib/rook/mon-* on the nodes;
Restore (if lost) the CRDs CephCluster, CephFilesystem, CephBlockPool, CephNFS, CephObjectStore;
Scale the Rook operator deployment back up to 1.

Helpful tips

Make backups!

And to avoid situations where restoration will be needed:

Before large-scale work with the cluster involving server reboots, scale down the Rook operator to zero so it doesn't perform unnecessary actions.
Anticipate and add nodeAffinity to the monitors.
Pay attention to advance timeout settings ROOK_MON_HEALTHCHECK_INTERVAL and ROOK_MON_OUT_TIMEOUT.

In conclusion

There is no point in arguing that Rook, as an additional "layer" in the overall scheme of storage organization in Kubernetes, simplifies many things while also introducing new complexities and potential issues in infrastructure. The task remains to make a measured, justified choice between these risks on one side and the benefits that the solution brings in your particular case on the other.

By the way, a section "Adopt an existing Rook Ceph cluster into a new Kubernetes cluster" was recently added to the Rook documentation. It outlines in more detail what needs to be done to migrate existing data to a new Kubernetes cluster or to restore the operation of a cluster that has collapsed for one reason or another. Longhorn, a distributed storage solution for K8s from Rancher, has been transferred to CNCF