🥇Tips & tricks for working with Ceph in demanding projects

Using Ceph as a network storage solution in projects with varying loads, we may encounter various challenges that at first glance do not seem simple or trivial. For example:

data migration from an old Ceph to a new one, partially utilizing previous servers in the new cluster;
addressing the issue of disk space distribution in Ceph.

When dealing with such tasks, we face the need to correctly extract OSD without data loss, which is particularly relevant when handling large volumes of data. This is what this article will discuss.

The methods described below are applicable to any version of Ceph. Additionally, it will be taken into account that Ceph may store a large amount of data: to prevent data loss and other issues, some actions will be 'broken down' into several others.

Preface about OSD

Since two out of the three recipes under consideration are dedicated to OSD (Object Storage Daemon), before diving into the practical part—let's briefly discuss what it actually is in Ceph and why it is so important.

First and foremost, it should be noted that the entire Ceph cluster consists of multiple OSDs. The more there are, the greater the free data capacity in Ceph. From this, it is easy to understand the main function of OSD: it stores Ceph object data on the file systems of all nodes in the cluster and provides network access to them (for reading, writing, and other requests).

At this level, replication parameters are set by copying objects between different OSDs. Here, various problems may arise, the solutions to which will be discussed next.

Case #1. Safe extraction of OSD from Ceph cluster without data loss

The need to extract OSD may arise from removing the server from the cluster— for example, to replace it with another server—which is what happened in our case and served as the impetus for writing this article. Thus, the ultimate goal of the manipulations is to extract all OSDs and monitors on this server so that it can be shut down.

For convenience and to avoid a situation where we mistakenly point to the wrong OSD during command execution, we will set a separate variable, the value of which will be the number of the OSD being removed. Let's name it ${ID} — here and below such a variable replaces the number of the OSD we are working with.

Let's look at the state before starting the work:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
-1       0.46857 root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
 1   ssd 0.15619      osd.1     up     1.00000  1.00000
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2     up     1.00000  1.00000

To initiate the removal of an OSD, it is necessary to smoothly execute reweight it down to zero. This reduces the amount of data in the OSD by redistributing it to other OSDs. The following commands are executed for this purpose:

ceph osd reweight osd.${ID} 0.98
ceph osd reweight osd.${ID} 0.88
ceph osd reweight osd.${ID} 0.78

… and so on down to zero.

Smooth rebalancing is necessary, to avoid data loss. This is especially relevant if there is a large volume of data in the OSD. To ensure that the commands reweight have executed successfully, you can run ceph -s or alternatively launch in a separate terminal window ceph -w to monitor changes in real-time.

When the OSD is 'emptied', you can proceed with the standard operation to remove it. For this, we will set the desired OSD to a state of down:

ceph osd down osd.${ID}

We will 'take out' the OSD from the cluster:

ceph osd out osd.${ID}

We will stop the OSD service and unmount its partition in the FS:

systemctl stop ceph-osd@${ID}
umount /var/lib/ceph/osd/ceph-${ID}

We will remove the OSD from CRUSH map:

ceph osd crush remove osd.${ID}

We will delete the OSD user:

ceph auth del osd.${ID}

And finally, we will delete the OSD itself:

ceph osd rm osd.${ID}

Note: if you are using Ceph Luminous version or higher, then the above actions for removing the OSD can be reduced to two commands:

ceph osd out osd.${ID}
ceph osd purge osd.${ID}

If after executing the above commands you run the command ceph osd tree, it should be visible that there are no longer any OSDs on the server where the operations were performed:

root@hv-1 ~ # ceph osd tree
ID CLASS WEIGHT  TYPE NAME     STATUS REWEIGHT PRI-AFF
-1       0.46857      root default
-3       0.15619      host hv-1
-5       0.15619      host hv-2
-7       0.15619      host hv-3
 2   ssd 0.15619      osd.2    up     1.00000  1.00000

Meanwhile, we will note that the state of the Ceph cluster will transition to HEALTH_WARN, and we will also see a decrease in the number of OSDs and available disk space.

Next, actions will be described that will be required if you want to completely stop the server and, accordingly, remove it from Ceph. In this case, it is important to remember that before disconnecting the server, you must remove all OSDs on this server.

If there are no remaining OSDs on this server, then after their removal, you need to exclude the OSD server from the map hv-2, by executing the following command:

ceph osd crush rm hv-2

Remove mon from the server hv-2, running the command below on another server (i.e., in this case — on hv-1):

ceph-deploy mon destroy hv-2

After this, you can stop the server and proceed to the next steps (its redeployment, etc.).

Case #2. Distributing disk space in an already created Ceph cluster

I'll start the second story with a preface about PG (Placement Groups). The primary role of PG in Ceph is primarily to aggregate Ceph objects and further replicate them in OSD. The formula to calculate the necessary number of PG can be found in the relevant section of the Ceph documentation. This issue is also discussed there with concrete examples.

So, one of the common problems during the operation of Ceph is an unbalanced number of OSD and PG among pools in Ceph.

Firstly, this may lead to a situation where too many PGs are specified in a small-sized pool, which is essentially an irrational use of disk space in the cluster. Secondly, in practice, a more serious problem arises: data overflow in one of the OSDs. This leads to the cluster first entering a state HEALTH_WARN, and then HEALTH_ERR. The cause is that Ceph, when calculating the available data volume (you can find out by using MAX AVAIL in the output of the command ceph df for each pool individually) relies on the volume of available data in OSD. If there is insufficient space in even one OSD, data cannot be written until it is properly distributed among all OSDs.

It should be clarified that these problems are mostly solved during the configuration stage of the Ceph cluster.One of the tools you can use is Ceph PGCalc. It visually calculates the necessary number of PG. However, it can also be used in situations where the Ceph cluster already is misconfigured. Here it should be noted that while fixing issues, you will likely need to reduce the number of PG, which is not possible in older versions of Ceph (this feature only appeared starting with version Nautilus).

. So, imagine the following picture: the cluster has a status HEALTH_WARN because one of the OSDs is running out of space. This will be indicated by the error HEALTH_WARN: 1 near full osd. Below is the algorithm for resolving such a situation.

First, it is necessary to redistribute the existing data among the other OSDs. We have already performed such an operation in the first case when we 'drained' the node — the only difference being that now we will need to slightly reduce the weight. reweightFor example, to 0.95:

ceph osd reweight osd.${ID} 0.95

This frees up disk space in the OSD and fixes the error in ceph health. However, as mentioned earlier, this problem mainly arises due to incorrect Ceph configuration in the initial stages: it is very important to reconfigure it to prevent this from occurring in the future.

In our specific case, everything boiled down to:

too high a value replication_count in one of the pools,
too high a number of PGs in one pool and too low in another.

We will use the previously mentioned calculator. It clearly shows what needs to be entered, and overall, it's not complicated. By setting the necessary parameters, we obtain the following recommendations:

Note: if you are setting up a Ceph cluster from scratch, another useful feature of the calculator will be the generation of commands that will create pools from scratch with the parameters specified in the table.

The last column helps orient: Suggested PG Count. In our case, the second one is also useful, which indicates the replication parameter, since we decided to change the replication factor.

So, first, we need to change the replication parameters — this should be done first, as reducing the factor will free up disk space. During the execution of the command, you may notice that the available disk space increases:

ceph osd pool $pool_name set $replication_size

And after its completion — we change the values of the parameters pg_num and pgp_num as follows:

ceph osd pool set $pool_name pg_num $pg_number
ceph osd pool set $pool_name pgp_num $pg_number

Important: we must sequentially change the number of PGs in each pool and not change the values in other pools until the warnings disappear ‘Degraded data redundancy’ and ‘n-number of PGs degraded’.

You can also verify that everything went successfully by examining the outputs of the commands ceph health detail and ceph -s.

Case #3. Migration of a virtual machine from LVM to Ceph RBD

In situations where virtual machines are used on rented bare-metal servers, questions about fault-tolerant storage often arise. Additionally, it is highly desirable to have enough space in this storage... Another common scenario is having a virtual machine with local storage on the server and needing to expand the disk, but there is nowhere to do so because there’s no free disk space left on the server.

This problem can be solved in various ways—such as migrating to another server (if one is available) or adding new disks to the server. However, this is not always feasible, so migration from LVM to Ceph can be an excellent solution to this issue. Choosing this option also simplifies the further migration process between servers since there will be no need to move local storage from one hypervisor to another. The only catch is that the VM will have to be stopped during the work.

The following recipe is taken from an article from this blog, whose instructions have been tested in practice. By the way, it also describes a method for simple migration, but in our case, it wasn't needed, so we didn't validate it. However, if this is critical for your project, we would be happy to hear the results in the comments.

Let's get to the practical part. In this example, we will use virsh and, accordingly, libvirt. First, make sure that the Ceph pool, into which the data will be migrated, is connected to libvirt:

virsh pool-dumpxml $ceph_pool

The pool description should contain connection details to Ceph with authorization data.

The next step involves converting the LVM image to Ceph RBD. The execution time primarily depends on the size of the image:

qemu-img convert -p -O rbd /dev/main/$vm_image_name rbd:$ceph_pool/$vm_image_name

After the conversion, the LVM image will remain, which will be useful if migrating the VM to RBD fails and we need to roll back the changes. Also, to allow for a quick rollback, we will create a backup of the virtual machine's configuration file:

virsh dumpxml $vm_name > $vm_name.xml
cp $vm_name.xml $vm_name_backup.xml

… and we will edit the original (vm_name.xml). We will find the block describing the disk (starting from the line <disk type='file' device='disk'> and ending with </disk>) and we will bring it to the following form:

\n
  \n 

  \n  \n
 
\n\n

Let's break down some details:

In the protocol source the address to the storage in Ceph RBD is specified (this is the address indicating the name of the Ceph pool and the RBD image defined in the first step).
In the section secret the type is specified as ceph, as well as the UUID of the secret for connecting to it. Its uuid can be obtained using the command virsh secret-list.
In the section host the addresses of the Ceph monitors are specified.

After editing the configuration file and completing the conversion of LVM to RBD, you can apply the modified configuration file and start the virtual machine:

virsh define $vm_name.xml\nvirsh start $vm_name

Now it's time to check that the virtual machine started correctly: you can find this out, for example, by connecting to it via SSH or through virsh.

If the virtual machine is running correctly and you have not found any other issues, you can remove the LVM image that is no longer in use:

lvremove main/$vm_image_name

Conclusion

We have encountered all the described cases in practice — we hope that the instructions will help other administrators solve similar problems. If you have comments or other similar stories from your experience with Ceph — we would be delighted to see them in the comments!