When the amount of data exceeds what fits on a single disk, it's time to consider RAID. In my childhood, I often heard from elders: "One day RAID will become a thing of the past; object storage will take over the world, and you won't even know what CEPH is," so the first thing I did in my independent life was to create my own cluster. The goal of the experiment was to familiarize myself with the internal structure of Ceph and understand the limits of its application. How justified is the implementation of Ceph in medium businesses, and in small ones? After several years of operation and a couple of irreversible data losses, I gained an understanding of the nuances, realizing that not everything is straightforward. The features of CEPH create obstacles to its widespread adoption, and these challenges have led to dead ends in experiments. Below is a description of all the steps taken, the results obtained, and the conclusions drawn. If knowledgeable people could share their experiences and clarify some points, I would be grateful.
Note: Commentators pointed out serious errors in some assumptions, requiring a revision of the entire article.
CEPH Strategy
The CEPH cluster combines an arbitrary number of K disks of arbitrary sizes and stores data on them, duplicating each piece (4 MB by default) a specified number of N times.
Let's consider the simplest case with two identical disks. From these, you can either create a RAID 1 or a cluster with N=2 — the result will be the same. If there are three disks of different sizes, creating a cluster with N=2 is straightforward: part of the data will be on disks 1 and 2, part on 1 and 3, and part on 2 and 3, whereas RAID would not allow this (you could create such a RAID, but it would be unconventional). If there are even more disks, it is possible to create a RAID 5; CEPH has an equivalent — erasure_code, which contradicts the earlier concepts of the developers and therefore is not considered. RAID 5 assumes that there is a small number of disks and that all of them are in good condition. In the event of one failure, the others must last until the disk is replaced and the data is restored onto it. However, CEPH, with N>=3, encourages the use of old disks; specifically, if you keep several good disks to store one copy of the data while keeping the remaining two or three copies on a large number of old disks, the information will be safe, because as long as the new disks are functioning, there are no issues. If one of them fails, then a simultaneous failure of three disks older than five years, ideally from different servers, is an extremely unlikely event.
There is a nuance in the distribution of copies. By default, it is assumed that the data is divided into a larger number (~100 per disk) of PG distribution groups, each of which is duplicated on certain disks. Suppose K=6, N=2, then in the event of the failure of any two disks, data is guaranteed to be lost, since, according to probability theory, there will be at least one PG located precisely on those two disks. The loss of one group makes all data in the pool inaccessible. However, if the disks are split into three pairs and data is allowed to be stored only on disks within one pair, then this distribution is also resistant to the failure of any single disk, but the probability of data loss on the failure of two disks is not 100%, but only 3/15, and even with the failure of three disks — only 12/20. Thus, entropy in data distribution does not contribute to fault tolerance. It is also noteworthy that for a file server, free RAM significantly increases response speed. The more memory in each node, and the more memory across all nodes, the faster it will be. This is undoubtedly an advantage of a cluster over a single server and, even more so, over a hardware NAS, which integrates a very small amount of memory.
It follows that CEPH is a good way to create a reliable data storage system for tens of TB with minimal investment using outdated equipment, with the possibility of scaling (of course, some costs will be required here, but they are small compared to commercial SANs).
Cluster Implementation
For the experiment, let's take an old computer Intel DQ57TM + Intel Core i3 540 + 16 GB RAM. We will organize four 2 TB disks into a RAID10-like setup, and after successful testing, we will add a second node and the same number of disks.
Installing Linux. The distribution needs to be customizable and stable. Debian and Suse meet these requirements. Suse has a more flexible installer that allows you to disable any package; unfortunately, I couldn't figure out which ones could be removed without harming the system. We're installing Debian using debootstrap buster. The min-base option installs a non-functional system that lacks drivers. The size difference compared to the full version isn't significant enough to worry about. Since we're working on a physical machine, we want to take snapshots like in virtual machines. This capability is provided either by LVM or btrfs (or xfs or zfs — the difference isn't substantial). LVM doesn’t handle snapshots well. We go with btrfs. The bootloader is installed in MBR. There's no point in cluttering the disk with a 50 MB FAT partition when it can be squeezed into a 1 MB area of the partition table, leaving all the space for the system. It took up 700 MB on the disk. I don't remember how much the base installation of SUSE takes — it seems around 1.1 or 1.4 GB.
Installing CEPH. We ignore version 12 in the debian repository and connect directly from the website version 15.2.3. Follow the instructions from the 'installing CEPH manually' section with the following notes:
- Before connecting the repository, you need to install gnupg wget ca-certificates.
- After connecting the repository, but before installing the cluster, the installation of packages was skipped: apt -y --no-install-recommends install ceph-common ceph-mon ceph-osd ceph-mds ceph-mgr.
- At the moment of installing CEPH, for some unknown reason, it will attempt to install lvm2. In principle, it’s not a problem, but the installation fails, so CEPH won’t install either.
This patch helped:
cat <> /var/lib/dpkg/status Package: lvm2 Status: install ok installed Priority: important Section: admin Installed-Size: 0 Maintainer: Debian Adduser Developers Architecture: all Multi-Arch: foreign Version: 113.118 Description: No-install EOF
Cluster Overview
ceph-osd – is responsible for data storage on disk. A network service is launched for each disk to accept and execute read or write requests for objects. Two partitions are created on the disk. One contains information about the cluster, disk number, and the cluster keys. This 1KB information is created once when the disk is added and I have never noticed it changing. The second partition does not have a file system and stores the CEPH binary data. Automatic installation in previous versions created an xfs partition of 100MB for service information. I converted the disk to MBR and allocated only 16MB – the service does not complain. I think xfs could easily be replaced with ext. This partition is mounted in /var/lib/…, where the service reads OSD information and finds the link to the block device that stores the binary data. Theoretically, auxiliary data could be placed in /var/lib/…, and the disk could be entirely allocated for data. When creating an OSD through ceph-deploy, a rule is automatically created for mounting the partition in /var/lib/…, and read permissions are assigned to the ceph user for the required block device. When installing manually, this must be done by yourself; it is not mentioned in the documentation. It is also advisable to specify the osd memory target parameter to ensure sufficient physical memory.
ceph-mds. At a low level, CEPH is an object storage system. The ability for block storage comes down to storing each 4MB block as an object. File storage works on the same principle. Two pools are created: one for metadata and another for data. They are combined into a file system. At this point, a kind of record is created, so if the file system is deleted but both pools are preserved, it cannot be recovered. There is a procedure for extracting files by blocks, which I have not tested. The ceph-mds service is responsible for access to the file system. A separate instance of the service is required for each file system. There is an 'index' option that allows creating a semblance of multiple file systems in one – this has also not been tested.
ceph-mon — this service keeps the cluster map. It includes information about all OSDs, the algorithm for distributing PGs across OSDs, and, most importantly, data about all objects (the details of this mechanism are unclear to me: there is a directory /var/lib/ceph/mon/.../store.db, which contains a large file — 26MB, and there are 105K objects in the cluster, resulting in just over 256 bytes per object — I think the monitor stores a list of all objects and the PGs they are in). Damage to this directory leads to the loss of all data in the cluster. Hence, it is concluded that CRUSH shows how PGs are located across OSDs, and how objects are located across PGs — they are stored centrally within the database, no matter how much the developers try to avoid using that word. As a consequence, firstly, we cannot install the system on a flash drive in RO mode, as the database undergoes constant writing; a separate disk is required for these (likely no more than 1 GB). Secondly, it is necessary to have a real-time copy of this database. If there are several monitors, redundancy is ensured automatically, but in our case, there is only one monitor, at most two. There is a theoretical procedure for restoring the monitor based on OSD data, which I have resorted to three times for various reasons, and each time there were no error messages, nor any data either. Unfortunately, this mechanism does not work. Either we exploit a miniature partition on OSD and assemble RAID for storing the database, which will surely have a very negative impact on performance, or we allocate at least two reliable physical media, preferably USB, to avoid occupying ports.
rados-gw — exports object storage via the S3 protocol and similar. Creates multiple pools, unclear why. I haven't experimented much.
ceph-mgr — when this service is installed, several modules are launched. One of them is an unremovable autoscale. It seeks to maintain the correct number of PGs/OSDs. If one wants to manage the ratio manually, scaling can be disabled for each pool, but in that case, the module crashes due to division by zero, and the cluster status becomes ERROR. The module is written in Python, and if you comment out the necessary line, it leads to its disabling. I’m too lazy to recall the details.
List of sources used:
Script listings:
Installing the system via debootstrap
blkdev=sdb1
mkfs.btrfs -f /dev/$blkdev
mount /dev/$blkdev /mnt
cd /mnt
for i in {@,@var,@home}; do btrfs subvolume create $i; done
mkdir snapshot @/{var,home}
for i in {var,home}; do mount -o bind @${i} @/$i; done
debootstrap buster @ http://deb.debian.org/debian; echo $?
for i in {dev,proc,sys}; do mount -o bind /$i @/$i; done
cp /etc/bash.bashrc @/etc/
chroot /mnt/@ /bin/bash
echo rbd1 > /etc/hostname
passwd
uuid=`blkid | grep $blkdev | cut -d """ -f 2`
cat < /etc/fstab
UUID=$uuid / btrfs noatime,nodiratime,subvol=@ 0 1
UUID=$uuid /var btrfs noatime,nodiratime,subvol=@var 0 2
UUID=$uuid /home btrfs noatime,nodiratime,subvol=@home 0 2
EOF
cat <> /var/lib/dpkg/status
Package: lvm2
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 0
Maintainer: Debian Adduser Developers
Architecture: all
Multi-Arch: foreign
Version: 113.118
Description: No-install
Package: sudo
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 0
Maintainer: Debian Adduser Developers
Architecture: all
Multi-Arch: foreign
Version: 113.118
Description: No-install
EOF
exit
grub-install --boot-directory=@/boot/ /dev/$blkdev
init 6
apt -yq install --no-install-recommends linux-image-amd64 bash-completion ed btrfs-progs grub-pc iproute2 ssh smartmontools ntfs-3g net-tools man
exit
grub-install --boot-directory=@/boot/ /dev/$blkdev
init 6Creating a cluster
apt -yq install --no-install-recommends gnupg wget ca-certificates
echo 'deb https://download.ceph.com/debian-octopus/ buster main' >> /etc/apt/sources.list
wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -
apt update
apt -yq install --no-install-recommends ceph-common ceph-mon
echo 192.168.11.11 rbd1 >> /etc/hosts
uuid=`cat /proc/sys/kernel/random/uuid`
cat < /etc/ceph/ceph.conf
[global]
fsid = $uuid
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
mon allow pool delete = true
mon host = 192.168.11.11
mon initial members = rbd1
mon max pg per osd = 385
osd crush update on start = false
#osd memory target = 2147483648
osd memory target = 1610612736
osd scrub chunk min = 1
osd scrub chunk max = 2
osd scrub sleep = .2
osd pool default pg autoscale mode = off
osd pool default size = 1
osd pool default min size = 1
osd pool default pg num = 1
osd pool default pgp num = 1
[mon]
mgr initial modules = dashboard
EOF
ceph-authtool --create-keyring ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
ceph-authtool --create-keyring ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
cp ceph.client.admin.keyring /etc/ceph/
ceph-authtool --create-keyring bootstrap-osd.ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
cp bootstrap-osd.ceph.keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
ceph-authtool ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
ceph-authtool ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
monmaptool --create --add rbd1 192.168.11.11 --fsid $uuid monmap
rm -R /var/lib/ceph/mon/ceph-rbd1/*
ceph-mon --mkfs -i rbd1 --monmap monmap --keyring ceph.mon.keyring
chown ceph:ceph -R /var/lib/ceph
systemctl enable ceph-mon@rbd1
systemctl start ceph-mon@rbd1
ceph mon enable-msgr2
ceph status
# dashboard
apt -yq install --no-install-recommends ceph-mgr ceph-mgr-dashboard python3-distutils python3-yaml
mkdir /var/lib/ceph/mgr/ceph-rbd1
ceph auth get-or-create mgr.rbd1 mon 'allow profile mgr' osd 'allow *' mds 'allow *' > /var/lib/ceph/mgr/ceph-rbd1/keyring
systemctl enable ceph-mgr@rbd1
systemctl start ceph-mgr@rbd1
ceph config set mgr mgr/dashboard/ssl false
ceph config set mgr mgr/dashboard/server_port 7000
ceph dashboard ac-user-create root 1111115 administrator
systemctl stop ceph-mgr@rbd1
systemctl start ceph-mgr@rbd1Adding OSD (part)
apt install ceph-osd
osdnum=`ceph osd create`
mkdir -p /var/lib/ceph/osd/ceph-$osdnum
mkfs -t xfs /dev/sda1
mount -t xfs /dev/sda1 /var/lib/ceph/osd/ceph-$osdnum
cd /var/lib/ceph/osd/ceph-$osdnum
ceph auth get-or-create osd.0 mon 'profile osd' mgr 'profile osd' osd 'allow *' > /var/lib/ceph/osd/ceph-$osdnum/keyring
ln -s /dev/disk/by-partuuid/d8cc3da6-02 block
ceph-osd -i $osdnum --mkfs
#chown ceph:ceph /dev/sd?2
chown ceph:ceph -R /var/lib/ceph
systemctl enable ceph-osd@$osdnum
systemctl start ceph-osd@$osdnumSummary
The main marketing advantage of CEPH is CRUSH — the algorithm for data placement. Monitors distribute this algorithm to clients, after which clients directly request the desired node and the required OSD. CRUSH ensures the absence of centralization. It is a small file that could even be printed and hung on a wall. Practice has shown that CRUSH is not a comprehensive map. If you destroy and recreate the monitors while preserving all OSDs and CRUSH, it is insufficient for cluster recovery. Thus, it is concluded that each monitor stores some metadata about the entire cluster. This small volume of metadata does not impose restrictions on cluster size, but it requires ensuring their preservation, which rules out saving disk space by installing the system on a flash drive and excludes clusters with fewer than three nodes. The developer's aggressive policy towards optional features is far from minimalism. The documentation is at a level of: 'for what exists, thank you, but it is very, very sparse.' The possibility of interacting with services at a low level is provided, but the documentation only superficially addresses this topic, so it’s more likely a no than a yes. There are practically no chances of recovering data from an emergency situation.
Options for further action: abandon CEPH and resort to a simple multi-disk btrfs (or xfs, zfs), find out new information about CEPH that will allow its utilization under the specified conditions, or try to develop your own storage system as a way to enhance skills.
Source: habr.com
