An engineer from Clyso summarized the experience of building a storage cluster based on a fault-tolerant distributed Ceph system with a throughput exceeding tebibytes per second. It is noted that this is the first Ceph-based cluster to achieve such a figure, but before reaching the presented result, the engineers had to overcome a series of non-obvious pitfalls.
For example, to increase performance by 10-20%, it was sufficient to enable only maximum performance mode in the BIOS power-saving settings on the servers and disable c-state (c-state alters power-saving parameters based on load, which affects Ceph). It was also found that using NVMe drives, the Linux kernel spends a significant amount of time processing spin locks during IOMMU mapping updates. Disabling IOMMU in the kernel led to a noticeable increase in performance in block read and write tests with a size of 4MB.
However, disabling IOMMU did not resolve the performance drop issue during random writes of blocks sized 4KB. While investigating the matter, engineers came across patches in the Ceph build scripts from Gentoo and Ubuntu projects that included building with the RelWithDebInfo option, as this enabled the «-O2» optimization mode in GCC, which significantly improved Ceph's performance. Performance degradation also resulted from compiling with the TCMalloc library. Modifying compilation flags and ceasing to use TCMalloc resulted in a threefold reduction in compaction time and doubled performance for random write operations with 4K blocks. Additionally, optimization of Reef RocksDB settings and placement groups (PG) was also carried out.
The cluster was formed from 68 nodes based on servers Dell PowerEdge R6615 with AMD EPYC 9454P 48C/96T CPU. Each node contains 10 Dell 15.36TB NVMe drives, two 100GbE Mellanox ConnectX-6 Ethernet adapters, and 192GB of RAM. The software is based on Ubuntu 20.04.6 and Ceph 17.2.7. The cluster, which consists of 63 nodes, has 630 OSDs (Object Storage Daemon, a background process managing data storage in local storage, one OSD per NVMe drive), three MON processes (monitor, which tracks the cluster's status), and one MGR process (Manager, managing service). The storage size is 8.2 PB.
The throughput for sequential block read operations of 4M was 1025 GiB/s, and for writes, it was 270 GiB/s. For random reads of 4KB blocks, the performance was 25.5 million read operations per second and 4.9 million for writes. Enabling encryption reduced the read throughput to approximately 750 GiB/s. With EC62 error correction codes in use, the throughput was 547 GiB/s for reads and 387 GiB/s for writes (the write speed was higher than without error correction), while random access achieved 3.4M IOPS for reads and 936K IOPS for writes.
Additionally, it can be noted that in September, a similar throughput milestone in tebibytes per second was reached in the CERN storage exabyte cluster, implemented on the basis of the open distributed storage EOS built on the XRootD protocol.
Source: opennet.ru
