Choosing CEPH. Part 1
We had five racks, ten optical switches, configured BGP, a couple dozen SSDs, and a bunch of SAS disks of all colors and sizes, along with Proxmox and the desire to store all static data in our own S3 storage. Not that all this was necessary for virtualization, but since I started using open-source, I wanted to fully pursue my hobby. The only thing that concerned me was BGP. There is no one more helpless, irresponsible, and unethical in the world than internal routing via BGP. And I knew that we would soon dive into it.
The task was simple: we had CEPH, but it wasn't working very well. We needed to make it 'good'.
The cluster I inherited was heterogeneous, hastily configured, and practically untuned. It consisted of two groups of different nodes, with one common network serving as both the cluster and public network. The nodes were populated with four types of disks: two types of SSDs, set up in two separate placement rules, and two types of HDDs of different sizes, grouped into a third category. The issue with varying sizes was addressed by using different weights for OSD.
We divided the setup into two parts — tuning the operating system and tuning CEPH itself and its settings.
Boosting the OS
Network
High latency affected both the writing and balancing processes. During writing — because the client does not receive confirmation of successful writing until data replicas in other placement groups confirm success. Since our replica distribution rules in the CRUSH map had one replica per host, the network was always in use.
Therefore, first of all, I decided to slightly adjust the current network while simultaneously trying to convince others to switch to separate networks.
To start, I tweaked the network card settings, beginning with queue settings:
What was configured:
ethtool -l ens1f1
root@ceph01:~# ethtool -l ens1f1
Channel parameters for ens1f1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 63
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 1
root@ceph01:~# ethtool -g ens1f1
Ring parameters for ens1f1:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 256
root@ceph01:~# ethtool -l ens1f1
Channel parameters for ens1f1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 63
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 1It's clear that the current parameters are far from the maximums. I increased them:
root@ceph01:~# ethtool -G ens1f0 rx 4096
root@ceph01:~# ethtool -G ens1f0 tx 4096
root@ceph01:~# ethtool -L ens1f0 combined 63Guided by an excellent article
, I increased the length of the sending queue. txqueuelen from 1000 to 10,000
root@ceph01:~#ip link set ens1f0 txqueuelen 10000And following the documentation of ceph
increased MTU to 9000.
root@ceph01:~#ip link set dev ens1f0 mtu 9000Added to /etc/network/interfaces, so that all the above loads at startup
cat /etc/network/interfaces
root@ceph01:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto ens1f0
iface ens1f0 inet manual
post-up /sbin/ethtool -G ens1f0 rx 4096
post-up /sbin/ethtool -G ens1f0 tx 4096
post-up /sbin/ethtool -L ens1f0 combined 63
post-up /sbin/ip link set ens1f0 txqueuelen 10000
mtu 9000
auto ens1f1
iface ens1f1 inet manual
post-up /sbin/ethtool -G ens1f1 rx 4096
post-up /sbin/ethtool -G ens1f1 tx 4096
post-up /sbin/ethtool -L ens1f1 combined 63
post-up /sbin/ip link set ens1f1 txqueuelen 10000
mtu 9000After that, following the same article, I began to thoughtfully tweak the kernel parameters 4.15. Considering that the nodes have 128G RAM, I created a configuration file for sysctl
cat /etc/sysctl.d/50-ceph.conf
net.core.rmem_max = 56623104
# Maximum size of the receive buffer for all connections 54M
net.core.wmem_max = 56623104
# Maximum size of the transmit buffer for all connections 54M
net.core.rmem_default = 56623104
# Default size of the receive buffer for all connections. 54M
net.core.wmem_default = 56623104
# Default size of the transmit buffer for all connections 54M
# per socket
net.ipv4.tcp_rmem = 4096 87380 56623104
# Vector (minimum, default, maximum) variable in the tcp_rmem file
# contains 3 integers that define the size of the TCP socket's receive buffer.
# Minimum: each TCP socket has the right to use this memory at
# the time of its creation. The ability to use such a buffer
# is guaranteed even when the limit threshold is reached (moderate memory pressure).
# The default minimum buffer size is 8 KB (8192).
# Default value: the amount of memory allowed for the default TCP socket
# receive buffer. This value is applied instead of
# the parameter /proc/sys/net/core/rmem_default, used by other protocols.
# The default buffer size used is typically (by default)
# 87830 bytes. This defines the window size of 65535 with
# the default value of tcp_adv_win_scale and tcp_app_win = 0,
# slightly lower than the default value defined for tcp_app_win.
# Maximum: the maximum size of the buffer that can be automatically
# allocated for receiving by a TCP socket. This value does not override the maximum,
# specified in the /proc/sys/net/core/rmem_max file. In "static"
# memory allocation using SO_RCVBUF, this parameter is irrelevant.
net.ipv4.tcp_wmem = 4096 65536 56623104
net.core.somaxconn = 5000
# Maximum number of open sockets waiting for connections.
net.ipv4.tcp_timestamps=1
# Allows the use of timestamps in accordance with RFC 1323.
net.ipv4.tcp_sack=1
# Allow selective acknowledgments for the TCP protocol
net.core.netdev_max_backlog=5000 (default 1000)
# maximum number of packets in the queue to be processed if
# the interface receives packets faster than the kernel can handle them.
net.ipv4.tcp_max_tw_buckets=262144
# Maximum number of sockets in the TIME-WAIT state at the same time.
# If this threshold is exceeded, the "extra" socket is destroyed and
# a message is written to the system log.
net.ipv4.tcp_tw_reuse=1
# Allow the reuse of TIME-WAIT sockets in cases,
# where the protocol considers it safe.
net.core.optmem_max=4194304
# Increase the maximum allocatable buffer space
# measured in page units (4096 bytes)
net.ipv4.tcp_low_latency=1
# Allows the TCP/IP stack to favor low latency
# over higher throughput.
net.ipv4.tcp_adv_win_scale=1
# This variable impacts the computation of the amount of memory in the socket buffer,
# allocated for the TCP window size and application buffer.
# If tcp_adv_win_scale is negative, the size is computed using the following expression:
# Bytes - bytes2 raised to the power of -tcp_adv_win_scale
# Where bytes is the window size in bytes. If tcp_adv_win_scale is positive,
# the size is defined using the following expression:
# Bytes - bytes2 raised to the power of tcp_adv_win_scale
# The variable takes an integer value. Default value – 2,
# meaning ¼ of the volume defined by the tcp_rmem variable is allocated for the application buffer.
net.ipv4.tcp_slow_start_after_idle=0
# mechanism for resetting slow start that resets the congestion window size
# if the connection has not been used for a specified period of time.
# It is better to disable SSR on the server to improve the performance of
# long-lived connections.
net.ipv4.tcp_no_metrics_save=1
# Do not save TCP connection measurement results in the cache upon its closure.
net.ipv4.tcp_syncookies=0
# Disable the syncookie sending mechanism
net.ipv4.tcp_ecn=0
# Explicit Congestion Notification in
# TCP connections. Used to inform of congestion occurring on the route to a specified host or network. Can inform the
# sender host of the need to reduce the packet transmission rate through
# a specific router or firewall.
net.ipv4.conf.all.send_redirects=0
# disables the sending of ICMP Redirects to other hosts. This option must be enabled if the host acts as a router of any kind.
# We have no routing.
net.ipv4.ip_forward=0
# Disables IP forwarding. We are not a gateway, Docker is not running on machines,
# we don't need this.
net.ipv4.icmp_echo_ignore_broadcasts=1
# Do not respond to ICMP ECHO requests sent via broadcast packets
net.ipv4.tcp_fin_timeout=10
# defines the time to keep a socket in the FIN-WAIT-2 state after being closed by the local side. Default is 60
net.core.netdev_budget=600 # (default 300)
# If the processing of software interrupts does not happen quickly enough,
# the rate of incoming data may exceed the kernel's ability
# to empty the buffer. As a result, NIC buffers will overflow, and traffic will be lost.
# Sometimes, it is necessary to increase the duration of the SoftIRQs
# (software interrupts) with the CPU. This is handled by netdev_budget.
# Default value is 300. This parameter will cause the SoftIRQ process to handle
# 300 packets from the NIC before relinquishing the CPU.
net.ipv4.tcp_fastopen=3
# TFO TCP Fast Open
# if both client and server support TFO, which is indicated by
# a special flag in the TCP packet. In our case, it's placebo, just
# looks nice)Cluster network was dedicated on separate 10Gbps network interfaces into a separate flat network. Each machine was equipped with dual-port network cards mellanox 10/25 Gbps, connected to two separate 10Gbps switches. Aggregation was achieved using OSPF, as bonding with LACP somehow showed a total bandwidth of only 16 Gbps, while OSPF fully utilized both tens on each machine. Future plans were to use ROCE on these Mellanox devices to reduce latency. Here's how we configured this part of the network:
- Since the machines themselves have external IP addresses on BGP, we needed the following software — (more specifically, at the time of writing this article it was ) already installed.
- In total, there were two network interfaces on each machine — four ports in total. One network card with two ports was connected to the factory and BGP was configured on it, while the other with two ports was connected to two different switches and had OSPF configured on it.
More details on OSPF setup: The main task is to aggregate two links and have fault tolerance.
two network interfaces configured in two simple flat networks — 10.10.10.0/24 and 10.10.20.0/24
1: ens1f0: mtu 9000 qdisc mq state UP group default qlen 1000
inet 10.10.10.2/24 brd 10.10.10.255 scope global ens1f0
2: ens1f1: mtu 9000 qdisc mq state UP group default qlen 1000
inet 10.10.20.2/24 brd 10.10.20.255 scope global ens1f1through which the machines can see each other.
DISK
The next step was to optimize disk performance. For SSDs, I changed the scheduler to noop, for HDDs — deadline. In simple terms, NOOP operates on the principle of "first come, first served," which in English is referred to as "FIFO (First In, First Out)." Requests stack up in the queue as they arrive. DEADLINE is more tailored for reads, plus processes in the queue get near-monopolistic access to the disk during their operation. This works excellently for our system—since only one process, the OSD daemon, works with each disk.
(Those interested in delving deeper into the I/O scheduler can read about it here:
For those who prefer to read in Russian: )
Linux tuning recommendations also suggest increasing nr_requests
nr_requests
The value of nr_requests determines the number of I/O requests that get buffered before the I/O scheduler sends or receives data to the block device. If you are using a RAID card or block device that can handle a larger queue than what the I/O scheduler is set to, increasing the value of nr_requests may help improve throughput and reduce server load during high I/O activity. If you are using Deadline or CFQ as the scheduler, it is recommended to set the nr_request value to twice the queue depth.
However, the developers of CEPH themselves assure us that their priority system works better.
WBThrottle and/or nr_requests
WBThrottle and/or nr_requests
The file storage utilizes buffered I/O operations for writing; this brings a range of advantages if the file storage journal is on a faster medium. Client requests are notified as soon as data is written to the journal, which is then flushed to the data disk at a later time using standard Linux functionality. This allows OSD spinning disks to provide write latency similar to SSDs for small packet writes. Such delayed write also enables the kernel to rearrange I/O operation requests to the disk in hopes of either merging them or allowing the existing heads to choose a more optimal path across their platters. The end effect is that you can squeeze slightly more I/O operations from each disk than would be possible with direct or synchronous I/O operations.
However, there is a specific problem if the volume of incoming writes to a Ceph cluster exceeds the capabilities of the underlying disks. In such a scenario, the total number of pending I/O operations waiting to be written to disk can uncontrollably grow, resulting in queues of I/O operations saturating the entire disk and the Ceph queues. Read requests are particularly affected as they get stuck between write requests, which can take several seconds to flush to the main disk.
To overcome this issue, Ceph has an integrated mechanism for writeback throttling in its file storage called WBThrottle. It is designed to limit the total volume of queued write-back I/O operations that may begin their flushing process earlier than would naturally occur due to kernel triggering. Unfortunately, testing shows that the default settings may still not curb the existing behavior to a level that reduces the impact on read operation latency. Tuning can alter this behavior and reduce overall write queue lengths, potentially minimizing such an impact. However, there is a trade-off: by decreasing the total maximum number of allowed queued writes, you may hinder the kernel's ability to maximize the efficiency of its request sorting. It is essential to consider what is more necessary for your specific use case and workloads and adjust accordingly.
To manage the depth of write-back queues, you can either lower the total maximum number of outstanding I/O operations using WBThrottle settings or reduce the maximum value for outstanding operations at the block level of your kernel. Both can effectively manage the same behavior, and your preferences will guide the implementation of this setting.
It is also worth noting that the existing priority system in Ceph is more effective for shorter disk-level requests. By reducing the overall queue to a given disk, the primary queuing location moves to Ceph, where it has greater control over the priority of the I/O operation. Consider the following example:
echo 8 > /sys/block/sda/queue/nr_requestsCOMMON
And a few more kernel settings to make your machine smooth and silky, squeezing out a bit more performance from the hardware
cat /etc/sysctl.d/60-ceph2.conf
kernel.pid_max = 4194303
# There are 25 disks in each machine, so we calculated that there would be many processes
kernel.threads-max=2097152
# Threads, of course, too.
vm.max_map_count=524288
# Increased the number of memory map areas for the process.
# As noted in the documentation regarding kernel variables,
# memory map areas are used as a side effect of calling
# malloc, directly via mmap, mprotect, and madvise, as well as when loading
# shared libraries.
fs.aio-max-nr=50000000
# Tuning input-output parameters
# The Linux kernel provides asynchronous non-blocking input-output (AIO) function,
# which allows the process to initiate multiple input-output operations
# simultaneously without waiting for any of them to complete.
# This helps improve the performance of applications
# that can overlap processing and input-output.
# The aio-max-nr parameter defines the maximum number of allowed
# simultaneous requests.
vm.min_free_kbytes=1048576
# The minimum size of free memory that needs to be maintained.
# Set to 1Gb, which is quite sufficient for the operating system to work,
# and helps avoid the OOM Killer for OSD processes. While there is already plenty of memory,
# it's always good to have a buffer.
vm.swappiness=10
# We say to use swap if 10% of the memory is free.
# On machines with 128G RAM, 10% is 12 Gigs. More than enough for operation.
# The default parameter of 60% would cause the system to lag, going into swap,
# when there was still plenty of free memory.
vm.vfs_cache_pressure=1000
# Increased from the default 100. We make the kernel more actively evict
# unused memory pages from the cache.
vm.zone_reclaim_mode=0
# Allows for setting more or less aggressive approaches to
# reclaiming memory when the zone runs out of memory.
# If set to zero, no zone reclaim occurs.
# For file servers or workloads,
# it is often beneficial to keep their data cached, hence keeping zone_reclaim_mode
# disabled, since the caching effect is
# likely to be more important than the location of the data.
vm.dirty_ratio=20
# Percentage of RAM that can be designated for "dirty" pages.
# Calculated from approximate reasoning:
# The system has 128 gigabytes of memory.
# Approximately 20 SSD disks, which are configured in CEPH to
# allocate 3G of RAM for caching.
# Approximately 40 HDD disks, for which this parameter equals 1G.
# 20% of 128 is 25.6 gigabytes. Thus, in the case of maximum memory utilization,
# the system should have 2.4G of memory left. That should be enough for it to survive and wait
# for the cavalry's hoofbeats - that is, the arrival of DevOps who will fix everything.
vm.dirty_background_ratio=3
# Percentage of system memory that can be filled with dirty pages before
# background processes pdflush/flush/kdmflush write them to disk.
fs.file-max=524288
# Well, we will likely have many more open files than specified by default. Diving into CEPH
Settings to delve into in more detail:
cat /etc/ceph/ceph.conf
osd:
journal_aio: true # Three parameters enabling
journal_block_align: true # direct i/o
journal_dio: true # to the journal
journal_max_write_bytes: 1073714824 # Let’s stretch the maximum size
# of a single write operation to the journal
journal_max_write_entries: 10000 # And the number of concurrent entries
journal_queue_max_bytes: 10485760000
journal_queue_max_ops: 50000
rocksdb_separate_wal_dir: true # We decided to create a separate WAL
# Even tried to set this up for it
# NVMe
bluestore_block_db_create: true # And a separate device for the journal
bluestore_block_db_size: '5368709120 #5G'
bluestore_block_wal_create: true
bluestore_block_wal_size: '1073741824 #1G'
bluestore_cache_size_hdd: '3221225472 # 3G'
# large memory capacity allows
# storing quite substantial volumes
bluestore_cache_size_ssd: '9663676416 # 9G'
keyring: /var/lib/ceph/osd/ceph-$id/keyring
osd_client_message_size_cap: '1073741824 #1G'
osd_disk_thread_ioprio_class: idle
osd_disk_thread_ioprio_priority: 7
osd_disk_threads: 2 # number of threads for the daemon per disk
osd_failsafe_full_ratio: 0.95
osd_heartbeat_grace: 5
osd_heartbeat_interval: 3
osd_map_dedup: true
osd_max_backfills: 2 # number of concurrent fill operations on one OSD.
osd_max_write_size: 256
osd_mon_heartbeat_interval: 5
osd_op_threads: 16
osd_op_num_threads_per_shard: 1
osd_op_num_threads_per_shard_hdd: 2
osd_op_num_threads_per_shard_ssd: 2
osd_pool_default_min_size: 1 # Features of greediness. Space became scarce very quickly
osd_pool_default_size: 2 # because the temporary solution was to reduce the number
# of data replicas
osd_recovery_delay_start: 10.000000
osd_recovery_max_active: 2
osd_recovery_max_chunk: 1048576
osd_recovery_max_single_start: 3
osd_recovery_op_priority: 1
osd_recovery_priority: 1 # parameter controlled as needed on the fly
osd_recovery_sleep: 2
osd_scrub_chunk_max: 4Some parameters tested on QA in version 12.2.12 are absent in version ceph 12.2.2, for example osd_recovery_threads. Therefore, the plan included upgrading to 12.2.12 in production. Practice showed compatibility within a cluster of versions 12.2.2 and 12.2.12, which allows for a rolling update.
Test cluster
Naturally, for testing, it was necessary to have the same version as in production, but at the start of my work with the cluster, only a newer version was available in the repository. Noticing that the differences in the minor version were not significant,1393 the lines in the configs versus 1436 the new version), we decided to start testing the new one (we would have to update anyway, so why stick with the old junk)
The only thing we tried to keep from the old version was the package ceph-deploy, since some utilities (and some staff) were accustomed to its syntax. The new version differed significantly, but it did not affect the operation of the cluster itself, so the version was retained. 1.5.39
Since the ceph-disk team clearly states that it is deprecated and advises users to switch to the ceph-volume command — we started creating OSDs using this command, without wasting time on the obsolete one.
The plan was as follows — to create a mirror from two SSD disks, on which we would place the OSD logs, while the OSDs themselves would be located on spindle SAS drives. This way, we would safeguard against data issues if a disk with a log failed.
The cluster was built following the documentation.
cat /etc/ceph/ceph.conf
root@ceph01-qa:~# cat /etc/ceph/ceph.conf # pre-prepared config
[client]
rbd_cache = true
rbd_cache_max_dirty = 50331648
rbd_cache_max_dirty_age = 2
rbd_cache_size = 67108864
rbd_cache_target_dirty = 33554432
rbd_cache_writethrough_until_flush = true
rbd_concurrent_management_ops = 10
rbd_default_format = 2
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster network = 10.10.10.0/24
debug_asok = 0/0
debug_auth = 0/0
debug_buffer = 0/0
debug_client = 0/0
debug_context = 0/0
debug_crush = 0/0
debug_filer = 0/0
debug_filestore = 0/0
debug_finisher = 0/0
debug_heartbeatmap = 0/0
debug_journal = 0/0
debug_journaler = 0/0
debug_lockdep = 0/0
debug_mon = 0/0
debug_monc = 0/0
debug_ms = 0/0
debug_objclass = 0/0
debug_objectcatcher = 0/0
debug_objecter = 0/0
debug_optracker = 0/0
debug_osd = 0/0
debug_paxos = 0/0
debug_perfcounter = 0/0
debug_rados = 0/0
debug_rbd = 0/0
debug_rgw = 0/0
debug_throttle = 0/0
debug_timer = 0/0
debug_tp = 0/0
fsid = d0000000d-4000-4b00-b00b-0123qwe123qwf9
mon_host = ceph01-q, ceph02-q, ceph03-q
mon_initial_members = ceph01-q, ceph02-q, ceph03-q
public network = 8.8.8.8/28 # address changed, of course ))
rgw_dns_name = s3-qa.mycompany.ru # and this address changed
rgw_host = s3-qa.mycompany.ru # and this one too
[mon]
mon allow pool delete = true
mon_max_pg_per_osd = 300 # more than three hundred placement groups
# not decided for disk
# although this parameter, of course, depends on the number of pools,
# their sizes, and the number of OSDs. Having few but healthy PGs
# is also not the best choice - balance accuracy suffers
mon_osd_backfillfull_ratio = 0.9
mon_osd_down_out_interval = 5
mon_osd_full_ratio = 0.95 # currently for SSD disks, the space for their
# journal is the same device as for OSDs
# we decided that 5% of the disk (which is 1.2Tb in size)
# should be enough, and correlates with the parameter
# bluestore_block_db_size plus variability for larger
# placement groups
mon_osd_nearfull_ratio = 0.9
mon_pg_warn_max_per_osd = 520
[osd]
bluestore_block_db_create = true
bluestore_block_db_size = 5368709120 #5G
bluestore_block_wal_create = true
bluestore_block_wal_size = 1073741824 #1G
bluestore_cache_size_hdd = 3221225472 # 3G
bluestore_cache_size_ssd = 9663676416 # 9G
journal_aio = true
journal_block_align = true
journal_dio = true
journal_max_write_bytes = 1073714824
journal_max_write_entries = 10000
journal_queue_max_bytes = 10485760000
journal_queue_max_ops = 50000
keyring = /var/lib/ceph/osd/ceph-$id/keyring
osd_client_message_size_cap = 1073741824 #1G
osd_disk_thread_ioprio_class = idle
osd_disk_thread_ioprio_priority = 7
osd_disk_threads = 2
osd_failsafe_full_ratio = 0.95
osd_heartbeat_grace = 5
osd_heartbeat_interval = 3
osd_map_dedup = true
osd_max_backfills = 4
osd_max_write_size = 256
osd_mon_heartbeat_interval = 5
osd_op_num_threads_per_shard = 1
osd_op_num_threads_per_shard_hdd = 2
osd_op_num_threads_per_shard_ssd = 2
osd_op_threads = 16
osd_pool_default_min_size = 1
osd_pool_default_size = 2
osd_recovery_delay_start = 10.0
osd_recovery_max_active = 1
osd_recovery_max_chunk = 1048576
osd_recovery_max_single_start = 3
osd_recovery_op_priority = 1
osd_recovery_priority = 1
osd_recovery_sleep = 2
osd_scrub_chunk_max = 4
osd_scrub_chunk_min = 2
osd_scrub_sleep = 0.1
rocksdb_separate_wal_dir = true# создаем мониторы
root@ceph01-qa:~#ceph-deploy mon create ceph01-q
# генерируем ключи для аутентификации нод в кластере
root@ceph01-qa:~#ceph-deploy gatherkeys ceph01-q
# Это если поштучно. Если у нас несколько машин доступны - те, которые описаны в конфиге в секции
# mon_initial_members = ceph01-q, ceph02-q, ceph03-q
# можно запустить эти две команды в виде одной
root@ceph01-qa:~#ceph-deploy mon create-initial
# Положим ключи в указанные в конфиге места
root@ceph01-qa:~#cat ceph.bootstrap-osd.keyring > /var/lib/ceph/bootstrap-osd/ceph.keyring
root@ceph01-qa:~#cat ceph.bootstrap-mgr.keyring > /var/lib/ceph/bootstrap-mgr/ceph.keyring
root@ceph01-qa:~#cat ceph.bootstrap-rgw.keyring > /var/lib/ceph/bootstrap-rgw/ceph.keyring
# создадим ключ для управления кластером
root@ceph01-qa:~#ceph-deploy admin ceph01-q
# и менеджер, плагинами управлять
root@ceph01-qa:~#ceph-deploy mgr create ceph01-qThe first issue I encountered with this version of ceph-deploy and the cluster version 12.2.12 was an error trying to create an OSD with a db on a software RAID —
root@ceph01-qa:~#ceph-volume lvm create --bluestore --data /dev/sde --block.db /dev/md0
blkid could not detect a PARTUUID for device: /dev/md1Indeed, blkid does not show a PARTUUID, so I had to create partitions manually:
root@ceph01-qa:~#parted /dev/md0 mklabel GPT
# there will be many partitions,
# without GPT they cannot be created
# the size of the partition is specified in the config above = bluestore_block_db_size: '5368709120 #5G'
# I have 20 disks for OSD, it’s tedious to create partitions by hand
# so I made a loop
root@ceph01-qa:~#for i in {1..20}; do echo -e "nnnn+5Gnw" | fdisk /dev/md0; doneIt seems everything is ready, let's try to create the OSD again and we get the following error (which, by the way, did not occur in production)
when creating a bluestore OSD without specifying the path for WAL, but with specifying db
root@ceph01-qa:~#ceph-volume lvm create --bluestore --data /dev/sde --block.db /dev/md0
stderr: 2019-04-12 10:39:27.211242 7eff461b6e00 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: 2019-04-12 10:39:27.213185 7eff461b6e00 -1 bdev(0x55824c273680 /var/lib/ceph/osd/ceph-0//block.wal) open open got: (22) Invalid argument
stderr: 2019-04-12 10:39:27.213201 7eff461b6e00 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db add block device(/var/lib/ceph/osd/ceph-0//block.wal) returned: (22) Invalid argument
stderr: 2019-04-12 10:39:27.999039 7eff461b6e00 -1 bluestore(/var/lib/ceph/osd/ceph-0/) mkfs failed, (22) Invalid argument
stderr: 2019-04-12 10:39:27.999057 7eff461b6e00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (22) Invalid argument
stderr: 2019-04-12 10:39:27.999141 7eff461b6e00 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0/: (22) Invalid argumentHowever, if on the same mirror (or elsewhere, as you choose) another partition is created for WAL and specified during OSD creation — then everything goes smoothly (except for the separate WAL, which you may not have wanted).
But, since it was planned to eventually move WAL to NVMe, the practice wasn’t wasted.
root@ceph01-qa:~#ceph-volume lvm create --bluestore --data /dev/sdf --block.wal /dev/md0p2 --block.db /dev/md1p2We created monitors, managers, and OSDs. Now I want to categorize them differently, as I plan to have disks of various types — fast pools on SSD and large, but slow on SAS disks.
Let's assume that there are 20 disks on the servers, the first ten are one type, the second ten are another.
The initial, default map looks like this:
ceph osd tree
root@ceph01-q:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 14.54799 root default
-3 9.09200 host ceph01-q
0 ssd 1.00000 osd.0 up 1.00000 1.00000
1 ssd 1.00000 osd.1 up 1.00000 1.00000
2 ssd 1.00000 osd.2 up 1.00000 1.00000
3 ssd 1.00000 osd.3 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
5 hdd 0.27299 osd.5 up 1.00000 1.00000
6 hdd 0.27299 osd.6 up 1.00000 1.00000
7 hdd 0.27299 osd.7 up 1.00000 1.00000
8 hdd 0.27299 osd.8 up 1.00000 1.00000
9 hdd 0.27299 osd.9 up 1.00000 1.00000
10 hdd 0.27299 osd.10 up 1.00000 1.00000
11 hdd 0.27299 osd.11 up 1.00000 1.00000
12 hdd 0.27299 osd.12 up 1.00000 1.00000
13 hdd 0.27299 osd.13 up 1.00000 1.00000
14 hdd 0.27299 osd.14 up 1.00000 1.00000
15 hdd 0.27299 osd.15 up 1.00000 1.00000
16 hdd 0.27299 osd.16 up 1.00000 1.00000
17 hdd 0.27299 osd.17 up 1.00000 1.00000
18 hdd 0.27299 osd.18 up 1.00000 1.00000
19 hdd 0.27299 osd.19 up 1.00000 1.00000
-5 5.45599 host ceph02-q
20 ssd 0.27299 osd.20 up 1.00000 1.00000
21 ssd 0.27299 osd.21 up 1.00000 1.00000
22 ssd 0.27299 osd.22 up 1.00000 1.00000
23 ssd 0.27299 osd.23 up 1.00000 1.00000
24 hdd 0.27299 osd.24 up 1.00000 1.00000
25 hdd 0.27299 osd.25 up 1.00000 1.00000
26 hdd 0.27299 osd.26 up 1.00000 1.00000
27 hdd 0.27299 osd.27 up 1.00000 1.00000
28 hdd 0.27299 osd.28 up 1.00000 1.00000
29 hdd 0.27299 osd.29 up 1.00000 1.00000
30 hdd 0.27299 osd.30 up 1.00000 1.00000
31 hdd 0.27299 osd.31 up 1.00000 1.00000
32 hdd 0.27299 osd.32 up 1.00000 1.00000
33 hdd 0.27299 osd.33 up 1.00000 1.00000
34 hdd 0.27299 osd.34 up 1.00000 1.00000
35 hdd 0.27299 osd.35 up 1.00000 1.00000
36 hdd 0.27299 osd.36 up 1.00000 1.00000
37 hdd 0.27299 osd.37 up 1.00000 1.00000
38 hdd 0.27299 osd.38 up 1.00000 1.00000
39 hdd 0.27299 osd.39 up 1.00000 1.00000
-7 6.08690 host ceph03-q
40 ssd 0.27299 osd.40 up 1.00000 1.00000
41 ssd 0.27299 osd.41 up 1.00000 1.00000
42 ssd 0.27299 osd.42 up 1.00000 1.00000
43 ssd 0.27299 osd.43 up 1.00000 1.00000
44 hdd 0.27299 osd.44 up 1.00000 1.00000
45 hdd 0.27299 osd.45 up 1.00000 1.00000
46 hdd 0.27299 osd.46 up 1.00000 1.00000
47 hdd 0.27299 osd.47 up 1.00000 1.00000
48 hdd 0.27299 osd.48 up 1.00000 1.00000
49 hdd 0.27299 osd.49 up 1.00000 1.00000
50 hdd 0.27299 osd.50 up 1.00000 1.00000
51 hdd 0.27299 osd.51 up 1.00000 1.00000
52 hdd 0.27299 osd.52 up 1.00000 1.00000
53 hdd 0.27299 osd.53 up 1.00000 1.00000
54 hdd 0.27299 osd.54 up 1.00000 1.00000
55 hdd 0.27299 osd.55 up 1.00000 1.00000
56 hdd 0.27299 osd.56 up 1.00000 1.00000
57 hdd 0.27299 osd.57 up 1.00000 1.00000
58 hdd 0.27299 osd.58 up 1.00000 1.00000
59 hdd 0.89999 osd.59 up 1.00000 1.00000
Let's create our own virtual racks and servers with blackjack and other things:
root@ceph01-q:~#ceph osd crush add-bucket rack01 root #created new root
root@ceph01-q:~#ceph osd crush add-bucket ceph01-q host #created new host
root@ceph01-q:~#ceph osd crush move ceph01-q root=rack01 #moved server to another rack
root@ceph01-q:~#osd crush add 28 1.0 host=ceph02-q # Added OSD to server
# If created incorrectly, you can remove it
root@ceph01-q:~# ceph osd crush remove osd.4
root@ceph01-q:~# ceph osd crush remove rack01Problems we faced in the production cluster when trying to create new host and move it to an existing rack — the command ceph osd crush move ceph01-host root=rack01 was hanging, and monitors started falling one by one. Interrupting the command with a simple CTRL+C brought the cluster back to life.
Searching revealed such a problem:
The solution turned out to be dumping the crushmap and removing the section rule replicated_ruleset
root@ceph01-prod:~#ceph osd getcrushmap -o crushmap.row #Dumping the map in raw format
root@ceph01-prod:~#crushtool -d crushmap.row -o crushmap.txt #Converting to readable format
root@ceph01-prod:~#vim crushmap.txt #Editing, removing rule replicated_ruleset
root@ceph01-prod:~#crushtool -c crushmap.txt -o new_crushmap.row #Compiling it back
root@ceph01-prod:~#ceph osd setcrushmap -i new_crushmap.row #Loading into the clusterAttention: this operation may cause a rebalance of the placement group among OSDs. It caused this for us, but it was very minor.
The oddity we encountered in the test cluster was that after rebooting the OSD servers forgot they were moved to new servers and racks, and returned to the root default.
As a result, after creating the final scheme where we set up a separate root for SSDs and a separate one for spinning disks, we distributed all the OSDs across racks and simply deleted the default root. After rebooting, the OSDs stayed in their places.
Later, digging through the documentation, we found a parameter that is responsible for this behavior. It's discussed in the second part.
How we created various groups by disk types.
First, we created two roots — for SSDs and for HDDs.
root@ceph01-q:~#ceph osd crush add-bucket ssd-root root
root@ceph01-q:~#ceph osd crush add-bucket hdd-root rootSince the servers are physically located in different racks — for convenience, we created racks and placed the servers in them.
# Стойки:
root@ceph01-q:~#ceph osd crush add-bucket ssd-rack01 rack
root@ceph01-q:~#ceph osd crush add-bucket ssd-rack02 rack
root@ceph01-q:~#ceph osd crush add-bucket ssd-rack03 rack
root@ceph01-q:~#ceph osd crush add-bucket hdd-rack01 rack
root@ceph01-q:~#ceph osd crush add-bucket hdd-rack01 rack
root@ceph01-q:~#ceph osd crush add-bucket hdd-rack01 rack
# Сервера
root@ceph01-q:~#ceph osd crush add-bucket ssd-ceph01-q host
root@ceph01-q:~#ceph osd crush add-bucket ssd-ceph02-q host
root@ceph01-q:~#ceph osd crush add-bucket ssd-ceph03-q host
root@ceph01-q:~#ceph osd crush add-bucket hdd-ceph01-q host
root@ceph01-q:~#ceph osd crush add-bucket hdd-ceph02-q host
root@ceph01-q:~#ceph osd crush add-bucket hdd-ceph02-q hostand allocated disks by their types across different servers.
root@ceph01-q:~# Disks 0 to 3 are SSDs, located in ceph01-q, assigning them to the server
root@ceph01-q:~# ssd-ceph01-q
root@ceph01-q:~#ceph osd crush add 0 1 host=ssd-ceph01-q
root@ceph01-q:~#ceph osd crush add 1 1 host=ssd-ceph01-q
root@ceph01-q:~#ceph osd crush add 2 1 host=ssd-ceph01-q
root@ceph01-q:~#ceph osd crush add 3 1 host=ssd-ceph01-q
root-ceph01-q:~# similarly with other serversAfter distributing disks across the ssd-root and hdd-root, we left the root-default empty, so we can delete it.
root-ceph01-q:~#ceph osd crush remove defaultNext, we need to create distribution rules that we will bind to the pools we create — in the rules, we will specify which roots data from our pool can be placed in and the level of replica uniqueness — for example, replicas must be on different servers or in different racks (even in different roots, if we have such a distribution).
Before choosing a type, it's better to read the documentation:
root-ceph01-q:~# ceph osd crush rule create-simple rule-ssd ssd-root host firstn
root-ceph01-q:~# ceph osd crush rule create-simple rule-hdd hdd-root host firstn
root-ceph01-q:~# We specified two rules for data replication
root-ceph01-q:~# between hosts — meaning the replica must reside on a different host,
root-ceph01-q:~# even if they are in the same rack.
root-ceph01-q:~# In production, if possible, it's better to distribute hosts
root-ceph01-q:~# across racks and indicate to distribute replicas across racks:
root-ceph01-q:~# ## ceph osd crush rule create-simple rule-ssd ssd-root rack firstnNow we create pools where we want to store our virtualization disk images in the future — PROXMOX:
root-ceph01-q:~# # ceph osd pool create {NAME} {pg_num} {pgp_num}
root-ceph01-q:~# ceph osd pool create ssd_pool 1024 1024
root-ceph01-q:~# ceph osd pool create hdd_pool 1024 1024And we tell these pools which placement rules to use
root-ceph01-q:~# ceph osd crush rule ls # view the list of rules
root-ceph01-q:~# ceph osd crush rule dump rule-ssd | grep rule_id # select the needed ID
root-ceph01-q:~# ceph osd pool set ssd_pool crush_rule 2
The choice of the number of placement groups should be approached with a pre-existing vision for your cluster — how many OSDs will be there, what percentage of the total volume will be in the pool, and how much data overall.
It is generally advisable to have no more than 300 placement groups per disk, and it will be easier to balance with smaller placement groups — that is, if your entire pool takes up 10 Tb and it has 10 PGs — balancing by moving terabyte-sized blocks (pg) will be problematic — it's easier and smoother to shift sand with smaller grain sizes using buckets.
But it should be remembered that the more PGs there are, the more resources are spent on calculating their placement — this starts to utilize memory and CPU.
A rough understanding can , provided by the developers of the CEPH documentation.
List of materials:
Source: habr.com
