What do LVM and a matryoshka doll have in common?

Good day.
I want to share practical experience with the community regarding building a storage system for KVM using md RAID + LVM.

The program will include:

• Building md RAID 1 from NVMe SSD.
• Building md RAID 6 from SATA SSD and regular disks.
• Features of TRIM/DISCARD operation on SSD RAID 1/6.
• Creating a bootable md RAID 1/6 array on a common set of disks.
• Installing the system on NVMe RAID 1 without NVMe support in BIOS.
• Using LVM cache and LVM thin.
• Using BTRFS snapshots and send/receive for backups.
• Using LVM thin snapshots and thin_delta for BTRFS-style backups.

If you are interested, please read more.

Statement

The author takes no responsibility for the consequences of using or not using the materials/examples/code/advice/data from this article. By reading or otherwise using this material, you assume responsibility for all consequences of these actions. Possible consequences include:

• NVMe SSDs burnt to a crisp.
• Complete depletion of write resources and failure of SSD drives.
• Total loss of all data on all drives, including backups.
• Faulty computer hardware.
• Wasted time, nerves, and money.
• Any other consequences not listed above.

Hardware

Available was:

Motherboard from around 2013 based on the Z87 chipset paired with Intel Core i7 / Haswell.

• 4-core, 8-thread processor
• 32 gigabytes of DDR3 RAM
• 1 x 16 or 2 x 8 PCIe 3.0
• 1 x 4 + 1 x 1 PCIe 2.0
• 6 x 6 GBps SATA 3 connectors

SAS adapter LSI SAS9211-8I flashed into IT / HBA mode. The firmware with RAID support was intentionally replaced with HBA firmware so that:

1. This adapter could be easily discarded and replaced with any other one on hand.
2. TRIM/Discard worked fine on the disks, as these commands are not supported in RAID firmware, and HBA doesn’t care what commands are sent over the bus.

Hard drives — 8 HGST Travelstar 7K1000 disks with a capacity of 1 TB in a 2.5-inch form factor, as used in laptops. These disks were previously in a RAID 6 array. They will also find use in the new system for local backups.

Additionally, the following was added:

6 pieces of SATA SSD model Samsung 860 QVO 2TB. These SSDs required a large capacity, the presence of SLC cache, desired reliability, and a low price. Support for discard/zero was mandatory, which can be verified with the line in dmesg:

kernel: ata1.00: Enabling discard_zeroes_data

2 pieces of NVMe SSD model Samsung SSD 970 EVO 500GB.

For these SSDs, random read/write speed and longevity for your needs are crucial. A heatsink is mandatory. Absolutely essential. Otherwise, you'll roast them to a crisp during the first RAID synchronization.

StarTech PEX8M2E2 adapter for 2 x NVMe SSDs installed in a PCIe 3.0 8x slot. This is, again, just an HBA, but for NVMe. It differs from cheap adapters by not requiring PCIe bifurcation support from the motherboard due to having a built-in PCIe switch. It will work even in the oldest system with PCIe, even if it's in an x1 PCIe 1.0 slot. Naturally, with the corresponding speed. No RAID there. There is no built-in BIOS on board. So, your system won't magically learn to boot from NVMe or, even more so, create an NVMe RAID thanks to this device.

This component was dictated solely by the presence of only one free 8x PCIe 3.0 slot in the system, and when 2 free slots are available, it can easily be replaced with two inexpensive PEX4M2E1 or similar models, which can be bought anywhere for around 600 rubles.

The refusal of all sorts of hardware or chipset/BIOS RAIDs was made deliberately, with the aim of being able to completely replace the entire system, except for the SSD/HDD, preserving all data. Ideally, to be able to keep even the installed operating system when moving to completely new/different hardware. The main thing is that SATA and PCIe ports are available. It's like a live CD or bootable flash drive, just very fast and a bit bulky.

HumorYou know how it goes — sometimes you need to take the entire array with you on the go. And you don't want to lose data. For this, all the mentioned drives are conveniently placed in brackets in the 5.25 standard case bays.

Well, and of course, for experiments with various methods of SSD caching in Linux.

Hardware RAIDs are boring. You turn it on. It either works, or it doesn’t. But with mdadm, there are always options.

Software

Previously, Debian 8 Jessie, which is close to EOL, was installed on the hardware. A RAID 6 was built using the aforementioned HDDs in combination with LVM. It was running virtual machines in kvm/libvirt.

Since the author has the relevant experience in creating portable bootable SATA/NVMe flash drives, and to avoid disrupting the familiar apt template, Ubuntu 18.04 was chosen as the target system, which has stabilized sufficiently but still has three years of support ahead.

The mentioned system includes all necessary hardware drivers out of the box. We won't need any third-party software or drivers.

Installation Preparation

To install the system, we will need the Ubuntu Desktop Image. The server version has a rather rigid installer that excessively imposes UEFI system partition on one of the disks, ruining the overall setup. Consequently, it can only be installed in UEFI mode. No alternative options are provided.

This is unacceptable for us.

Why?Unfortunately, UEFI boot is extremely poorly compatible with software RAID, as no one offers redundancy for the UEFI ESP partition. There are recipes online suggesting placing the ESP partition on a flash drive in the USB port, but this is a point of failure. There are also recipes using software mdadm RAID 1 with version 0.9 metadata that do not interfere with UEFI BIOS seeing this partition, but this works until the lucky moment when the BIOS or another OS on the hardware writes something to the ESP without synchronizing to the other mirrors.

Moreover, UEFI boot depends on NVRAM, which will not transfer with the disks to the new system, as it is part of the motherboard.

So, we won’t invent a new bicycle. We already have an old, tried-and-true bicycle known today as Legacy/BIOS boot, proudly bearing the name CSM on UEFI-compatible systems. We just need to take it off the shelf, lubricate it, pump up the tires, and wipe it with a damp cloth.

The Desktop version of Ubuntu also doesn’t install properly with a Legacy bootloader, but here, at least, there are options.

So, let's gather the hardware and load the system from the Ubuntu Live bootable flash drive. We will need to download packages, so let’s configure the network, whichever one has started working. If it hasn’t, the necessary packages can be preloaded onto the flash drive.

Let’s enter the Desktop environment, launch the terminal emulator, and off we go:

#sudo bash

How...?The line above is the canonical trigger for wars about sudo. With the...higher level of isolation, as if one controller is broken, the problem is confined to that specific context).With greater capabilities comes greater responsibility.higher level of isolation, as if one controller is broken, the problem is confined to that specific context).The question is, can you take it on? Many believe that using sudo this way is, at least, not cautious. However:

#apt-get install mdadm lvm2 thin-provisioning-tools btrfs-tools util-linux lsscsi nvme-cli mc

Why not ZFS…?When we install software on our computer, we are essentially lending our hardware to the developers of that software.
When we trust this software with the safety of our data, we are essentially taking a loan equal to the cost of recovering that data, which we will eventually have to pay off.

From this perspective, ZFS is like a Ferrari, while mdadm+lvm is more like a bicycle.

Subjectively, the author prefers to lend an unknown person a bicycle taken on credit instead of a Ferrari. The costs are lower, there are no rights to worry about, rules are simpler, parking is free, and maneuverability is better. You can always add parts to a bicycle, and you can even repair it yourself.

So why BTRFS…?To boot the operating system, we will need a file system supported by Legacy/BIOS GRUB out of the box, and at the same time, supporting live snapshots. We will use it for the /boot partition. Besides that, the author prefers to use this filesystem for / (root), keeping in mind that separate partitions can be created on LVM for any other software and mounted to the necessary directories.

We will not store either images of virtual machines, or databases on this filesystem.
This filesystem will only be used for creating system snapshots without shutting it down, followed by transferring these snapshots to a backup disk using send/receive.

Additionally, the author generally prefers to keep minimal software directly on the hardware and run all other software in virtual machines, using things like GPU passthrough and PCI-USB host controllers in KVM through IOMMU.

On the hardware, there are only — data storage, virtualization, and backup.

If you trust ZFS more, then in principle, for the specified application, they are interchangeable.

Nevertheless, the author consciously ignores the built-in mirroring/RAID and redundancy features present in ZFS, BTRFS, and LVM.

As an additional argument, BTRFS has the property of turning random writes into sequential ones, which positively affects the speed of synchronizing snapshots/backups on HDDs.

We will rescan all devices:

#udevadm control --reload-rules && udevadm trigger

Let's take a look:

#lsscsi && nvme list
[0:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sda
[1:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdb
[2:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdc
[3:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdd
[4:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sde
[5:0:0:0] disk ATA Samsung SSD 860 2B6Q /dev/sdf
[6:0:0:0] disk ATA HGST HTS721010A9 A3J0 /dev/sdg
[6:0:1:0] disk ATA HGST HTS721010A9 A3J0 /dev/sdh
[6:0:2:0] disk ATA HGST HTS721010A9 A3J0 /dev/sdi
[6:0:3:0] disk ATA HGST HTS721010A9 A3B0 /dev/sdj
[6:0:4:0] disk ATA HGST HTS721010A9 A3B0 /dev/sdk
[6:0:5:0] disk ATA HGST HTS721010A9 A3B0 /dev/sdl
[6:0:6:0] disk ATA HGST HTS721010A9 A3J0 /dev/sdm
[6:0:7:0] disk ATA HGST HTS721010A9 A3J0 /dev/sdn
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 S466NXXXXXXX15L Samsung SSD 970 EVO 500GB 1 0,00 GB / 500,11 GB 512 B + 0 B 2B2QEXE7
/dev/nvme1n1 S5H7NXXXXXXX48N Samsung SSD 970 EVO 500GB 1 0,00 GB / 500,11 GB 512 B + 0 B 2B2QEXE7


Disk partitioning

NVMe SSD

However, we will not partition them at all. Our BIOS still does not see these drives, so they will go entirely into a software RAID. We won't even create partitions there. If you want to do it by the 'canon' or 'principle' — create one large partition like on an HDD.

SATA HDD

Here, we don't need to get too creative. We will create one partition for everything. We create a partition because these drives are visible to the BIOS and may even attempt to boot from them. We will even install GRUB on these drives later so that the system can boot from them unexpectedly.

#cat >hdd.part << EOF
label: dos
label-id: 0x00000000
device: /dev/sdg
unit: sectors

/dev/sdg1 : start= 2048, size= 1953523120, type=fd, bootable
EOF
#sfdisk /dev/sdg < hdd.part
#sfdisk /dev/sdh < hdd.part
#sfdisk /dev/sdi < hdd.part
#sfdisk /dev/sdj < hdd.part
#sfdisk /dev/sdk < hdd.part
#sfdisk /dev/sdl < hdd.part
#sfdisk /dev/sdm < hdd.part
#sfdisk /dev/sdn < hdd.part

SATA SSD

This is where it gets more interesting.

Firstly, our drives are 2 TB in size. This is within the acceptable limits for MBR, which we will use. If necessary, it can be changed to GPT. GPT disks have a compatibility layer that allows MBR-compatible systems to see the first 4 partitions if they are located within the first 2 terabytes. The main thing is that the boot partition and the bios_grub partition on these disks are at the beginning. This allows for Legacy/BIOS booting from GPT disks.

But this is not our case.

Here, we will create two partitions. The first will be 1 GB in size and used for RAID 1 /boot.

The second will be used for RAID 6 and take up all remaining free space except for a small unallocated area at the end of the drive.

What is the unallocated area?According to online sources, our SATA SSDs have a dynamically expandable SLC cache of 6 to 78 gigabytes. We get 6 gigabytes 'for free' due to the difference between 'gigabytes' and 'gibibytes' in the drive's technical passport. The remaining 72 gigabytes are allocated from unused space.

It should be noted that our cache is SLC, while the space is taken up in 4-bit MLC mode. This effectively means that for every 4 gigabytes of free space we get only 1 gigabyte of SLC cache.

We multiply 72 gigabytes by 4 and get 288 gigabytes. This is the free space that we will not partition in order to allow the drives to fully utilize the SLC cache.

Thus, we will effectively achieve up to 312 gigabytes of SLC cache in total from six drives. Of all the drives, 2 will be used in RAID for redundancy.

This amount of cache will allow us to rarely encounter a situation in live practice where the write operation does not go to cache. This significantly compensates for the most unfortunate drawback of QLC memory — extremely low write speed when data is written bypassing the cache. If your workloads do not match this, I recommend seriously considering how long your SSDs will last under such load, taking into account the TBW from the specification.

#cat >ssd.part << EOF
label: dos
label-id: 0x00000000
device: /dev/sda
unit: sectors

/dev/sda1 : start= 2048, size= 2097152, type=fd, bootable
/dev/sda2 : start= 2099200, size= 3300950016, type=fd
EOF
#sfdisk /dev/sda < ssd.part
#sfdisk /dev/sdb < ssd.part
#sfdisk /dev/sdc < ssd.part
#sfdisk /dev/sdd < ssd.part
#sfdisk /dev/sde < ssd.part
#sfdisk /dev/sdf < ssd.part

Creating arrays

First, we need to rename the machine. This is necessary because the hostname is part of the array name somewhere within mdadm and affects other elements. Arrays can of course be renamed later, but it is unnecessary work.

#mcedit /etc/hostname
#mcedit /etc/hosts
#hostname
vdesk0


NVMe SSD

#mdadm --create --verbose --assume-clean /dev/md0 --level=1 --raid-devices=2 /dev/nvme[0-1]n1

Why —assume-clean…?To avoid initializing the arrays. This is acceptable for both RAID levels 1 and 6. Everything can work without initialization if it is a new array. Moreover, initializing the SSD array upon creation is a waste of TBW resources. We use TRIM/DISCARD wherever possible on assembled SSD arrays to 'initialize' them.

For SSD RAID 1 arrays, DISCARD is supported out of the box.

For SSD RAID 6 arrays, DISCARD needs to be enabled in the kernel module parameters.

This should only be done if all SSDs used in RAID levels 4/5/6 in this system have working support for discard_zeroes_data. Occasionally, strange drives appear that report to the kernel support for this feature, but in reality, it may not exist, or the feature may not always work. Currently, support is almost universal, however, older drives and faulty firmware do occur. For this reason, DISCARD support is disabled by default for RAID 6.

Attention, the following command will destroy all data on NVMe drives by 'initializing' the array with 'zeros'.

#blkdiscard /dev/md0

If something goes wrong, try specifying the step.

#blkdiscard --step 65536 /dev/md0

SATA SSD

#mdadm --create --verbose --assume-clean /dev/md1 --level=1 --raid-devices=6 /dev/sd[a-f]1
#blkdiscard /dev/md1
#mdadm --create --verbose --assume-clean /dev/md2 --chunk-size=512 --level=6 --raid-devices=6 /dev/sd[a-f]2

Why so large…?Increasing the chunk size positively affects the speed of random reads in blocks up to the chunk size. This happens because one operation of the corresponding size or smaller can be fully executed on one device. Therefore, the IOPS from all devices are summed up. Statistically, 99% of IO does not exceed 512K.

In RAID 6, the write IOPS is less than or equal to the IOPS of a single drive. However, for random reads, the IOPS can exceed that of a single drive by several times, and here the block size is crucial. recommendations only The author sees no point in trying to optimize a parameter that is inherently poor in RAID 6 by design and instead optimizes what RAID 6 performs well.
We will compensate for the poor random writes of RAID 6 with NVMe caching and thin provisioning tricks.
We have not yet enabled DISCARD for RAID 6. So we will not 'initialize' this array just yet. We will do this later, after the OS installation.

LVM on NVMe RAID

SATA HDD

#mdadm --create --verbose --assume-clean /dev/md3 --chunk-size=512 --level=6 --raid-devices=8 /dev/sd[g-n]1

For speed, we want to place the root filesystem on NVMe RAID 1 at /dev/md0.

However, this fast array will also be needed for other purposes, such as swap, LVM-cache metadata, and LVM-thin metadata, so we will create an LVM VG on this array.
Let's create a partition for the root filesystem.

#pvcreate /dev/md0
#vgcreate root /dev/md0

Let's create a partition for swap the size of the RAM.

#lvcreate -L 128G --name root root

OS Installation

#lvcreate -L 32G --name swap root

So, we have everything necessary to install the system.

We launch the installation wizard from the Ubuntu Live environment. Standard installation. Only at the disk selection stage do we need to specify the following:

Install the bootloader on /dev/sda

• /dev/md1, — точка монтирования /boot, ФС — BTRFS
• /dev/root/root (a.k.a /dev/mapper/root-root), — точка монтирования / (корень), ФС — BTRFS
• /dev/root/swap (a.k.a /dev/mapper/root-swap), — использовать как раздел подкачки
• When selecting BTRFS as the root filesystem, the installer will automatically create two BTRFS volumes named '@' for / (root) and '@home' for /home.

Let's start the installation...

The installation will end with a modal dialog reporting an installation bootloader error. Unfortunately, you won't be able to exit this dialog normally and continue the installation. Log out of the system and log back in, landing on a clean Ubuntu Live desktop. Open the terminal and again:

Let's create a chroot environment to continue the installation:

#sudo bash

We will set up the network and hostname in the chroot:

#mkdir /mnt/chroot
#mount -o defaults,space_cache,noatime,nodiratime,discard,subvol=@ /dev/mapper/root-root /mnt/chroot
#mount -o defaults,space_cache,noatime,nodiratime,discard,subvol=@home /dev/mapper/root-root /mnt/chroot/home
#mount -o defaults,space_cache,noatime,nodiratime,discard /dev/md1 /mnt/chroot/boot
#mount --bind /proc /mnt/chroot/proc
#mount --bind /sys /mnt/chroot/sys
#mount --bind /dev /mnt/chroot/dev

Enter the chroot environment:

#cat /etc/hostname >/mnt/chroot/etc/hostname
#cat /etc/hosts >/mnt/chroot/etc/hosts
#cat /etc/resolv.conf >/mnt/chroot/etc/resolv.conf

First, we will deliver the packages:

#chroot /mnt/chroot

apt-get install --reinstall mdadm lvm2 thin-provisioning-tools btrfs-tools util-linux lsscsi nvme-cli mc debsums hdparm

We will check and fix all packages that were incorrectly installed due to the incomplete system installation:

If something doesn't seem right, you may need to edit /etc/apt/sources.list beforehand.

#CORRUPTED_PACKAGES=$(debsums -s 2>&1 | awk '{print $6}' | uniq)
#apt-get install --reinstall $CORRUPTED_PACKAGES

Let's adjust the parameters for the RAID 6 module to enable TRIM/DISCARD:

We will tweak our arrays a bit:

#cat >/etc/modprobe.d/raid456.conf << EOF
options raid456 devices_handle_discard_safely=1
EOF

What was that..?

#cat >/etc/udev/rules.d/60-md.rules << EOF
SUBSYSTEM=="block", KERNEL=="md*", ACTION=="change", TEST=="md/stripe_cache_size", ATTR{md/stripe_cache_size}="32768"
SUBSYSTEM=="block", KERNEL=="md*", ACTION=="change", TEST=="md/sync_speed_min", ATTR{md/sync_speed_min}="48000"
SUBSYSTEM=="block", KERNEL=="md*", ACTION=="change", TEST=="md/sync_speed_max", ATTR{md/sync_speed_max}="300000"
EOF
#cat >/etc/udev/rules.d/62-hdparm.rules << EOF
SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", RUN+="/sbin/hdparm -B 254 /dev/%k"
EOF
#cat >/etc/udev/rules.d/63-blockdev.rules << EOF
SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", RUN+="/sbin/blockdev --setra 1024 /dev/%k"
SUBSYSTEM=="block", ACTION=="add|change", KERNEL=="md*", RUN+="/sbin/blockdev --setra 0 /dev/%k"
EOF

We created a set of udev rules that will do the following:We have created a set of udev rules that will do the following:

• Set an adequate block cache size for RAID 6 for the year 2020. The default value appears to have remained unchanged since the inception of Linux and is no longer appropriate.
• Reserve minimum IO during checks/synchronizations of arrays. This is necessary to prevent your arrays from becoming stuck in a perpetual synchronization state under load.
• Limit maximum IO during checks/synchronizations of arrays. This is needed to ensure that SSD RAID synchronization/checks do not overheat your drives. This is especially relevant for NVMe. (Remember the heatsink? I wasn’t joking.)
• Prohibit disks from stopping spindle rotation (HDD) via APM and set a sleep timeout for disk controllers to 7 hours. APM can be completely disabled if your drives support it (-B 255). With the default setting, disks will stop spinning after five seconds. Then the OS will want to flush the disk cache, the disks will spin up again, and it all starts over. Disks have a limited maximum number of spindle spins. This simple default cycle can easily kill your drives in a few years. Not all disks are affected, but our 'laptop' drives, with the corresponding default settings, turn a RAID into a mangled version of a mini-MAID.
• Set the readahead on spinning disks to 1 megabyte — two consecutive blocks/chunks of RAID 6.
• Prohibit readahead on the arrays themselves.

Let's edit /etc/fstab:

#cat >/etc/fstab << EOF
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
# file-system mount-point type options dump pass
/dev/mapper/root-root / btrfs defaults,space_cache,noatime,nodiratime,discard,subvol=@ 0 1
UUID=$(blkid -o value -s UUID /dev/md1) /boot btrfs defaults,space_cache,noatime,nodiratime,discard 0 2
/dev/mapper/root-root /home btrfs defaults,space_cache,noatime,nodiratime,discard,subvol=@home 0 2
/dev/mapper/root-swap none swap sw 0 0
EOF
Why is that..?We will look for the /boot partition by UUID since the naming of arrays may theoretically change.

We will look for the other partitions by LVM names in the notation /dev/mapper/vg-lv, as they uniquely identify partitions.

Do not use UUID for LVM because the UUIDs of LVM volumes and their snapshots may overlap.Mount /dev/mapper/root-root twice..?Yes. Exactly. This is a feature of BTRFS. This filesystem can be mounted multiple times with different subvolumes.

Due to this same feature, I recommend never creating LVM snapshots of active BTRFS volumes. You may get a surprise upon reboot.

Regenerate the mdadm config:

#/usr/share/mdadm/mkconf | sed 's/#DEVICE/DEVICE/g' >/etc/mdadm/mdadm.conf

Let's adjust the LVM settings:

#cat >>/etc/lvm/lvmlocal.conf << EOF

activation {
thin_pool_autoextend_threshold=90
thin_pool_autoextend_percent=5
}
allocation {
cache_pool_max_chunks=2097152
}
devices {
global_filter=["r|^/dev/.*_corig$|","r|^/dev/.*_cdata$|","r|^/dev/.*_cmeta$|","r|^/dev/.*gpv$|","r|^/dev/images/.*$|","r|^/dev/mapper/images.*$|","r|^/dev/backup/.*$|","r|^/dev/mapper/backup.*$|"]
issue_discards=1
}
EOF

We created a set of udev rules that will do the following:We have enabled automatic expansion of LVM thin pools once 90% of the storage is utilized by 5% of the volume.

We have increased the maximum number of cache blocks for LVM cache.

We have prohibited LVM from searching for LVM volumes (PV) on:

• devices containing LVM cache (cdata)
• devices cached using LVM cache bypassing the cache (<lv_name>_corig). However, the cached device will still be scanned through the cache (just <lv_name>).
• devices containing LVM cache metadata (cmeta)
• all devices in the VG named images. Here we will have disk images of virtual machines, and we do not want LVM on the host to activate volumes belonging to the guest OS.
• all devices in the VG named backup. Here we will have backup images of virtual machines.
• all devices whose names end with "gpv" (guest physical volume)

We have enabled DISCARD support when freeing up unused space on LVM VG. Be careful. This will make removing LV on SSD quite long, especially regarding SSD RAID 6. However, by design, we will use thin provisioning, so this won't hinder us at all.

We will update the initramfs image:

#update-initramfs -u -k all

We will install and configure grub:

#apt-get install grub-pc
#apt-get purge os-prober
#dpkg-reconfigure grub-pc

Which disks to choose?All those starting with sd*. The system should be able to boot from any functioning SATA disk or SSD.

Why did we disable os-prober..?For excessive independence and mischievous hands.

It does not work correctly if one of the RAIDs is in a degraded state. It attempts to search for the OS on partitions that are used in virtual machines running on this hardware.

If you need it, you can keep it, but keep in mind all of the above. I recommend looking for recipes to get rid of mischievous hands online.

At this point, we have finished the initial installation. It's time to reboot into the freshly installed OS. Don't forget to remove the bootable Live CD/USB.

#exit
#reboot

For the boot device, we choose any SATA SSD.

LVM on SATA SSD

By this point, we have booted into the new OS, set up the network, apt, opened the terminal emulator, and started:

#sudo bash

Let's continue.

We are "initializing" the array from the SATA SSD:

#blkdiscard /dev/md2

If it doesn't work, try:

#blkdiscard --step 65536 /dev/md2
Creating LVM VG on SATA SSD:

#pvcreate /dev/md2
#vgcreate data /dev/md2

Why another VG..?In fact, we already have a VG named root. Why not add everything into one VG?

If there are multiple PVs in VG, all PVs must be present (online) for VG to activate correctly. An exception is LVM RAID, which we intentionally do not use.

We strongly want the operating system to boot normally and allow us to address issues in the event of a failure (read: data loss) on any of the RAID 6 arrays.

To achieve this, at the first level of abstraction, we will isolate each type of physical 'medium' into a separate VG.

Scientifically speaking, different RAID arrays belong to different 'reliability domains.' We should not create an additional common point of failure by putting them into one VG.

Having LVM at the 'hardware' level will allow us to slice parts of different RAID arrays differently by combining them. For example, to run simultaneously bcache + LVM thin, bcache + BTRFS, LVM cache + LVM thin, a complex ZFS configuration with caches, or any other hellish mixture to touch and compare them.

At the 'hardware' level, we will not use anything apart from the old-fashioned 'thick' LVM volumes. An exception to this rule may be the backup partition.

I believe that by this point, many readers have already started to suspect something regarding the nesting doll.

LVM on SATA HDD

#pvcreate /dev/md3
#vgcreate backup /dev/md3

Another new VG..?We strongly want our operating system to continue functioning normally when a disk array, which we will use for data backup, fails, while still maintaining access to non-reserved data. Therefore, to avoid VG activation issues, we create a separate VG.

Configuring LVM cache

We will create an LV on NVMe RAID 1 to use it as a caching device.

#lvcreate -L 70871154688B --name cache root

Why so few…?The thing is, our NVMe SSDs also have SLC cache. 4 gigabytes of 'free' and 18 gigabytes dynamic due to the free space occupied in 3-bit MLC. Once this cache is exhausted, the NVMe SSDs will not be much faster than our SATA SSD with cache. For this reason, it makes no sense to make the LVM cache partition significantly larger than twice the volume of the SLC cache of the NVMe drive. For the NVMe drives we use, the author considers it reasonable to make a 32-64 gigabyte cache.

The specified partition size is necessary to organize 64 gigabytes of cache, store cache metadata, and create a backup of the metadata.

Additionally, I note that after a dirty system shutdown, LVM will mark the entire cache as dirty and will synchronize it again. Moreover, this will recur with each use of lvchange on this device until the system is rebooted. Therefore, I recommend recreating the cache immediately using the corresponding script.

Let's create an LV on SATA RAID 6 to use it as a cacheable device.

#lvcreate -L 3298543271936B --name cache data

Why only three terabytes..?So that, if necessary, it can be used SATA SSD RAID 6 for other purposes. The size of the cacheable space can be dynamically increased on-the-fly, without stopping the system. For this, it is necessary to temporarily stop and re-enable the cache, but a distinctive advantage of LVM-cache over, for instance, bcache, is that this can be done on-the-fly.

Let's create a new VG for caching.

#pvcreate /dev/root/cache
#pvcreate /dev/data/cache
#vgcreate cache /dev/root/cache /dev/data/cache

Let's create an LV on the cacheable device.

#lvcreate -L 3298539077632B --name cachedata cache /dev/data/cache

Here we immediately occupied all the free space on /dev/data/cache so that all other necessary partitions are created directly on /dev/root/cache. If something was created in the wrong place, it can be moved using pvmove.

Let's create and enable the cache:

#lvcreate -y -L 64G -n cache cache /dev/root/cache
#lvcreate -y -L 1G -n cachemeta cache /dev/root/cache
#lvconvert -y --type cache-pool --cachemode writeback --chunksize 64k --poolmetadata cache/cachemeta cache/cache
#lvconvert -y --type cache --cachepool cache/cache cache/cachedata

Why such a chunksize..?Through practical experiments, the author found that the best result is achieved when the block size of LVM cache matches the block size of LVM thin. Moreover, the smaller the size, the better the configuration performs in random writes.

64k is the minimum block size allowed for LVM thin.

Caution writeback..!Yes. This type of cache delays the synchronization of writes to the cacheable device. This means that, in the event of a cache loss, data on the cacheable device may be lost. Later, the author will discuss what measures, aside from NVMe RAID 1, can be taken to mitigate this risk.

This type of cache is deliberately chosen to compensate for the low performance of RAID 6 in random writes.

Let's check what we have:

#lvs -a -o lv_name,lv_size,devices --units B cache
LV LSize Devices
[cache] 68719476736B cache_cdata(0)
[cache_cdata] 68719476736B /dev/root/cache(0)
[cache_cmeta] 1073741824B /dev/root/cache(16384)
cachedata 3298539077632B cachedata_corig(0)
[cachedata_corig] 3298539077632B /dev/data/cache(0)
[lvol0_pmspare] 1073741824B /dev/root/cache(16640)

On /dev/data/cache there should only be [cachedata_corig]. If something is wrong, use pvmove.

You can disable the cache if necessary with one command:

#lvconvert -y --uncache cache/cachedata

This is done online. LVM simply synchronizes the cache to the disk, removes it, and renames cachedata_corig back to cachedata.

Configuring LVM thin

Let's estimate how much space we will need for LVM thin metadata:

#thin_metadata_size --block-size=64k --pool-size=6terabytes --max-thins=100000 -u bytes
thin_metadata_size - 3,385,794,560 bytes estimated metadata area size for "--block-size=64kibibytes --pool-size=6terabytes --max-thins=100000"

Round to 4 gigabytes: 4294967296B

Multiply by two and add 4194304B for LVM PV metadata: 8594128896B
Let's create a separate partition on NVMe RAID 1 to store LVM thin metadata and its backup:

#lvcreate -L 8594128896B --name images root

Why..?This raises the question of why to store LVM thin metadata separately if they will still be cached on NVMe and will work quickly.

Speed is important here, but it is far from the main reason. The issue is that the cache can be a point of failure. Something can happen to it, and if LVM thin metadata is cached, this will lead to complete data loss. Without intact metadata, it will be practically impossible to recreate thin volumes.

By moving the metadata to a separate non-cached but fast volume, we ensure the preservation of the metadata in case of cache loss or damage. In this case, any damage caused by cache loss will be localized within the thin volumes, significantly simplifying the recovery process. With a high probability, this damage can be restored using FS logs.

Moreover, if a snapshot of the thin volume was previously taken, and after that, the cache was fully synchronized at least once, then due to the internal structure of LVM thin, the integrity of the snapshot will be guaranteed in case of cache loss.

Let's create a new VG responsible for thin-provisioning:

#pvcreate /dev/root/images
#pvcreate /dev/cache/cachedata
#vgcreate images /dev/root/images /dev/cache/cachedata
Let's create a pool:

#lvcreate -L 274877906944B --poolmetadataspare y --poolmetadatasize 4294967296B --chunksize 64k -Z y -T images/thin-pool
Why -Z yIn addition to its intended purpose — preventing data from one virtual machine from leaking into another during space redistribution — zeroing is also used to increase the speed of random writes of blocks smaller than 64k. Any write smaller than 64k to a previously unallocated area of the thin volume will be converted into 64K aligned with the cache boundary. This allows the operation to be performed entirely through the cache, bypassing the cached device.

We will move the LV to the corresponding PVs:

#pvmove -n images/thin-pool_tdata /dev/root/images /dev/cache/cachedata
#pvmove -n images/lvol0_pmspare /dev/cache/cachedata /dev/root/images
#pvmove -n images/thin-pool_tmeta /dev/cache/cachedata /dev/root/images

Let's check:

#lvs -a -o lv_name,lv_size,devices --units B images
LV LSize Devices
[lvol0_pmspare] 4294967296B /dev/root/images(0)
thin-pool 274877906944B thin-pool_tdata(0)
[thin-pool_tdata] 274877906944B /dev/cache/cachedata(0)
[thin-pool_tmeta] 4294967296B /dev/root/images(1024)

Let's create a thin volume for testing:

#lvcreate -V 64G --thin-pool thin-pool --name test images

We will install packages for testing and monitoring:

#apt-get install sysstat fio

This way, we can observe the behavior of our storage configuration in real-time:

#watch 'lvs --rows --reportformat basic --quiet -ocache_dirty_blocks,cache_settings cache/cachedata && (lvdisplay cache/cachedata | grep Cache) && (sar -p -d 2 1 | grep -E "sd|nvme|DEV|md1|md2|md3|md0" | grep -v Average | sort)'

This way, we can test our configuration:

#fio --loops=1 --size=64G --runtime=4 --filename=/dev/images/test --stonewall --ioengine=libaio --direct=1
--name=4kQD32read --bs=4k --iodepth=32 --rw=randread
--name=8kQD32read --bs=8k --iodepth=32 --rw=randread
--name=16kQD32read --bs=16k --iodepth=32 --rw=randread
--name=32KQD32read --bs=32k --iodepth=32 --rw=randread
--name=64KQD32read --bs=64k --iodepth=32 --rw=randread
--name=128KQD32read --bs=128k --iodepth=32 --rw=randread
--name=256KQD32read --bs=256k --iodepth=32 --rw=randread
--name=512KQD32read --bs=512k --iodepth=32 --rw=randread
--name=4Kread --bs=4k --rw=read
--name=8Kread --bs=8k --rw=read
--name=16Kread --bs=16k --rw=read
--name=32Kread --bs=32k --rw=read
--name=64Kread --bs=64k --rw=read
--name=128Kread --bs=128k --rw=read
--name=256Kread --bs=256k --rw=read
--name=512Kread --bs=512k --rw=read
--name=Seqread --bs=1m --rw=read
--name=Longread --bs=8m --rw=read
--name=Longwrite --bs=8m --rw=write
--name=Seqwrite --bs=1m --rw=write
--name=512Kwrite --bs=512k --rw=write
--name=256write --bs=256k --rw=write
--name=128write --bs=128k --rw=write
--name=64write --bs=64k --rw=write
--name=32write --bs=32k --rw=write
--name=16write --bs=16k --rw=write
--name=8write --bs=8k --rw=write
--name=4write --bs=4k --rw=write
--name=512KQD32write --bs=512k --iodepth=32 --rw=randwrite
--name=256KQD32write --bs=256k --iodepth=32 --rw=randwrite
--name=128KQD32write --bs=128k --iodepth=32 --rw=randwrite
--name=64KQD32write --bs=64k --iodepth=32 --rw=randwrite
--name=32KQD32write --bs=32k --iodepth=32 --rw=randwrite
--name=16KQD32write --bs=16k --iodepth=32 --rw=randwrite
--name=8KQD32write --bs=8k --iodepth=32 --rw=randwrite
--name=4kQD32write --bs=4k --iodepth=32 --rw=randwrite
| grep -E 'read|write|test' | grep -v ioengine

Caution! Resource!This code will run 36 different tests, each lasting 4 seconds. Half of these tests are for writing. In 4 seconds on NVMe, a significant amount can be written—up to 3 gigabytes per second. Therefore, each run of write tests can consume up to 216 gigabytes of SSD resources.

Reading and writing intermixed?Yes. Read and write tests should be run separately. Moreover, it's essential to ensure that all caches are synchronized so that previous writes do not affect the reads.

Results will vary significantly between the first run and subsequent runs as the cache and thin volume fill up, and depending on whether the system had time to synchronize caches filled during the last run.

Additionally, I recommend measuring speed on a thin volume that has just been snapshot. The author observed that random writes can accelerate sharply right after creating the first snapshot, especially when the cache is not yet fully filled. This happens due to copy-on-write semantics, block alignment of the cache and thin volume, and the fact that random writing on RAID 6 turns into random reading with RAID 6 followed by writing to the cache. In our configuration, random reading on RAID 6 is up to 6 times (the number of SATA SSDs in the array) faster than writing. Since blocks for CoW are allocated sequentially from the thin pool, writes are mostly transformed into sequential ones.

Both of these features can be effectively utilized.

Cache-‘coherent’ snapshots

To reduce the risk of data loss in case of cache corruption/loss, the author suggests adopting a snapshot rotation practice to ensure their integrity in such cases.

Firstly, because the metadata for thin volumes is located on a non-cached device, the metadata will be intact, and any potential losses will be isolated within the data blocks.

The next snapshot rotation cycle guarantees data integrity within the snapshots in case of cache loss:

1. For each thin volume named <name>, create a snapshot named <name>.cached
2. Set the migration threshold to a reasonably high value: #lvchange --quiet --cachesettings "migration_threshold=16384" cache/cachedata
3. In the loop, check the number of dirty blocks in the cache: #lvs --rows --reportformat basic --quiet -ocache_dirty_blocks cache/cachedata | awk '{print $2}' until we get to zero. If there is no zero for too long, it can be temporarily created by switching the cache to writethrough mode. However, given the speed characteristics of our SATA and NVMe SSD arrays, as well as their TBW resource, you will either be able to catch the moment quickly without changing the cache mode, or your hardware will completely consume its resource in a matter of days. Due to resource limitations, the system cannot be under 100% write load constantly. Our NVMe SSDs will fully exhaust their resource under 100% write load in 3-4 days. SATA SSDs will last about twice as long. Therefore, we will assume that most of the load is on read, and on write, we have relatively short bursts of extremely high activity combined with low average load.
4. As soon as we catch (or create) a zero — we rename .cached to .committed. The old .committed is then deleted.
5. Optionally, if the cache is 100% full, it can be recreated via a script, thus clearing it. The system works much faster on write with a partially empty cache.
6. We will set the migration threshold to zero: #lvchange --quiet --cachesettings "migration_threshold=0" cache/cachedata This will temporarily prohibit syncing the cache to the main storage.
7. We wait until enough changes accumulate in the cache #lvs --rows --reportformat basic --quiet -ocache_dirty_blocks cache/cachedata | awk '{print $2}' or the timer goes off.
8. We repeat again.

Why complicate things with a migration threshold…?The thing is, in real practice, 'random' writes are not actually so random. If we wrote something to a 4 kilobyte sector, there is a high probability that a write will be made to this or one of the neighboring (+- 32K) sectors in the next couple of minutes.

By setting the migration threshold to zero, we delay the synchronization of writes to the SATA SSD and aggregate several changes of a 64K block in the cache. This significantly conserves the resource of the SATA SSD.

And where's the code..?Unfortunately, the author considers himself not competent enough in bash script development, as he is 100% self-taught and practices 'google'-driven development, therefore he believes that the terrible code that comes from his hands is better left unused by anyone else.

I think that professionals in this field could independently implement all the logic described above if necessary and perhaps even beautifully format it as a systemd service, as the author tried to do.

This simple snapshot rotation scheme will allow us to not only always have a fully synchronized snapshot on the SATA SSD, but also, using the thin_delta utility, identify which blocks have changed since its creation, thus localizing damage on the primary volumes and greatly simplifying restoration.

TRIM/DISCARD in libvirt/KVM

Since the storage will be used for KVM managed by libvirt, it would be good to teach our VMs not only to occupy free space but also to release space that is no longer needed.

This is done by emulating TRIM/DISCARD support on virtual disks. To do this, you need to change the controller type to virtio-scsi and edit the xml.

#virsh edit vmname
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='writethrough' io='threads' discard='unmap'/>
<source dev='/dev/images/vmname'/>
<backingStore/>
<target dev='sda' bus='scsi'/>
<alias name='scsi0-0-0-0'/>