🥇RAID Arrays on NVMe

In this article, we will discuss various ways to organize RAID arrays and showcase one of the first hardware RAID controllers with NVMe support.

The diverse applications of RAID technology are prominent in the server segment. In the client segment, software RAID0 or RAID1 with two disks is most commonly used.

This article will provide a brief overview of RAID technology, a small guide on creating RAID arrays using three different tools, as well as a performance comparison of virtual disks using each method.

What is RAID?

Wikipedia provides a comprehensive definition of RAID technology:

RAID (English, Redundant Array of Independent Disks — a redundant array of independent disks) — a data virtualization technology that combines multiple physical disk devices into a logical module to enhance fault tolerance and performance.

The configuration of disk arrays and the technologies used depend on the selected RAID level. RAID levels are standardized in the specification Common RAID Disk Data Format. It describes many RAID levels, but the most commonly accepted are RAID0, RAID1, RAID5, and RAID6.

RAID0, or Stripes, is a RAID level that combines two or more physical disks into a single logical one. The capacity of the logical disk is the sum of the capacities of the physical disks in the array. At this level, RAID has no redundancy, and the failure of one disk may result in the loss of all data in the virtual disk.

The RAID1, or Mirror, creates identical copies of data on two or more disks. The capacity of the virtual disk does not exceed the capacity of the smallest physical disk. Data on a RAID1 virtual disk will be accessible as long as at least one physical disk from the array is operational. Using RAID1 adds redundancy but is a relatively expensive solution, as only the capacity of one disk is available in arrays of two or more disks.

The RAID5 solves the problem of high cost. To create an array with RAID5 level, at least 3 disks are required, making the array resilient to the failure of one disk. Data in RAID5 is stored in blocks with checksums. There is no strict division between data disks and checksum disks. Checksums in RAID5 are the result of the XOR operation applied to N-1 blocks, each taken from its own disk.

Although RAID arrays enhance redundancy and provide failover, they are not suitable for storing backups.

After a brief overview of the types of RAID arrays, we can move on to the devices and programs that allow us to assemble and use disk arrays.

Types of RAID Controllers

There are two ways to create and use RAID arrays: hardware and software. We will review the following solutions:

Linux Software RAID.
Intel® Virtual RAID On CPU.
LSI MegaRAID 9460-8i.

Note that the Intel® solution operates on the chipset, which raises the question of whether it is a hardware or software solution. For example, the hypervisor VMware ESXi considers VROC to be software and does not officially support it.

Linux Software RAID

Software RAID arrays in the Linux operating system family are a fairly common solution both in the client segment and in servers. All that is needed to create an array is the mdadm utility and a few block devices. The only requirement that Linux Software RAID imposes on the storage devices used is that they must be block devices accessible to the system.

The lack of costs for hardware and software is an obvious advantage of this approach. Linux Software RAID organizes disk arrays at the cost of processor time. The list of supported RAID levels and the status of current disk arrays can be viewed in the mdstat file, located at the root of procfs:

root@grindelwald:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid10]
unused devices:

Support for RAID levels is added by loading the corresponding kernel module, for example:

root@grindelwald:~# modprobe raid456
root@grindelwald:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
unused devices:

All operations with disk arrays are performed through the command-line utility mdadm. The assembly of the disk array is done in one command:

mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/nvme1n1 /dev/nvme2n1

After executing this command, a block device /dev/md0 will appear in the system, representing a virtual disk.

Intel® Virtual RAID On CPU

Intel® VROC Standard Hardware Key
Intel® Virtual RAID On CPU (VROC) is a software-hardware technology for creating RAID arrays based on Intel® chipsets. This technology is primarily available for motherboards that support Intel® Xeon® Scalable processors. By default, VROC is not available. To activate it, a hardware VROC license key must be installed.

The standard VROC license allows the creation of disk arrays with RAID levels 0, 1, and 10. The premium version expands this list with support for RAID 5.

Intel® VROC technology on modern motherboards works in conjunction with the Intel® Volume Management Device (VMD), enabling hot-swapping for NVMe interface drives.

Intel® VROC with standard license Array configurations are done through the Setup Utility during server boot. On the tab Advanced the Intel® Virtual RAID on CPU option appears, where disk arrays can be configured.

Creating a RAID1 array with two drives
The Intel® VROC technology has its 'tricks up its sleeve.' Disk arrays built with VROC are compatible with Linux Software RAID. This means the status of the arrays can be monitored in /proc/mdstat, and administered through mdadm. This 'feature' is officially supported by Intel. After assembling RAID1 in the Setup Utility, you can observe the synchronization of drives in the OS:

root@grindelwald:~# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md126 : active raid1 nvme2n1[1] nvme1n1[0]
      1855832064 blocks super external:/md127/0 [2/2] [UU]
      [>....................]  resync =  1.3% (24207232/1855832064) finish=148.2min speed=205933K/sec

md127 : inactive nvme1n1[1](S) nvme2n1[0](S)
      10402 blocks super external:imsm
       
unused devices:

Note that arrays cannot be assembled on VROC through mdadm (the assembled arrays will be Linux SW RAID), but disks can be changed and arrays can be dismantled.

LSI MegaRAID 9460-8i

Appearance of the LSI MegaRAID 9460-8i controller
The RAID controller is a standalone hardware solution. The controller only works with drives connected directly to it. This RAID controller supports up to 24 drives with NVMe interface. The support for NVMe distinguishes this controller from many others.

Main Menu of the Hardware Controller
When using UEFI mode, controller settings are integrated into the Setup Utility. Compared to VROC, the hardware controller menu appears significantly more complex.

Creating RAID1 on two disks
Explaining disk array configurations on a hardware controller is quite a nuanced topic and could serve as the basis for a full article. Here, we will simply limit ourselves to creating RAID0 and RAID1 with the default settings.

Drives connected to the hardware controller are not visible to the operating system. Instead, the controller 'masks' all RAID arrays as SAS drives. Drives connected to the controller but not part of a disk array will not be accessible to the OS.

root@grindelwald:~# smartctl -i /dev/sda
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-48-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               AVAGO
Product:              MR9460-8i
Revision:             5.14
Compliance:           SPC-3
User Capacity:        1,999,844,147,200 bytes [1.99 TB]
Logical block size:   512 bytes
Rotation Rate:        Solid State Device
Logical Unit id:      0x000000000000000000000000000000
Serial number:        00000000000000000000000000000000
Device type:          disk
Local Time is:        Sun Oct 11 16:27:59 2020 MSK
SMART support is:     Unavailable - device lacks SMART capability.

Despite being masked as SAS drives, NVMe arrays will operate at PCIe speeds. However, this feature allows booting from NVMe in Legacy mode.

Test Stand

Each method of organizing disk arrays has its own physical advantages and disadvantages. But is there a performance difference when working with disk arrays?

To achieve maximum fairness, all tests will be conducted on the same server. Its configuration is as follows:

2x Intel® Xeon® 6240;
12x DDR4-2666 16 GB;
LSI MegaRAID 9460-8i;
Intel® VROC Standard Hardware Key;
4x Intel® SSD DC P4510 U.2 2TB;
1x Samsung 970 EVO Plus M.2 500GB.

The tested items are P4510, half of which are connected to the motherboard and the other half to the RAID controller. The operating system Ubuntu 20.04 is installed on M.2, and tests will be conducted using fio version 3.16.

Testing

First, we will check the latencies when working with the disk. The test is performed in a single thread with a block size of 4 KB. Each test lasts 5 minutes. Before starting, 'none' is set as the I/O scheduler for the corresponding block device. The fio command is as follows:

fio --name=test --blocksize=4k --direct=1 --buffered=0 --ioengine=libaio --iodepth=1 --loops=1000 --runtime=300 --rw= --filename=

From the fio results, we take clat 99.00%. The results are shown in the table below.

Random read, µs
Random write, µs

Disk
112
78

Linux SW RAID, RAID0
113
45

VROC, RAID0
112
46

LSI, RAID0
122
63

Linux SW RAID, RAID1
113
48

VROC, RAID1
113
45

LSI, RAID1
128
89

In addition to latency when accessing data, we want to see the performance of virtual drives and compare it with the performance of physical disks. Command to run fio:

fio --name=test --blocksize=4k --direct=1 --buffered=0 --ioengine=libaio  --loops=1000 --runtime=300  --iodepth= --rw= --filename=

Performance is measured in I/O operations. The results are presented in the table below.

Random read 1 stream, IOPS
Random write 1 stream, IOPS
Random read 128 streams, IOPS
Random write 128 streams, IOPS

Disk
11300
40700
453000
105000

Linux SW RAID, RAID0
11200
52000
429000
232000

VROC, RAID0
11200
52300
441000
162000

LSI, RAID0
10900
44200
311000
160000

Linux SW RAID, RAID1
10000
48600
395000
147000

VROC, RAID1
10000
54400
378000
244000

LSI, RAID1
11000
34300
229000
248000

It is clear that using a hardware controller increases latency and decreases performance compared to software solutions.

Conclusion

Using hardware solutions to create disk arrays from two disks seems unwise. However, there are tasks where the use of RAID controllers is justified. With the emergence of controllers supporting the NVMe interface, users can use faster SSDs in their projects.