🥇Why is my NVMe slower than SSD?

In this article, we will discuss some nuances of the input-output subsystem and their impact on performance.

A couple of weeks ago, I encountered the question of why NVMe on one server is slower than SATA on another. I looked at the server specifications and realized this was a trick question: the NVMe was from the consumer segment, while the SSD was from the server segment.

Clearly, comparing products from different segments in different environments is not fair, but this is not a comprehensive technical answer. We will study the basics, conduct experiments, and provide an answer to the posed question.

What is fsync and where is it used?

To speed up operations with drives, data is buffered, meaning that it is saved in volatile memory until there is a convenient opportunity to write the buffer's content to the drive. The criteria for a "convenient opportunity" are determined by the operating system and the characteristics of the drive. In the event of a power loss, all data in the buffer will be lost.

There are a number of tasks where it is essential to be sure that changes to a file are written to the drive and not just sitting in an intermediate buffer. This assurance can be obtained by using the POSIX-compliant system call fsync. The fsync call initiates a forced write from the buffer to the drive.

We will demonstrate the impact of buffers with an artificial example in the form of a short program written in C.

#include <fcntl.h>
#include <unistd.h>
#include <sys/stat.h>
#include <sys/types.h>

int main(void) {
    /* Открываем файл answer.txt на запись, если его нет -- создаём */
    int fd = open("answer.txt", O_WRONLY | O_CREAT);
    /* Записываем первый набор данных */
    write(fd, "Answer to the Ultimate Question of Life, The Universe, and Everything: ", 71);
    /* Делаем вид, что проводим вычисления в течение 10 секунд */
    sleep(10);
    /* Записываем результат вычислений */
    write(fd, "42n", 3); 

    return 0;
}

Comments clearly explain the sequence of actions in the program. The text "the answer to the ultimate question of life, the universe, and everything" will be buffered by the operating system, and if the server is rebooted by pressing the Reset button during the "calculations," the file will be empty. In our example, the loss of text is not an issue, so fsync is not needed. Databases do not share this optimism.

Databases are complex programs that handle multiple files simultaneously, so they need to ensure that the data they record will be preserved on the storage device, as this affects the consistency of the data within the DB. Databases are designed to log all completed transactions and be ready for power loss at any moment. This behavior necessitates the use of fsync constantly and in large quantities.

How frequent use of fsync affects performance

During regular input-output operations, the operating system tries to optimize communication with the disks, as external storage devices are the slowest in the memory hierarchy. Therefore, the operating system attempts to write as much data as possible in a single access to the storage device.

Let’s demonstrate the impact of using fsync with a specific example. Our subjects are the following solid-state drives:

Intel® DC SSD S4500 480 GB, connected via SATA 3.2, 6 Gb/s;
Samsung 970 EVO Plus 500GB, connected via PCIe 3.0 x4, ~31 Gb/s.

The tests are conducted on an Intel® Xeon® W-2255 running Ubuntu 20.04. The disk testing uses sysbench 1.0.18. One partition formatted as ext4 has been created on the disks. The preparation for the test consists of creating files totaling 100 GB:

sysbench --test=fileio --file-total-size=100G prepare

Starting tests:

# Без fsync
sysbench --num-threads=16 --test=fileio --file-test-mode=rndrw --file-fsync-freq=0 run

# С fsync после каждой записи
sysbench --num-threads=16 --test=fileio --file-test-mode=rndrw --file-fsync-freq=1 run

The test results are presented in the table.

Test
Intel® S4500
Samsung 970 EVO+

Read without fsync, MiB/s
5734.89
9028.86

Write without fsync, MiB/s
3823.26
6019.24

Read with fsync, MiB/s
37.76
3.27

Write with fsync, MiB/s
25.17
2.18

It is easy to notice that the NVMe drives from the client segment lead confidently when the operating system decides how to work with the disks, and they perform worse when fsync is used. This raises two questions:

Why does the reading speed in the test without fsync exceed the physical bandwidth of the channel?
Why do SSDs from the server segment handle a large number of fsync requests better?

The answer to the first question is simple: sysbench generates files filled with zeros. Thus, the test was conducted over 100 gigabytes of zeros. Since the data is quite uniform and predictable, various OS optimizations come into play, and they significantly speed up execution.

If you doubt all the results from sysbench, you can use fio.

# Без fsync
fio --name=test1 --blocksize=16k --rw=randrw --iodepth=16 --runtime=60 --rwmixread=60 --fsync=0 --filename=/dev/sdb

# С fsync после каждой записи
fio --name=test1 --blocksize=16k --rw=randrw --iodepth=16 --runtime=60 --rwmixread=60 --fsync=1 --filename=/dev/sdb

Test
Intel® S4500
Samsung 970 EVO+

Read without fsync, MiB/s
45.5
178

Write without fsync, MiB/s
30.4
119

Read with fsync, MiB/s
32.6
20.9

Write with fsync, MiB/s
21.7
13.9

The trend of performance degradation in NVMe when using fsync is quite noticeable. We can move on to the answer to the second question.

Optimization or bluff

Earlier we mentioned that data is stored in a buffer, but we didn't specify which one, as it was not fundamental. We still won't delve into the intricacies of operating systems and will highlight two general types of buffers:

Software;
Hardware.

By software buffer, we mean buffers that exist in the operating system, while hardware refers to the volatile memory of the disk controller. The system call fsync sends a command to the storage to write data from its buffer to the main storage but cannot monitor the correctness of the command's execution.

Since SSDs show better results, we can make two assumptions:

The drive is designed for this type of load;
The drive is 'bluffing' and ignoring the command.

Dishonest behavior of the storage can be observed if a test is conducted with power loss. This can be checked with a script diskchecker.pl, which was created in 2005.

This script requires two physical machines — a 'server' and a 'client'. The client writes a small amount of data to the test disk, calls fsync, and sends information to the server about what has been written.

# Запускается на сервере
./diskchecker.pl -l [port]

# Запускается на клиенте
./diskchecker.pl -s <server[:port]> create <file> <size_in_MB>

After starting the script, it is necessary to cut power to the 'client' and not restore it for several minutes. It's important to specifically cut off the tested machine from electricity, rather than just performing a hard shutdown. After some time, the server can be connected and booted into the operating system. After the OS has booted, it is necessary to run diskchecker.pl, but with the argument verify.

.\/diskchecker.pl -s  verify

At the end of the check, you will see the number of errors. If it's 0, it means the disk passed the test. To eliminate a fortunate coincidence for the disk, the experiment can be repeated several times.

Our S4500 showed no errors during power loss, so we can assert that it is ready for loads with a high number of fsync calls.

Conclusion

When choosing disks or complete ready-made configurations, it is essential to consider the specifics of the tasks that need to be solved. At first glance, it may seem obvious that NVMe, meaning SSD with PCIe interface, is faster than a "classic" SATA SSD. However, as we learned today, this may not be the case under specific conditions and with certain tasks.

How do you test server components when renting from an IaaS provider?
We look forward to your comments.

Source: habr.com