🥇How to use fio to check disks for adequate performance for etcd

Note: translation.: this article summarizes a mini-research conducted by IBM Cloud engineers in search of a solution to a real problem related to the operation of the etcd database. We faced a similar task, but the authors' thought process and actions may be interesting in a broader context.

A brief summary of the entire article: fio and etcd

The performance of an etcd cluster heavily depends on the speed of the underlying storage. To monitor performance, etcd exports various Prometheus metrics. One of these is wal_fsync_duration_seconds. According to the etcd documentation , it can be considered that the storage is fast enough if the 99th percentile of this metric does not exceed 10 ms…If you are considering setting up an etcd cluster on machines running Linux and want to check whether your storage devices (e.g., SSDs) are fast enough, we recommend using a popular I/O tester called

fio . Just run the following command (the directorytest-data should be located in the mounted partition of the tested storage): fio --rw=write --ioengine=sync --fdatasync=1 --directory=test-data --size=22m --bs=2300 --name=mytest

You just need to look at the output and check if the 99th percentile of

fdatasync is within 10 ms. If so, your storage is performing fast enough. Here is an example output: fsync/fdatasync/sync_file_range: sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70 sync percentiles (usec): | 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627], | 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549], | 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278], | 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795], | 99.99th=[15795]

A few remarks:

In the example above, we adjusted the parameters

--size --bs and to fit the specific case. To obtain meaningful results from , specify values appropriate for your scenario. We will discuss how to choose them below. . Just run the following command (the directoryDuring testing, only
loads the disk subsystem. In real life, it is likely that other processes will be writing to the disk (in addition to those related to . Just run the following command (the directory ). Such additional load can lead to an increase in wal_fsync_duration_seconds. In other words, if the 99th percentile obtained from testing with wal_fsync_duration_seconds. In other words, if the 99th percentile obtained from the testing is . Just run the following command (the directory, just slightly less than 10 ms, there is a high likelihood that the storage performance is insufficient.
For the test, you will need version . Just run the following command (the directory at least 3.5, as older versions do not aggregate results is within 10 ms. If so, your storage is performing fast enough. Here is an example output: in the form of percentiles.
The output above represents only a small excerpt from the overall output . Just run the following command (the directory.

Detailed information about fio and etcd

A few words about etcd WALs

Typically, databases use write-ahead logging (write-ahead logging, WAL). This also applies to etcd. The discussion of WAL goes beyond the scope of this article, but for our purposes, it is important to know that each member of the etcd cluster stores its WAL in persistent storage. etcd writes certain operations with the key-value store (such as updates) to the WAL before executing them. If a node fails and restarts between snapshots, etcd can recover transactions that occurred since the previous snapshot by referring to the contents of the WAL.

Thus, every time a client adds a key to the KV store or updates the value of an existing key, etcd adds a description of the operation to the WAL, which is a standard file in persistent storage. Before continuing to operate, etcd MUST be 100% sure that the entry in the WAL is actually saved. To achieve this in Linux, it is not enough to use the system call exit, as the operation of writing to physical media itself may be delayed. For example, Linux may hold the WAL write in the kernel cache in memory for some time (for instance, in the page cache). To ensure that data is written to the media, after writing, a system call must be invoked is within 10 ms. If so, your storage is performing fast enough. Here is an example output: — this is how etcd behaves (as seen in the following output strace; here 8 — the file descriptor of the WAL):

21:23:09.894875 lseek(8, 0, SEEK_CUR)   = 12808 
21:23:09.894911 write(8, ".      20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296 
21:23:09.895041 fdatasync(8)            = 0

Unfortunately, writing to persistent storage takes some time. Prolonged execution of the fdatasync call may affect the performance of etcd. The documentation for the storage states, that for adequate performance, the 99th percentile of the duration of all calls must be is within 10 ms. If so, your storage is performing fast enough. Here is an example output: The WAL file write took less than 10 ms. There are other metrics related to storage, but this article will focus specifically on this one.

Evaluating storage with fio

You can determine whether a storage is suitable for use with etcd using the utility . Just run the following command (the directory — a popular I/O tester. Keep in mind that disk I/O can happen in various ways: sync/async, many different classes of system calls, etc. The downside is that . Just run the following command (the directory it is extremely complex to use. The utility has many parameters, and different combinations of their values lead to completely different results. To get a reasonable estimate in the case of etcd, you must ensure that the write load generated by fio closely resembles the load on etcd when writing to WAL files:

This means that the generated . Just run the following command (the directory load should at least represent a series of sequential writes to a file, where each write operation consists of a system call exit, followed by is within 10 ms. If so, your storage is performing fast enough. Here is an example output:.
To enable sequential writing, you need to specify the flag --rw=write.
To . Just run the following command (the directory writes using calls exit (rather than other system calls — for example, pwrite), use the flag --ioengine=sync.
Finally, the flag --fdatasync=1 ensures that after each exit you should is within 10 ms. If so, your storage is performing fast enough. Here is an example output:.
Two other parameters in our example: --bs and to fit the specific case. To obtain meaningful results from — may vary depending on the specific use case. Their configuration will be described in the next section.

Why we chose fio and how we learned to configure it

This note arose from a real case we encountered. We had a cluster on Kubernetes v1.13 with monitoring on Prometheus. Solid state drives served as storage for etcd v3.2.24. Metrics from etcd showed excessively high latencies is within 10 ms. If so, your storage is performing fast enough. Here is an example output:, even when the cluster was idle. These metrics seemed quite questionable to us, and we were unsure of what exactly they represented. Additionally, the cluster consisted of virtual machines, so it was unclear whether the latency was related to virtualization or if the SSDs were to blame.

Moreover, we considered various changes in hardware and software configurations, so a way to evaluate them was needed. Of course, we could run etcd in each configuration and look at the corresponding Prometheus metrics, but that would require significant effort. We needed a simple way to evaluate a specific configuration. We wanted to test our understanding of the Prometheus metrics coming from etcd.

To accomplish this, two problems needed to be solved:

First, what does the I/O load generated by etcd look like when writing to WAL files? What system calls are used? What is the size of the write blocks?
Second, suppose we have answers to the above questions. How to reproduce the corresponding load with . Just run the following command (the directory? Ведь . Just run the following command (the directory — an extremely flexible utility with a wealth of parameters (this can be easily verified, for example, here — trans. note).

We solved both problems using the same approach based on the commands lsof and strace:

Using lsof can be used to view all file descriptors used by the process as well as the files they refer to.
Using strace you can analyze an already running process or start a process and observe it. The command outputs all system calls made by this process and, if necessary, its descendants. The latter is important for processes that fork, and etcd is one of those processes.

The first thing we did was use strace to study the etcd server in the Kubernetes cluster while it was idle.

It was discovered that the write blocks in the WAL are very tightly grouped, with most sizes ranging from 2200 to 2400 bytes. This is why the command at the beginning of this article uses the flag --bs=2300 (bs — the size in bytes of each write block in . Just run the following command (the directory).

Note that the size of the etcd write blocks can vary depending on the version, deployment, parameter values, etc. — this affects the duration is within 10 ms. If so, your storage is performing fast enough. Here is an example output:. If you have a similar usage scenario, analyze your etcd processes to get relevant values. strace Then, to get a clear and comprehensive view of how etcd interacts with the file system, we ran it using

with the flags strace -ffttT -ffttTThis allowed us to cover child processes and record the output of each in a separate file. Additionally, detailed information was obtained about the start time and duration of each system call.

We also utilized the command lsof, to confirm our understanding of the output strace regarding which file descriptor was used for which purpose. The output was strace, similar to that provided above. Statistical manipulations of synchronization times confirmed that the metric wal_fsync_duration_seconds from etcd corresponds to calls is within 10 ms. If so, your storage is performing fast enough. Here is an example output: with WAL file descriptors.

To generate a workload similar to that from etcd, we studied the utility's documentation and selected parameters suitable for our task. We ensured that the necessary system calls were involved and confirmed their duration by running . Just run the following command (the directory (as was done in the case of etcd). . Just run the following command (the directory from strace Particular attention was given to determining the value of the parameter

. It represents the total I/O load generated by the fio utility. In our case, it is the total number of bytes written to the storage medium. It is directly proportional to the number of calls --bs). For a certain exit Volume Provisioning. is within 10 ms. If so, your storage is performing fast enough. Here is an example output:number of calls bs size / bs is within 10 ms. If so, your storage is performing fast enough. Here is an example output: is equal to Since we were interested in the percentile, we aimed for the number of samples to be sufficiently large for statistical significance. We decided that.

(which corresponds to a size of 22 MB) would be sufficient. Smaller parameter values 10^4 generated more pronounced noise (for example, calls --bs , which take significantly longer than usual and affect the 99th percentile). is within 10 ms. If so, your storage is performing fast enough. Here is an example output:The ball is in your court

The article shows how to assess whether a storage medium intended for use with etcd is fast enough. Now it’s your turn! You can explore virtual machines with SSD-based storage in the

IBM Cloud . Just run the following command (the directory You can find ready-to-use examples for solving other tasks in.

P.S. from the translator

or directly in the . Just run the following command (the directory project repository the documentation (there are many more examples presented there than mentioned in the documentation). P.P.S. from the translator etcd 3.4.3: exploring the reliability and security of the storage