A Brief History of fio and etcd
Cluster Performance largely depends on the performance of its storage. etcd exports some metrics to , to provide necessary information about the storage performance. For example, the metric wal_fsync_duration_seconds. : for the storage to be considered fast enough, the 99th percentile of this metric should be under 10 ms. If you plan to run an etcd cluster on Linux machines and want to assess whether your storage (like SSD) is fast enough, you can use — a popular tool for testing input/output operations. Run the following command where test-data is the directory under the storage mount point:
You just need to look at the output and check if the 99th percentile ofYou just need to look at the results and check that the 99th percentile duration is under 10 ms. If so, you have sufficiently fast storage. Here’s an example of the results:
sync (usec): min=534, max=15766, avg=1273.08, stdev=1084.70
sync percentiles (usec):
| 1.00th=[ 553], 5.00th=[ 578], 10.00th=[ 594], 20.00th=[ 627],
| 30.00th=[ 709], 40.00th=[ 750], 50.00th=[ 783], 60.00th=[ 1549],
| 70.00th=[ 1729], 80.00th=[ 1991], 90.00th=[ 2180], 95.00th=[ 2278],
| 99.00th=[ 2376], 99.50th=[ 9634], 99.90th=[15795], 99.95th=[15795],
| 99.99th=[15795]Notes
- We configured the parameters —size and —bs for our specific scenario. To get useful results from fio, specify your values. Where to find them? Read .
- During testing, all input/output load comes from fio. In a real scenario, there will likely be other write requests to the storage apart from those associated with wal_fsync_duration_seconds. Additional load will increase the wal_fsync_duration_seconds value. So if the 99th percentile is nearly hitting 10 ms, your storage is not fast enough.
- Use version at least 3.5 (previous versions do not display the percentiles of fdatasync duration).
- The above shows only a snippet of results from fio.
A Long Story About fio and etcd
What is WAL in etcd
Typically, databases use ; etcd also uses it. Here, we will not discuss the write-ahead log (WAL) in detail. It suffices to know that each member of the etcd cluster maintains it in persistent storage. etcd logs every operation with key-value pairs (for instance, updates) in the WAL before applying them to the storage. If one of the members of the storage crashes and restarts between snapshots, it can locally recover transactions from the last snapshot based on the contents of the WAL.
When a client adds a key to the key-value storage or updates the value of an existing key, etcd records this operation in the WAL, which is a regular file in persistent storage. Before proceeding, etcd MUST be completely sure that the entry in the WAL has indeed occurred. In Linux, a single system call is not sufficient for this. , as the actual write to physical storage may be delayed. For example, Linux may hold the WAL record in the kernel’s memory cache (such as page cache) for a while. To ensure that the data is correctly written to persistent storage, a fdatasync system call is required after writing, and etcd uses it (as can be seen from the output of , where 8 is the file descriptor for the WAL):
21:23:09.894875 lseek(8, 0, SEEK_CUR) = 12808
21:23:09.894911 write(8, ". 20210220361223255266632$10 20103026"34"rn3fo"..., 2296) = 2296
21:23:09.895041 fdatasync(8) = 0Unfortunately, writing to permanent storage does not occur instantly. If the fdatasync call operates slowly, the performance of the etcd system decreases. , that storage is considered fast enough if in the 99th percentile, fdatasync calls while writing to the WAL file take less than 10 ms. There are other useful metrics for storage, but in this post, we are only discussing this metric.
Evaluating the storage with fio
If you need to assess whether your storage is suitable for etcd, use fio — a very popular I/O load testing tool. It is important to remember that disk operations can vary widely: synchronous and asynchronous, many classes of system calls, etc. Consequently, fio is quite complex to use. It has numerous parameters, and different combinations of their values produce completely different I/O workloads. To obtain accurate figures for etcd, ensure that the write workload from fio closely resembles the actual write workload from etcd when writing WAL files.
Therefore, fio must, at a minimum, create a workload consisting of a series of sequential write operations to a file, each write consisting of a system call , followed by a system call fdatasync. For sequential write operations, fio needs the parameter —rw=write. To ensure fio uses the write system call rather than , you should specify the parameter —ioengine=sync. Finally, to invoke fdatasync after each write, you need to add the parameter —fdatasync=1. The other two parameters in this example (—size and —bs) depend on the specific scenario. In the next section, we will discuss how to configure them.
Why fio and how we learned to configure it
In this post, we describe a real case. We had a cluster v1.13, which we monitored with Prometheus. etcd v3.2.24 was running on SSD. The metrics for etcd indicated excessively high latencies for fdatasync, even when the cluster was idle. The metrics were strange, and we didn't quite understand what they meant. The cluster consisted of virtual machines, and we needed to determine whether the issue was with the physical SSDs or the virtualization layer. Additionally, we often made changes to the hardware and software configuration, and we needed a way to assess their impact. We could run etcd with each configuration and observe the Prometheus metrics, but that was too cumbersome. We were looking for a sufficiently simple way to evaluate a specific configuration. We wanted to verify if we understood the Prometheus metrics from etcd correctly.
But to do this, two problems needed to be solved. First, what does the I/O load look like that etcd generates when writing to the WAL? What system calls are used? What is the size of the records? Second, if we answer these questions, how can we reproduce a similar workload with fio? Keep in mind that fio is a very flexible tool with many parameters. We solved both problems with one approach — through commands. and . lsof displays all file descriptors used by the process and the associated files. Using strace, one can examine an already running process or start a process and study it. strace outputs all system calls from the examined process (and its child processes). The latter is quite important, as etcd applies a similar approach.
First, we used strace to examine the etcd server for Kubernetes when there was no load on the cluster. We saw that almost all WAL records were approximately the same size: 2200–2400 bytes. Therefore, in the command at the beginning of the post, we specified the parameter —bs=2300 (bs stands for the size in bytes for each fio record). Note that the size of the etcd record depends on the version of etcd, the distribution, parameter values, etc., and affects the duration of fdatasync. If you have a similar scenario, study your etcd processes using strace to find the exact figures.
Then, to clearly visualize the actions in the etcd file system, we ran it with strace and parameters -ffttT. This way, we tried to examine the child processes and log the output of each of them into separate files, as well as obtain detailed reports on the start and duration of each system call. We used lsof to confirm our analysis of strace output and see which file descriptor was used for what purpose. Thus, using strace, we obtained the results shown above. The synchronization time statistics confirmed that the wal_fsync_duration_seconds metric from etcd corresponds to fdatasync calls with WAL file descriptors.
We studied the documentation for fio and selected parameters for our scenario so that fio could generate a load similar to etcd. We also checked the system calls and their duration by running fio from strace, similar to etcd.
We carefully defined the parameter value —size, which represents the entire I/O load from fio. In our case, this is the total number of bytes written to the storage. It proved to be directly proportional to the number of system calls write (and fdatasync). For a specific value of bs, the number of fdatasync calls = size/bs. Since we were interested in the percentile, we needed enough samples for reliability, and we calculated that 10^4 (which equals 22 mebibytes) would be sufficient. If —size is smaller, outliers may occur (for instance, several fdatasync calls taking longer than usual and affecting the 99th percentile).
Try it for yourself
We have demonstrated how to use fio to determine if the storage speed is sufficient for high performance with etcd. Now you can try this on your own, using, for example, virtual machines with SSD storage in .
Source: habr.com
