🥇Durable data storage and Linux file APIs

As I explore the resilience of data storage in cloud systems, I decided to test myself to ensure that I understand the basics. I started by reading the NVMe specification to understand what guarantees regarding persistent data storage (i.e., guarantees that data will be available after a system failure) are provided by NVMe drives. I drew the following key conclusions: data should be considered corrupted from the moment the command to write data is issued until the moment it is fully written to the storage medium. However, most data writing programs comfortably use system calls.

In this material, I explore the mechanisms of persistent data storage provided by Linux file APIs. It seems that everything should be straightforward: the program calls the command write(), and after the completion of this command, the data will be securely saved on the disk. But write() it only copies the application data to the kernel cache located in RAM. To force the system to write data to disk, some additional mechanisms need to be employed.

Overall, this material is a set of notes concerning what I've learned about the topic that interests me. If I were to summarize the most important point, it would be that to ensure persistent data storage, one should use the command fdatasync() or open files with the flag O_DSYNC. If you are interested in the details of what happens to the data on its journey from the program code to the disk, take a look at this article.

Characteristics of using the write() function

The system call write() is defined in the IEEE POSIX as an attempt to write data to a file descriptor. After successful completion, write() read operations should return exactly those bytes that were previously written, doing so even if the data is accessed from other processes or threads (here the relevant section of the POSIX standard). Here, in the section dedicated to stream interaction with regular file operations, there is a note stating that if each of the two threads calls these functions, then each call must see either all the designated effects that result from the execution of the other call or not see any effects at all. This implies that all input/output file operations must hold a lock on the resource they are working with.

Does this mean that the operation write() is atomic? From a technical standpoint — yes. Data reading operations should return either everything or nothing of what was written using write(). However, the operation write(), according to the standard, does not necessarily have to complete by writing all that it was requested to write. It is allowed to write only part of the data. For example, we may have two threads, each appending 1024 bytes to a file described by the same file descriptor. From the standard's perspective, a permissible outcome would be for each of the write operations to append only one byte to the file. These operations would remain atomic, but after they are completed, the data they wrote to the file would be intermingled. Here There's a very interesting discussion on this topic on Stack Overflow.

The fsync() and fdatasync() functions

The simplest way to flush data to disk is to call the function fsync(). This function requests the operating system to transfer all modified blocks from the cache to the disk. This includes all file metadata (access time, modification time, and so on). I believe the need for this metadata arises rarely, so if you know that it is not important for you, you can use the function fdatasync(). In in the documentation by fdatasync() it is mentioned that during the operation of this function, the volume of metadata necessary for the correct execution of subsequent data reading operations is saved to the disk. And this is exactly what concerns most applications.

One of the issues that can arise here is that these mechanisms do not guarantee that the file can be discovered after a potential failure. Specifically, when creating a new file, one needs to call fsync() for the directory that contains it. Otherwise, after a crash, it may happen that this file does not exist. The reason for this is that in UNIX, due to the use of hard links, a file may exist in multiple directories. Therefore, when called fsync() there is no way for the file to know which directory's data should also be flushed to disk (here you can read more about this). It seems that the ext4 file system is capable of automatically applying fsync() to the directories containing the corresponding files, but with other file systems, this may not be the case.

This mechanism can be implemented differently in different file systems. I used blktrace to learn about the disk operations used in ext4 and XFS file systems. Both emit standard write commands to the disk for both file content and file system journal, flush the cache, and finish the operation with FUA write (Force Unit Access, writing data directly to the disk, bypassing cache) in the journal. They probably do this to confirm the fact of operation completion. On disks that do not support FUA, this causes two cache flushes. My experiments showed that fdatasync() a bit faster fsync(). Utility blktrace indicates that fdatasync() usually writes less data to disk (in ext4 fsync() writes 20 KiB, while fdatasync() writes 16 KiB). Additionally, I found that XFS is slightly faster than ext4. And here, with the help of blktrace I was able to find out that fdatasync() flushes less data to disk (4 KiB in XFS).

Ambiguous situations arising from using fsync()

I can recall three ambiguous situations related to fsync()that I encountered in practice.

The first such case occurred in 2008. At that time, the interface of Firefox 3 would 'freeze' if a large number of files were being written to disk. The problem was that the interface's state information was stored using an SQLite database. After each change made in the interface, the function was called fsync()which provided good guarantees of reliable data storage. In the ext3 file system used at that time, the function fsync() reset all "dirty" pages in the system to the disk, not just those related to the corresponding file. This meant that a click of a button in Firefox could trigger writing megabytes of data to the hard disk, which could take many seconds. The solution to the problem, as I understood from this one the material, was to move database operations to asynchronous background tasks. This means that earlier versions of Firefox implemented stricter requirements for data storage durability than were actually necessary, and the peculiarities of the ext3 file system only exacerbated this problem.

The second discrepancy occurred in 2009. Then, after a system crash, users of the new ext4 file system faced the issue that many recently created files had zero length, while the older ext3 file system did not experience similar issues. In the previous paragraph, I mentioned that ext3 reset too much data to the disk, which significantly slowed down performance fsync(). To improve the situation, ext4 only writes back to the disk the "dirty" pages related to a specific file. Other file data remains in memory for a much longer period than with ext3. This was done to enhance performance (by default, data remains in this state for 30 seconds, which can be adjusted using dirty_expire_centisecs; here , where you can find additional materials on this). This means that a large volume of data may be irretrievably lost after a crash. The solution to this problem involves using fsync() in applications that need to ensure persistent data storage and maximize protection against the consequences of crashes. The feature fsync() works much more efficiently with ext4 than with ext3. The downside of this approach is that, as before, it slows down some operations, such as program installation. For details on this, see here and here.

The third issue regarding fsync(), arose in 2018. At that time, as part of the PostgreSQL project, it was discovered that if the function fsync() encounters an error, it marks "dirty" pages as "clean." As a result, subsequent calls fsync() nothing is done with such pages. Because of this, modified pages are kept in memory and are never written to disk. This is a real disaster, as the application will think that some data has been written to the disk, but in reality, this is not the case. Such failures fsync() are rare, and the application can do almost nothing to combat the problem in such situations. Nowadays, when this happens, PostgreSQL and other applications crash. Here, in the article "Can Applications Recover from fsync Failures?", this issue is explored in detail. Currently, the best solution to this problem is to use Direct I/O with the flag O_SYNC or with the flag O_DSYNC. In this approach, the system will report errors that may occur during specific data write operations, but this approach requires the application to manage buffers itself. Read more about this here and here.

Opening files using the O_SYNC and O_DSYNC flags

Let’s return to discussing the Linux mechanisms that provide robust data storage. Specifically, we are talking about the use of the flag O_SYNC or the flag O_DSYNC when opening files using the system call open(). With this approach, each data write operation is executed as if after each command write() the system is given, respectively, commands fsync() and fdatasync(). In of the POSIX specifications this is referred to as "Synchronized I/O File Integrity Completion" and "Data Integrity Completion". The main advantage of this approach is that ensuring data integrity requires only one system call rather than two (for example - write() and fdatasync()). The main disadvantage of this approach is that all write operations using the corresponding file descriptor will be synchronized, which may limit the application's code structuring capabilities.

Using Direct I/O with the O_DIRECT flag

The system call open() supports the flag O_DIRECT, which is designed to bypass the operating system cache and perform I/O operations directly with the disk. This often means that the write commands issued by the program will be directly translated into commands directed at disk operations. However, in general, this mechanism does not replace the functions fsync() or fdatasync(). The fact is that the disk itself can defer or cache the appropriate data write commands. Furthermore, in certain specific cases, the input/output operations performed using the flag O_DIRECT, are translated into traditional buffered operations. The easiest way to resolve this issue is by opening files with the additional flag O_DSYNC, which means that every write operation will be followed by a call fdatasync().

It turns out that a 'fast path' was recently added to the XFS file system for O_DIRECT|O_DSYNC-data writes. If a block is rewritten using O_DIRECT|O_DSYNC, then XFS, instead of flushing the cache, will execute the FUA (Force Unit Access) write command if the device supports it. I confirmed this by using the utility blktrace on Linux 5.4/Ubuntu 20.04. This approach should be more efficient, as it writes the minimum amount of data to disk and uses a single operation instead of two (write and flush the cache). I found a link to patch a kernel from 2018, where this mechanism was implemented. There is discussion about applying this optimization in other file systems, but as far as I know, XFS is currently the only file system that supports this.

The function sync_file_range()

In Linux, there is a system call sync_file_range(), which allows flushing only a portion of a file to disk, rather than the entire file. This call initiates an asynchronous data flush and does not wait for its completion. However, the documentation for sync_file_range() states that this command is 'very dangerous'. It is not recommended to use it. The specifics and dangers sync_file_range() are well described in this the material. In particular, it seems that this call uses RocksDB to manage when the kernel flushes 'dirty' data to disk. However, for ensuring data durability, it also utilizes fdatasync(). In the code RocksDB has interesting comments on this topic. For example, it appears that the call sync_file_range() when using ZFS does not lead to a flush of data to disk. My experience suggests that rarely used code may contain bugs. Therefore, I would advise against using this system call unless absolutely necessary.

System calls that help ensure data durability

I have come to the conclusion that there are three approaches for performing input/output operations that ensure durable data storage. All of them require calling a function fsync() for the directory where the file is created. Here are these approaches:

Calling the function fdatasync() or fsync() after the function write() (better to use fdatasync()).
Working with a file descriptor opened with the flag O_DSYNC or O_SYNC (better — with the flag O_DSYNC).
Using the command pwritev2() with the flag RWF_DSYNC or RWF_SYNC (preferably — with the flag RWF_DSYNC).

Notes on Performance

I have not conducted thorough measurements of the performance of the various mechanisms I have explored. The differences in speed that I noticed are quite small. This means that I could be mistaken, and that under different conditions the same might yield other results. First, I will discuss what has a stronger impact on performance, and then, what affects performance less.

Overwriting file data is faster than appending data to a file (the performance gain can be 2-100%). Appending data to a file requires making additional changes to the file's metadata, even after a system call fallocate(), but the scale of this effect can vary. I recommend, for the best performance, to call fallocate() for pre-allocating the necessary space. Then this space needs to be explicitly filled with zeros and call fsync(). This will mark the corresponding blocks in the file system as 'allocated' rather than 'unallocated'. This gives a slight (about 2%) performance improvement. Additionally, some disks may perform the first block access operation slower than others. This means that zero-filling the space can lead to a significant (about 100%) performance improvement. In particular, this can happen with disks AWS EBS (this is unofficial data, I could not verify it). The same applies to storage GCP Persistent Disk (this is already official information, confirmed by tests). Other specialists have made similar observations, related to various disks.
The fewer system calls, the higher the performance (the gain can be about 5%). It seems that calling open() with the flag O_DSYNC or calling pwritev2() with the flag RWF_SYNC is faster than calling fdatasync()I suspect that the reason lies in the fact that with this approach, fewer system calls are needed to accomplish the same task (one call instead of two). However, the performance difference is minimal, so you can safely ignore it and use what won't complicate your application's logic.

If you're interested in the topic of durable data storage, here are some useful materials:

I/O Access methods — an overview of the basic mechanisms of input/output.
Ensuring data reaches disk — a description of what happens to data as it travels from the application to the disk.
When should you fsync the containing directory — the answer to the question of when to apply fsync() to directories. In short, this should be done when creating a new file, and the reason for this recommendation is that in Linux, there can be many links to the same file.
SQL Server on Linux: FUA Internals — this provides a description of how durable data storage is implemented in SQL Server on the Linux platform. It includes some interesting comparisons between system calls in Windows and Linux. I’m almost sure that it was this material that made me aware of XFS FUA optimization.

Have you ever lost data you thought was securely stored on disk?