Have a great weekend, everyone! Join us for a free Demo class , led by Andrey Buranov — a UNIX systems specialist at Mail.Ru Group. We are also publishing an article by Jonathan Corbet — Executive Editor at LWN.net.
Journaling file systems promise to free system administrators from disk corruption issues during system crashes, even without running filesystem integrity checks. However, in reality, things are quite a bit more complicated. Recent discussions suggest it may be even more intricate than many of us think, as ensuring the integrity of journaling file systems affects performance.
A file system like ext3 uses a separate area on the disk called a journal. When changes are made to the file system's metadata, these changes are first recorded in the journal without modifying the rest of the file system. After all changes are logged, a 'commit block' is added to the journal to indicate the completion of the transaction. Only after the commit block is recorded is the transaction finalized, and the changed metadata is written to the disk. If the system fails at any point, the information in the journal allows for a safe shutdown, preventing file system corruption due to only part of the metadata being updated.
However, there's a catch: the file system's code must be absolutely certain that all transaction information is in the journal before writing the commit block. Simply recording operations in the correct order is not enough — modern drives have large internal caches and reorder operations for better performance. Therefore, before the commit block, it's essential to explicitly indicate that all journal data has been transferred to the disk. If the commit block is recorded too early, the journal could become corrupted. This problem is addressed by using barriers. Essentially, a barrier prevents any blocks from being written after it until all blocks written before the barrier have been transferred to the disk. By using barriers, file systems ensure the consistency of file structures.
But there is another problem: ext3 and ext4 file systems do not use barriers by default. There is an option, but if the administrator has not explicitly enabled them, these file systems operate without barriers, although some distributions (like SUSE) have different default values. Eric Sandeen recently decided that this situation needed to change and , modifying the default settings for ext3 and ext4. And thus began a lively discussion.
Andrew Morton responded in detail why the default value is as it is:
The last time we tried to change this, performance on many workloads deteriorated by 30%, so I was horrified and threw out all those patches. I think we can't afford to do this and slow down all machines so seriously…
There are no perfect solutions here, and I tend to think it's best not to wake that sleeping dog and leave the default settings to the discretion of the distribution developers.
Thus, by default, barriers are disabled, as they seriously affect performance. Additionally, file systems are quite successfully used without barriers. Reports of ext3 file system corruption are few and far between.
But it's not just luck. Ted Ts’o this by saying that the ext3/ext4 journal is usually laid out continuously. First, the file system driver tries to create it continuously. Second, the journal is typically created alongside the file system, when it is easy to find continuous space. Continuity and order are beneficial not only for performance but also for preventing reordering. Usually, the commit block will be placed right after other data in the journal, so the disk has no reason to reorder. The commit block is naturally written to the disk immediately after the other journal entries.
However, nobody claims that this will always be the case. Disk drives can behave differently. Moreover, the journal functions as a circular buffer. Therefore, when a transaction is written to the end of the journal, the commit block may be located in an earlier block, preceding other journal entries. Thus, the risk of corruption is always present. In fact, Chris Mason has reasons for this. There is no doubt that barrier-free operation is less secure than operation with them.
If you're willing to take a performance hit, you can enable barriers. This, of course, is provided that your filesystem is not based on LVM (as some distributions are by default). It turns out that the device mapper does not support barriers. In other cases, it would be good to minimize performance degradation. And it seems this can be done.
The current implementation of ext3 (with barriers enabled) performs the following sequence of operations for each transaction:
Data is logged to the journal
A barrier is executed
The commit block is written
The next barrier is executed
Later, metadata is flushed to disk
In ext4, the first barrier (step 2) can be skipped, as the ext4 filesystem supports checksums in the journal.
If the journal data and commit block are reordered, and the operation is interrupted due to a failure, the journal checksum will not match the one stored in the commit block, resulting in a transaction rejection.
Chris Mason , that it would generally be "safe" to remove this barrier in ext3, with the possible exception when the journal reaches its end and starts writing from the beginning.
Another idea to enhance performance is to delay barrier operations when possible. If there’s no pressing need to immediately flush data to disk, multiple transactions can be created in the journal, and a single barrier can be used to flush them to disk.
There is also some potential for improvement by carefully ordering operations so that barriers (which are usually implemented as requests to "flush all pending operations to disk") do not force the writing of blocks that do not require ordering.
It seems the time has come to think about how to make the cost of barriers acceptable. Ted Ts’o seems to :
I think we should include barriers in ext3/ext4, and then work on reducing overhead in ext4/jbd2. Most likely, the overwhelming majority of systems do not operate under conditions similar to those Chris used to demonstrate the problem, and the filesystem security should be a priority.
Common sense tells me that this dog is already awake and will probably bark for a while. This may cause some neighbors to be uneasy, but it's better than letting it bite.
Interested in developing in this direction? Sign up for a free demo lesson and participate in the broadcast , hosted by Pavel Vikyryuk — MVNO communications operator, DevOps engineer.
Source: habr.com
