{"id":83582,"date":"2020-06-01T19:42:21","date_gmt":"2020-06-01T17:42:21","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/osnovy-zfs-sistema-hraneniya-i-proizvoditelnost"},"modified":"2020-06-01T19:42:21","modified_gmt":"2020-06-01T17:42:21","slug":"osnovy-zfs-sistema-hraneniya-i-proizvoditelnost","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/osnovy-zfs-sistema-hraneniya-i-proizvoditelnost","title":{"rendered":"Basics of ZFS: storage system and performance","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Basics
\n
\nThis spring, we have already discussed some introductory topics, such as how to check your disk speed<\/a><\/noindex> and what RAID is<\/a><\/noindex>. In the second part, we even promised to continue exploring the performance of various multi-disk topologies in ZFS. This is a next-generation file system that is being implemented everywhere: from Apple<\/a><\/noindex> up to Ubuntu<\/a><\/noindex>.
\n<\/a><\/noindex>
\nWell, today is the perfect day to get to know ZFS, curious readers. Just know that according to OpenZFS developer Matt Ahrens, \u201cit\u2019s really complicated.\u201d<\/p>\n

But before we get to the numbers \u2014 and I promise there will be, covering all options of the eight-disk configuration of ZFS \u2014 we need to talk about as<\/i> how ZFS actually stores data on disk.<\/p>\n

Zpool, vdev, and device<\/h1>\n

\n\"Basics
\nThis full pool diagram includes three auxiliary vdevs, one of each class, and four for RAIDz2.<\/i><\/p>\n

\"Basics
\nUsually, there\u2019s no reason to create a pool from mismatched types and sizes of vdev \u2014 but if you want, nothing prevents you from doing so<\/i><\/p>\n

To truly understand the ZFS file system, you need to take a close look at its actual structure. First, ZFS combines traditional levels of volume management and file system. Second, it uses a copy-on-write transactional mechanism. These features mean that the system is structurally very different from regular file systems and RAID arrays. The first set of fundamental building blocks for understanding is: storage pool (zpool), virtual device (vdev), and physical device (device).<\/p>\n

zpool<\/h3>\n

\nThe storage pool zpool is the topmost structure of ZFS. Each pool contains one or more virtual devices. In turn, each of these contains one or more physical devices (device). Virtual pools are standalone blocks. One physical computer can contain two or more separate pools, but each one is completely independent of the others. Pools cannot share virtual devices.<\/p>\n

ZFS redundancy exists at the level of virtual devices, not at the level of pools. At the pool level, there is absolutely no redundancy \u2014 if any vdev or special vdev storage is lost, the entire pool is lost along with it.<\/p>\n

Modern storage pools can withstand the loss of cache or log from a virtual device\u2014though they may lose a small amount of dirty data if the vdev log is lost during a power outage or system failure.<\/p>\n

There is a common misconception that ZFS data stripes are written across the entire pool. This is incorrect. Zpool is not simply a fun RAID0; it is rather complex. JBOD<\/a><\/noindex> with a complex variable distribution mechanism.<\/p>\n

For the most part, writes are distributed across available virtual devices according to the free space available, so theoretically, they will all fill up simultaneously. In later versions of ZFS, the current utilization of the vdev is taken into account\u2014if one virtual device is significantly busier than another (for example, due to a heavy read load), it will be temporarily skipped for writes despite having the highest free space ratio.<\/p>\n

The utilization determination mechanism built into modern ZFS write distribution methods can reduce latency and increase throughput during periods of unusually high load\u2014but this is not a blank check<\/i> for unintentional mixing of slow HDDs and fast SSDs in one pool. Such an imbalanced pool will still operate at the speed of the slowest device, as if it were entirely made up of those devices.<\/p>\n

vdev<\/h3>\n

\nEach storage pool consists of one or more virtual devices (vdev). Each vdev, in turn, contains one or more physical devices. Most virtual devices are used for simple data storage, but there are several auxiliary classes of vdev, including CACHE, LOG, and SPECIAL. Each of these types of vdev can have one of five topologies: single-device, RAIDz1, RAIDz2, RAIDz3, or mirror.<\/p>\n

RAIDz1, RAIDz2, and RAIDz3 are special types of what old-timers might call double (diagonal) parity RAID. The 1, 2, and 3 refer to how many parity blocks are allocated for each data stripe. Instead of using separate disks for parity, RAIDz virtual devices distribute this parity semi-evenly across disks. A RAIDz array can lose as many disks as it has parity blocks; if it loses one more, it will fail and take the storage pool with it.<\/p>\n

In mirror virtual devices (mirror vdev), each block is stored on every device in the vdev. While the most common configuration is a two-wide mirror, any arbitrary number of devices can be included in a mirror\u2014triple mirrors are often used in larger installations to enhance read performance and fault tolerance. A vdev mirror can survive any failure as long as at least one device in the vdev is still operational.<\/p>\n

Single vdevs are inherently dangerous. Such a virtual device will not survive any failure\u2014if used as storage or a special vdev, its failure will result in the destruction of the entire pool. Exercise extreme caution here.<\/p>\n

CACHE, LOG, and SPECIAL virtual devices can be created using any of the aforementioned topologies\u2014but remember that losing a SPECIAL virtual device means losing the pool, so redundant topology is highly recommended.<\/p>\n

device<\/h3>\n

\nProbably the easiest term to understand in ZFS\u2014it's literally a block device of random access. Remember, virtual devices are made up of individual devices, and a pool is made up of virtual devices.<\/p>\n

Disks\u2014magnetic or solid-state\u2014are the most common block devices used as building blocks for vdev. However, any device with a descriptor in \/dev will work\u2014so entire hardware RAID arrays can be used as individual devices.<\/p>\n

A simple raw file is one of the most important alternative block devices from which a vdev can be constructed. Test pools from sparse files<\/a><\/noindex>\u00a0are a very convenient way to test pool commands and see how much space is available in the pool or virtual device of this topology.<\/p>\n

\"Basics
\nYou can create a test pool from sparse files in just a few seconds\u2014 but don't forget to delete the entire pool and its components afterward.<\/i> <\/p>\n

Let's say you want to set up a server with eight disks and plan to use 10 TB (~9300 GiB) disks\u2014 but you're not sure which topology best meets your needs. In the example above, we construct a test pool from sparse files in a matter of seconds\u2014 and now we know that a RAIDz2 vdev with eight 10 TB disks provides 50 TiB of usable capacity.<\/p>\n

Another special class of devices is SPARE. Hot-swappable devices, unlike regular devices, belong to the entire pool rather than a single virtual device. If any vdev in the pool fails and a spare device is connected and available, it will automatically attach to the affected vdev.<\/p>\n

Once connected to the affected vdev, the spare device begins receiving copies or reconstructions of the data that should be on the missing device. In traditional RAID, this is called rebuilding, while in ZFS, it is termed \"resilvering.\"<\/p>\n

It is important to note that spare devices do not permanently replace failed devices. They are a temporary substitute to reduce the time during which vdev is degraded. After the administrator replaces the failed vdev device, the resilvering occurs on that permanent device, and the SPARE detaches from vdev and returns to being a spare for the entire pool.<\/p>\n

Datasets, blocks, and sectors<\/h1>\n

\nThe next set of building blocks to understand in our journey through ZFS relates not so much to hardware as to how the data itself is organized and stored. We are skipping several levels\u2014 such as metaslab\u2014 to avoid overwhelming details while maintaining an understanding of the overall structure.<\/p>\n

Dataset<\/h3>\n

\n\"Basics
\nWhen we first create a dataset, it shows all the available space in the pool. We then set a quota\u2014 and change the mount point. Magic!<\/i> <\/p>\n

\"Basics
\nZvol is essentially just a dataset devoid of its filesystem layer, which we replace here with a perfectly normal ext4 filesystem.<\/i> <\/p>\n

A ZFS dataset is roughly analogous to a standard mounted filesystem. Like a regular filesystem, at first glance, it seems like \"just another folder.\" However, much like standard mounted filesystems, each ZFS dataset has its own set of base properties.<\/p>\n

First of all, a dataset can have an assigned quota. If you set zfs set quota=100G poolname\/datasetname<\/code>, then you will not be able to write to the mounted folder \/poolname\/datasetname<\/code> more than 100 GiB.<\/p>\n

Did you notice the presence\u2014and absence\u2014of slashes at the beginning of each line? Each dataset has its place in both the ZFS hierarchy and the system mounting hierarchy. In the ZFS hierarchy, there is no leading slash\u2014you start with the pool name and then the path from one dataset to the next. For example, pool\/parent\/child<\/code> for a dataset named child<\/code> under the parent dataset parent<\/code> in a creatively named pool. pool<\/code>.<\/p>\n

By default, the mount point of a dataset will be equivalent to its name in the ZFS hierarchy, with a slash at the beginning\u2014the pool named pool<\/code> will be mounted as \/pool<\/code>, the dataset parent<\/code> is mounted at \/pool\/parent<\/code>, and the child dataset child<\/code> is mounted at \/pool\/parent\/child<\/code>. However, the system mount point of a dataset can be changed.<\/p>\n

If we specify zfs set mountpoint=\/lol pool\/parent\/child<\/code>, then the dataset pool\/parent\/child<\/code> will be mounted in the system as \/lol<\/code>.<\/p>\n

In addition to datasets, we must mention volumes (zvols). A volume is somewhat analogous to a dataset, except that it actually doesn\u2019t have a filesystem\u2014it\u2019s just a block device. For example, you can create zvol<\/code> named mypool\/myzvol<\/code>, then format it with the ext4 filesystem, and then mount this filesystem\u2014now you have an ext4 filesystem, but with all the security features of ZFS! This may seem silly on a single computer, but it makes much more sense as a backend when exporting iSCSI devices.<\/p>\n

Units<\/h3>\n

\n\"Basics
\nA file is represented by one or more blocks. Each block is stored on one virtual device. The block size is usually equal to the parameter recordsize<\/b>, but can be reduced to 2^ashift<\/b>, if it contains metadata or a small file.<\/i><\/p>\n

\"Basics
\nWe really really<\/b> aren't joking about the enormous performance penalty if you set ashift too small.<\/i><\/p>\n

In a ZFS pool, all data, including metadata, is stored in blocks. The maximum block size for each dataset is defined by the property recordsize<\/code> (record size). The record size can change, but this does not affect the size or location of any blocks that have already been written to the dataset\u2014it only applies to new blocks as they are written.<\/p>\n

If not specified otherwise, the default record size is 128 KiB. This is somewhat of a compromise, where the performance will not be ideal, but it won't be terrible in most cases. Recordsize<\/code> can be set to any value from 4K to 1M (with additional settings recordsize<\/code> it can be set even larger, but this is rarely a good idea).<\/p>\n

Any block refers to the data of only one file\u2014you cannot squeeze two different files into one block. Each file consists of one or more blocks, depending on its size. If the file size is smaller than the record size, it will be stored in a smaller block\u2014 for example, a block with a 2 KiB file will occupy only one 4 KiB sector on the disk.<\/p>\n

Zvols do not have the property recordsize<\/code>\u00a0\u2014 instead, they have an equivalent property volblocksize.<\/code>.<\/p>\n

Sectors<\/h3>\n

\nThe last, most basic building block is the sector. This is the smallest physical unit that can be written or read from the underlying device. For several decades, sectors of 512 bytes were used in most disks. Recently, most disks have been configured for 4 KiB sectors, and in some\u2014especially SSDs\u20148 KiB sectors or even larger.<\/p>\n

In the ZFS system, there is a property that allows you to manually set the sector size. This property ashift<\/code>. It is somewhat confusing that ashift is a power of two. For example, ashift=9<\/code> means a sector size of 2^9, or 512 bytes.<\/p>\n

ZFS queries the operating system for detailed information about each block device when it is added to a new vdev, and theoretically automatically sets ashift correctly based on this information. Unfortunately, many disks lie about their sector size to maintain compatibility with Windows XP (which was unable to understand disks with other sector sizes).<\/p>\n

This means that it is highly recommended for ZFS administrators to know the actual sector size of their devices and set it manually. ashift<\/code>. If ashift is set too low, the number of read\/write operations increases astronomically. For example, writing 512-byte 'sectors' to an actual 4 KiB sector requires writing the first 'sector', then reading the 4 KiB sector, modifying it with the second 512-byte 'sector', writing it back to a new 4 KiB sector, and so on for each write.<\/p>\n

In the real world, such a penalty affects Samsung EVO solid-state drives, for which it should be ashift=13<\/code>, but these SSDs lie about their sector size, which is why it is set by default to ashift=9<\/code>. If an experienced system administrator does not change this parameter, the SSD operates slower.<\/i> like a regular magnetic HDD.<\/p>\n

In comparison, there is practically no penalty for setting too large a size. ashift<\/code> There is no real performance degradation, and the increase in unused space is infinitesimal (or zero when compression is enabled). Therefore, we strongly recommend even for those disks that actually use 512-byte sectors to set ashift=12<\/code> or even ashift=13<\/code>, to confidently look to the future.<\/p>\n

window.name ashift<\/code> This is set for each virtual device vdev, and not for the pool<\/i>, as many mistakenly believe \u2014 and does not change after it is set. If you accidentally misconfigured ashift<\/code> when adding a new vdev to the pool, you have irrevocably polluted that pool with a low-performance device and generally there is no other option but to destroy the pool and start over. Even removing the vdev won't save you from the misconfiguration. ashift<\/code>!<\/p>\n

The copy-on-write mechanism<\/h3>\n

\n\"Basics
\nIf a regular file system needs to rewrite data, it modifies each block where it is located.<\/i><\/p>\n

\"Basics
\nThe copy-on-write file system writes a new version of the block and then unlocks the old version.<\/i><\/p>\n

\"Basics
\nIn abstract terms, ignoring the actual physical location of the blocks, our 'data comet' simplifies to a 'data worm' that moves left to right across the map of available space.<\/i><\/p>\n

\"Basics
\nNow we can get a good understanding of how copy-on-write snapshots work \u2014 each block can belong to multiple snapshots and will remain until all associated snapshots are destroyed.<\/i><\/p>\n

The copy-on-write mechanism (CoW) is the fundamental basis of what makes ZFS such an amazing system. The core concept is simple \u2014 if you ask a traditional file system to change a file, it will do exactly what you asked. If you ask a copy-on-write file system to do the same, it will say 'sure' \u2014 but it will lie to you.<\/p>\n

Instead, the copy-on-write file system writes a new version of the modified block and then updates the file's metadata to break the link with the old block and associate it with the new block you just wrote.<\/p>\n

The disconnection of the old block and linking of the new one occurs in a single operation, so it cannot be interrupted \u2014 if you lose power after it happens, you have a new version of the file, and if you lose power before, you have the old version. In either case, no conflicts will arise in the file system.<\/p>\n

Copy-on-write in ZFS occurs not only at the file system level but also at the disk management level. This means that ZFS is not subject to a write hole (a RAID hole<\/a><\/noindex>) \u2014 a phenomenon where the stripe was only partially written before the system failure, resulting in data corruption after reboot. Here, the stripe is written atomically, the vdev is always consistent, and Bob's your uncle.<\/a><\/noindex>.<\/p>\n

ZIL: ZFS Intent Log<\/h3>\n

\n\"Basics
\nThe ZFS system handles synchronous writes specially \u2014 it temporarily but immediately saves them in the ZIL, before later committing them permanently along with asynchronous writes.<\/i><\/p>\n

\"Basics
\nUsually, data written to the ZIL is never read again. But it is possible after a system failure.<\/i><\/p>\n

\"Basics
\nSLOG, or secondary LOG device, is simply a special \u2014 and ideally very fast \u2014 vdev where the ZIL can be stored separately from the main storage.<\/i><\/p>\n

\"Basics
\nAfter a failure, all dirty data in the ZIL is replayed \u2014 in this case, the ZIL is on the SLOG, so it is replayed from there.<\/i><\/p>\n

There are two main categories of write operations \u2014 synchronous (sync) and asynchronous (async). For most workloads, the overwhelming majority of write operations are asynchronous \u2014 the file system allows them to be aggregated and issued in batches, reducing fragmentation and significantly increasing throughput.<\/p>\n

Synchronous writes, however, are a completely different matter. When an application requests a synchronous write, it tells the file system: 'You need to commit this to non-volatile memory, right now<\/i>, and until then, I can't do anything else.' Therefore, synchronous writes must be immediately committed to disk \u2014 and if this increases fragmentation or reduces throughput, so be it.<\/p>\n

ZFS handles synchronous writes differently than traditional file systems \u2014 instead of immediately writing them to conventional storage, ZFS commits them to a special storage area known as the ZFS Intent Log, or ZIL. The trick is that these writes also<\/i> remain in memory, being aggregated together with regular asynchronous write requests, to later be flushed to storage as completely normal TXG (Transaction Groups).<\/p>\n

During normal operation, the ZIL is written to and never read again. When, after a few moments, the records from the ZIL are committed to the main storage in regular TXG from RAM, they are detached from the ZIL. The only time anything is read from the ZIL is during pool import.<\/p>\n

If a ZFS failure occurs \u2014 whether it\u2019s an OS failure or a power loss \u2014 when there is data in the ZIL, that data will be read during the next pool import (for example, after a system crash restart). Everything in the ZIL will be read, grouped into TXG, committed to the main storage, and then detached from the ZIL during the import process.<\/p>\n

One of the auxiliary classes vdev is called LOG or SLOG, a secondary LOG device. Its sole purpose is to provide the pool with a separate and, preferably, much faster vdev that has very high write resilience for storing ZIL, instead of storing ZIL on the main storage vdev. The ZIL itself behaves the same regardless of its storage location, but if the vdev with LOG has very high write performance, synchronous writes will occur faster.<\/p>\n

Adding a vdev with LOG to the pool does not cannot<\/b> improve the performance of asynchronous writes \u2014 even if you force all writes to ZIL using zfs set sync=always<\/code>, they will still be tied to the main storage in TXG in the same way and at the same pace as without the log. The only direct performance improvement is the latency of synchronous writes (as a higher log speed accelerates the execution of operations. sync<\/code>).<\/p>\n

However, in an environment that already requires a lot of synchronous writes, the vdev LOG can indirectly speed up asynchronous writes and uncached reads. Offloading ZIL writes to a separate vdev LOG means less competition for IOPS in the primary storage, which somewhat boosts the performance of all read and write operations.<\/p>\n

Snapshots<\/h3>\n

\nThe copy-on-write mechanism is also a necessary foundation for atomic ZFS snapshots and incremental asynchronous replication. In an active file system, there is a tree of pointers marking all records with the current data \u2014 when you take a snapshot, you simply create a copy of this pointer tree.<\/p>\n

When a record is overwritten in an active file system, ZFS first writes a new version of the block to unused space. It then detaches the old version of the block from the current file system. But if any snapshot references the old block, it remains unchanged. The old block actually will not be reclaimed as free space until all snapshots referencing that block are destroyed!<\/p>\n

Replication<\/h3>\n

\n\"Basics
\nMy Steam library in 2015 occupied 158 GiB and included 126,927 files. This is quite close to the optimal situation for rsync \u2014 ZFS replication over the network was 'only' 750% faster.<\/i><\/p>\n

\"Basics
\nIn the same network, replicating a 40-gigabyte Windows 7 virtual machine image is a completely different story. ZFS replication occurs 289 times faster than rsync\u2014or 'only' 161 times faster if you're savvy enough to invoke rsync with the --inplace key.<\/i><\/p>\n

\"Basics
\nWhen a virtual machine image scales, rsync issues scale with it. A size of 1.9 TiB isn't that large for a modern virtual machine image\u2014but it's large enough that ZFS replication turns out to be 1148 times faster than rsync, even with the rsync --inplace argument.<\/i><\/p>\n

Once you understand how snapshots work, grasping the essence of replication becomes straightforward. Since a snapshot is just a tree of pointers to records, it follows that when we create a zfs send<\/code> snapshot, we send both this tree and all related records. When we pass this zfs send<\/code> downward API support (simultaneously with this in zfs receive<\/code> to the target object, it writes both the actual block contents and the tree of pointers referencing the blocks into the target dataset.<\/p>\n

Things get even more interesting on the second zfs send<\/code>. Now we have two systems, each containing poolname\/datasetname@1<\/code>, and you take a new snapshot poolname\/datasetname@2<\/code>. Thus, in the source pool, you have datasetname@1<\/code> and datasetname@2<\/code>, while the target pool only has the first snapshot for now. datasetname@1<\/code>.<\/p>\n

Since we have a shared snapshot between the source and the target, we can create an datasetname@1<\/code>incremental on top of it. When we tell the system<\/i> zfs send<\/code> zfs send -i poolname\/datasetname@1 poolname\/datasetname@2 , it compares the two trees of pointers. Any pointers that exist only in<\/code>, obviously point to new blocks \u2014 hence we will need the content of those blocks. @2<\/code>On the remote system, processing the incremental<\/p>\n

is equally simple. First, we write down all the new records included in the stream send<\/code> , and then we add pointers to these blocks. Voila, we have send<\/code>in the new system! @2<\/code> Asynchronous incremental replication with ZFS is a massive improvement over earlier methods that did not use snapshots, such as rsync. In both cases, only changed data is transferred \u2014 but rsync has to gather data from both sides to check the checksum and compare it first. In contrast, ZFS replication reads nothing except the trees of pointers \u2014 and any blocks that are not represented in the shared snapshot.<\/p>\n

Asynchronous incremental ZFS replication is a huge improvement over earlier methods not based on snapshots, such as rsync. In both cases, only the changed data is transferred\u2014but rsync has to read from disk on both sides to check the checksum and compare it. read<\/i> In contrast, ZFS replication reads nothing but the pointer trees\u2014and any blocks not represented in the common snapshot.<\/p>\n

Built-in Compression<\/h3>\n

\nThe copy-on-write mechanism also simplifies the built-in compression system. In traditional file systems, compression is problematic\u2014both the old and new versions of modified data reside in the same space.<\/p>\n

If we consider a fragment of data in the middle of a file that starts its life as a megabyte of zeros from 0x00000000 onward, it can be easily compressed down to one sector on disk. But what happens if we replace that megabyte of zeros with a megabyte of incompressible data, such as JPEG or pseudorandom noise? Surprisingly, this megabyte of data would require not one, but 256 sectors of 4 KiB each, while only one sector is reserved at that point on the disk.<\/p>\n

ZFS does not have this problem, as modified records are always written to unused space\u2014the original block occupies only one 4 KiB sector, while the new record will take 256, but this is not an issue\u2014the recently modified fragment from the 'middle' of the file would be written to unused space regardless of whether its size changed or not, so this is a normal situation for ZFS.<\/p>\n

Built-in compression in ZFS is disabled by default, and the system offers pluggable algorithms\u2014currently including LZ4, gzip (1-9), LZJB, and ZLE.<\/p>\n

If the file is large enough and requires multiple blocks, then all records with this file will have a size recordsize<\/code>\u00a0\u2014 including the last record, the main part of which may turn out to be unused space.<\/a><\/noindex>.<\/p>\n