When dealing with large volumes of data, storage space issues can often arise. One way to address this problem is through compression, which allows for increased storage capacity on the same hardware. In this article, we will explore how data compression works in Apache Ignite. We will focus solely on the methods of disk compression implemented within the product. Other forms of data compression (network, in-memory), whether implemented or not, will not be covered.
So, with persistence mode enabled, when data changes occur in the caches, Ignite begins writing to disk:
- Cache contents
- Write Ahead Log (WAL)
There has long been a mechanism for compressing WAL, known as WAL compaction. In the recently released Apache Ignite 2.8, there are two additional mechanisms to compress data on disk: disk page compression for compressing cache contents and WAL page snapshot compression for compressing certain WAL entries. More details about these three mechanisms below.
Disk Page Compression
How it works
To begin, let’s briefly discuss how Ignite stores data. It uses page-based memory for storage. The page size is set when the node starts and cannot be changed at later stages, and it must also be a power of two and a multiple of the file system's block size. Pages are loaded into RAM from disk as needed, and the amount of data on disk can exceed the allocated RAM. If there is insufficient RAM to load a page from disk, older, unused pages will be evicted from RAM.
Data on disk is stored as follows: for each partition of each cache group, a separate file is created, with pages stored sequentially in increasing order of their index within that file. The complete page identifier consists of the cache group identifier, partition number, and page index within the file. Thus, from the complete page identifier, we can unambiguously determine the file and offset within the file for each page. More detailed information about the structure of page memory can be found in the article on Apache Ignite Wiki: .
The disk page compression mechanism, as the name suggests, operates on a page level. When this mechanism is enabled, operations on data in RAM are carried out as is, without any compression, but at the moment of saving pages from RAM to disk, compression is applied.
However, compressing each page individually does not solve the issue; we need to find a way to reduce the size of the final data files. If the size of a page is no longer fixed, we can no longer write pages to a file one after another, as this could lead to a series of problems:
- We will not be able to calculate the offset of a page's position in the file using the page index.
- It’s unclear what to do with pages that are not at the end of the file and are changing size. If the size of a page decreases, the space it freed is lost. If the size of a page increases, we will need to find a new location in the file for it.
- If a page is shifted by a number of bytes that is not a multiple of the file system block size, reading or writing it will require accessing one additional file system block, which may lead to performance degradation.
To avoid dealing with these problems on its own, disk page compression in Apache Ignite utilizes a file system mechanism called sparse files. A sparse file is one in which some regions filled with zeros can be marked as 'holes'. As a result, no file system blocks are allocated for storing these holes, thus saving disk space.
Logically, to free a file system block, the size of the hole must be greater than or equal to the file system block size, which imposes an additional restriction on the page size in Apache Ignite: for compression to be effective, the page size must be strictly greater than the file system block size. If the page size equals the block size, we will never be able to free any blocks, since to free a single block, the compressed page needs to occupy 0 bytes. If the page size is equal to the size of 2 or 4 blocks, we can free at least one block if our page compresses to at least 50% or 75%, respectively.
Thus, the final description of the mechanism's operation: When saving a page to disk, an attempt is made to compress the page. If the size of the compressed page allows for one or more file system blocks to be freed, the page is saved in compressed form, creating a "hole" where the freed blocks were (a system call is executed fallocate() with the "punch hole" flag). If the size of the compressed page does not allow for blocks to be freed, the page is saved as is, in uncompressed form. All page offsets are calculated the same way as without compression, by multiplying the page index by the page size. No relocation of pages is required. Page offsets, like in the case without compression, align with the boundaries of file system blocks.
In the current implementation, Ignite can work with sparse files only on Linux OS, therefore disk page compression can only be enabled when using Ignite on this operating system.
Compression algorithms that can be used for disk page compression include: ZSTD, LZ4, Snappy. In addition, there is a mode of operation (SKIP_GARBAGE), where only the unused space in the page is discarded without applying compression to the remaining data, which reduces CPU load compared to the previously mentioned algorithms.
Impact on performance
Unfortunately, I did not conduct actual performance measurements on real setups, as we do not plan to use this mechanism in production, but we can theoretically discuss where we might lose and where we might gain.
To do this, we need to recall how the reading and writing of pages occurs when accessing them:
- When performing a read operation, the system first searches for it in RAM; if the search is unsuccessful, the page is loaded into RAM from disk by the same thread that performs the reading.
- When writing, the page in RAM is marked as dirty; however, the physical saving of the page to disk does not occur immediately in the thread performing the write. All dirty pages are saved to disk later during the checkpointing process by separate threads.
Thus, the impact on read operations is:
- Positive (disk IO), due to the reduction in the number of file system blocks read.
- Negative (CPU) due to the additional load required by the operating system to work with sparse files. Additionally, there may implicitly be extra I/O operations required to save a more complex structure of the sparse file (I am unfortunately not familiar with all the details of sparse file operation).
- Negative (CPU) due to the need for page decompression.
- No impact on write operations.
- Impact on the checkpoint process (similar to read operations):
- Positive (disk I/O) due to the reduction in the number of blocks written to the filesystem.
- Negative (CPU, possibly disk I/O) due to working with sparse files.
- Negative (CPU) due to the need for page compression.
Which side of the scale will tip? This largely depends on the environment, but I tend to believe that disk page compression will likely lead to performance degradation on most systems. Furthermore, tests on other DBMSs utilizing similar approaches with sparse files indicate performance drops with compression enabled.
How to enable and configure
As mentioned above, the minimum version of Apache Ignite that supports disk page compression is 2.8 and it is only supported on the Linux operating system. Enabling and configuring is done as follows:
- The class-path must include the ignite-compression module. By default, it is located in the Apache Ignite distribution in the libs/optional directory and is not included in the class-path. You can simply move the directory one level up into libs and then it will be automatically included when starting via ignite.sh.
- Persistence must be enabled (enabled through
DataRegionConfiguration.setPersistenceEnabled(true)). - The page size must be greater than the filesystem block size (this can be set using
DataStorageConfiguration.setPageSize()). - For each cache whose data needs to be compressed, the configuration must be set to define the compression method and (optionally) the compression level (methods
CacheConfiguration.setDiskPageCompression(), CacheConfiguration.setDiskPageCompressionLevel()).
WAL compaction
How it works
What is WAL and why is it needed? In short, it is a log that captures all events that ultimately change the page storage. It is primarily needed for the ability to recover in case of failure. Any operation must first log the event in the WAL before relinquishing control to the user, ensuring that in the event of a crash, it can replay the log and restore all operations for which the user received a successful response, even if those operations did not yet reflect in the page storage on disk (as described earlier, the actual write to the page storage is performed in a process known as 'checkpointing' with some delay by separate threads).
Entries in the WAL are divided into logical and physical. Logical entries are the keys and values themselves. Physical entries reflect changes to the pages in the page storage. While logical entries may still be useful for some cases, physical entries are only needed for recovery in case of failure and are required only from the last successful checkpoint. We will not delve into the details here or explain why this works this way, but those interested can refer to the previously mentioned article on the Apache Ignite Wiki: .
Often, one logical entry corresponds to several physical entries. For example, one put operation in the cache affects several pages in the page memory (the page with the actual data, pages with indices, pages with free lists). In some synthetic tests, I found that physical entries accounted for up to 90% of the WAL file's size. They are required for only a brief period (by default, the interval between checkpoints is 3 minutes). It would make sense to discard this data once they lose their relevance. This is precisely what the WAL compaction mechanism does: it removes physical entries and compresses the remaining logical entries using zip, significantly reducing the file size (sometimes by factors of ten).
The physical WAL consists of several segments (default is 10) of a fixed size (default is 64MB), which are overwritten in a circular manner. Once the current segment is filled, it is assigned the next segment in line, and the filled segment is archived in a separate thread. WAL compaction already works with archived segments. It also tracks the completion of the checkpoint in a separate thread and begins compression on the archived segments for which physical records are no longer needed.
Impact on performance
Since WAL compaction operates in a separate thread, it should not directly affect the operations being performed. However, it does impose additional background load on the CPU (compression) and disk (reading each WAL segment from the archive and writing the compressed segments), so if the system is operating at its limits, it will also lead to performance degradation.
How to enable and configure
WAL compaction can be enabled using the property WalCompactionEnabled downward API support (simultaneously with this in DataStorageConfiguration (DataStorageConfiguration.setWalCompactionEnabled(true)). Additionally, with the DataStorageConfiguration.setWalCompactionLevel() method, you can specify the level of compression if you are not satisfied with the default value (BEST_SPEED).
WAL page snapshot compression
How it works
It was previously established that WAL records are divided into logical and physical types. For every change made to each page in memory, a physical WAL record is created. Physical records are further divided into two subtypes: page snapshot record and delta record. Each time we change something on a page and transition it from a clean to a dirty state, a complete copy of that page (page snapshot record) is saved in WAL. Even if we change just one byte, the WAL will store a record slightly larger than the page size. If we change something on an already dirty page, a delta record is formed in WAL, reflecting only the changes compared to the previous state of the page, not the entire page. Since the transition from dirty to clean state occurs during a checkpoint, right after the checkpoint begins, almost all physical records will consist solely of page snapshots (since all pages are clean right after the checkpoint starts); as we approach the next checkpoint, the proportion of delta records starts to increase and resets again at the beginning of the next checkpoint. Measurements from some synthetic tests showed that the proportion of page snapshots in the total volume of physical records reaches up to 90%.
The idea of WAL page snapshot compression involves compressing page snapshots using an existing tool for page compression (see disk page compression). Here, WAL records are stored sequentially in an append-only mode, with no need to bind records to the boundaries of filesystem blocks. Therefore, unlike the disk page compression mechanism, we do not require sparse files, which means this mechanism will work not only on Linux OS. Moreover, we no longer care how much we have managed to compress a page. Even if we’ve freed just 1 byte, it’s already a positive outcome, and we can store compressed data in WAL, unlike disk page compression, where we only save a compressed page if we have freed more than 1 filesystem block.
Pages are highly compressible data, and their share in the overall volume of WAL is very high. Thus, by not changing the format of the WAL file, we can achieve a significant reduction in its size. Compressing logical records would require a change in format and loss of compatibility, for example, for external consumers who might be interested in logical records, without providing a significant reduction in file size.
Similar to disk page compression, compression algorithms such as ZSTD, LZ4, Snappy, and the SKIP_GARBAGE mode can be used for WAL page snapshot compression.
Impact on performance
As can be seen, directly enabling WAL page snapshot compression only affects the streams that write data to page memory, meaning those streams that modify data in caches. Reading from WAL physical records occurs only once, when the node is brought up after a failure (and only in the case of failure during a checkpoint process).
For streams that modify data, this affects them as follows: we experience a negative effect (CPU) due to the necessity to compress the page before writing to disk each time, and a positive effect (disk IO) due to the reduced amount of data written. Accordingly, it’s straightforward: if the system's performance is limited by CPU, we get slight degradation; if it’s limited by disk input/output, we achieve an improvement.
Indirectly, the reduction in WAL size also positively impacts threads that archive WAL segments and WAL compaction threads.
Performance tests in our environment with synthetic data showed a slight increase (throughput grew by 10%-15%, latency decreased by 10%-15%).
How to enable and configure
Minimum version of Apache Ignite: 2.8. Enabling and configuring is done as follows:
- The class-path must include the ignite-compression module. By default, it is located in the Apache Ignite distribution in the libs/optional directory and is not included in the class-path. You can simply move the directory one level up into libs and then it will be automatically included when starting via ignite.sh.
- Persistence must be enabled (enabled through
DataRegionConfiguration.setPersistenceEnabled(true)). - The compression mode must be set using the method
DataStorageConfiguration.setWalPageCompression(), by default compression is disabled (DISABLED mode). - Optionally, the degree of compression can be set using the method
DataStorageConfiguration.setWalPageCompression(), acceptable values for each mode can be found in the javadoc for the method.
Conclusion
The data compression mechanisms reviewed in Apache Ignite can be used independently of one another, but any combinations of them are also permissible. Understanding the principles of their operation will help determine how well they fit your tasks in your environment and what compromises may be required when using them. Disk page compression is intended for compressing the primary storage and can provide a moderate level of compression. WAL page snapshot compression will offer a moderate level of compression for the WAL files, and is likely to even improve performance. WAL compaction will not positively affect performance, but will significantly reduce the size of WAL files by removing physical records.
Source: habr.com
