Meta* has introduced the OpenZL toolkit for data compression and decompression, demonstrating a higher compression level and speed compared to Zstd and XZ formats. OpenZL is designed for efficiently compressing structured datasets, such as those used in machine learning, as well as storage containing fields with various repetitive types of information. OpenZL is written in C/C++ and is open-sourced under the BSD license.
When compressing the database with the astronomical star catalog SAO, the OpenZL toolkit reduced the data size by 2.06 times, while the zstd algorithm compressed the information by 1.31 times, and XZ by 1.64 times. In terms of compression speed, OpenZL outperformed zstd by two times (203 MB/s vs. 115 MB/s) and XZ by 65 times (203 MB/s vs. 3.1 MB/s). The decompression speed in OpenZL was slightly slower than zstd (822 MB/s vs. 890 MB/s) but 27 times faster than XZ.
OpenZL is not a general-purpose algorithm and performs well only for data with a predefined structure. The operation of OpenZL involves adaptive generation of a packer based on the provided data description. As a result, code is generated for compression, optimized for a specific data format. A universal decompressor compatible with all generated packers is used for decompression.
Packaging and unpacking are carried out using the single utility "zli" or the libopenzl library. The data structure is described in the form of profiles. It already includes a set of predefined profiles that describe typical storage formats, such as a profile for the CSV format or data stored as an array of 64-bit numbers. Compression involves selecting a profile with the command "zli list-profiles" and initiating the packaging process with the command "zli compress —profile profile_name". To decompress, simply run "zli decompress".
For specific formats, you need to create your own profile using the command 'zli train', which identifies patterns in the data and generates a profile with an optimal compression level. Using the '—pareto-frontier' option, the created profile can be enhanced for faster packing or unpacking at the cost of reduced compression levels. For complex formats with nested structures and to define the arrangement of data formats in structures, the SDDL (Simple Data Description Language) may be used.
The method for creating optimal packers is based on the use of a set of primitive encoders, each of which is most effective for specific types and sequences of data. To compress, a directed acyclic graph of data processing is formed, where the nodes are codecs and the edges are data variations in the processed format. Depending on the type of data input, a sequence of codecs is selected that optimally compresses the incoming data element. In this organization, the file header is compressed by one codec, the integer data field by another, the field with an increasing counter by a third, and the field with string data by a fourth.
Source: opennet.ru
