After seven months of development, version 0.2.0 of the framework has been released. OpenZL, designed for creating lossless data compressors.
The framework consists of a core library and tools for creating specialized compressors, described in the language SDDL.
To create a good specialized compressor, there are two phases:
- Data analysis to extract the structure.
- Using good backend compressors that utilize the obtained structure to achieve effective compression.
OpenZL provides tools for both phases.
The project is written in C and C++ and is distributed under the BSD license.
Key changes
SDDL2
SDDL has been completely redesigned from the ground up to achieve the stated design goals. While the initial demo version provided a simplified runtime environment, SDDL2 is a full-fledged compiler: the parser feeds data to the semantic analyzer, which in turn passes a typed abstract syntax tree (AST) to the optimizer, while the optimizer manages the code generator that produces bytecode for the virtual machine.
The key result is instant parsing. When the location of a record can be fully determined by parameters and constants, the engine jumps directly to any field without scanning previous bytes, enabling access without copying and throughput of several GB/s.
The language itself has evolved alongside the toolkit. It now supports when blocks for conditional placements, parameterized and anonymous records, access to record field members, as well as bitwise and logical operators.
As for developer convenience, the semantic analysis phase now identifies ambiguous references, type mismatches, and arity errors at compile time—with location indications in the source code—rather than at runtime, and a VS Code extension has been released for syntax highlighting of .sddl files.
The new built-in LZ codec
OpenZL now includes its own LZ codec, presented as ZL_GRAPH_LZ, as well as a sequential compression profile in the zli utility. Development on the codec continues: the feature set is being expanded and the performance for processing small input data is being improved. Currently, it supports functionality equivalent to zstd level 1, with a compression window size of 64 KB.
OpenZL allows for the redesign of each stage of the LZ pipeline to enhance speed. Its graph architecture also enables the combination of entropy coding stages instead of using a single pipeline that works equally well for all use cases. Multiple stages can then be combined into a single operation for increased processing speed. This allows OpenZL to achieve 10% higher compression speed and 70% higher decompression speed compared to Zstandard level 1 on the Silesia corpus in our tests:
| Compressor | Compression Ratio | Compression Speed | Decompression Speed |
|---|---|---|---|
| OpenZL LZ level 1 | 2.74 | 466 MB/s | 2288 MB/s |
| Zstd level 1 with 64K window size | 2.74 | 419 MB/s | 1254 MB/s |
| Zstd level 1 | 2.89 | 424 MB/s | 1345 MB/s |
Support for very large input data
zli now supports processing enormous input data (in several gigabytes). Such data is now automatically split into manageable-sized chunks (defaulting to about 16MB) before compression, which helps limit memory usage, improve data locality, and opens up opportunities for parallel processing. A similar automatic chunking function is implemented in SDDL2 when working under the schema. New or updated segmenters have been created for CSV, Parquet, and standard numeric data, and all segmenters are now serializable and configurable, allowing the selected layout to be saved in the compressor and reused later.
This is transparently applied during compression. Note that the learning pipeline is different and remains untouched, so it is not intended to handle gigantic input data as training material.
Improvements in the online graph visualizer (try)
Now the visualizer recognizes compression and decompression traces from start to finish.
The stream preview panel allows you to view the bytes actually passing through each edge, and thanks to the trimming controls, even large streams remain manageable.
The settings panel combines all display options in one place, and the complete set of keyboard shortcuts – navigation directions, ordered traversal, expanding and collapsing, node selection – allows you to work with the tool conveniently without a mouse.
Now the traces have versions, block-compressed rendering is displayed correctly, and zli can finally generate its own traces using the new flags --trace and --trace-streams-dir.
Miscellaneous
- Several codecs have been added to the directory. The Partition and bitpack codecs now use a combined decoder. The Floating-point bitsplit codec has received special encoders and decoders for the fp16, fp32, fp64, and bf16 formats with specialized accelerations. Range-aware splitting (split_byrange), length multiplexer, sentinel codec, lz4 graph, and small helper functions like tryParseInt and splitByParam have been added.
- API has been organized.
- Fuzzing testing has been improved.
- The build and packaging process has been enhanced for a greater number of platforms.
Source: linux.org.ru
