{"id":170618,"date":"2026-05-11T18:24:52","date_gmt":"2026-05-11T16:24:53","guid":{"rendered":"https:\/\/prohoster.info\/blog\/novosti-interneta\/openzl-0-2-0"},"modified":"2026-05-11T18:24:52","modified_gmt":"2026-05-11T16:24:53","slug":"openzl-0-2-0","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/openzl-0-2-0","title":{"rendered":"OpenZL 0.2.0","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"OpenZL <\/a> \"OpenZL <\/a> <\/a><\/noindex> <\/a> <\/p>\n

After seven months of development, version 0.2.0 of the framework has been released. OpenZL<\/a>, designed for creating lossless data compressors.<\/p>\n

<\/noindex> <\/p>\n

The framework consists of a core library and tools for creating specialized compressors, described in the language SDDL<\/a>.
To create a good specialized compressor, there are two phases:<\/p>\n

    \n
  1. Data analysis to extract the structure.<\/li>\n
  2. Using good backend compressors that utilize the obtained structure to achieve effective compression.<\/li>\n<\/ol>\n

    OpenZL provides tools for both phases.<\/p>\n

    The project is written in C and C++ and is distributed under the BSD license.<\/p>\n

    Key changes<\/h5>\n

    SDDL2<\/strong><\/p>\n

    SDDL has been completely redesigned from the ground up to achieve its intended design goals. While the initial demo version was a simplified execution environment, SDDL2 is a full-fledged compiler: the parser passes data to the semantic analyzer, which in turn passes the typed abstract syntax tree (AST) to the optimizer, and the optimizer manages the code generator, which generates bytecode for the virtual machine.<\/p>\n

    The key result is instant parsing. When the location of a record can be fully determined by parameters and constants, the engine jumps directly to any field without scanning previous bytes, enabling access without copying and throughput of several GB\/s.<\/p>\n

    The language itself has evolved alongside the toolkit. It now supports when blocks for conditional placements, parameterized and anonymous records, access to record field members, as well as bitwise and logical operators.<\/p>\n

    In terms of developer convenience, the semantic analysis stage now detects undefined references, type mismatches, and arity errors at compile time\u2014with location indicators in the source code\u2014rather than at runtime. Additionally, a VS Code extension for syntax highlighting of .sddl files has been released.<\/p>\n

    The new built-in LZ codec<\/strong><\/p>\n

    OpenZL now includes its own LZ codec, presented as ZL_GRAPH_LZ, as well as a sequential compression profile in the zli utility. Development on the codec continues: the feature set is being expanded and the performance for processing small input data is being improved. Currently, it supports functionality equivalent to zstd level 1, with a compression window size of 64 KB.<\/p>\n

    <\/p>\n

    OpenZL allows for the redesign of each stage of the LZ pipeline to enhance speed. Its graph architecture also enables the combination of entropy coding stages instead of using a single pipeline that works equally well for all use cases. Multiple stages can then be combined into a single operation for increased processing speed. This allows OpenZL to achieve 10% higher compression speed and 70% higher decompression speed compared to Zstandard level 1 on the Silesia corpus in our tests<\/a>:<\/p>\n

    <\/noindex> <\/p>\n\n\n\n\n\n\n\n
    Compressor<\/strong><\/th>\nCompression Ratio<\/strong><\/th>\nCompression Speed<\/strong><\/th>\nDecompression Speed<\/strong><\/th>\n<\/tr>\n<\/thead>\n
    OpenZL LZ level 1<\/td>\n2.74<\/td>\n466 MB\/s<\/td>\n2288 MB\/s<\/td>\n<\/tr>\n
    Zstd level 1 with 64K window size<\/td>\n2.74<\/td>\n419 MB\/s<\/td>\n1254 MB\/s<\/td>\n<\/tr>\n
    Zstd level 1<\/td>\n2.89<\/td>\n424 MB\/s<\/td>\n1345 MB\/s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Support for very large input data<\/strong><\/p>\n

    zli now supports processing massive input data (in the gigabyte range). Before compression, such data is now automatically split into managed-size chunks (defaulting to about 16MB), limiting memory usage, enhancing data locality, and allowing for parallel processing. SDDL2 implements a similar automatic fragmentation feature when working with the framework. During this process, new or updated segmenters have been created for CSV, Parquet, and standard numeric data, all of which are now serializable and configurable, allowing the chosen layout to be saved in the compressor and reused later.<\/p>\n

    This is transparently applied during compression. Note that the learning pipeline is different and remains untouched, so it is not intended to handle gigantic input data as training material.<\/p>\n

    Improvements in the online graph visualizer (try<\/a>)<\/strong>
    Now the visualizer recognizes compression and decompression traces from start to finish.<\/p>\n

    The stream preview panel allows you to view the bytes actually passing through each edge, and thanks to the trimming controls, even large streams remain manageable.<\/p>\n

    The settings panel combines all display options in one place, and the complete set of keyboard shortcuts\u2014navigation through directions, ordered traversal, expanding and collapsing, node selection\u2014allows for convenient operation of the tool without a mouse.<\/p>\n

    Tracing now has versions, block-compressed tracing is displayed correctly, and zli can finally generate its own traces using the new flags --trace and --trace-streams-dir.<\/p>\n

    Miscellaneous<\/strong><\/p>\n