The Pack project attempted to create a file archiving format based on the SQLite library and the ZSTD (Zstandard) compression algorithm. The prepared prototype, written in Pascal and distributed under the Apache 2.0 license, outpaced the most common archivers in terms of archive creation speed, as its operations involved reading data, compressing with the libzstd library, and performing SQL operations to add compressed data to a file with the SQLite database.
When compressing a directory with 81,000 files, totaling 1.25 GB, Pack proved to be 112 times faster than the ZIP utility, completing the operation in 1.3 seconds compared to 146 seconds for ZIP. The archive size was also 23% smaller with Pack (194 MB for Pack and 253 MB for ZIP). For comparison, the tar utility took 4.7 seconds for packaging without compression and 28.5 seconds with gzip compression, the RAR archiver completed the test in 27.5 seconds, and 7z took 54.2 seconds. The sizes of the archives were: tar.gz — 214 MB, RAR — 235 MB, 7z — 135 MB. It is noted that in terms of extraction speed and random file access, Pack also outperforms other archivers while consuming less RAM. ZIP: 253 MB, 146 s 7z: 135 MB, 54.2 s faster than ZIP by 2.7 times tar.gz: 214 MB, 28.5 s x 5.1 RAR: 235 MB, 27.5 s x 5.3 tar: 1345 MB, 4.7 s x 31 Pack: 194 MB, 1.3 s x 112
The influence of the file cache on the test results is not mentioned. It is likely that the low speed of ZIP is due to the order of test execution without regard to data caching in memory—the zip test was run with a cold cache, while the other tests were conducted with a warm cache. Under normal conditions, Zstandard shows 3-5 times higher compression speed compared to zlib and twice as fast decompression, with a compression level 10-15% higher.
Addition: A similar idea of storing compressed files as blobs in an SQLite database was implemented in 2014 in the sqlar archiver, created by SQLite developers as an experiment to evaluate the effectiveness of storing blobs in SQLite. In sqlar, zlib is used for compression, and the file sizes are approximately 2% larger than those generated by the ZIP utility.
Source: opennet.ru
