{"id":114563,"date":"2024-03-25T18:26:02","date_gmt":"2024-03-25T16:26:02","guid":{"rendered":"https:\/\/prohoster.info\/blog\/novosti-interneta\/eksperiment-s-ispolzovaniem-sqlite-v-kachestve-kontejnera-dlya-arhivirovaniya-fajlov"},"modified":"2024-03-25T18:26:02","modified_gmt":"2024-03-25T16:26:02","slug":"eksperiment-s-ispolzovaniem-sqlite-v-kachestve-kontejnera-dlya-arhivirovaniya-fajlov","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/eksperiment-s-ispolzovaniem-sqlite-v-kachestve-kontejnera-dlya-arhivirovaniya-fajlov","title":{"rendered":"An experiment using SQLite as a container for archiving files","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"
The Pack project attempted to create a file archiving format based on the SQLite library and the ZSTD (Zstandard) compression algorithm. The prepared prototype, written in Pascal and distributed under the Apache 2.0 license, outpaced the most common archivers in terms of archive creation speed, as its operations involved reading data, compressing with the libzstd library, and performing SQL operations to add compressed data to a file with the SQLite database. <\/p>\n
When compressing a directory with 81,000 files totaling 1.25 GB, pack proved to be 112 times faster than the ZIP utility, completing the operation in 1.3 seconds compared to 146 seconds for ZIP. The archive size with pack was 23% smaller (194 MB for Pack and 253 MB for ZIP). In comparison, the tar utility packaged without compression in 4.7 seconds and with gzip compression in 28.5 seconds, the RAR archiver completed the test in 27.5 seconds, and 7z took 54.2 seconds. The sizes of the archives were: tar.gz \u2014 214 MB, RAR \u2014 235 MB, 7z \u2014 135 MB. It is noted that regarding decompression speed and random access to files, Pack also outperforms other archivers while consuming less memory.\n ZIP: 253 MB, 146 s 7z: 135 MB, 54.2 s 2.7 times faster than ZIP tar.gz: 214 MB, 28.5 s x 5.1 RAR: 235 MB, 27.5 s x 5.3 tar: 1345 MB, 4.7 s x 31 Pack: 194 MB, 1.3 s x 112 <\/p>\n
The influence of the file cache on the test results is not mentioned. It is likely that the low speed of ZIP is due to the order of test execution without consideration for data caching in memory \u2014 the test with zip was run with a cold cache, while the others were run with a warm cache. Under normal conditions, Zstandard shows 3-5 times higher compression speed compared to zlib and performs decompression twice as fast, with a compression level that is 10-15% higher. <\/p>\n
Addition: A similar idea of storing compressed files as blobs in an SQLite database was implemented in 2014 in the sqlar archiver, created by SQLite developers as an experiment to evaluate the effectiveness of storing blobs in SQLite. In sqlar, zlib is used for compression, and the file sizes are approximately 2% larger than those generated by the ZIP utility. \n Source: opennet.ru<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"