I-OpenZL 0.2.0

Emva kweenyanga ezisixhenxe zophuhliso, inguqulelo 0.2.0 yesakhelo yakhululwa. I-OpenZL, yenzelwe ukudala ii-compressor zedatha ezingenalahleko.

Esi sikhokelo siqulathe ithala leencwadi elisisiseko kunye nezixhobo zokwenza ii-compressor ezikhethekileyo ezichazwe ngolwimi. SDDL.
Kukho amanyathelo amabini okudala i-compressor elungileyo:

1. Uhlalutyo lwedatha ukuze kukhutshwe isakhiwo.
2. Ukusebenzisa ii-backend compressors ezilungileyo ezisebenzisa ulwakhiwo oluvelayo ukuze kufezekiswe ucinezelo olufanelekileyo.

I-OpenZL ibonelela ngezixhobo kuzo zombini izigaba.

Le projekthi ibhalwe ngo-C no-C++ kwaye isasazwa phantsi kwelayisenisi ye-BSD.

Utshintsho olukhulu

I-SDDL2

I-SDDL yabhalwa ngokutsha ngokupheleleyo ukususela ekuqaleni ukuze ifezekise iinjongo zayo zoyilo lokuqala. Nangona idemo yokuqala yayiyindawo yokusebenza elula, i-SDDL2 yi-compiler epheleleyo: i-parser idlulisela idatha kwi-semantic analyzer, ethi yona idlulise umthi we-syntax ochwetheziweyo (AST) kwi-optimizer, kwaye i-optimizer ilawula i-code generator, evelisa i-bytecode yomatshini obonakalayo.

Isiphumo esiphambili kukuhlalutya ngoko nangoko. Xa indawo yerekhodi inokuqinisekiswa ngokupheleleyo kusetyenziswa iiparameter kunye nee-constants zodwa, injini itsiba ngqo kuyo nayiphi na intsimi ngaphandle kokuskena ii-byte zangaphambili, ivumela ukufikelela ngaphandle kokukopa kunye nokuphuma kwee-GB/s ezininzi.

Ulwimi ngokwalo luye lwavela kunye nezixhobo zalo. Ngoku luxhasa amabinzana athi xa kuthethwa ngezitatimende ezinemiqathango, iirekhodi ezibekwe ngokweparameter nezingaziwayo, ukufikelela kumalungu entsimi yerekhodi, kunye nabaqhubi be-bitwise kunye ne-logical.

Kwicala lomphuhlisi, inyathelo lohlalutyo lwesemantic ngoku lichonga iireferensi ezingachazwanga, ukungafani kohlobo, kunye neempazamo ze-arity ngexesha lokuqokelela-nendawo yekhowudi yomthombo-kunokuba ngexesha lokusebenza, kwaye ulwandiso lwe-VS Code lokuqaqambisa i-syntax yeefayile ze-.sddl lukhutshiwe.

Ikhowudi entsha ye-LZ eyakhelwe ngaphakathi

I-OpenZL ngoku iquka i-LZ codec yayo, emelwe yi-ZL_GRAPH_LZ, kunye neprofayili yoxinzelelo olulandelelanayo kwisixhobo se-zli. Umsebenzi kwi-codec uyaqhubeka, wandisa iseti yeempawu zayo kwaye uphucula ukusebenza xa kusetyenzwa idatha encinci yokufaka. Okwangoku, ixhasa ukusebenza okulingana nenqanaba loku-1 le-zstd, kunye nefestile yoxinzelelo lwe-64 KB.

I-OpenZL ivumela isigaba ngasinye sombhobho we-LZ ukuba sihlengahlengiswe ngokwesantya. Uyilo lwayo lwegrafu lukwavumela ukudibanisa izigaba zokufaka ikhowudi ze-entropy, endaweni yokusebenzisa umbhobho omnye ofanelekileyo ngokulinganayo kuzo zonke iimeko zokusetyenziswa. Izigaba ezininzi zinokudityaniswa zibe ngumsebenzi omnye ukuphucula isantya sokucubungula. Oku kuvumela i-OpenZL ukuba ifezekise ucinezelo olukhawulezileyo lwe-10% kunye nokunciphisa ucinezelo olukhawulezileyo lwe-70% xa kuthelekiswa nenqanaba le-Zstandard 1 kwi-Silesia corpus. iimvavanyo zethu:

Inkxaso yedatha enkulu kakhulu yokufaka

I-zli ngoku ixhasa ukucutshungulwa kwedatha enkulu yokufaka (iigigabytes ezininzi ngobukhulu). Ngaphambi kokucinezelwa, idatha enjalo ngoku yahlulwe ngokuzenzekelayo ibe ziinxalenye ezinobukhulu obulawulekayo (malunga ne-16 MB ngokuzenzekelayo), inciphisa ukusetyenziswa kwememori, iphucula indawo yedatha, kwaye ivumela ukucutshungulwa ngaxeshanye. I-SDDL2 isebenzisa uphawu olufanayo lokucutshungulwa ngokuzenzekelayo xa isebenza nge-schema. Ii-segmenters ezintsha zenziwe okanye zahlaziywa kule nkqubo—ye-CSV, iParquet, kunye nedatha yenombolo eqhelekileyo—kwaye zonke ii-segmenters ngoku zinokulandelelwaniswa kwaye zinokucwangciswa, ngoko ke uyilo olukhethiweyo lunokugcinwa kwi-compressor kwaye lusetyenziswe kwakhona kamva.

Oku kusetyenziswa ngokucacileyo ngexesha lokucinezelwa. Qaphela ukuba umbhobho woqeqesho wahlukile kwaye awuchaphazeleki, ngoko awudalelwanga ukwamkela idatha enkulu yokufaka njengezixhobo zoqeqesho.

Uphuculo kwi-visualizer yegrafu ekwi-intanethi (ukuzama)
Isibonisi ngoku siyabona imikhondo yokucinezela kunye nokunciphisa ucinezelo ukususela ekuqaleni ukuya ekugqibeleni.

Iphaneli yokujonga kwangaphambili ukusasazwa kwemithombo ikuvumela ukuba ubone ii-byte zihamba ngqo kumda ngamnye, kwaye ulawulo lokunciphisa lugcina imijelo emikhulu kulula ukusebenza ngayo.

Iphaneli yezicwangciso idibanisa zonke iindlela zokubonisa kwindawo enye, kwaye iseti epheleleyo yamaqhosha ashushu—ukuhamba kwicala, ukunqumla okucwangcisiweyo, ukwandisa nokudilika, kunye nokukhetha amaqhuqhuva—ikuvumela ukuba usebenze ngokulula nesixhobo ngaphandle kwemawusi.

Iitrace ngoku ziguqulelwe, ucinezelo olusekelwe kwiibhloko luboniswa ngokuchanekileyo, kwaye i-zli ekugqibeleni inokuvelisa iitrace zayo isebenzisa iiflegi ezintsha ze--trace kunye ne--trace-streams-dir.

Разное

• Iikhowudi ezininzi zongezwe kwikhathalogu. Iikhowudi zePartition kunye ne-bitpack ngoku zisebenzisa i-united decoder. I-floating-point bitsplit codec ngoku iquka ii-encoders kunye nee-decoders ezizinikeleyo zefomathi ye-fp16, fp32, fp64, kunye ne-bf16 ene-acceleration ekhethekileyo. Ukwahlulahlula okuqaphelayo (split_byrange), i-length multiplexer, i-sentinel codec, i-lz4 graph, kunye nemisebenzi emincinci yomncedisi efana ne-tryParseInt kunye ne-splitByParam zongeziwe.
• I-API yenziwe lula.
• Uvavanyo lwe-fuzz oluphuculweyo.
• Inkqubo yokwakha nokupakisha ephuculweyo yamaqonga amaninzi.

umthombo: linux.org.ru