Nkhaniyi ndi yachiwiri pamndandanda wokhudza kuphatikizika kwa data mwachangu. Nkhani yoyamba idafotokoza za kompresa yomwe ikugwira ntchito pa 10 GB/sec pa purosesa pachimake (kupanikizika kochepa, RTT-Min).
Compressor iyi yakhazikitsidwa kale kukhala zobwereza zazamalamulo kuti azipondereza kwambiri zotayira zosungiramo zinthu zosungira komanso kulimbitsa mphamvu ya cryptographic. Itha kugwiritsidwanso ntchito kufinya zithunzi zamakina ndi mafayilo osinthira a RAM mukawasunga pama drive othamanga kwambiri a SSD.
Nkhani yoyamba idalengezanso za kukhazikitsidwa kwa algorithm yophatikizira yophatikizira ma HDD ndi ma SSD (RTT-Mid) okhala ndi magawo owongolera a data. Compressor iyi yapangidwa kwathunthu, ndipo nkhaniyi ikunena za izi.
Compressor yomwe ikugwiritsa ntchito RTT-Mid algorithm imapereka milingo yoponderezedwa yofananira ndi zakale monga WinRar ndi 7-Zip yomwe imagwira ntchito mothamanga kwambiri. Pa nthawi yomweyo, liwiro lake ndi osachepera dongosolo la magnitude apamwamba.
Kuthamanga kwa data / decompression ndi gawo lofunikira lomwe limazindikiritsa kukula kwa matekinoloje a compression. Ndizokayikitsa kuti aliyense angayesere kufinya terabyte ya data pa 10-15 megabytes pamphindikati (liwiro la ma archives mumayendedwe wamba), chifukwa izi zitha kutengera pafupifupi maola makumi awiri akukonza nthawi pansi pa kuchuluka kwa CPU.
Komano, ma terabyte omwewo amatha kukopera pa liwiro la 2-3 Gigabytes pamphindikati pafupifupi mphindi khumi.
Choncho, kupondereza kuchuluka kwa deta kumakhala kothandiza kokha ngati kuchitidwa pamlingo wosacheperapo kusiyana ndi kulowetsa / kutulutsa kwenikweni. Kwa machitidwe amakono, izi ndi zosachepera 100 megabytes pamphindikati.
Ma compressor amakono amatha kukwanitsa kuthamanga kotere mu "mofulumira". Ndi momwe zilili pano pomwe tidzafanizira algorithm ya RTT-Mid ndi ma compressor achikhalidwe.
Kuyesa kofananiza kwa algorithm yatsopano yophatikizira
Compressor ya RTT-Mid idagwiritsidwa ntchito ngati gawo la pulogalamu yoyesera. Mu ntchito yeniyeni, "yopanga", imayenda mwachangu kwambiri, chifukwa imagwiritsa ntchito ma multithreading mwanzeru ndikugwiritsa ntchito compiler "yabwinobwino", osati C #.
Popeza ma compressor omwe amagwiritsidwa ntchito poyesa kufananitsa amamangidwa pazikhalidwe zosiyanasiyana ndikupondereza mitundu yosiyanasiyana ya data mosiyanasiyana, njira ya "kutentha kwapakati pachipatala" idagwiritsidwa ntchito kuwonetsetsa kuti mayesowo anali ndi cholinga.
Fayilo yochotsera gawo lililonse ya disk ya logical yomwe ili ndi makina ogwiritsira ntchito idapangidwa. Windows 10Uku ndi kusakaniza kwachilengedwe kwa mitundu yosiyanasiyana ya deta komwe kumapezeka pa kompyuta iliyonse. Kukanikiza fayiloyi kudzatithandiza kuyerekeza liwiro ndi kukanikiza kwa njira yatsopanoyi ndi ma compressor apamwamba kwambiri omwe amagwiritsidwa ntchito m'ma archiver amakono.
Nayi fayilo yotaya:
Fayilo yotaya idapanikizidwa pogwiritsa ntchito PTT-Mid, 7-zip, ndi WinRar. WinRar ndi 7-zip adayikidwa pa liwiro lalikulu.
Compressor ikugwira ntchito 7-zip:
Imanyamula purosesa pa 100%, pomwe liwiro lapakati powerenga kutaya koyambirira ndi pafupifupi 60 Megabytes / sec.
Compressor ikugwira ntchito Zowonjezera:
Zomwe zilili ndizofanana, kuchuluka kwa purosesa kuli pafupifupi 100%, kuthamanga kwapakati pakutaya ndi pafupifupi 125 Megabytes / sec.
Monga momwe zinalili m'mbuyomu, kuthamanga kwa archive kumakhala kochepa ndi mphamvu za purosesa.
Pulogalamu yoyesera ya kompresa tsopano ikugwira ntchito. RTT-Mid:
Chithunzicho chikuwonetsa kuti purosesa ikugwiritsidwa ntchito ndi 50% ndipo imagwira ntchito nthawi yonseyi chifukwa palibe pomwe mungatayire deta yoponderezedwa. Dambo la disk (Disk 0) latsala pang'ono kugwiritsidwa ntchito. Kuthamanga kwa data (Disk 1) kumasintha kwambiri, koma pafupifupi 200 MB / mphindi.
Kuthamanga kwa kompresa kumakhala kochepa pankhaniyi ndikutha kulemba deta yoponderezedwa ku Disk 0.
Tsopano chiŵerengero cha kuponderezedwa kwa zosungidwazo:
Zikuwonekeratu kuti kompresa ya RTT-Mid idachita bwino kwambiri pakupondereza; malo osungira omwe adapanga anali ang'onoang'ono a 1,3 Gigabytes kuposa malo osungirako zakale a WinRar ndi ma Gigabytes 2,1 ocheperako kuposa 7z archive.
Nthawi yogwiritsidwa ntchito popanga zolemba zakale:
- 7-zip - 26 mphindi 10 masekondi;
- WinRar - Mphindi 17 masekondi 40;
- RTT-Mid - 7 mphindi 30 masekondi.
Chifukwa chake, ngakhale pulogalamu yoyeserera, yopanda kukhathamiritsa, pogwiritsa ntchito algorithm ya RTT-Mid, idakwanitsa kupanga zosungirako kuposa nthawi ziwiri ndi theka mwachangu, pomwe zosungirako zidakhala zocheperako kuposa za omwe akupikisana nawo ...
Iwo omwe sakhulupirira zowonera amatha kutsimikizira okha. Pulogalamu yoyeserera ikupezeka pa , koperani ndi kufufuza.
Koma pa mapurosesa omwe ali ndi chithandizo cha AVX-2; popanda thandizo la malangizo awa, kompresa sigwira ntchito. Osayesa ma aligorivimu pa mapurosesa akale a AMD; amachedwa pakuchita malangizo a AVX...
Njira yogwiritsira ntchito compression
Algorithm imagwiritsa ntchito njira yolozera magawo obwerezabwereza pogwiritsa ntchito byte granularity. Njira yophatikizirayi yakhala ikudziwika kwa nthawi yayitali, koma siinagwiritsidwe ntchito kwambiri chifukwa kufananiza kunali kogwiritsa ntchito kwambiri ndipo kunatenga nthawi yayitali kuposa kupanga mtanthauzira mawu. Chifukwa chake, algorithm ya RTT-Mid ndi chitsanzo chapamwamba cha "kubwerera m'tsogolo."
Compressor ya PTT imagwiritsa ntchito chojambulira chapadera, chothamanga kwambiri, chomwe chimathandizira kuphatikizika. Chojambuliracho ndi chopangidwa kunyumba, "chamtengo wanga ..." komanso "ndichokwera mtengo chifukwa ndi chopangidwa ndi manja" (cholembedwa m'chinenero cha msonkhano).
Chojambulira cha machesi chimakhazikitsidwa pogwiritsa ntchito njira ya magawo awiri: choyamba, kukhalapo kwa "chizindikiro" cha machesi kumafufuzidwa, ndipo pokhapokha "chizindikiro" chitadziwika pamalo ano ndi njira yodziwira machesi enieni omwe ayambitsidwa.
Zenera lofufuzira lili ndi kukula kosayembekezereka, kutengera entropy ya block block yomwe ikukonzedwa. Kwa data yachisawawa (yosasinthika), ndi megabyte kukula kwake; pa data yomwe ili ndi kubwereza, nthawi zonse imakhala yayikulu kuposa megabyte.
Komabe, mitundu yambiri yamakono ya deta ndi yosasunthika, ndipo kuyendetsa makina opangira zinthu pogwiritsa ntchito izo sikuthandiza komanso kuwononga, kotero scanner imagwiritsa ntchito njira ziwiri zogwirira ntchito. Choyamba, magawo a mawu oyambira omwe ali ndi kubwereza kotheka amafufuzidwa. Opaleshoniyi imachitikanso mwachidwi ndipo imathamanga kwambiri (pa liwiro la 4-6 Gbytes/mphindi). Kenako, magawo omwe ali ndi machesi otheka amakonzedwa ndi scanner yayikulu.
Kuphatikizika kwa index sikothandiza kwambiri, ndikofunikira kusintha magawo mobwerezabwereza ndi ma index, ndipo mndandanda wazomwe umachepetsa kwambiri kupsinjika.
Kuti muwonjezere kuponderezana, sikuti machesi a zingwe athunthu amalembedwa, komanso ochepa, pomwe chingwe chili ndi ma byte ofananira ndi osagwirizana. Pachifukwa ichi, mtundu wa index umaphatikizapo gawo la chigoba cha machesi lomwe limawonetsa ma byte ofananira a midadada iwiri. Pakuponderezana kwakukulu, kulondolera kumagwiritsidwa ntchito ndikukuta midadada ingapo yofananira pa block yomwe ilipo.
Zonsezi zidapangitsa kuti athe kupeza kuchuluka kwa compression mu RTT-Mid kompresa yofanana ndi ma compressor opangidwa pogwiritsa ntchito njira ya mtanthauzira mawu, koma imagwira ntchito mwachangu kwambiri.
Liwiro la algorithm yatsopano yophatikizira
Ngati kompresa ukugwira ntchito ndi ntchito yekha posungira kukumbukira (4 megabytes chofunika pa mtsinje), ndiye ntchito liwiro zimasinthasintha mu osiyanasiyana 700-2000 megabytes/sec pa purosesa pachimake, malingana ndi mtundu wa deta wothinikizidwa ndi pang'ono amadalira purosesa ntchito pafupipafupi.
Ndi compressor yamitundu yambiri, scalability yabwino imatsimikiziridwa ndi kukula kwa cache L3. Mwachitsanzo, ndi 9 MB ya cache, kuyendetsa ulusi wopondereza wopitilira awiri sikungawongolere magwiridwe antchito. Komabe, ndi posungira 20 MB, mukhoza kuthamanga ulusi psinjika zisanu.
RAM latency ndichinthu chofunikira kwambiri chomwe chimatsimikizira magwiridwe antchito a kompresa. Algorithm imagwiritsa ntchito zopempha za RAM mwachisawawa, zina zomwe (pafupifupi 10%) zimaphonya posungira, kukakamiza algorithm kudikirira deta kuchokera ku RAM, zomwe zimachepetsa magwiridwe antchito.
Dongosolo la I/O limakhudza kwambiri liwiro la compressor. Ma I/O amapempha deta ku RAM block CPU, zomwe zimachepetsanso liwiro la compression. Vutoli ndi lofunika kwambiri pa ma laptops ndi ma desktops. maseva Sichinthu chofunikira kwenikweni chifukwa cha chipangizo chowongolera mabasi chomwe chili ndi njira zamakono komanso RAM ya njira zambiri.
M'nkhani yonseyi, kupanikizika kumakambidwa; decompression ili pamwamba pa nkhaniyi, monga "zonse zili mu chokoleti." Decompression imathamanga kwambiri ndipo imachepetsedwa ndi liwiro la I/O. Pachimake pamutu umodzi mu ulusi umodzi amapeza mosavuta kuthamanga kwa 3-4 GB / sec.
Izi ndichifukwa chakusowa kwa ntchito yosaka machesi panthawi yotsegulira, yomwe "imadya" zinthu zazikulu za purosesa ndi kukumbukira kwa cache panthawi ya kupanikizana.
Kudalirika kwa kusungidwa kwa data kothinikizidwa
Monga dzina la gulu lonse la zida zamapulogalamu zomwe zimagwiritsa ntchito kuponderezana kwa data (zosungidwa zakale) zikusonyezera, zidapangidwa kuti zisungidwe kwanthawi yayitali, osati zaka, koma kwazaka zambiri…
Panthawi yosungirako, zosungirako zosungirako zimataya zina mwazinthu zawo, nachi chitsanzo:
Wonyamula zidziwitso wa "analogi" uyu ali ndi zaka chikwi chimodzi, zidutswa zina zatayika, koma zonsezo "ndizowerengeka" ...
Palibe amene amapanga makina amakono osungira deta ndi makanema apa digito kwa iwo omwe amawatsimikizira chitetezo chokwanira kwazaka zopitilira 75.
Ndipo ili ndi vuto, koma vuto lochedwetsedwa, mbadwa zathu ziyenera kulithetsa ...
Machitidwe osungira deta a digito amatha kutaya deta osati pambuyo pa zaka 75, komanso nthawi iliyonse, ngakhale akujambula. Zolakwa izi zimachepetsedwa pogwiritsa ntchito njira zowongolera zolakwika komanso zolakwika. Njira zowongolera zolakwika ndi zolakwika sizimatha nthawi zonse kubwezeretsa zidziwitso zotayika, ndipo ngakhale zitatero, palibe chitsimikizo kuti ntchito yobwezeretsayo idapambana.
Ndipo ilinso ndi vuto lalikulu, koma osati lochedwetsedwa, koma lopitilira.
Ma compressor amakono omwe amagwiritsidwa ntchito posunga zidziwitso za digito amamangidwa pakusintha kosiyanasiyana kwa njira ya mtanthauzira mawu, ndipo pazosungidwa zotere, kutayika kwachidutswa chazidziwitso kumatha kufa. Pali ngakhale liwu lokhazikitsidwa la izi: malo osungira "osweka" ...
Kutsika kodalirika kwa kusunga zidziwitso mu zolemba zakale zotengera mtanthauzira mawu ndi chifukwa cha kapangidwe ka data yopanikizidwa. Zambiri zomwe zili munkhokwe yotere mulibe mawu oyamba; imasunga manambala olowera mtanthauzira mawu, omwe amasinthidwa mwamphamvu ndi mawu opanikizidwa apano. Ngati gawo lina lazosungidwalo litatayika kapena litawonongeka, zonse zomwe zalembedwa munkhokweyo sizingadziwike ndi zomwe zili kapena kutalika kwa mtanthauzira mawu, chifukwa sizikudziwika kuti nambala yolembera mtanthauzira mawu ikugwirizana ndi chiyani.
Sizingatheke kupezanso zambiri kuchokera kumalo osungira zakale "wosweka".
Ma algorithm a RTT amatengera njira yodalirika yosungira deta yoponderezedwa. Imagwiritsa ntchito njira yolondolera kuwerengera kubwereza zidutswa. Njira iyi yophatikizira imachepetsa zotsatira za chivundi cha data pa sing'anga yosungiramo ndipo, nthawi zambiri, imangokonza ziphuphu zomwe zimachitika panthawi yosungira.
Izi ndichifukwa choti fayilo yosungidwa mukamakanikiza index ili ndi magawo awiri:
- gawo lolemba gwero ndi magawo obwerezabwereza achotsedwapo;
- index munda.
Gawo lolozera, lofunika kwambiri pakubwezeretsa deta, ndilaling'ono kukula ndipo litha kubwerezedwanso kuti lisungidwe zodalirika. Chifukwa chake, ngakhale chidutswa chazolemba choyambirira kapena mndandanda wazolozera chitayika, zidziwitso zina zonse zitha kubwezeredwa popanda mavuto, monga momwe tawonetsera pachithunzichi ndi "analogi" yosungirako media.
Zoyipa za algorithm
Ubwino uliwonse uli ndi zovuta zake. Njira yopanikizira yochokera ku index simapanikiza kalozera kakang'ono, kobwerezabwereza. Izi ndichifukwa cha malire a njira yotengera index. Ma index ndi osachepera 3 byte kukula kwake ndipo amatha mpaka 12 byte. Ngati kubwereza kukumana komwe kuli kocheperako poyerekeza ndi index yofotokoza, sikunyalanyazidwa, mosasamala kanthu kuti kubwereza kotereku kumapezeka kangati mufayilo yomwe ikupanikizidwa.
Njira yopanikizira yotengera mtanthauzira mawu momveka bwino imapondereza kubwereza kwautali wamfupi kangapo motero imakwaniritsa kuphatikizika kwapamwamba kuposa kukakamiza kotengera index. Komabe, izi zimabwera pamtengo wokwera kwambiri wa CPU. Kuti njira yotengera mtanthauzira mawu ikwaniritse kuphatikizika kwa data kogwira mtima kuposa njira yotengera index, ikuyenera kuchepetsa liwiro la kukonza deta mpaka 10-20 megabytes pa sekondi imodzi pamakompyuta enieni pansi pa kuchuluka kwa CPU.
Kuthamanga kochepa kotereku sikuvomerezeka kuzinthu zamakono zosungiramo deta ndipo ndizofunika kwambiri "zamaphunziro" kuposa momwe zimakhalira.
Kuchuluka kwa kupsinjika kwa chidziwitso kudzawonjezeka kwambiri pakusinthidwa kotsatira kwa algorithm ya RTT (RTT-Max), yomwe ikupangidwa kale.
Kotero, monga mwa nthawi zonse, kuti zipitirire ...
Source: www.habr.com
