Gcina ngokufanelekileyo amakhulu ezigidi zeefayile ezincinci. Isisombululo Self-Hosting

Uluntu oluthandekayo, Eli nqaku liza kugxila ekugcineni ngokufanelekileyo kunye nokubuyisela amakhulu ezigidi zeefayile ezincinci. Kweli nqanaba, isisombululo sokugqibela sicetywayo kwiinkqubo zefayile ezihambelanayo zePOSIX ezinenkxaso epheleleyo yezitshixo, kubandakanywa izitshixo zeqela, kwaye zibonakala zingenazo iintonga.

Ndiye ndabhala eyam iseva yesiko ngale njongo.
Ngethuba lokuphumeza lo msebenzi, sakwazi ukucombulula ingxaki ephambili, kwaye kwangaxeshanye sifezekise ukonga kwindawo yediski kunye ne-RAM, apho inkqubo yethu yefayile yeqela yayidla ngokungenalusini. Ngokwenyani, inani elinjalo leefayile liyingozi kuyo nayiphi na inkqubo yefayile edibeneyo.

Ingcamango yile:

Ngamagama alula, iifayile ezincinci zilayishwa ngomncedisi, zigcinwa ngqo kwi-archive, kwaye zifunde kwakhona kuyo, kwaye iifayile ezinkulu zibekwe ecaleni. Inkqubo: Ifolda e-1 = i-archive e-1, sisonke sinezigidi ezininzi zoovimba ezineefayile ezincinci, kwaye azikho iifayili ezingamakhulu ezigidi. Kwaye konke oku kuphunyezwe ngokupheleleyo, ngaphandle kwezikripthi okanye ukubeka iifayile kwindawo yokugcina yetar/zip.

Ndiza kuzama ukuyigcina imfutshane, ndicela uxolo kwangethuba ukuba isithuba sinde.

Yonke yaqala ngento yokuba andizange ndifumane umncedisi ofanelekileyo kwihlabathi onokugcina idatha efunyenwe nge-HTTP protocol ngokuthe ngqo kwii-archives, ngaphandle kokungalungi okukhoyo kwiindawo eziqhelekileyo kunye nokugcinwa kwezinto. Kwaye isizathu sokukhangela sasiyi-Origin cluster yeeseva ze-10 eziye zanda kwinqanaba elikhulu, apho iifayile ezincinci ze-250,000,000 sele ziqokelele, kwaye ukukhula kwakungayeki.

Kwabo bangathandi ukufunda amanqaku, amaxwebhu amancinci alula:

apha и apha.

Kwaye i-docker ngaxeshanye, ngoku kukho inketho kuphela nge-nginx ngaphakathi kwimeko nje:

docker run -d --restart=always -e host=localhost -e root=/var/storage 
-v /var/storage:/var/storage --name wzd -p 80:80 eltaline/wzd

Okulandelayo:

Ukuba kukho iifayile ezininzi, izibonelelo ezibalulekileyo ziyafuneka, kwaye eyona nto imbi kakhulu kukuba ezinye zazo zichithwa. Ngokomzekelo, xa usebenzisa i-clustered file system (kule meko, i-MooseFS), ifayile, kungakhathaliseki ubungakanani bayo bokwenene, ihlala ithatha ubuncinane i-64 KB. Oko kukuthi, kwiifayili eziyi-3, 10 okanye 30 KB ngobukhulu, kufuneka i-64 KB kwidiski. Ukuba kukho ikota yeebhiliyoni zeefayile, silahlekelwa ukusuka kwi-2 ukuya kwi-10 terabytes. Akunakwenzeka ukwenza iifayile ezintsha ngokungenasiphelo, ekubeni i-MooseFS inomda: akukho ngaphezu kwe-1 yezigidigidi kunye ne-replica enye yefayile nganye.

Njengoko inani leefayile lisanda, i-RAM eninzi iyadingeka kwimetadata. Ukulahla i-metadata enkulu rhoqo kunegalelo ekugugisweni kwee-SSD drives.

iseva ye-wZD. Sibeka izinto ngolungelelwano kwiidiski.

Umncedisi ubhalwe kwi Go. Okokuqala, bendidinga ukunciphisa inani leefayile. Ukwenza njani? Ngenxa yogcino, kodwa kule meko ngaphandle koxinzelelo, kuba iifayile zam ziyimifanekiso ecinezelweyo. I-BoltDB yeza kuhlangula, okwakusafuneka ikhutshwe kwiintsilelo zayo, oku kubonakaliswa kumaxwebhu.

Lilonke, endaweni yekota yeebhiliyoni zeefayile, kwimeko yam kwakukho kuphela i-archives ye-Bolt yezigidi ezili-10 ezisele. Ukuba bendinethuba lokutshintsha ulwakhiwo lwefayile yolawulo lwangoku, bekunokwenzeka ukulehlisa libe malunga nesigidi seefayile.

Zonke iifayile ezincinci zipakishwe kwii-archives ze-Bolt, ezifumana ngokuzenzekelayo amagama abalawuli apho zikhoyo, kwaye zonke iifayile ezinkulu zihlala zisecaleni kweendawo zokugcina, akukho nqaku lokupakisha, oku kunokwenziwa ngokwezifiso. Ezincinci zigcinwe, ezinkulu zishiywe zingatshintshi. Umncedisi usebenza ngokuphandle ngazo zombini.

Uyilo kunye neempawu zeseva ye-wZD.

Iseva isebenza phantsi kolawulo lweenkqubo zokusebenza Linux, BSD, Solaris, kunye ne OSX. Ndivavanye kuphela uyilo lwe-AMD64 phantsi Linux, kodwa kufuneka isebenze nakwi-ARM64, PPC64, MIPS64.

Iimpawu eziphambili:

• Imisonto emininzi;
• I-Multiserver, ukubonelela ukunyamezela impazamo kunye nokulinganisa umthwalo;
• Ubuninzi bokungafihli kumsebenzisi okanye umphuhlisi;
• Iindlela ze-HTTP ezixhaswayo: GET, HEAD, PUT and DELETE;
• Ukulawulwa kokuziphatha kokufunda nokubhala kusetyenziswa izihloko zabathengi;
• Inkxaso yeenginginya eziguquguqukayo;
• Xhasa ukuthembeka kwedatha ye-CRC xa ubhala/ufunda;
• I-semi-dynamic buffers yokusetyenziswa kwememori encinci kunye nokulungiswa kokusebenza kakuhle kwenethiwekhi;
• Ukudityaniswa kwedatha emisiweyo;
• Ukongeza, i-archiver enemisonto emininzi i-wZA inikezelwa ngokufuduka kweefayile ngaphandle kokumisa inkonzo.

Amava Okwenyani:

Ndiphuhlisa kwaye ndivavanya iseva kunye nomgcini wedatha kwidatha ephilayo ixesha elide, ngoku isebenza ngempumelelo kwiqela elibandakanya iifayile ezincinci ezingama-250,000,000 ezikuluhlu lwe-15,000,000 kwiidrive zeSATA ezahlukeneyo. Iqela labancedisi abali-10 yiOrigin server efakwe emva kwenethiwekhi yeCDN. Ukuyisebenzela, iiseva ezi-2 ze-Nginx + iiseva ezi-2 ze-wZD zisetyenziswa.

Kwabo bathatha isigqibo sokusebenzisa le seva, kuya kuba bubulumko ukucwangcisa ulwakhiwo lolawulo, ukuba lukhona, ngaphambi kokusetyenziswa. Makhe ndenze ugcino ngoko nangoko ukuba umncedisi ayenzelwanga ukukrazula yonke into kwi-1 Bolt yokugcina.

Uvavanyo lokusebenza:

Ubungakanani obuncinci befayile evaliweyo, imisebenzi ye-GET kunye ne-PUT ekhawulezayo yenziwa kuyo. Makhe sithelekise ixesha elipheleleyo lokubhala umxumi we-HTTP kwiifayile eziqhelekileyo kunye nokugcinwa kwe-Bolt, kunye nokufunda. Ukusebenza ngeefayili ezinobukhulu obuyi-32 KB, 256 KB, 1024 KB, 4096 KB no-32768 KB kuthelekiswa.

Xa usebenza kunye ne-archives ye-Bolt, ingqibelelo yedatha yefayile nganye ihlolwe (i-CRC isetyenzisiwe), ngaphambi kokurekhoda kwaye emva kokurekhoda, ukufundwa kwe-fly kunye nokubala kwakhona kwenzeka, oku kwazisa ngokwemvelo ukulibaziseka, kodwa into ephambili kukhuseleko lwedatha.

Ndiqhube iimvavanyo zokusebenza kwii-SSD drives, kuba iimvavanyo kwii-SATA drives azibonisi umahluko ocacileyo.

Iigrafu ezisekelwe kwiziphumo zovavanyo:

Njengoko ubona, kwiifayile ezincinci umahluko kumaxesha okufunda nokubhala phakathi kweefayile ezigciniweyo nezingagcinwanga mncinci.

Sifumana umfanekiso owahluke ngokupheleleyo xa sivavanya ukufunda nokubhala iifayile ze-32 MB ngobukhulu:

Umahluko wexesha phakathi kweefayile zokufunda ungaphakathi kwe-5-25 ms. Ngokurekhoda, izinto zimbi kakhulu, umahluko umalunga ne-150 ms. Kodwa kulo mzekelo akukho mfuneko yokulayisha iifayile ezinkulu;

*Ngobuchule, ungasebenzisa le seva kwimisebenzi efuna iNoSQL.

Iindlela ezisisiseko zokusebenza ngeseva ye-wZD:

Ilayisha ifayile eqhelekileyo:

curl -X PUT --data-binary @test.jpg http://localhost/test/test.jpg

Ukulayisha ifayile kwindawo yogcino lweBolt (ukuba iparamitha yomncedisi fmaxsize, emisela ubungakanani befayile enkulu enokufakwa kwindawo yokugcina, ayigqithwanga; ukuba igqithile, ifayile iya kulayishwa njengesiqhelo ecaleni kwendawo yokugcina izinto):

curl -X PUT -H "Archive: 1" --data-binary @test.jpg http://localhost/test/test.jpg

Ukukhuphela ifayile (ukuba kukho iifayile ezinamagama afanayo kwidiski nakwindawo yokugcina, ngoko xa ukhuphela, okuphambili kunikwa ngokungagqibekanga kwifayile engagcinwanga):

curl -o test.jpg http://localhost/test/test.jpg

Ikhuphela ifayile kwindawo yokugcina yeBolt (inyanzelisiwe):

curl -o test.jpg -H "FromArchive: 1" http://localhost/test/test.jpg

Iinkcazo zezinye iindlela zikumaxwebhu.

wZD Amaxwebhu
wZA Uxwebhu

Umncedisi okwangoku uxhasa kuphela umthetho olandelwayo weHTTP. Indlela ye-POST nayo ayixhaswanga (ayikagqitywa ukuba iyafuneka okanye hayi).

Nabani na ogrumba kwikhowudi yomthombo uya kufumana i-butterscotch apho, ayinguye wonke umntu oyithandayo, kodwa andizange ndibophe ikhowudi ephambili kwimisebenzi yesakhelo sewebhu, ngaphandle komphathi ophazamisayo, ngoko kwixesha elizayo ndingaphinda ndiyibhale ngokukhawuleza malunga nantoni na. injini.

Ukwenza:

• Ukuphuhliswa kwereplicator yakho kunye nomsasazi + geo ukuba nokwenzeka kokusetyenziswa kwiinkqubo ezinkulu ngaphandle kwenkqubo yefayile yeqela (Yonke into yabantu abadala)
• Ukubanakho ukuphinda kubuyiswe ngokupheleleyo imetadata ukuba ilahlekile ngokupheleleyo (ukuba usebenzisa umsasazi)
• Iprothokholi yeNative yokukwazi ukusebenzisa uqhagamshelwano oluqhubekayo lwenethiwekhi kunye nabaqhubi kwiilwimi ezahlukeneyo zokucwangcisa
• Amathuba aphezulu okusebenzisa icandelo le-NoSQL
• Uxinzelelo lweentlobo ezahlukeneyo (i-gzip, i-zstd, i-snappy) yeefayile okanye amaxabiso ngaphakathi koovimba be-Bolt kunye neefayile eziqhelekileyo.
• Ufihlo lweentlobo ezahlukeneyo zeefayile okanye amaxabiso ngaphakathi kogcino lweBolt kunye neefayile eziqhelekileyo
• Ilibazisekile uguqulo lwevidiyo esecaleni kweseva, kubandakanya neGPU

Ndinento yonke, ndiyathemba ukuba lo mncedisi uya kuba luncedo kumntu, ilayisenisi ye-BSD-3, i-copyright ephindwe kabini, ekubeni ukuba bekungekho nkampani apho ndisebenza khona, umncedisi akayi kubhalwa. Ndim ndedwa umakhi. Ndingavuya ngazo naziphi na iimpazamo kunye nezicelo zesici ozifumanayo.

umthombo: www.habr.com