Gcina kahle amakhulu ezigidi zamafayela amancane. Isixazululo sokuzisingatha ngokwakho

Mphakathi Othandekayo, Lesi sihloko sizogxila ekugcineni kahle nasekubuyiseni amakhulu ezigidi zamafayela amancane. Kulesi sigaba, isisombululo sokugcina sihlongozwa kumasistimu wefayela ahambisana ne-POSIX asekelwa ngokugcwele izingidi, okuhlanganisa izingidi zeqoqo, futhi ezibonakala zingenazo izinduku.

Ngakho ngibhale iseva yami yangokwezifiso ngale njongo.
Ngesikhathi senza lo msebenzi, sikwazile ukuxazulula inkinga enkulu, futhi ngasikhathi sinye sizuze ukonga endaweni yediski kanye ne-RAM, uhlelo lwethu lwamafayela eqoqo lwaludla ngokungenasihawu. Empeleni, inani elinjalo lamafayela liyingozi kunoma iyiphi isistimu yefayela ehlanganisiwe.

Umqondo yilokhu:

Ngamagama alula, amafayela amancane alayishwa ngeseva, agcinwa ngqo ku-archive, futhi aphinde afundwe kuwo, futhi amafayela amakhulu abekwe eceleni. Uhlelo: Ifolda engu-1 = ingobo yomlando engu-1, isiyonke sinezigidi ezimbalwa zomlando ezinamafayili amancane, hhayi amafayela ayizigidi ezingamakhulu ambalwa. Futhi konke lokhu kwenziwa ngokugcwele, ngaphandle kwanoma iyiphi imibhalo noma ukufaka amafayela kungobo yomlando ye-tar/zip.

Ngizozama ukukugcina kufushane, ngiyaxolisa kusenesikhathi uma okuthunyelwe kukude.

Konke kwaqala ngokuthi angikwazanga ukuthola iseva efanelekile emhlabeni engalondoloza idatha etholwe ngephrothokholi ye-HTTP ngokuqondile kungobo yomlando, ngaphandle kobubi obukhona kuzingobo zomlando ezivamile nokugcinwa kwento. Futhi isizathu sokusesha kwakuyi-Origin cluster yamaseva angu-10 ayekhule abe yisilinganiso esikhulu, lapho amafayela amancane angu-250,000,000 asevele eqoqwe, futhi umkhuba wokukhula wawungeke ume.

Kulabo abangathandi ukufunda izindatshana, imibhalo encane ilula:

lapha и lapha.

Futhi i-docker ngasikhathi sinye, manje kunenketho kuphela nge-nginx ngaphakathi uma kwenzeka:

docker run -d --restart=always -e host=localhost -e root=/var/storage 
-v /var/storage:/var/storage --name wzd -p 80:80 eltaline/wzd

Okulandelayo:

Uma kunamafayela amaningi, izinsiza ezibalulekile ziyadingeka, futhi ingxenye embi kakhulu ukuthi amanye awo ayamoshwa. Isibonelo, uma usebenzisa isistimu yefayela ehlanganisiwe (kulokhu, i-MooseFS), ifayela, kungakhathaliseki ukuthi lingakanani ubukhulu balo, lihlala lithatha okungenani u-64 KB. Okusho ukuthi, kumafayela angu-3, ​​10 noma 30 KB ngosayizi, u-64 KB uyadingeka kudiski. Uma kukhona ingxenye yesine yebhiliyoni yamafayela, silahlekelwa kusuka ku-2 kuya ku-10 terabytes. Ngeke kwenzeke ukudala amafayela amasha unomphela, njengoba i-MooseFS inomkhawulo: akukho ngaphezu kwebhiliyoni elingu-1 ngekhophi eyodwa yefayela ngalinye.

Njengoba inani lamafayela likhula, i-RAM eningi iyadingeka ukuze uthole imethadatha. Ukulahlwa kwemethadatha okukhulu okuvamisile nakho kunomthelela ekugugekeni nasekudabukeni kwamadrayivu e-SSD.

iseva ye-wZD. Sibeka izinto ngendlela kumadiski.

Iseva ibhalwe kokuthi Go. Okokuqala, ngangidinga ukunciphisa inani lamafayela. Kwenziwa kanjani? Ngenxa yokugcinwa kwengobo yomlando, kodwa kulokhu ngaphandle kokucindezelwa, njengoba amafayela ami ayizithombe ezicindezelwe nje. I-BoltDB yasiza, okwakusafanele isuswe ekushiyekeni kwayo, lokhu kubonakala emibhalweni.

Sekukonke, esikhundleni sekota yezigidigidi zamafayela, esimweni sami kwakusele izingobo zomlando ze-Bolt eziyizigidi eziyi-10 kuphela. Ukube benginethuba lokushintsha ukwakheka kwefayela lohla lwemibhalo lwamanje, bengizokwazi ukulehlisa libe cishe amafayela ayisigidi esingu-1.

Wonke amafayela amancane apakishwe ezinqolobaneni ze-Bolt, ezithola ngokuzenzakalelayo amagama ezinkomba lapho atholakala khona, futhi wonke amafayela amakhulu ahlala eduze nezingobo zomlando; asikho isidingo sokuwapakisha, lokhu kungenziwa ngokwezifiso. Ezincane zigcinwa kungobo yomlando, ezinkulu zishiywa zingashintshiwe. Iseva isebenza ngokusobala ngazo zombili.

Izakhiwo nezici zeseva ye-wZD.

Iseva isebenza ngaphansi kwe-Linux, BSD, Solaris kanye nezinhlelo zokusebenza ze-OSX. Ngihlole kuphela ukwakheka kwe-AMD64 ngaphansi kweLinux, kodwa kufanele kusebenzele i-ARM64, PPC64, MPS64.

Izici eziyinhloko:

• Multithreading;
• I-Multiserver, ehlinzeka ngokubekezelelana kwamaphutha nokulinganisa komthwalo;
• Ubukhulu bokucaca komsebenzisi noma unjiniyela;
• Izindlela ze-HTTP ezisekelwe: GET, HEAD, PUT and DELETE;
• Ukulawulwa kokuziphatha kokufunda nokubhala ngokusebenzisa izihloko zamaklayenti;
• Ukusekelwa kwababungazi abavumelana nezimo;
• Sekela ubuqotho bedatha ye-CRC lapho ubhala/ufunda;
• Amabhafa ashintshashintsha kancane okusetshenziswa kwememori okuncane kanye nokulungisa ukusebenza kahle kwenethiwekhi;
• Ukuhlanganiswa kwedatha okuhlehlisiwe;
• Ngaphezu kwalokho, i-wZA ye-archiver enezintambo eziningi inikezwa ukuthutha amafayela ngaphandle kokumisa isevisi.

Umuzwa Wangempela:

Bengilokhu ngithuthukisa futhi ngivivinya iseva nesilondolozi kudatha ebukhoma isikhathi eside impela, manje isisebenza ngempumelelo kuqoqo elihlanganisa amafayela amancane (izithombe) angu-250,000,000 atholakala kunkomba ye-15,000,000 kumadrayivu e-SATA ahlukene. Iqoqo lamaseva ayi-10 yiseva ye-Origin efakwe ngemuva kwenethiwekhi ye-CDN. Ukuyisevisa, kusetshenziswa amaseva angu-2 we-Nginx + 2 wZD amaseva asetshenziswa.

Kulabo abanquma ukusebenzisa lesi siphakeli, kungaba ukuhlakanipha ukuhlela uhlaka lwemibhalo, uma lukhona, ngaphambi kokusetshenziswa. Ake ngenze ukubhuka ngokushesha ukuthi iseva ayihloselwe ukuminyanisa yonke into kungobo yomlando ye-1 Bolt.

Ukuhlolwa kokusebenza:

Uma umncane usayizi wefayela eliziphuwe, wenza imisebenzi ye-GET ne-PUT esheshayo eyenziwa kulo. Ake siqhathanise isikhathi esiphelele sokubhala kweklayenti le-HTTP namafayela avamile nezingobo zomlando ze-Bolt, kanye nokufunda. Sebenza ngamafayela osayizi 32 KB, 256 KB, 1024 KB, 4096 KB kanye no-32768 KB kuyaqhathaniswa.

Lapho usebenza nezingobo zomlando ze-Bolt, ubuqotho bedatha yefayela ngalinye buyahlolwa (i-CRC isetshenziswa), ngaphambi kokurekhoda futhi nangemva kokurekhoda, ukufundwa okundizayo nokubala kabusha kwenzeka, lokhu ngokwemvelo kwethula ukubambezeleka, kodwa into eyinhloko ukuphepha kwedatha.

Ngenze izivivinyo zokusebenza kumadrayivu e-SSD, njengoba ukuhlolwa kumadrayivu we-SATA akubonisi umehluko ocacile.

Amagrafu asuselwe emiphumeleni yokuhlola:

Njengoba ubona, kumafayela amancane umehluko ezikhathini zokufunda nokubhala phakathi kwamafayela agcinwe kungobo yomlando nalawo angagciniwe mncane.

Sithola isithombe esihluke ngokuphelele lapho sihlola amafayela okufunda nokubhala angu-32 MB ngosayizi:

Umehluko wesikhathi phakathi kwamafayela okufunda ungaphakathi kuka-5-25 ms. Ngokurekhoda, izinto zimbi kakhulu, umehluko umayelana ne-150 ms. Kodwa kulokhu asikho isidingo sokulayisha amafayela amakhulu; akunasizathu sokwenza kanjalo; angahlala ngokuhlukana nezingobo zomlando.

*Ngobuchwepheshe, ungasebenzisa le seva emisebenzini edinga i-NoSQL.

Izindlela eziyisisekelo zokusebenza neseva ye-wZD:

Ilayisha ifayela elijwayelekile:

curl -X PUT --data-binary @test.jpg http://localhost/test/test.jpg

Ukulayisha ifayela kungobo yomlando ye-Bolt (uma ipharamitha yeseva i-fmaxsize, enquma umkhawulo kasayizi wefayela ongafakwa kungobo yomlando, aweqiwa; uma yeqiwe, ifayela lizolayishwa njengokujwayelekile eduze kwengobo yomlando):

curl -X PUT -H "Archive: 1" --data-binary @test.jpg http://localhost/test/test.jpg

Ukulanda ifayela (uma kukhona amafayela anamagama afanayo kudiski nakungobo yomlando, lapho-ke ulanda, okubalulekile kunikezwa ngokuzenzakalelayo kufayela elingagciniwe):

curl -o test.jpg http://localhost/test/test.jpg

Ukulanda ifayela kusuka kungobo yomlando ye-Bolt (kuyaphoqelelwa):

curl -o test.jpg -H "FromArchive: 1" http://localhost/test/test.jpg

Izincazelo zezinye izindlela zikumadokhumenti.

wZD Amadokhumenti
wZA Amadokhumenti

Iseva okwamanje isekela iphrothokholi ye-HTTP kuphela; okwamanje ayisebenzi ne-HTTPS. Indlela ye-POST nayo ayisekelwe (ayikanqunywa ukuthi iyadingeka noma cha).

Noma ubani omba ikhodi yomthombo uzothola i-butterscotch lapho, akuwona wonke umuntu oyithandayo, kodwa angizange ngibophe ikhodi eyinhloko emisebenzini yohlaka lwewebhu, ngaphandle kwesibambi esiphazamisayo, ngakho esikhathini esizayo ngingakwazi ukuyibhala kabusha ngokushesha cishe noma yikuphi. injini.

Ukwenza:

• Ukuthuthukiswa kwe-replicator yakho kanye nomsabalalisi + i-geo ukuze ukwazi ukusetshenziswa ezinhlelweni ezinkulu ngaphandle kwezinhlelo zamafayela eqoqo (Yonke into eyabantu abadala)
• Amathuba okuthola kabusha okuphelele kokuhlehla kwemethadatha uma ilahleke ngokuphelele (uma kusetshenziswa umsabalalisi)
• Iphrothokholi yomdabu yekhono lokusebenzisa ukuxhumana kwenethiwekhi okuphikelelayo nezishayeli zezilimi zokuhlela ezihlukile
• Amathuba athuthukile okusebenzisa ingxenye ye-NoSQL
• Ukucindezelwa kwezinhlobo ezahlukene (gzip, zstd, snappy) kwamafayela noma amanani angaphakathi kwengobo yomlando ye-Bolt kanye namafayela avamile
• Ukubethela kwezinhlobo ezahlukene zamafayela noma amanani angaphakathi kwengobo yomlando ye-Bolt kanye namafayela avamile
• Ukuguqulwa kwevidiyo eseceleni kweseva okubambezelekile, okuhlanganisa ne-GPU

Nginakho konke, ngithemba ukuthi le seva izoba usizo kumuntu, ilayisense ye-BSD-3, i-copyright ephindwe kabili, ngoba ukube bekungekho inkampani lapho ngisebenza khona, iseva ibingeke ibhalwe. Yimi ngedwa unjiniyela. Ngingabonga nganoma yiziphi iziphazamisi nezicelo zesici ozitholayo.

Source: www.habr.com