Chengeta nemazvo mazana emamiriyoni emafaira madiki. Self-Hosted solution

Vanodiwa Nharaunda, Chinyorwa ichi chinotarisa pakuchengeta nekudzoreredza mazana emamiriyoni emafaira madiki. Panguva ino, mhinduro yekupedzisira inopihwa POSIX-inoenderana faira masisitimu ane rutsigiro rwakakwana rwekiyi, kusanganisira makiyi emasumbu, uye anoita kunge asina madondoro.

Saka ndakanyora yangu yega tsika server nekuda kwechinangwa ichi.
Mukati mekuita basa iri, takakwanisa kugadzirisa dambudziko guru, uye panguva imwe chete kuwana mari mu disk space uye RAM, iyo yedu cluster file system yaidya zvisina tsitsi. Chaizvoizvo, nhamba yakadaro yemafaira inokuvadza kune chero akaunganidzwa faira system.

Pfungwa ndeiyi:

Mumashoko akareruka, mafaira maduku akaiswa kuburikidza nevhavha, anochengetwa zvakananga mudura, uye anoverenga kubva pairi, uye mafaira makuru anoiswa padivi. Scheme: 1 dhairekitori = 1 archive, mune yakazara tine mamirioni akati wandei akachengetwa ane madiki mafaira, uye kwete akati wandei emamiriyoni mafaira. Uye zvese izvi zvinoitwa zvizere, pasina chero zvinyorwa kana kuisa mafaera mu tar/zip archives.

Ndichaedza kuichengeta ipfupi, ndinokumbira ruregerero pachine nguva kana chinyorwa chakareba.

Izvo zvese zvakatanga nenyaya yekuti handina kuwana sevha yakakodzera munyika iyo inogona kuchengetedza data yakagamuchirwa kuburikidza neHTTP protocol yakananga mudura, pasina zvipingamupinyi zviri muzvakajairwa zvinyorwa uye kuchengetedza chinhu. Uye chikonzero chekutsvaga chaiva cheOrigin cluster ye 10 servers yakakura kusvika pamwero mukuru, umo 250,000,000 mafaira maduku akanga atounganidza, uye kukura kwekukura kwaisazomira.

Kune avo vasingade kuverenga zvinyorwa, zvinyorwa zvishoma zviri nyore:

pano и pano.

Uye docker panguva imwe chete, ikozvino pane sarudzo chete ne nginx mukati kuti zvingoitika:

docker run -d --restart=always -e host=localhost -e root=/var/storage 
-v /var/storage:/var/storage --name wzd -p 80:80 eltaline/wzd

Zvadaro:

Kana paine akawanda mafaera, zvakakosha zviwanikwa zvinodiwa, uye chakaipisisa ndechekuti mamwe acho anoraswa. Semuenzaniso, kana uchishandisa clustered faira system (munyaya iyi, MooseFS), iyo faira, zvisinei nehukuru hwayo chaihwo, inogara ichitora kanenge 64 KB. Kureva, mafaera e3, 10 kana 30 KB muhukuru, 64 KB inodiwa padhisiki. Kana paine chikamu chechina chebhiriyoni mafaera, tinorasikirwa kubva pa2 kusvika ku10 terabytes. Izvo hazvizogoneke kugadzira mafaera matsva nekusingaperi, sezvo MooseFS ine painogumira: isingasviki bhiriyoni imwe neimwe replica yefaira rega rega.

Sezvo huwandu hwemafaira hunowedzera, yakawanda RAM inodiwa kune metadata. Anogara achiraswa metadata hombe anobatsirawo mukupfekwa uye kubvaruka kweSSD madhiraivha.

wZD server. Tinoisa zvinhu zvakarongeka pamadisiki.

Sevha yakanyorwa muna Go. Chokutanga pane zvose, ndaifanira kuderedza nhamba yemafaira. Kuzviita sei? Nekuda kwekuchengetedza, asi mune iyi kesi pasina kudzvanywa, sezvo mafaera angu angori akamanikidzwa mapikicha. BoltDB yakauya kuzonunura, iyo ichiri kufanira kubviswa kubva mukukanganisa kwayo, izvi zvinoratidzwa mune zvinyorwa.

Pakazara, pane chikamu chechina chebhiriyoni mafaira, mune yangu pakanga paine 10 miriyoni Bolt archives yakasara. Dai ndanga ndiine mukana wekushandura ikozvino dhairekitori faira chimiro, zvaizogoneka kuidzikisa kusvika kunosvika miriyoni mafaera.

Ese mafaera madiki akazara muBolt archives, ayo anogashira mazita emadhairekitori maanowanikwa, uye ese makuru mafaera anoramba ari padyo neakachengetwa; hapana chikonzero chekuarongedza, izvi zvinogoneka. Madiki akachengetwa, makuru anosiiwa asina kuchinjwa. Sevha inoshanda pachena nezvose zviri zviviri.

Architecture uye maficha eiyo wZD server.

Sevha inoshanda pasi peLinux, BSD, Solaris uye OSX masisitimu anoshanda. Ini ndakangoyedza AMD64 architecture pasi peLinux, asi inofanirwa kushandira ARM64, PPC64, MIPS64.

Main features:

• Multithreading;
• Multiserver, inopa kukanganisa kushivirira uye kuyera mutoro;
• Kunyanya kujeka kune mushandisi kana mugadziri;
• Inotsigirwa nzira dzeHTTP: GET, HEAD, PUT uye DELETE;
• Kudzora kwekuverenga nekunyora maitiro kuburikidza nevatengi misoro;
• Tsigiro kune anochinjika chaiwo mahosi;
• Tsigira CRC data kutendeseka paunenge uchinyora / kuverenga;
• Semi-dynamic buffers yekushomeka ndangariro yekushandisa uye yakakwana network performance tuning;
• Deferred data compaction;
• Uye zvakare, iyo yakawanda-yakarukwa archiver wZA inopihwa yekufambisa mafaera pasina kumisa sevhisi.

Chiitiko Chaicho:

Ndanga ndichigadzira uye kuyedza sevha uye archiver pane mhenyu data kwenguva yakati rebei, iko zvino iri kushanda zvakanaka pane sumbu rinosanganisira 250,000,000 madiki mafaera (mifananidzo) ari mu15,000,000 madhairekitori pane akasiyana maSATA madhiraivha. Cluster yemaseva gumi iOrigin server yakaiswa kuseri kweCDN network. Kuishandira, 10 Nginx maseva + 2 wZD maseva anoshandiswa.

Kune avo vanosarudza kushandisa sevha iyi, zvingave zvine hungwaru kuronga dhairekitori chimiro, kana zvichibvira, usati washandisa. Rega ndiite chengetedzo ipapo ipapo kuti sevha haina kuitirwa kupinza zvese mu 1 Bolt archive.

Performance test:

Iyo diki saizi yezip faira, iyo inokurumidza GET nePUT mashandiro anoitwa pairi. Ngatienzanise iyo yakazara nguva yeHTTP mutengi kunyora kune akajairwa mafaera uye Bolt archives, pamwe nekuverenga. Shanda nemafaira ehukuru 32 KB, 256 KB, 1024 KB, 4096 KB uye 32768 KB inofananidzwa.

Paunenge uchishanda neBolt archives, data kutendeseka kwefaira imwe neimwe inotariswa (CRC inoshandiswa), isati yarekodha uye zvakare mushure mekurekodha, pa-the-fly kuverenga uye recalculation inoitika, izvi zvinowanzounza kunonoka, asi chinhu chikuru ndechekuchengetedza data.

Ndakaita bvunzo dzekuita paSSD madhiraivha, sezvo bvunzo paSATA dhiraivha dzisingaratidzi mutsauko wakajeka.

Magirafu anoenderana nemhinduro dzekuyedza:

Sezvauri kuona, kune madiki mafaira musiyano wekuverenga nekunyora nguva pakati peakachengetwa uye asina-akachengetwa mafaera idiki.

Isu tinowana mufananidzo wakasiyana zvachose pakuedza kuverenga nekunyora mafaera e32 MB muhukuru:

Musiyano wenguva pakati pekuverenga mafaira uri mukati me5-25 ms. Nekurekodha, zvinhu zvakanyanya kuipa, mutsauko unosvika 150 ms. Asi mune iyi kesi hapana chikonzero chekurodha mahombe mafaera; hapana chikonzero chekuita kudaro; vanogona kugara vakaparadzana kubva mudura.

*Nehunyanzvi, unogona kushandisa sevha iyi kumabasa anoda NoSQL.

Nzira dzekutanga dzekushanda ne wZD server:

Kuisa faira renguva dzose:

curl -X PUT --data-binary @test.jpg http://localhost/test/test.jpg

Kuisa faira kuBolt archive (kana server parameter fmaxsize, iyo inosarudza iyo yakakura faira saizi inogona kuverengerwa mudura, haina kupfuudzwa; kana ikapfuudzwa, iyo faira ichaiswa semazuva ese padivi peiyo archive):

curl -X PUT -H "Archive: 1" --data-binary @test.jpg http://localhost/test/test.jpg

Kudhawunirodha faira (kana paine mafaera ane mazita akafanana padhisiki uye mudura, zvino kana uchirodha, zvakakosha zvinopihwa nekusarudzika kune faira risina kuchengetwa):

curl -o test.jpg http://localhost/test/test.jpg

Kurodha faira kubva kuBolt archive (kumanikidzwa):

curl -o test.jpg -H "FromArchive: 1" http://localhost/test/test.jpg

Tsananguro dzedzimwe nzira dziri muzvinyorwa.

wZD Zvinyorwa
wZA Zvinyorwa

Sevha parizvino inongotsigira HTTP protocol; haisati yashanda neHTTPS. Iyo POST nzira hainawo kutsigirwa (haisati yasarudzwa kuti iri kudiwa here kana kuti kwete).

Ani naani anochera mukati meiyo kodhi kodhi anowana butterscotch ipapo, kwete munhu wese anoifarira, asi ini handina kusungira iyo huru kodhi kumabasa ewebhu framework, kunze kweanovhiringidza anobata, saka mune ramangwana ndinogona kukurumidza kuinyora zvakare kune chero chero. injini.

Kuita:

• Kuvandudza yako wega replicator uye mugovera + geo mukana wekushandisa mumahombe masisitimu asina cluster faira masisitimu (Zvese zvevakuru)
• Mikana yekudzoreredza kudzoreredza metadata kana yakarasika zvachose (kana uchishandisa muparidzi)
• Native protocol yekugona kushandisa inoenderera netiweki yekubatanidza uye madhiraivha emitauro yakasiyana yekuronga
• Mikana yepamberi yekushandisa iyo NoSQL chikamu
• Madimikira emhando dzakasiyana (gzip, zstd, snappy) yemafaira kana kukosha mukati meBolt archives uye yenguva dzose mafaera.
• Encryption yemhando dzakasiyana dzemafaira kana kukosha mukati meBolt archives uye yenguva dzose mafaera
• Yakanonoka server-padivi vhidhiyo shanduko, kusanganisira paGPU

Ndine zvese, ndinovimba sevha iyi ichave yakakosha kune mumwe munhu, BSD-3 rezinesi, kaviri copyright, sezvo dai pasina kambani kwandinoshanda, sevha ingadai isina kunyorwa. Ndini ndega mugadzirisi. Ndingave nekutenda kune chero tsikidzi uye maficha zvikumbiro zvaunowana.

Source: www.habr.com