I-GitLab ikusiza kanjani ukuthi wenze isipele isitoreji esikhulu se-NextCloud

Sawubona Habr!

Namuhla ngifuna ukukhuluma ngokuhlangenwe nakho kwethu ekwenzeni isipele sedatha enkulu kusuka ku-Nextcloud storages ekucushweni okuhlukene. Ngisebenza njengesiteshi sesevisi e-Molniya AK, lapho senza khona ukuphathwa kokucushwa kwezinhlelo ze-IT; i-Nextcloud isetshenziselwa ukugcinwa kwedatha. Kuhlanganisa, nesakhiwo esabalalisiwe, esinokweqa.

Izinkinga ezivela ezicini zokufakwa ukuthi kunedatha eminingi. Ukuhumusha okuhlinzekwe yi-Nextcloud, ukuphindaphindeka, izizathu ezicabangelayo, nokuningi kudala izimpinda eziningi.

prehistory

Lapho uphatha i-Nextcloud, kuphakama inkinga yokuhlela isipele esisebenzayo, okufanele sibethelwe, ngoba idatha ibalulekile.

Sinikeza izinketho zokugcina izipele endaweni yethu noma kumakhasimende emishinini ehlukene evela ku-Nextcloud, edinga indlela ezenzakalelayo eguquguqukayo yokuphatha.

Kunamakhasimende amaningi, wonke anezinhlelo ezihlukene, futhi wonke kumasayithi awo kanye nezici zawo. Lena indlela evamile lapho isayithi lonke lingelakho, futhi izipele zenziwa ngemiqhele; akuhlali kahle.

Okokuqala, ake sibheke idatha yokufaka. Sidinga:

• I-scalability ngokuya nge-node eyodwa noma eziningana. Ekufakeni okukhulu sisebenzisa i-minio njengesitoreji.
• Thola ngezinkinga ngokwenza izipele.
• Udinga ukugcina ikhophi yasenqolobaneni namakhasimende akho kanye/noma nathi.
• Bhekana nezinkinga ngokushesha futhi kalula.
• Amaklayenti nokufakwa kuhluke kakhulu komunye nomunye - ukufana akukwazi ukufezwa.
• Ijubane lokutakula kufanele libe lincane ezimweni ezimbili: ukutakula okugcwele (inhlekelele), ifolda eyodwa isulwe ngephutha.
• Kudingeka umsebenzi wokudonsela phansi.

Ukuxazulula inkinga yokuphatha izipele, sifake i-GitLab. Imininingwane eyengeziwe nge-tackle.

Yebo, asibona abokuqala ukuxazulula inkinga enjalo, kodwa kubonakala kithi ukuthi isipiliyoni sethu esisebenzayo, esisebenze kanzima singathakazelisa futhi sikulungele ukwabelana ngakho.

Njengoba inkampani yethu inenqubomgomo yomthombo ovulekile, besifuna isisombululo somthombo ovulekile. Ngokulandelayo, sabelana ngentuthuko yethu futhi siyithumele. Isibonelo, ku-GitHub kukhona i-plugin yethu ye-Nextcloud, esinikeza amaklayenti, sithuthukisa ukuvikeleka kwedatha uma kwenzeka ukususwa ngephutha noma ngamabomu.

Amathuluzi ekhophi yasenqolobaneni

Siqale ukusesha kwethu izindlela zesixazululo ngokukhetha ithuluzi lokudala eliyisipele.

I-tar + gzip evamile ayisebenzi kahle - idatha iyimpinda. Ukukhuphuka kuvame ukuqukatha izinguquko zangempela ezimbalwa kakhulu, futhi idatha eningi engaphakathi kwefayela elilodwa iyaphindwa.
Kukhona enye inkinga - ukuphinda kusetshenziswe ukugcinwa kwedatha esabalalisiwe. Sisebenzisa i-minio futhi idatha yayo ngokuyisisekelo ayinalutho. Noma bekufanele wenze ikhophi yasenqolobaneni nge-minio ngokwayo - uyilayishe futhi usebenzise zonke izikhala phakathi kwesistimu yefayela, futhi, okungabalulekile kangako, kunengozi yokukhohlwa ngamanye amabhakede nolwazi lwe-meta. Noma sebenzisa ukuphindaphinda.

Amathuluzi ekhophi yasenqolobaneni anempinda ayatholakala kumthombo ovulekile (ku-Habré kwakukhona izindatshana mayelana nale ngqikithi) kanye nabaphumelele bethu Borg и I-restic. Ukuqhathanisa kwethu lezi zicelo ezimbili kungezansi, kodwa okwamanje sizokutshela ukuthi siluhlele kanjani lonke uhlelo.

Ukuphatha izipele

I-Borg ne-Restic zinhle, kodwa awukho umkhiqizo onomshini wokulawula ophakathi nendawo. Ngenhloso yokuphatha nokulawula, sikhethe ithuluzi esesivele silisebenzisile, ngaphandle kwalo esingeke siwucabange umsebenzi wethu, okuhlanganisa ne-automation - lena eyaziwa kakhulu i-CI/CD - GitLab.

Umbono umi kanje: i-gitlab-runner ifakwe endaweni ngayinye egcina idatha ye-Nextcloud. Umgijimi uqhuba iskripthi kushejuli eqapha inqubo yokwenza isipele, futhi sethula i-Borg noma i-Restic.

Sitholeni? Impendulo evela ekusebenzeni, ukulawula okulula phezu kwezinguquko, imininingwane uma kwenzeka iphutha.

Lapha lapha ku-GitHub sithumele izibonelo zeskripthi semisebenzi ehlukahlukene, futhi sagcina siyinamathisele kusipele se-Nextcloud hhayi kuphela, kodwa nezinye izinsiza eziningi. Kukhona nesihleli lapho uma ungafuni ukusilungiselela mathupha (futhi asifuni) kanye .gitlab-ci.yml

Ayikho indlela yokushintsha ukuphela kwesikhathi kwe-CI/CD ku-Gitlab API okwamanje, kodwa kuncane. Idinga ukwanda, kusho 1d.

I-GitLab, ngenhlanhla, ingakwazi ukwethula hhayi kuphela ngokuzibophezela, kodwa kuphela ngokohlelo, yilokhu kanye esikudingayo.

Manje mayelana neskripthi se-wrapper.

Setha izimo ezilandelayo zalesi script:

• Kufanele yethulwe kokubili ngumgijimi nangesandla kusuka kukhonsoli ngokusebenza okufanayo.
• Kufanele kube nezibambi zamaphutha:
• ikhodi yokubuyisela.
• sesha iyunithi yezinhlamvu kulogi. Isibonelo, kithi iphutha kungaba umlayezo uhlelo olungawuthathi njengobulalayo.
• Icubungula ukuphela kwesikhathi. Isikhathi sokuhola kufanele sibe esinengqondo.
• Sidinga ilogi enemininingwane eminingi. Kodwa kuphela uma kwenzeka iphutha.
• Izivivinyo eziningi nazo ziyenziwa ngaphambi kokuqala.
• Amabhonasi amancane esiwathole ewusizo phakathi nenqubo yosekelo:
• Isiqalo nesiphetho zirekhodwa ku-syslog yomshini wendawo. Lokhu kusiza ukuxhuma amaphutha esistimu nokusebenza kwekhophi yasenqolobaneni.
• Ingxenye yelogi yephutha, uma ikhona, iphuma ku-stdout, yonke ilogu ibhalwa efayeleni elihlukile. Kulula ukubheka i-CI ngokushesha futhi uhlole iphutha uma lincane.
• Izindlela zokususa iphutha.

Ilogi egcwele igcinwa njenge-artifact ku-GitLab; uma lingekho iphutha, ilogu iyasuswa. Sibhala umbhalo ku-bash.

Sizokujabulela ukucabangela noma yiziphi iziphakamiso namazwana mayelana nomthombo ovulekile - wamukelekile.

Kanjani lo msebenzi

Umgijimi one-Bash executor yethulwa endaweni yokusekelayo. Ngokusho komhleli, umsebenzi we-CI/CD wethulwa ngetheniphu ekhethekile. Umgijimi wethula iskripthi se-wrapper yendawo yonke semisebenzi enjalo, sihlola ukufaneleka kwendawo yokugcina ikhophi, amaphuzu okukhweza nakho konke esikufunayo, bese senza isipele futhi sihlanze endala. Isipele esiqediwe ngokwaso sithunyelwa ku-S3.

Sisebenza ngokwalolu hlelo - luwumhlinzeki we-AWS wangaphandle noma okulingana nesiRashiya (kuyashesha futhi idatha ayishiyi e-Russian Federation). Noma sifakela iklayenti iqoqo elihlukile le-minio esizeni salo ngalezi zinhloso. Ngokuvamile sikwenza lokhu ngenxa yezizathu zokuphepha, lapho iklayenti lingafuni ukuthi idatha ishiye isifunda salo nhlobo.

Asizange sisebenzise isici sokuthumela isipele nge-ssh. Lokhu akukwengezi ukuphepha, futhi amandla enethiwekhi omhlinzeki we-S3 aphakeme kakhulu kunomshini wethu owodwa we-ssh.

Ukuze uvikele umshini wakho wendawo ku-hacker, njengoba engakwazi ukusula idatha ku-S3, kufanele unike amandla ukwenza inguqulo.
Ikhophi yasenqolobaneni ihlala ibhala ngemfihlo isipele.

I-Borg inemodi engabethelwe none, kodwa asincomi ngokuqinile ukuyivula. Kule modi, ngeke kube khona ukubethela kuphela, kodwa i-checksum yalokho okubhalwayo ayibalwa, okusho ukuthi ubuqotho bungabhekwa kuphela ngokungaqondile, kusetshenziswa izinkomba.

Umhleli ohlukile uhlola izipele ukuze uthole ubuqotho bezinkomba nokuqukethwe. Isheke lihamba kancane futhi lide, ngakho silisebenzisa ngokuhlukile kanye ngenyanga. Kungathatha izinsuku ezimbalwa.

Fundame in Russian

Imisebenzi eyinhloko

• prepare ukuqeqeshwa
• testcheck ukuhlola ukulungela
• maincommand iqembu eliwumgogodla
• forcepostscript umsebenzi owenziwe ekugcineni noma ngephutha. Siyisebenzisela ukwehlisa i-partition.

Imisebenzi yesevisi

• cleanup Sirekhoda amaphutha noma sisule ifayela lokungena.
• checklog hlaziya ilogu ukuze kuvele umugqa onephutha.
• ret phuma isibambi.
• checktimeout hlola isikhathi sokuvala.

Environment

• VERBOSE=1 Sibonisa amaphutha esikrinini ngokushesha (stdout).
• SAVELOGSONSUCCES=1 gcina ilogi phezu kwempumelelo.
• INIT_REPO_IF_NOT_EXIST=1 Dala indawo yokugcina uma ingekho. Ikhutshazwe ngokuzenzakalela.
• TIMEOUT isikhathi esiphezulu sokusebenza okuyinhloko. Ungayisetha ngokuthi 'm', 'h' noma 'd' ekugcineni.

Imodi yesitoreji yamakhophi amadala. Okuzenzakalelayo:

• KEEP_DAILY=7
• KEEP_WEEKLY=4
• KEEP_MONTHLY=6

Okuguquguqukayo ngaphakathi kweskripthi

• ERROR_STRING - Uchungechunge lwephutha lokungena ngemvume.
• EXTRACT_ERROR_STRING — isisho seyunithi yezinhlamvu uma kunephutha.
• KILL_TIMEOUT_SIGNAL - isignali yokubulala uma isikhathi siphelile.
• TAIL — zingaki izintambo ezinamaphutha esibukweni.
• COLORMSG — umbala womlayezo (ophuzi okuzenzakalelayo).

Leso script, esibizwa nge-wordpress, sinegama elinemibandela, iqhinga laso ukuthi siphinde sisekele i-mysql database. Lokhu kusho ukuthi ingasetshenziselwa ukufakwa kwe-Nexcloud yenodi eyodwa, lapho ungase futhi wenze isipele sesizindalwazi. Okulula akukhona nje ukuthi yonke into isendaweni eyodwa, kodwa futhi okuqukethwe kwedatha kuseduze nokuqukethwe kwamafayela, njengoba umehluko wesikhathi uncane.

I-Restic vs Borg

Kukhona futhi ukuqhathanisa phakathi kwe-Borg ne-Restic lapha ku-Habre, futhi sasingenawo umsebenzi wokwenza omunye nje, kodwa owethu. Bekubalulekile kithi ukuthi izobukeka kanjani kudatha yethu, nemininingwane yethu. Siyaziletha.

Indlela yethu yokukhetha, ngaphezu kwalezo esezibaluliwe kakade (ukukhipha imali, ukutakula ngokushesha, njll.):

• Ukumelana nomsebenzi ongaqediwe. Hlola ukubulala -9.
• Usayizi kudiski.
• Isidingo sezinsiza (CPU, inkumbulo).
• Usayizi wama-blobs agciniwe.
• Ukusebenza ne-S3.
• Ukuhlola ubuqotho.

Ukuze sihlolwe, sithathe iklayenti elilodwa elinedatha yangempela kanye nosayizi ophelele ongu-1,6 TB.
Izimo.

U-Borg akazi ukuthi asebenze kanjani ngokuqondile ne-S3, futhi safaka idiski njenge-fuse, nge goofys. U-Restic uyithumele ku-S3 uqobo.

I-Goofys isebenza ngokushesha okukhulu futhi kahle, futhi kukhona imodyuli yenqolobane yediski, okusheshisa umsebenzi nakakhulu. Kusesigabeni se-beta, futhi, ngokungananazi, siphahlazeke ngokulahleka kwedatha phakathi nokuhlolwa (okunye). Kodwa okuhle ukuthi inqubo yokusekelayo ngokwayo ayidingi ukufunda okuningi, kodwa ikakhulukazi ukubhala, ngakho sisebenzisa inqolobane kuphela ngesikhathi sokuhlolwa kobuqotho.

Ukunciphisa ithonya lenethiwekhi, sasebenzisa umhlinzeki wendawo - Yandex Cloud.

Imiphumela yokuhlolwa kokuqhathanisa.

• Ukubulala -9 ngokuphinda uqalise kabusha zombili zibe yimpumelelo.
• Usayizi kudiski. I-Borg ingacindezela, ngakho-ke imiphumela ilindelekile.

Isipele
usayizi

Borg
I-562Gb

I-restic
I-628Gb

• Nge-CPU
  I-Borg ngokwayo idla kancane, ngokucindezelwa okuzenzakalelayo, kodwa kufanele ihlolwe kanye nenqubo ye-goofys. Sekukonke, ayaqhathaniswa futhi asebenzisa cishe ama-cores angu-1,2 emshinini ofanayo wokuhlola.
• Inkumbulo. I-Restic cishe i-0,5GB, i-Borg cishe i-200MB. Kodwa konke lokhu akubalulekile uma kuqhathaniswa nenqolobane yefayela lesistimu. Ngakho-ke kuhle ukwaba inkumbulo eyengeziwe.
• Umehluko ngosayizi we-blob wawumangalisa.

Isipele
usayizi

Borg
cishe 500MB

I-restic
cishe 5MB

• Okuhlangenwe nakho nge-Restic's S3 kuhle kakhulu. Ukusebenza no-Borg ngokusebenzisa ama-goofys akuphakamisi noma yimiphi imibuzo, kodwa kuye kwaphawulwa ukuthi kuyatuseka ukwehlisa ngemva kokuba isipele sesiqedile ukusetha kabusha inqolobane ngokuphelele. Okukhethekile kwe-S3 ukuthi izingcezu ezipompa ngaphansi ngeke zithunyelwe ebhakedeni, okusho ukuthi idatha egcwaliswe ngokungaphelele iholela emonakalweni omkhulu.
• Ukuhlolwa kobuqotho kusebenza kahle kuzo zombili izimo, kodwa isivinini sihluka kakhulu.
  I-Restic Amahora we-3,5.
  Borg, enenqolobane yefayela le-100GB SSD - Amahora we-5.Cishe umphumela wesivinini ofanayo uma idatha ikudiski yendawo.
  I-Borg ifunda ngokuqondile ku-S3 ngaphandle kwe-cache Amahora we-33. Mude ngokumangalisayo.

Okubalulekile ukuthi i-Borg ingaminyanisa futhi inama-blobs amakhulu - okwenza ukugcinwa kanye nokusebenza kwe-GET/PUT ku-S3 kushibhe. Kodwa lokhu kuza ngezindleko zokuqinisekisa okuyinkimbinkimbi nokunensa. Mayelana nesivinini sokululama, asibonanga mehluko. I-Restic ithatha ama-backups alandelayo (emva kokokuqala) isikhashana, kodwa hhayi kakhulu.

Okokugcina ekukhetheni ubukhulu bomphakathi.

Futhi sakhetha i-borg.

Amagama ambalwa mayelana nokucindezelwa

I-Borg ine-algorithm entsha yokucindezelwa okuhle kakhulu ku-arsenal yayo - zstd. Ikhwalithi yokucindezela ayimbi kune-gzip, kodwa ishesha kakhulu. Futhi iqhathaniswa ngesivinini ne-lz4 ezenzakalelayo.

Isibonelo, indawo yokulahla isizindalwazi se-MySQL icindezelwe izikhathi ezimbili kangcono kune-lz4 ngesivinini esifanayo. Kodwa-ke, isipiliyoni esinedatha yangempela sibonisa ukuthi kunomehluko omncane kakhulu esilinganisweni sokucindezela se-Nextcloud node.

I-Borg inemodi yokucindezela ibhonasi kunalokho - uma ifayela line-entropy ephezulu, khona-ke ukucindezelwa akusetshenzisiwe nhlobo, okwandisa isivinini. Inikwe amandla ngenketho uma udala
-C auto,zstd
ye-algorithm ye-zstd
Ngakho ngale nketho, uma kuqhathaniswa nokucindezelwa okuzenzakalelayo, sithole
I-560Gb kanye ne-562Gb ngokulandelanayo. Idatha evela kusibonelo esingenhla, ake ngikukhumbuze, ngaphandle kokucindezelwa umphumela ungu-628Gb. Umphumela womehluko we-2GB uke wasimangaza, kodwa sacabanga ukuthi sizowukhetha phela. auto,zstd.

Indlela yokuqinisekisa

Ngokusho komhleli, umshini obonakalayo wethulwa ngokuqondile kumhlinzeki noma kuklayenti, okunciphisa kakhulu umthwalo wenethiwekhi. Okungenani ishibhile kunokuzikhulisela wena nokushayela ithrafikhi.

goofys --cache "--free:5%:/mnt/cache" -o allow_other --endpoint https://storage.yandexcloud.net --file-mode=0666 --dir-mode=0777 xxxxxxx.com /mnt/goofys
export BORG_PASSCOMMAND="cat /home/borg/.borg-passphrase"
borg list /mnt/goofys/borg1/
borg check --debug -p --verify-data /mnt/goofys/borg1/

Sisebenzisa uhlelo olufanayo, sibheka amafayela nge-antivirus (ngemuva kweqiniso). Phela, abasebenzisi balayisha izinto ezahlukene ku-Nextcloud futhi akuwona wonke umuntu one-antivirus. Ukwenza izivivinyo ngesikhathi sokuthulula kuthatha isikhathi esiningi futhi kuphazamisa ibhizinisi.

I-scalability ifinyelelwa ngokugijima abagijimi kumanodi ahlukene anamathegi ahlukene.
Ukuqapha kwethu kuqoqa izimo eziyisipele nge-GitLab API efasiteleni elilodwa; uma kunesidingo, izinkinga zibonakala kalula futhi zenziwe ezasendaweni kalula.

isiphetho

Ngenxa yalokho, siyazi ngokuqinisekile ukuthi senza izipele, ukuthi izipele zethu zivumelekile, izinkinga eziphakama ngazo zithatha isikhathi esincane futhi zixazululwa ezingeni lomphathi womsebenzi. Izipele zithatha isikhala esincane kakhulu uma kuqhathaniswa ne-tar.gz noma i-Bacula.

Source: www.habr.com