Momwe GitLab imakuthandizireni kusunga zosungira zazikulu za NextCloud

Pa Habr!

Lero ndikufuna kulankhula za zomwe takumana nazo pakusunga zosunga zobwezeretsera zazikulu kuchokera ku Nextcloud storages mumasinthidwe osiyanasiyana. Ndimagwira ntchito ngati malo othandizira ku Molniya AK, komwe timakonza masinthidwe a IT; Nextcloud imagwiritsidwa ntchito posungira deta. Kuphatikizapo, ndi dongosolo logawidwa, ndi redundancy.

Mavuto omwe amabwera chifukwa cha mawonekedwe oyikapo ndikuti pali zambiri. Kusintha koperekedwa ndi Nextcloud, redundancy, zifukwa zodziwikiratu, ndi zina zimapanga zobwereza zambiri.

prehistory

Mukamapereka Nextcloud, vuto lokonzekera zosunga zobwezeretsera limakhala, lomwe liyenera kubisidwa, popeza deta ndiyofunikira.

Timapereka zosankha zosunga zosunga zobwezeretsera kwathu kapena kwa kasitomala pamakina osiyana kuchokera ku Nextcloud, zomwe zimafuna njira yosinthika yokhazikika pakuwongolera.

Pali makasitomala ambiri, onse ali ndi masinthidwe osiyanasiyana, ndipo onse pamasamba awo komanso mawonekedwe awo. Iyi ndi njira yokhazikika pomwe tsamba lonseli ndi lanu, ndipo zosunga zobwezeretsera zimapangidwa kuchokera ku korona; sizikukwanira bwino.

Choyamba, tiyeni tiwone zomwe zalowa. Tikufuna:

• Scalability malinga ndi mfundo imodzi kapena zingapo. Pamakhazikitsidwe akuluakulu timagwiritsa ntchito minio ngati yosungirako.
• Dziwani zamavuto ndikuchita ma backups.
• Muyenera kusunga zosunga zobwezeretsera ndi makasitomala anu ndi/kapena nafe.
• Yang'anani ndi zovuta mwachangu komanso mosavuta.
• Makasitomala ndi makhazikitsidwe amasiyana kwambiri wina ndi mnzake - kufanana sikungakwaniritsidwe.
• Kuthamanga kuchira kuyenera kukhala kochepa muzochitika ziwiri: kuchira kwathunthu (tsoka), chikwatu chimodzi chofufutidwa molakwika.
• Ntchito yochotsa ndiyofunikira.

Kuti tithane ndi vuto lakusamalira zosunga zobwezeretsera, tidayika GitLab. Zambiri mwa tackle.

Zoonadi, sife oyamba kuthetsa vuto loterolo, koma zikuwoneka kwa ife kuti zochitika zathu zothandiza, zomwe tapeza movutikira zingakhale zosangalatsa ndipo ndife okonzeka kugawana nawo.

Popeza kampani yathu ili ndi ndondomeko yotseguka, tinali kuyang'ana njira yotsegula. Kenako, timagawana zomwe tikuchita ndikuzitumiza. Mwachitsanzo, pa GitHub pali pulogalamu yowonjezera yathu ya Nextcloud, zomwe timapereka kwa makasitomala, kupititsa patsogolo chitetezo cha data pakachotsedwa mwangozi kapena mwadala.

Zida zosunga zobwezeretsera

Tinayamba kusaka kwathu njira zothetsera posankha chida chopangira zosunga zobwezeretsera.

Nthawi zonse tar + gzip siigwira ntchito bwino - deta imabwerezedwa. Zowonjezera nthawi zambiri zimakhala ndi zosintha zenizeni zochepa, ndipo zambiri zomwe zili mkati mwa fayilo imodzi zimabwerezedwa.
Palinso vuto lina - redundancy ya kugawa deta yosungirako. Timagwiritsa ntchito minio ndipo zambiri zake ndizosowa. Kapena mumayenera kupanga zosunga zobwezeretsera kudzera mu minio yokha - kuyiyika ndikugwiritsa ntchito ma spacers onse pakati pa fayilo, ndipo, mocheperapo, pali chiopsezo choyiwala za zidebe zina ndi chidziwitso cha meta. Kapena gwiritsani ntchito deduplication.

Zida zosunga zobwezeretsera zomwe zili ndi kubwereza zimapezeka poyera (pa Habré panali zolemba za mutuwu) ndipo omaliza athu anali Borg и Zoletsa. Kuyerekeza kwathu kwa mapulogalamu awiriwa kuli pansipa, koma pakadali pano tikuuzani momwe tidakonzera dongosolo lonselo.

Kuwongolera ma backups

Borg ndi Restic ndi zabwino, koma palibe mankhwala omwe ali ndi makina owongolera. Ndi cholinga cha kasamalidwe ndi kulamulira, tinasankha chida chomwe takhazikitsa kale, popanda zomwe sitingathe kulingalira ntchito yathu, kuphatikizapo automation - iyi ndi CI / CD yodziwika bwino - GitLab.

Lingaliro ndi motere: gitlab-runner imayikidwa pa node iliyonse kusunga deta ya Nextcloud. Wothamanga amayendetsa script pa ndandanda yomwe imayang'anira zosunga zobwezeretsera, ndipo imayambitsa Borg kapena Restic.

Tinapeza chiyani? Ndemanga kuchokera pakuchita, kuwongolera kosavuta pakusintha, tsatanetsatane pakachitika cholakwika.

pano apa pa GitHub tinatumiza zitsanzo za zolemba za ntchito zosiyanasiyana, ndipo tinamaliza kuziyika ku zosunga zobwezeretsera osati Nextcloud, komanso mautumiki ena ambiri. Palinso ndandanda pamenepo ngati simukufuna kuyikonza pamanja (ndipo sitikufuna) ndi .gitlab-ci.yml

Palibe njira yosinthira nthawi ya CI / CD mu Gitlab API pano, koma ndi yaying'ono. Iyenera kuwonjezeredwa, kunena 1d.

GitLab, mwamwayi, ikhoza kuyambitsa osati mongodzipereka, koma molingana ndi ndandanda, izi ndizomwe tikufuna.

Tsopano za script wrapper.

Tidakhazikitsa zotsatirazi pazolemba izi:

• Iyenera kuyambitsidwa ndi wothamanga komanso ndi dzanja kuchokera ku console ndi ntchito zomwezo.
• Payenera kukhala zowongolera zolakwika:
• kodi kubwerera.
• fufuzani chingwe mu chipika. Mwachitsanzo, kwa ife cholakwika chikhoza kukhala uthenga womwe pulogalamuyo siiwona ngati yakupha.
• Kukonza nthawi kutha. Nthawi yotsogolera iyenera kukhala yokwanira.
• Tikufuna chipika chatsatanetsatane. Koma pokhapokha pakakhala cholakwika.
• Mayeso angapo amachitidwanso asanayambe.
• Mabonasi ang'onoang'ono osavuta omwe tidapeza othandiza panthawi yothandizira:
• Chiyambi ndi mapeto amalembedwa mu syslog ya makina am'deralo. Izi zimathandiza kulumikiza zolakwika za dongosolo ndi ntchito zosunga zobwezeretsera.
• Gawo la chipika cholakwa, ngati chilipo, chimachokera ku stdout, chipika chonsecho chimalembedwa ku fayilo yosiyana. Ndikwabwino kuyang'ana CI nthawi yomweyo ndikuwunika zolakwika ngati ndizochepa.
• Debugging modes.

chipika chonsecho chimasungidwa ngati chopangidwa mu GitLab; ngati palibe cholakwika, chipikacho chimachotsedwa. Timalemba script mu bash.

Tidzakhala okondwa kulingalira malingaliro ndi ndemanga zilizonse zokhudzana ndi gwero lotseguka - kulandiridwa.

Kodi ntchito

Wothamanga yemwe ali ndi Bash executor amayambitsidwa pa malo osungira. Malinga ndi ndandanda, ntchito CI/CD imayambitsidwa mu mpiru wapadera. Wothamanga amayambitsa script ya chilengedwe chonse cha ntchito zoterezi, imayang'ana kutsimikizika kwa malo osungirako zosungirako, malo okwera ndi chirichonse chomwe tikufuna, kenako ndikubwezeretsanso ndikuyeretsa chakale. Kusunga komaliza komweko kumatumizidwa ku S3.

Timagwira ntchito molingana ndi chiwembu ichi - ndi wopereka AWS wakunja kapena wofanana waku Russia (ndiwofulumira komanso zomwe sizikuchokera ku Russian Federation). Kapena timayika gulu losiyana la minio kwa kasitomala patsamba lake pazifukwa izi. Nthawi zambiri timachita izi pazifukwa zachitetezo, pomwe kasitomala safuna kuti deta ichoke mdera lawo konse.

Sitinagwiritse ntchito potumiza zosunga zobwezeretsera kudzera pa ssh. Izi sizikuwonjezera chitetezo, ndipo mphamvu za netiweki za S3 ndizokwera kwambiri kuposa makina athu amodzi a ssh.

Pofuna kuteteza makina anu am'deralo kuchokera kwa owononga, popeza akhoza kuchotsa deta pa S3, muyenera kuloleza kumasulira.
Zosunga zobwezeretsera nthawi zonse zimasunga zosunga zobwezeretsera.

Borg ili ndi njira yosasinthika none, koma sitikulangiza mwamphamvu kuyatsa. Munjira iyi, sipadzakhala kokha kubisa, koma checksum ya zomwe zikulembedwa sizinawerengedwe, zomwe zikutanthauza kuti umphumphu ukhoza kufufuzidwa molakwika, pogwiritsa ntchito ma index.

Wosankha wina amawunika zosunga zobwezeretsera kuti awonetse kukhulupirika kwa ma index ndi zomwe zili. Chekeyo imachedwa komanso yayitali, kotero timayiyendetsa padera kamodzi pamwezi. Zitha kutenga masiku angapo.

Readme in Russian

Ntchito zazikulu

• prepare kukonzekera
• testcheck kukonzekera cheke
• maincommand core team
• forcepostscript ntchito yomwe imachitidwa pamapeto kapena molakwika. Timagwiritsa ntchito kutsitsa magawo.

Ntchito zautumiki

• cleanup Timalemba zolakwika kapena kufufuta fayilo ya chipika.
• checklog fotokozani chipikacho kuti mzere ukhale ndi cholakwika.
• ret tulukani chogwirira.
• checktimeout fufuzani nthawi.

Environment

• VERBOSE=1 Timawonetsa zolakwika pazenera nthawi yomweyo (stdout).
• SAVELOGSONSUCCES=1 sunga chipika pa kupambana.
• INIT_REPO_IF_NOT_EXIST=1 Pangani nkhokwe ngati kulibe. Zayimitsidwa mwachisawawa.
• TIMEOUT nthawi yochuluka ya ntchito yaikulu. Mutha kuyiyika ngati 'm', 'h' kapena 'd' pamapeto.

Momwe mungasungire makope akale. Zofikira:

• KEEP_DAILY=7
• KEEP_WEEKLY=4
• KEEP_MONTHLY=6

Zosintha mkati mwa script

• ERROR_STRING - chingwe cha cheke cholembera cholakwika.
• EXTRACT_ERROR_STRING - Kufotokozera kwa chingwe chowonetsera ngati cholakwika.
• KILL_TIMEOUT_SIGNAL - chizindikiro chakupha ngati nthawi yatha.
• TAIL - ndi zingwe zingati zomwe zili ndi zolakwika pazenera.
• COLORMSG - mtundu wa uthenga (wosasinthika wachikasu).

Script imeneyo, yomwe imatchedwa wordpress, ili ndi dzina lokhazikika, chinyengo chake ndikuti imathandiziranso database ya mysql. Izi zikutanthauza kuti zitha kugwiritsidwa ntchito pakuyika kwa Nexcloud kwa node imodzi, komwe mutha kusungitsanso database. Zosavuta sikuti zonse zili pamalo amodzi, komanso zomwe zili m'dawunilodi zili pafupi ndi zomwe zili m'mafayilo, popeza kusiyana kwa nthawi kumakhala kochepa.

Restic vs Borg

Palinso kufananitsa pakati pa Borg ndi Restic apa pa Habre, ndipo tinalibe ntchito yongopanga ina, koma yathu yathu. Zinali zofunikira kwa ife momwe zingawonekere pa data yathu, ndi zenizeni zathu. Timawabweretsa.

Zosankha zathu, kuphatikiza zomwe zatchulidwa kale (kuchotsa, kuchira mwachangu, ndi zina zotero):

• Kukana ntchito yosamalizidwa. Onani kupha -9.
• Kukula pa disk.
• Zofunikira pazachuma (CPU, kukumbukira).
• Kukula kwa mabulosi osungidwa.
• Kugwira ntchito ndi S3.
• Kuwona kukhulupirika.

Kuti tiyesedwe, tinatenga kasitomala mmodzi wokhala ndi deta yeniyeni komanso kukula kwake kwa 1,6 TB.
Zoyenera.

Borg sadziwa mmene ntchito mwachindunji ndi S3, ndipo ife wokwera litayamba monga fusesi, kudzera goofys. Restic adatumiza ku S3 yokha.

Goofys amagwira ntchito mwachangu komanso bwino, ndipo alipo disk cache module, zomwe zimafulumizitsa ntchitoyo kwambiri. Ili pagawo la beta, ndipo, kunena zoona, tidawonongeka ndikutayika kwa data pakuyesa (ena). Koma chosavuta ndichakuti njira yosunga zobwezeretsera yokha sikufuna kuwerenga kwambiri, koma makamaka kulemba, chifukwa chake timagwiritsa ntchito posungira pokha pofufuza kukhulupirika.

Kuti tichepetse mphamvu ya intaneti, tidagwiritsa ntchito wothandizira wamba - Yandex Cloud.

Kuyerekeza zotsatira zoyesa.

• Kill -9 ndikuyambiranso kwina onse adapambana.
• Kukula pa disk. Borg akhoza compress, kotero zotsatira zake ndi kuyembekezera.

Backuper
kukula

Borg
562Gb

Zoletsa
628Gb

• Pa CPU
  Borg yokha imadya pang'ono, ndi kuponderezedwa kosasintha, koma iyenera kuyesedwa pamodzi ndi ndondomeko ya goofys. Pazonse, amafanana ndipo amagwiritsa ntchito pafupifupi 1,2 cores pamakina omwewo amayeso.
• Memory. Restic ndi pafupifupi 0,5GB, Borg ndi pafupifupi 200MB. Koma zonsezi ndizochepa poyerekeza ndi cache ya fayilo ya dongosolo. Choncho m'pofunika kugawa zambiri kukumbukira.
• Kusiyana kwa kukula kwa blob kunali kochititsa chidwi.

Backuper
kukula

Borg
pafupifupi 500MB

Zoletsa
pafupifupi 5MB

• Zochitika ndi Restic's S3 ndizabwino kwambiri. Kugwira ntchito ndi Borg kudzera ku goofys sikudzutsa mafunso, koma zadziwika kuti ndikofunikira kuti muchotse zosunga zobwezeretserazo kuti mukhazikitsenso cache. Chodabwitsa cha S3 ndikuti ma chunks osapopedwa sangatumizidwe ku chidebe, zomwe zikutanthauza kuti kusadzaza deta kumabweretsa kuwonongeka kwakukulu.
• Cheke chaumphumphu chimagwira ntchito bwino pazochitika zonsezi, koma liwiro limasiyana kwambiri.
  Wokhazikika Maola 3,5.
  Borg, yokhala ndi 100GB SSD file cache - Maola 5.Pafupifupi zotsatira zothamanga zomwezo ngati deta ili pa disk yapafupi.
  Borg amawerenga mwachindunji kuchokera ku S3 popanda posungira Maola 33. Mochititsa mantha.

Chofunikira ndichakuti Borg imatha kufinya ndipo imakhala ndi mabulosi okulirapo - zomwe zimapangitsa kuti ntchito zosungirako ndi GET/PUT mu S3 zikhale zotsika mtengo. Koma izi zimabwera pamtengo wotsimikizika wovuta komanso wocheperako. Ponena za liwiro la kuchira, sitinazindikire kusiyana kulikonse. Restic imatenga zosunga zobwezeretsera (pambuyo poyambirira) motalikirapo, koma osati kwambiri.

Chomaliza koma chocheperako pakusankha chinali kukula kwa anthu ammudzi.

Ndipo tinasankha borg.

Mawu ochepa za compression

Borg ili ndi algorithm yatsopano yophatikizira mu zida zake - zstd. Kuphatikizika kwabwino sikuli koyipa kuposa gzip, koma mwachangu kwambiri. Ndipo kufananizidwa mwachangu ndi lz4 yosasinthika.

Mwachitsanzo, kutayira kwa database ya MySQL kumapanikizidwa kawiri kuposa lz4 pa liwiro lomwelo. Komabe, zokumana nazo ndi deta zenizeni zikuwonetsa kuti pali kusiyana kochepa kwambiri pakuponderezana kwa node ya Nextcloud.

Borg ili ndi mawonekedwe a bonasi - ngati fayiloyo ili ndi entropy yayikulu, ndiye kuti kuponderezana sikumagwiritsidwa ntchito konse, komwe kumawonjezera liwiro. Yathandizidwa ndi mwayi popanga
-C auto,zstd
kwa algorithm ya zstd
Chifukwa chake ndi njira iyi, poyerekeza ndi kusakhazikika kokhazikika, tili ndi
560Gb ndi 562Gb motsatana. Zomwe zili m'chitsanzo pamwambapa, ndiloleni ndikukumbutseni, popanda kukanikiza zotsatira zake ndi 628Gb. Zotsatira za kusiyana kwa 2GB zidatidabwitsa, koma tidaganiza kuti tingasankhe. auto,zstd.

Njira yotsimikizira zosunga zobwezeretsera

Malinga ndi ndandanda, makina enieni amayambitsidwa mwachindunji kuchokera kwa wothandizira kapena kwa kasitomala, zomwe zimachepetsa kwambiri katundu wa netiweki. Osachepera ndizotsika mtengo kuposa kudzikweza nokha ndikuyendetsa magalimoto.

goofys --cache "--free:5%:/mnt/cache" -o allow_other --endpoint https://storage.yandexcloud.net --file-mode=0666 --dir-mode=0777 xxxxxxx.com /mnt/goofys
export BORG_PASSCOMMAND="cat /home/borg/.borg-passphrase"
borg list /mnt/goofys/borg1/
borg check --debug -p --verify-data /mnt/goofys/borg1/

Pogwiritsa ntchito chiwembu chomwecho, timayang'ana mafayilo ndi antivayirasi (pambuyo pake). Kupatula apo, ogwiritsa ntchito amakweza zinthu zosiyanasiyana ku Nextcloud ndipo si aliyense amene ali ndi antivayirasi. Kuchita kuyendera panthawi yothira kumatenga nthawi yambiri ndikusokoneza bizinesi.

Scalability imatheka pothamanga othamanga pama node osiyanasiyana okhala ndi ma tag osiyanasiyana.
Kuyang'anira kwathu kumasonkhanitsa zidziwitso zosunga zobwezeretsera kudzera pa GitLab API pawindo limodzi; ngati kuli kofunikira, zovuta zimazindikirika mosavuta komanso kupezeka komweko.

Pomaliza

Chotsatira chake, tikudziwa motsimikiza kuti timapanga zosunga zobwezeretsera, kuti zosunga zobwezeretsera zathu ndizovomerezeka, mavuto omwe amabwera nawo amatenga nthawi yochepa ndipo amathetsedwa pamlingo wa woyang'anira ntchito. Zosunga zobwezeretsera zimatenga malo ochepa poyerekeza ndi tar.gz kapena Bacula.

Source: www.habr.com