GitLab inokubatsira sei kuchengetedza yakakura NextCloud storages

Hei Habr!

Nhasi ndinoda kutaura nezve chiitiko chedu mukuita otomatiki backup yedata hombe kubva kuNextcloud storages mune akasiyana masisitimu. Ini ndinoshanda sesevhisi chiteshi kuMolniya AK, kwatinoita kumisikidza manejimendi eIT masisitimu; Nextcloud inoshandiswa kuchengetedza data. Kusanganisira, neyakagoverwa chimiro, ine redundancy.

Matambudziko anomuka kubva kune zvimiro zvezvimisikidzo ndezvekuti kune data rakawanda. Shanduro yakapihwa neNextcloud, redundancy, subjective zvikonzero, uye nezvimwe zvinogadzira akawanda makopi.

prehistory

Paunenge uchitonga Nextcloud, dambudziko rekuronga backup inoshanda inomuka, iyo inofanirwa kuvharirwa, sezvo iyo data yakakosha.

Isu tinopa sarudzo dzekuchengeta ma backups panzvimbo yedu kana kumutengi pamakina akasiyana kubva kuNextcloud, izvo zvinoda inoshanduka otomatiki maitiro ekutonga.

Kune vatengi vazhinji, vese vane masisitimu akasiyana, uye ese ari pasaiti yavo uye nehunhu hwavo. Iyi inzira yakajairwa kana saiti yese iri yako, uye ma backups anogadzirwa kubva kumakorona; haikwane mushe.

Kutanga, ngatitarisei data rekuisa. Tinoda:

• Scalability maererano neimwe node kana akati wandei. Pakuisa kukuru tinoshandisa minio sekuchengetedza.
• Tsvaga nezve matambudziko nekuita backups.
• Iwe unofanirwa kuchengeta backup nevatengi vako uye / kana nesu.
• Kugadzirisa matambudziko nekukurumidza uye nyore.
• Vatengi uye kuiswa kwakasiyana zvakanyanya kubva kune mumwe nemumwe - kufanana hakugone kuwanikwa.
• Iko kumhanya kudzoreredza kunofanirwa kuve kushoma muzviitiko zviviri: kudzoreredza kuzere (njodzi), imwe folda yakadzimwa nekukanganisa.
• Deduplication basa rinodiwa.

Kugadzirisa dambudziko rekugadzirisa ma backups, takaisa GitLab. Mamwe mashoko ne tackle.

Zvechokwadi, isu hatisi vekutanga kugadzirisa dambudziko rakadaro, asi zvinoratidzika kwatiri kuti ruzivo rwedu runoshanda, rwakawana zvakaoma runogona kufadza uye takagadzirira kugovera.

Sezvo kambani yedu ine yakavhurika sosi mutemo, isu taitsvaga yakavhurika sosi mhinduro. Nesuwo, isu tinogovana zviitiko zvedu uye tinozvitumira. Semuenzaniso, paGitHub pane yedu plugin yeNextcloud, iyo yatinopa kune vatengi, inosimudzira kuchengetedzeka kwedata kana netsaona kana nemaune kudzima.

Zvishandiso zvekuchengetedza

Takatanga kutsvaga kwedu nzira dzekugadzirisa nekusarudza chishandiso chekugadzira chekuchengetedza.

Yenguva dzose tar + gzip haishande zvakanaka - iyo data inodzokororwa. Kuwedzera kunowanzo kuve neshanduko shoma shoma, uye yakawanda yedata mukati mefaira rimwe inodzokororwa.
Pane rimwe dambudziko - redundancy yekuparadzira data data. Isu tinoshandisa minio uye data yayo haina basa. Kana kuti iwe waifanira kuita backup kuburikidza ne minio pachayo - itakure uye shandisa ese ma spacers pakati peiyo faira system, uye, zvisinganyanyi kukosha, kune njodzi yekukanganwa nezve mamwe mabhakiti uye meta-ruzivo. Kana kushandisa deduplication.

Maturusi ekuchengetedza ane dudziro anowanikwa mune yakavhurika sosi (paHabré pakanga paine zvinyorwa nezvedingindira iri) uye vedu vakapedzisa vaive Borg и Restic. Kuenzanisa kwedu kwezvikumbiro zviviri zviri pazasi, asi parizvino tichakuudza kuti takaronga sei chirongwa chese.

Kugadzirisa backups

Borg uye Restic yakanaka, asi hapana chigadzirwa chine centralized control mechanism. Nechinangwa chekutungamira uye kutonga, takasarudza chishandiso chatakatoshandisa, pasina icho isu hatigone kufungidzira basa redu, kusanganisira otomatiki - iyi ndiyo inonyanya kuzivikanwa CI/CD - GitLab.

Pfungwa yacho ndeiyi inotevera: gitlab-runner yakaiswa pane imwe neimwe node kuchengetedza Nextcloud data. Mumhanyi anomhanyisa script pane chirongwa chinotarisisa maitiro ekuchengetedza, uye inotangisa Borg kana Restic.

Takawanei? Mhinduro kubva pakuitwa, kutonga kuri nyore pane shanduko, ruzivo kana paine chikanganiso.

pano pano paGitHub takatumira mienzaniso yezvinyorwa zvemabasa akasiyana-siyana, uye takazopedzisira tazvibatanidza nekuchengetedza kwete chete Nextcloud, asiwo mamwe akawanda masevhisi. Panewo anoronga ipapo kana usingadi kuigadzirisa nemaoko (uye isu hatidi) uye .gitlab-ci.yml

Iko hakuna nzira yekushandura iyo CI / CD nguva yekubuda muGitlab API parizvino, asi idiki. Inoda kuwedzerwa, toti 1d.

GitLab, nerombo rakanaka, inogona kuvhura kwete chete maererano nekuzvipira, asi chete zvinoenderana nehurongwa, izvi ndizvo chaizvo zvatinoda.

Iye zvino nezve wrapper script.

Isu tinoseta anotevera mamiriro eiyi script:

• Inofanirwa kutangwa zvese nemumhanyi uye neruoko kubva kuconsole ine mashandiro akafanana.
• Panofanira kunge paine vanobata kukanganisa:
• return code.
• tsvaga tambo mulog. Semuenzaniso, kwatiri kukanganisa kunogona kunge iri meseji iyo chirongwa hachioni sechinouraya.
• Kugadzirisa nguva yekupera. Nguva yekugadzirira inofanira kuva inonzwisisika.
• Tinoda danda rakadzama. Asi chete kana pane kukanganisa.
• Maedzo akati wandei anoitwa asati atanga.
• Mabhonasi madiki ekurerutsira atakawana anobatsira panguva yekutsigira maitiro:
• Kutanga uye kupera kwakanyorwa mu syslog yemuchina wemuno. Izvi zvinobatsira kubatanidza zvikanganiso zvehurongwa uye kuchengetedza kushanda.
• Chikamu chegiyo chekukanganisa, kana chiripo, chinoburitswa kune stdout, irogi rese rinonyorerwa kune rakaparadzana faira. Zviri nyore kuti ungotarisa CI uye woongorora chikanganiso kana chiri chidiki.
• Debugging modes.

Iyo yakazara log inochengetwa seyakagadzirwa muGitLab; kana pasina kukanganisa, irogi rinobviswa. Isu tinonyora script mu bash.

Isu tichafara kufunga chero mazano uye makomendi maererano neakavhurika sosi - kugamuchirwa.

Sei basa iri

Mumhanyi ane Bash executor anotangwa pane backup node. Sekureva kweanoronga, basa CI/CD rinotangwa mune yakakosha turnip. Mumhanyi anotangisa universal wrapper script yemabasa akadaro, inotarisa kutendeseka kweiyo backup repository, mapoinzi uye zvese zvatinoda, wozodzosera kumusoro uye kuchenesa yekare. Iyo yakapedzwa backup pachayo inotumirwa kuS3.

Isu tinoshanda zvinoenderana nechirongwa ichi - ndeyekunze AWS mupi kana yeRussia yakaenzana (inokurumidza uye data haibvi muRussian Federation). Kana isu tinoisa yakaparadzana minio cluster yemutengi pane yake saiti nekuda kweizvi zvinangwa. Isu tinowanzo tinoita izvi nekuda kwezvikonzero zvekuchengetedza, kana mutengi asingadi kuti data risiye dunhu ravo zvachose.

Hatina kushandisa chimiro chekutumira backup kuburikidza nessh. Izvi hazviwedzere kuchengetedzeka, uye netiweki kugona kweS3 mupi akakwira zvakanyanya kupfuura yedu imwe ssh muchina.

Kuti udzivirire muchina wako wemuno kubva kune hacker, sezvo achigona kudzima data paS3, unofanirwa kugonesa shanduro.
Iyo backup inogara ichivharira iyo backup.

Borg ine uncrypted mode none, asi isu hatikurudzire kuibatidza. Mune iyi modhi, kwete chete pachave pasina encryption, asi cheki yezviri kunyorwa haina kuverengerwa, zvinoreva kuti kuperera kunogona kungotariswa zvisina kunanga, uchishandisa indexes.

Imwe yakaparadzana scheduler inotarisa backups yekuvimbika kwema indexes uye zvirimo. Cheki inononoka uye yakareba, saka tinoimhanya zvakasiyana kamwe pamwedzi. Zvinogona kutora mazuva akati wandei.

Readme in Russian

Mabasa makuru

• prepare kudzidziswa
• testcheck readyness check
• maincommand core team
• forcepostscript basa rinoitwa pakupedzisira kana nekukanganisa. Isu tinoishandisa kudzikisa chikamu.

Sevhisi mabasa

• cleanup Isu tinorekodha zvikanganiso kana kudzima iyo log file.
• checklog nananisa irogi kuitira kuitika kwemutsara uine kukanganisa.
• ret exit handler.
• checktimeout tarisa nguva.

mhepo mvura nenzvimbo

• VERBOSE=1 Isu tinoratidza zvikanganiso pachiratidziro nekukasira (stdout).
• SAVELOGSONSUCCES=1 chengetedza danda pamusoro pekubudirira.
• INIT_REPO_IF_NOT_EXIST=1 Gadzira repository kana isipo. Yakavharwa nekusingaperi.
• TIMEOUT nguva yakawanda yekushanda kukuru. Unogona kuiisa se 'm', 'h' kana 'd' pakupedzisira.

Kuchengetedza maitiro emakopi ekare. Default:

• KEEP_DAILY=7
• KEEP_WEEKLY=4
• KEEP_MONTHLY=6

Kusiyana mukati mezvinyorwa

• ERROR_STRING - tambo yecheki yelogi yekukanganisa.
• EXTRACT_ERROR_STRING - kutaura kwetambo yekuratidzira kana kukanganisa.
• KILL_TIMEOUT_SIGNAL - chiratidzo chekuuraya kana nguva yapera.
• TAIL - matani matambo ane zvikanganiso pachiratidziri.
• COLORMSG - ruvara rwemeseji (default yero).

Iyo script, iyo inonzi wordpress, ine zita rine mamiriro, hunyengeri hwayo ndeyekuti zvakare inodzosera iyo mysql dhatabhesi. Izvi zvinoreva kuti inogona kushandiswa kune imwe-node Nexcloud kumisikidzwa, kwaunogona zvakare kutsigira dhatabhesi. Izvo zviri nyore hazvisi chete kuti zvese zviri munzvimbo imwe chete, asiwo zviri mukati medhatabhesi zviri padyo nezviri mumafaira, sezvo musiyano wenguva uri mudiki.

Restic vs Borg

Kune zvakare kuenzanisa pakati peBorg neRestic pano paHabré, uye takanga tisina basa rokungogadzira rimwe chete, asi redu pachedu. Zvaive zvakakosha kwatiri kuti yaizotaridzika sei pane yedu data, nemaitiro edu. Tinovaunza.

Maitiro edu ekusarudza, kuwedzera kune ayo atotaurwa (kubvisa, kukurumidza kupora, nezvimwewo):

• Kupokana nebasa risina kupera. Tarisa kuuraya -9.
• Saizi pa diski.
• Zvinodiwa zvezviwanikwa (CPU, ndangariro).
• Saizi yemabhubhu akachengetwa.
• Kushanda neS3.
• Kutendeseka cheki.

Kuongororwa, takatora mutengi mumwe aine data chaiyo uye saizi yakazara ye1,6 TB.
Conditions.

Borg haazive kushanda zvakananga neS3, uye isu takaisa dhisiki se fuse, kuburikidza goofys. Restic yakatumira kuS3 pachayo.

Goofys inoshanda nekukurumidza uye zvakanaka, uye iripo disk cache module, izvo zvinokurumidzisa basa zvakanyanya. Iri muchikamu chebeta, uye, kutaura chokwadi, takapunzika nekurasikirwa nedata panguva yekuyedzwa (vamwe). Asi zviri nyore ndezvekuti iyo backup maitiro pachayo haidi kuverenga kwakawanda, asi kunyanya kunyora, saka isu tinoshandisa cache chete panguva yekutendeseka cheki.

Kuti tideredze simba retiweki, takashandisa mupi wepanzvimbo - Yandex Cloud.

Kuenzanisa bvunzo mhinduro.

• Kuuraya -9 nekumwe kutangazve zvese zvakabudirira.
• Saizi pa diski. Borg inogona kumanikidza, saka mhedzisiro inotarisirwa.

Backuper
zera

Borg
562GB

Restic
628GB

• Kubva CPU
  Borg pachayo inodya zvishoma, ine default compression, asi inofanirwa kuongororwa pamwe chete neiyo goofys maitiro. Pakazara, iwo anofananidzwa uye anoshandisa anenge 1,2 cores pamushini wakafanana webvunzo chaiwo.
• Memory. Restic inenge 0,5GB, Borg inenge 200MB. Asi izvi zvese hazvina kukosha kana zvichienzaniswa neye system file cache. Saka zvinokurudzirwa kugovera imwe ndangariro.
• Musiyano muhukuru hwebhurobhu waishamisa.

Backuper
zera

Borg
anenge 500MB

Restic
anenge 5MB

• Chiitiko neRestic's S3 chakanakisa. Kushanda naBorg kuburikidza negoofys hakumutsi chero mibvunzo, asi zvakacherechedzwa kuti zvinokurudzirwa kuita umount mushure mekunge backup yapera kuti ugadzirise zvachose cache. Iyo peculiarity yeS3 ndeyekuti under-pumped chunks haizombofa yakatumirwa kubhaketi, izvo zvinoreva kuti isina kukwana yakazara data inotungamira mukukuvara kukuru.
• Kutendeseka cheki kunoshanda zvakanaka mune ese ari maviri, asi kumhanya kunosiyana zvakanyanya.
  Restic 3,5 maawa.
  Borg, ine 100GB SSD faira cache - 5 maawa.Inenge yakafanana nekumhanya mhedzisiro kana data iri pane dhisiki renzvimbo.
  Borg inoverenga zvakananga kubva kuS3 isina cache 33 maawa. Nguva yakareba kwazvo.

Chinonyanya kukosha ndechekuti Borg inogona kumanikidza uye ine mabhurobhu akakura - anoita kuti kuchengetedza uye GET/PUT mashandiro muS3 adhure. Asi izvi zvinouya pamutengo wekuomesesa uye kunonoka kwechokwadi. Kana zviri zvekukurumidza kupora, hatina kuona musiyano. Restic inotora anotevera backups (mushure mekutanga) zvishoma, asi kwete zvakanyanya.

Chekupedzisira asi chisiri chidiki pasarudzo yaive saizi yenharaunda.

Uye takasarudza borg.

Mazwi mashoma nezve compression

Borg ine yakanakisa compression algorithm mune yayo arsenal - zstd. Iyo yekumanikidza mhando haina kuipa kupfuura gzip, asi nekukurumidza zvakanyanya. Uye inofananidzwa nekumhanya kune yakasarudzika lz4.

Semuenzaniso, MySQL dhatabhesi dump inomanikidzwa kaviri zvirinani pane lz4 nekumhanya kwakafanana. Nekudaro, chiitiko chine data chaiyo chinoratidza kuti pane musiyano mudiki muhuwandu hwekumanikidza weNextcloud node.

Borg ine bhonasi yekumanikidza modhi - kana iyo faira ine yakakwira entropy, ipapo compression haina kuiswa zvachose, iyo inowedzera kumhanya. Inogoneswa nesarudzo paunenge uchigadzira
-C auto,zstd
ye zstd algorithm
Saka neiyi sarudzo, mukuenzanisa neiyo default compression, isu tine
560Gb uye 562Gb zvichiteerana. Iyo data kubva kumuenzaniso uri pamusoro, rega ndikuyeuchidze, pasina kumanikidza mhedzisiro ndeye 628Gb. Mhedzisiro yemusiyano we2GB yakatishamisa, asi isu takafunga kuti taizozvisarudza mushure mezvose. auto,zstd.

Backup yekuongorora nzira

Sekureva kwemugadziri, iwo chaiwo muchina unotangwa zvakananga kubva kune mupi kana kubva kumutengi, izvo zvinoderedza zvakanyanya mutoro wetiweki. Zvirinani zvakachipa pane kuzvisimudza iwe uye kutyaira traffic.

goofys --cache "--free:5%:/mnt/cache" -o allow_other --endpoint https://storage.yandexcloud.net --file-mode=0666 --dir-mode=0777 xxxxxxx.com /mnt/goofys
export BORG_PASSCOMMAND="cat /home/borg/.borg-passphrase"
borg list /mnt/goofys/borg1/
borg check --debug -p --verify-data /mnt/goofys/borg1/

Tichishandisa chirongwa chimwechete, tinotarisa mafaira ane antivirus (mushure mechokwadi). Mushure mezvose, vashandisi vanorodha zvinhu zvakasiyana kuNextcloud uye havasi vese vane antivirus. Kuita ongororo panguva yekudurura kunotora nguva yakawandisa uye kunokanganisa bhizinesi.

Scalability inowanikwa nekumhanya vanomhanya pane dzakasiyana node dzine ma tag akasiyana.
Yedu yekutarisa inounganidza masisitimu ekuchengetedza kuburikidza neGitLab API muhwindo rimwe; kana zvichidikanwa, matambudziko anoonekwa zviri nyore uye nyore nyore kuwanikwa.

mhedziso

Nekuda kweizvozvo, isu tinoziva zvechokwadi kuti tinoita ma backups, kuti mabhegi edu anoshanda, matambudziko anomuka navo anotora nguva shoma uye anogadziriswa pamwero webasa maneja. Zvigadziriso zvinotora nzvimbo shoma chaizvo zvichienzaniswa netar.gz kana Bacula.

Source: www.habr.com