Yuav ua li cas GitLab pab koj backup loj NextCloud cia

Hlo Habr!

Hnub no kuv xav tham txog peb cov kev paub dhau los hauv automating lub thaub qab ntawm cov ntaub ntawv loj los ntawm Nextcloud cia nyob rau hauv ntau configurations. Kuv ua haujlwm ua chaw pabcuam ntawm Molniya AK, qhov uas peb ua kev tswj hwm kev tswj hwm ntawm IT systems; Nextcloud yog siv los khaws cov ntaub ntawv. Xws li, nrog rau cov qauv faib, nrog redundancy.

Cov teeb meem tshwm sim los ntawm cov yam ntxwv ntawm kev teeb tsa yog tias muaj ntau cov ntaub ntawv. Versioning muab los ntawm Nextcloud, redundancy, subjective yog vim li cas, thiab ntau dua tsim ntau duplicates.

prehistory

Thaum tswj hwm Nextcloud, qhov teeb meem ntawm kev teeb tsa kev thaub qab zoo tshwm sim, uas yuav tsum tau encrypted, vim cov ntaub ntawv tseem ceeb.

Peb muab cov kev xaiv rau khaws cia thaub qab ntawm peb qhov chaw lossis ntawm cov neeg siv khoom ntawm cov tshuab cais los ntawm Nextcloud, uas yuav tsum muaj kev hloov pauv hloov pauv mus rau kev tswj hwm.

Muaj ntau tus neeg siv khoom, txhua tus ntawm lawv nrog kev teeb tsa sib txawv, thiab tag nrho ntawm lawv tus kheej qhov chaw thiab nrog lawv tus yam ntxwv. Qhov no yog txheej txheem txheej txheem thaum tag nrho lub xaib yog koj, thiab cov ntaub ntawv thaub qab yog tsim los ntawm cov yas; nws tsis haum zoo.

Ua ntej, cia saib cov ntaub ntawv nkag. Peb xav tau:

• Scalability nyob rau hauv cov nqe lus ntawm ib tug node los yog ob peb. Rau kev teeb tsa loj peb siv minio ua chaw cia.
• Tshawb nrhiav cov teeb meem nrog kev ua haujlwm thaub qab.
• Koj yuav tsum khaws cov thaub qab nrog koj cov neeg siv khoom thiab / lossis nrog peb.
• Nrog cov teeb meem sai thiab yooj yim.
• Cov neeg siv khoom thiab kev teeb tsa yog qhov sib txawv ntawm ib leeg - tsis tuaj yeem ua tiav tsis tiav.
• Qhov ceev rov qab yuav tsum yog tsawg kawg nkaus hauv ob qhov xwm txheej: kev rov qab tag nrho (kev puas tsuaj), ib daim nplaub tshev tshem tawm yuam kev.
• Deduplication muaj nuj nqi yuav tsum tau.

Txhawm rau daws qhov teeb meem ntawm kev tswj cov thaub qab, peb tau nruab GitLab. Xav paub ntau ntxiv los ntawm tackle.

Tau kawg, peb tsis yog thawj zaug los daws qhov teeb meem no, tab sis nws zoo li peb tias peb qhov kev ua tau zoo, kev khwv tau los nyuaj tuaj yeem nthuav thiab peb npaj siab los qhia nws.

Txij li thaum peb lub tuam txhab muaj txoj cai qhib, peb tab tom nrhiav kev qhib qhov kev daws teeb meem. Nyob rau hauv lem, peb qhia peb txoj kev loj hlob thiab tshaj tawm lawv. Piv txwv li, ntawm GitHub muaj peb plugin rau Nextcloud, uas peb muab rau cov neeg siv khoom, txhim kho cov ntaub ntawv kev ruaj ntseg nyob rau hauv cov ntaub ntawv ntawm kev huam yuaj los yog txhob txwm deletion.

Cov cuab yeej thaub qab

Peb pib peb tshawb nrhiav cov kev daws teeb meem los ntawm kev xaiv cov cuab yeej tsim thaub qab.

Tsis tu ncua tar + gzip tsis ua hauj lwm zoo - cov ntaub ntawv yog duplicated. Ib qho kev nce ntxiv feem ntau muaj qee qhov kev hloov pauv tiag tiag, thiab ntau cov ntaub ntawv hauv ib cov ntaub ntawv tau rov ua dua.
Muaj lwm qhov teeb meem - redundancy ntawm faib cov ntaub ntawv cia. Peb siv minio thiab nws cov ntaub ntawv yog qhov pib redundant. Los yog koj yuav tsum tau ua ib qho thaub qab los ntawm minio nws tus kheej - thauj nws thiab siv tag nrho cov spacers ntawm cov ntaub ntawv kaw lus, thiab, tsis muaj qhov tseem ceeb, muaj kev pheej hmoo ntawm tsis nco qab txog qee cov thoob thiab cov ntaub ntawv meta. Los yog siv deduplication.

Cov cuab yeej thaub qab nrog kev theej tawm muaj nyob rau hauv qhov chaw qhib (ntawm Habré muaj lus txog lub ntsiab lus no) thiab peb cov finalists yog ua borg и Rawg. Peb qhov kev sib piv ntawm ob daim ntawv thov yog hauv qab no, tab sis tam sim no peb yuav qhia koj seb peb npaj tag nrho cov tswv yim li cas.

Tswj cov thaub qab

Borg thiab Restic yog qhov zoo, tab sis tsis muaj cov khoom lag luam muaj lub hauv paus tswj kev tswj hwm. Rau lub hom phiaj ntawm kev tswj hwm thiab kev tswj hwm, peb tau xaiv cov cuab yeej uas peb twb tau siv lawm, yam tsis muaj qhov uas peb tsis tuaj yeem xav txog peb txoj haujlwm, suav nrog automation - qhov no yog qhov paub zoo CI / CD - GitLab.

Lub tswv yim yog raws li nram no: gitlab-khiav yog ntsia rau ntawm txhua qhov ntawm cov ntaub ntawv Nextcloud. Tus khiav khiav ib tsab ntawv ntawm lub sijhawm uas saib xyuas cov txheej txheem thaub qab, thiab nws pib Borg lossis Restic.

Peb tau txais dab tsi? Kev tawm tswv yim los ntawm kev ua tiav, yooj yim tswj kev hloov pauv, cov ntsiab lus thaum muaj qhov yuam kev.

no ntawm GitHub peb tau tshaj tawm cov piv txwv ntawm tsab ntawv rau ntau yam dej num, thiab peb tau mus txuas nws mus rau lub thaub qab ntawm Nextcloud, tab sis kuj muaj ntau lwm yam kev pab. Kuj tseem muaj lub sijhawm teem sijhawm nyob ntawd yog tias koj tsis xav teeb tsa nws tus kheej (thiab peb tsis xav tau) thiab .gitlab-ci.yml

Tsis muaj txoj hauv kev los hloov CI / CD timeout hauv Gitlab API tsis tau, tab sis nws yog me me. Nws yuav tsum tau nce, hais rau 1d.

GitLab, hmoov zoo, tuaj yeem tso tawm tsis yog raws li kev cog lus, tab sis tsuas yog raws li lub sijhawm, qhov no yog qhov peb xav tau.

Tam sim no hais txog wrapper tsab ntawv.

Peb teeb tsa cov xwm txheej hauv qab no rau tsab ntawv no:

• Nws yuav tsum tau launched ob qho tib si los ntawm tus khiav thiab los ntawm txhais tes los ntawm lub console nrog tib functionality.
• Yuav tsum muaj qhov yuam kev handlers:
• rov code.
• nrhiav ib txoj hlua hauv lub cav. Piv txwv li, rau peb qhov yuam kev yuav yog cov lus uas qhov kev zov me nyuam tsis xav txog kev tuag.
• Lub sijhawm ua haujlwm. Lub sijhawm ua haujlwm yuav tsum tsim nyog.
• Peb xav tau ib daim ntawv qhia ntxaws heev. Tab sis tsuas yog thaum muaj qhov yuam kev.
• Ntau qhov kev sim kuj tau ua ua ntej pib.
• Cov nyiaj tshwj xeeb me me rau kev yooj yim uas peb pom muaj txiaj ntsig thaum lub sijhawm txhawb nqa:
• Qhov pib thiab xaus yog kaw rau hauv syslog ntawm lub tshuab hauv zos. Qhov no pab txhawm rau txuas qhov yuam kev thiab kev ua haujlwm thaub qab.
• Ib feem ntawm lub cav yuam kev, yog tias muaj, yog tso tawm rau stdout, tag nrho cov cav raug sau rau hauv cov ntaub ntawv cais. Nws yog qhov yooj yim los saib tam sim ntawm CI thiab ntsuas qhov yuam kev yog tias nws tsis tseem ceeb.
• Debugging hom.

Daim ntawv teev tag nrho tau txais kev cawmdim raws li cov khoom cuav hauv GitLab; yog tias tsis muaj qhov yuam kev, lub cav raug tshem tawm. Peb sau tsab ntawv hauv bash.

Peb yuav zoo siab los txiav txim siab txog cov lus qhia thiab cov lus hais txog kev qhib qhov chaw - txais tos.

Yuav ua li cas li no ua hauj lwm

Ib tug khiav nrog ib tug Bash executor yog launched ntawm lub thaub qab ntawm. Raws li tus teem sijhawm, txoj haujlwm CI / CD tau pib ua haujlwm tshwj xeeb. Tus neeg khiav dej num nthuav tawm cov ntawv sau thoob ntiaj teb rau cov haujlwm zoo li no, nws tshawb xyuas qhov siv tau ntawm lub chaw khaws ntaub ntawv thaub qab, mount cov ntsiab lus thiab txhua yam peb xav tau, tom qab ntawd rov qab thiab ntxuav qhov qub. Lub tiav thaub qab nws tus kheej raug xa mus rau S3.

Peb ua hauj lwm raws li cov tswv yim no - nws yog ib tug lwm tus neeg muab kev pab AWS los yog ib tug Lavxias teb sab sib npaug (nws yog ceev thiab cov ntaub ntawv tsis tawm hauv Lavxias teb sab Federation). Los yog peb nruab ib lub minio cais rau tus neeg siv khoom ntawm nws qhov chaw rau cov hom phiaj no. Peb feem ntau ua qhov no rau kev nyab xeeb vim li cas, thaum tus neeg siv khoom tsis xav kom cov ntaub ntawv tawm hauv lawv lub voj voog txhua.

Peb tsis tau siv lub feature ntawm xa thaub qab ntawm ssh. Qhov no tsis ntxiv kev ruaj ntseg, thiab lub peev xwm network ntawm S3 tus neeg zov me nyuam muaj ntau dua li peb lub tshuab ssh.

Txhawm rau tiv thaiv koj lub tshuab hauv zos los ntawm hacker, vim nws tuaj yeem tshem tawm cov ntaub ntawv ntawm S3, koj yuav tsum ua kom muaj kev hloov pauv.
Cov thaub qab ib txwm encrypts cov thaub qab.

Borg muaj ib hom unencrypted none, tab sis peb xav kom tsis txhob tig nws. Nyob rau hauv hom no, tsis tsuas yog yuav tsis muaj encryption, tab sis lub checksum ntawm dab tsi yog sau yog tsis xam, uas txhais tau hais tias kev ncaj ncees tsuas yog yuav tsum tau soj ntsuam indirectly, siv indexes.

Ib tus neeg teem caij sib cais txheeb xyuas cov ntaub ntawv khaws tseg rau kev ncaj ncees ntawm cov indexes thiab cov ntsiab lus. Daim tshev yog qeeb thiab ntev, yog li peb khiav nws nyias ib hlis ib zaug. Nws yuav siv li ob peb hnub.

Readme hauv Russian

Lub luag haujlwm tseem ceeb

• prepare kev kawm
• testcheck kev kuaj xyuas
• maincommand pawg tub ntxhais
• forcepostscript ib qho kev ua haujlwm uas tau ua tiav thaum kawg lossis los ntawm kev ua yuam kev. Peb siv nws los unmount lub muab faib.

Kev pabcuam muaj nuj nqi

• cleanup Peb sau qhov yuam kev lossis tshem tawm cov ntaub ntawv teev tseg.
• checklog parse lub cav rau qhov tshwm sim ntawm ib kab nrog qhov yuam kev.
• ret tawm handler.
• checktimeout tshawb xyuas lub sijhawm.

ib puag ncig

• VERBOSE=1 Peb tso saib yuam kev ntawm qhov screen tam sim ntawd (stdout).
• SAVELOGSONSUCCES=1 txuag lub cav thaum ua tiav.
• INIT_REPO_IF_NOT_EXIST=1 Tsim ib lub chaw cia khoom yog tias nws tsis muaj. Disabled los ntawm lub neej ntawd.
• TIMEOUT lub sijhawm siab tshaj plaws rau kev ua haujlwm tseem ceeb. Koj tuaj yeem teem nws ua 'm', 'h' lossis 'd' thaum kawg.

Cia hom rau cov ntawv qub. Default:

• KEEP_DAILY=7
• KEEP_WEEKLY=4
• KEEP_MONTHLY=6

Hloov pauv hauv tsab ntawv

• ERROR_STRING - txoj hlua rau daim tshev hauv cav rau qhov yuam kev.
• EXTRACT_ERROR_STRING - kev qhia rau qhia txoj hlua yog tias yuam kev.
• KILL_TIMEOUT_SIGNAL - teeb liab tua yog tias ncua sijhawm.
• TAIL - Muaj pes tsawg txoj hlua nrog qhov yuam kev ntawm qhov screen.
• COLORMSG - xim ntawm cov lus (yeem daj).

Tsab ntawv ntawd, uas yog hu ua wordpress, muaj lub npe raws li txoj cai, nws ua kom yuam kev yog tias nws tseem rov qab mus rau mysql database. Qhov no txhais tau tias nws tuaj yeem siv rau ib qho ntawm Nexcloud kev teeb tsa, qhov twg koj tuaj yeem rov qab tau cov ntaub ntawv. Qhov yooj yim tsis yog tsuas yog tias txhua yam nyob hauv ib qho chaw, tab sis kuj tseem muaj cov ntsiab lus ntawm cov ntaub ntawv nyob ze rau cov ntsiab lus ntawm cov ntaub ntawv, txij li lub sijhawm sib txawv yog tsawg.

Restic vs Borg

Kuj tseem muaj kev sib piv ntawm Borg thiab Restic ntawm no ntawm Habre, thiab peb tsis muaj txoj haujlwm los ua lwm tus, tab sis peb tus kheej. Nws yog ib qho tseem ceeb rau peb yuav ua li cas nws yuav saib ntawm peb cov ntaub ntawv, nrog rau peb qhov tshwj xeeb. Peb coj lawv.

Peb cov txheej txheem xaiv, ntxiv rau cov uas twb tau hais lawm (deduplication, rov qab sai, thiab lwm yam):

• Kev ua haujlwm tsis tiav. Tshawb nrhiav tua -9.
• Loj ntawm disk.
• Kev xav tau rau cov peev txheej (CPU, nco).
• Qhov loj ntawm cov blobs khaws cia.
• Ua haujlwm nrog S3.
• Kev txheeb xyuas kev ncaj ncees.

Rau kev sim, peb coj ib tus neeg siv nrog cov ntaub ntawv tiag tiag thiab tag nrho qhov loj ntawm 1,6 TB.
Cov xwm txheej.

Borg tsis paub yuav ua li cas ncaj qha nrog S3, thiab peb mounted lub disk raws li ib tug fuse, ntawm ruam. Restic xa nws mus rau S3 nws tus kheej.

Goofys ua haujlwm sai heev thiab zoo, thiab muaj disk cache module, uas ua rau kev ua haujlwm nrawm dua. Nws yog nyob rau theem beta, thiab, hais ncaj ncees, peb poob nrog cov ntaub ntawv poob thaum kuaj (lwm tus). Tab sis qhov yooj yim yog tias cov txheej txheem thaub qab nws tus kheej tsis xav tau kev nyeem ntawv ntau, tab sis feem ntau sau ntawv, yog li peb siv lub cache tsuas yog thaum kuaj xyuas kev ncaj ncees.

Txhawm rau txo qis kev cuam tshuam ntawm lub network, peb siv lub chaw muab kev pabcuam hauv zos - Yandex Cloud.

Cov txiaj ntsig kev sib piv.

• Tua -9 nrog kev rov pib dua ntxiv tau ua tiav ob qho tib si.
• Loj ntawm disk. Borg tuaj yeem compress, yog li cov txiaj ntsig tau raws li qhov xav tau.

Backuper
loj

ua borg
562Gb

Rawg
628Gb

• Los ntawm CPU
  Borg nws tus kheej siv me ntsis, nrog kev ua haujlwm tsis zoo, tab sis nws yuav tsum tau soj ntsuam ua ke nrog cov txheej txheem goofys. Nyob rau hauv tag nrho, lawv sib piv thiab siv txog 1,2 cores ntawm tib lub tshuab virtual.
• Nco. Restic yog kwv yees li 0,5GB, Borg yog kwv yees li 200MB. Tab sis qhov no yog tag nrho insignificant piv rau cov ntaub ntawv system cache. Yog li nws pom zoo kom faib ntau lub cim xeeb.
• Qhov sib txawv ntawm blob loj tau tawm tsam.

Backuper
loj

ua borg
txog 500 MB

Rawg
txog 5 MB

• Kev paub nrog Restic's S3 yog qhov zoo heev. Ua hauj lwm nrog Borg los ntawm goofys tsis tsa ib lo lus nug, tab sis nws tau raug sau tseg tias nws yog ib advisable mus ua umount tom qab lub backup ua tiav kom tag rov pib dua lub cache. Qhov peculiarity ntawm S3 yog tias cov chunks under-pumped yuav tsis raug xa mus rau lub thoob, uas txhais tau hais tias cov ntaub ntawv tsis tiav ua rau muaj kev puas tsuaj loj.
• Kev kuaj xyuas kev ncaj ncees ua haujlwm zoo nyob rau hauv ob qho tib si, tab sis qhov ceev sib txawv heev.
  Restic 3,5 teev.
  Borg, nrog 100GB SSD cov ntaub ntawv cache - 5 teev. Kwv yees li qhov ceev tib yam yog tias cov ntaub ntawv nyob rau hauv lub zos disk.
  Borg nyeem ncaj qha los ntawm S3 yam tsis muaj cache 33 teev. Ntev ntev heev.

Cov kab hauv qab yog tias Borg tuaj yeem compress thiab muaj blobs loj dua - uas ua rau kev cia thiab GET / PUT cov haujlwm hauv S3 pheej yig dua. Tab sis qhov no los ntawm tus nqi ntawm kev tshawb xyuas ntau dua thiab qeeb dua. Raws li kev rov qab ceev, peb tsis pom qhov txawv. Restic siv cov thaub qab tom qab (tom qab thawj zaug) ntev me ntsis, tab sis tsis loj.

Qhov kawg tab sis tsis tsawg kawg hauv kev xaiv yog qhov loj ntawm lub zej zog.

Thiab peb xaiv borg.

Ob peb lo lus hais txog compression

Borg muaj qhov zoo tshaj plaws tshiab compression algorithm hauv nws cov arsenal - zstd. Lub compression zoo tsis yog phem tshaj gzip, tab sis sai npaum li cas. Thiab piv rau hauv ceev rau lub neej ntawd lz4.

Piv txwv li, MySQL database dump yog compressed ob zaug zoo dua li lz4 ntawm tib qhov ceev. Txawm li cas los xij, kev paub nrog cov ntaub ntawv tiag tiag qhia tau tias muaj qhov sib txawv me ntsis hauv qhov sib piv ntawm qhov Nextcloud node.

Borg muaj qhov zoo dua compression hom - yog tias cov ntaub ntawv muaj entropy siab, ces compression tsis siv txhua, uas ua rau kom ceev. Enabled los ntawm kev xaiv thaum tsim
-C auto,zstd
rau zstd algorithm
Yog li nrog qhov kev xaiv no, nyob rau hauv kev sib piv nrog lub neej ntawd compression, peb tau txais
560Gb thiab 562Gb feem. Cov ntaub ntawv los ntawm cov piv txwv saum toj no, cia kuv ceeb toom koj, tsis muaj compression qhov tshwm sim yog 628Gb. Qhov tshwm sim ntawm 2GB qhov sib txawv me ntsis ua rau peb xav tsis thoob, tab sis peb xav tias peb yuav xaiv nws tom qab tag nrho. auto,zstd.

Thaub qab pov thawj txoj kev

Raws li lub sijhawm teem sijhawm, lub tshuab virtual tau pib ncaj qha los ntawm tus neeg muab kev pabcuam lossis los ntawm tus neeg siv khoom, uas txo qis lub network thauj khoom zoo heev. Yam tsawg kawg nws pheej yig dua li tsa koj tus kheej thiab tsav tsheb.

goofys --cache "--free:5%:/mnt/cache" -o allow_other --endpoint https://storage.yandexcloud.net --file-mode=0666 --dir-mode=0777 xxxxxxx.com /mnt/goofys
export BORG_PASSCOMMAND="cat /home/borg/.borg-passphrase"
borg list /mnt/goofys/borg1/
borg check --debug -p --verify-data /mnt/goofys/borg1/

Siv tib lub tswv yim, peb tshawb xyuas cov ntaub ntawv nrog antivirus (tom qab qhov tseeb). Tom qab tag nrho, cov neeg siv upload ntau yam rau Nextcloud thiab tsis yog txhua tus muaj ib qho antivirus. Kev tshuaj xyuas thaum lub sij hawm nchuav yuav siv sij hawm ntau dhau thiab cuam tshuam rau kev lag luam.

Scalability yog ua tiav los ntawm kev sib tw khiav ntawm cov nodes sib txawv nrog cov cim sib txawv.
Peb qhov kev saib xyuas khaws cov xwm txheej thaub qab ntawm GitLab API hauv ib lub qhov rais; yog tias tsim nyog, cov teeb meem tau pom tau yooj yim thiab ib qho yooj yim hauv zos.

xaus

Raws li qhov tshwm sim, peb paub tseeb tias peb ua cov thaub qab, tias peb cov ntaub ntawv thaub qab siv tau, cov teeb meem tshwm sim nrog lawv siv sijhawm me ntsis thiab raug daws nyob rau theem ntawm tus thawj tswj hwm lub luag haujlwm. Backups siv qhov chaw me me piv rau tar.gz lossis Bacula.

Tau qhov twg los: www.hab.com