{"id":86904,"date":"2020-07-01T07:42:30","date_gmt":"2020-07-01T05:42:30","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud"},"modified":"2020-07-01T07:42:30","modified_gmt":"2020-07-01T05:42:30","slug":"kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","title":{"rendered":"Cum ajut\u0103 GitLab la realizarea copiilor de rezerv\u0103 ale marii stoc\u0103ri NextCloud","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Salut, Habr!<\/p>\n<p><\/p>\n<p>Ast\u0103zi vreau s\u0103 v\u0103 povestesc despre experien\u021ba noastr\u0103 \u00een automatizarea copiilor de siguran\u021b\u0103 a stoc\u0103rii de date mari Nextcloud \u00een diferite configura\u021bii. Lucrez ca CTO la \"Molnia AK\", unde ne ocup\u0103m cu managementul configur\u0103rilor sistemelor IT, folosind Nextcloud pentru stocarea datelor. Acesta include, de asemenea, o structur\u0103 distribuit\u0103, cu rezervare.<\/p>\n<p><\/p>\n<p>Problemele care decurg din particularit\u0103\u021bile instal\u0103rilor sunt c\u0103 sunt multe date. Versiunile pe care le ofer\u0103 Nextcloud, rezervarea, motivele subiective \u0219i altele creeaz\u0103 multe duplicate.<\/p>\n<p><\/p>\n<h2 id=\"predystoriya\">Povestea<\/h2>\n<p><\/p>\n<p>\u00cen administrarea Nextcloud, apare cu acuitate problema organiz\u0103rii unei copii de siguran\u021b\u0103 eficiente, care trebuie obligatoriu criptat\u0103, deoarece datele sunt valoroase.<\/p>\n<p><\/p>\n<p>Oferim op\u021biuni de stocare a copiilor de siguran\u021b\u0103 fie la noi, fie la client pe ma\u0219ini separate de Nextcloud, ceea ce necesit\u0103 un abord flexibil automatizat \u00een administrare.<\/p>\n<p><\/p>\n<p>Avem mul\u021bi clien\u021bi, to\u021bi cu configura\u021bii diferite, \u0219i fiecare pe propriile platforme \u0219i cu particularit\u0103\u021bile lor. Metodologia standard, c\u00e2nd toat\u0103 platforma \u00ee\u021bi apar\u021bine, \u0219i copiile de siguran\u021b\u0103 sunt f\u0103cute din cron, se potrive\u0219te slab.<\/p>\n<p><\/p>\n<p>Pentru \u00eenceput, s\u0103 ne uit\u0103m la datele introductive. Avem nevoie de:<\/p>\n<p><\/p>\n<ul>\n<li>Scalabilitate \u00een ceea ce prive\u0219te o nod\u0103 sau mai multe. Pentru instal\u0103ri mari, folosim minio ca stocare.<\/li>\n<li>Afl\u0103m despre problemele cu executarea copiilor de siguran\u021b\u0103.<\/li>\n<li>Trebuie s\u0103 stoc\u0103m copia de siguran\u021b\u0103 la clien\u021bi \u0219i\/sau la noi.<\/li>\n<li>S\u0103 putem rezolva rapid \u0219i u\u0219or problemele.<\/li>\n<li>Clien\u021bii \u0219i instala\u021biile difer\u0103 semnificativ unul de altul - uniformitatea nu poate fi ob\u021binut\u0103.<\/li>\n<li>Viteza de recuperare trebuie s\u0103 fie minim\u0103 \u00een dou\u0103 scenarii: recuperare complet\u0103 (catastrof\u0103), un director - \u0219ters din gre\u0219eal\u0103.<\/li>\n<li>Func\u021bia de deduplicare este obligatorie.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cum ajut\u0103 GitLab la realizarea copiilor de rezerv\u0103 ale marii stoc\u0103ri NextCloud\" src=\"\/wp-content\/uploads\/2020\/07\/1a0048e3295fef94ddca28e50612e3cb.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pentru a rezolva problema gestion\u0103rii copiilor de siguran\u021b\u0103, am integrat GitLab. Detalii \u00een continuare. <\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Desigur, nu suntem primii care rezolv\u0103m o astfel de problem\u0103, dar ni se pare c\u0103 experien\u021ba noastr\u0103 practic\u0103 \u0219i \u00eendelungat\u0103 poate fi interesant\u0103 \u0219i suntem gata s\u0103 o \u00eemp\u0103rt\u0103\u0219im.<\/p>\n<p><\/p>\n<p>Deoarece \u00een compania noastr\u0103 este adoptat\u0103 politica open-source, solu\u021bia c\u0103utat\u0103 a fost cu cod surs\u0103 deschis. La r\u00e2ndul nostru, \u00eemp\u0103rt\u0103\u0219im dezvolt\u0103rile noastre \u0219i le public\u0103m. De exemplu, pe GitHub exist\u0103 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/molnia-com\/nextcloud-safe-trash\">pluginul nostru pentru Nextcloud<\/a><\/noindex>, pe care \u00eel instal\u0103m clien\u021bilor, care \u00eent\u0103re\u0219te siguran\u021ba datelor \u00een cazul \u0219tergerii accidentale sau inten\u021bionate.<\/p>\n<p><\/p>\n<h2 id=\"sredstva-bekapirovaniya\">Mijloace de copiere de siguran\u021b\u0103<\/h2>\n<p><\/p>\n<p>Am \u00eenceput c\u0103utarea metodelor de solu\u021bionare aleg\u00e2nd un instrument pentru crearea copiilor de siguran\u021b\u0103.<\/p>\n<p><\/p>\n<p>Tar \u0219i gzip func\u021bioneaz\u0103 prost \u2014 datele sunt duplicate. Incrementul con\u021bine adesea foarte pu\u021bine modific\u0103ri, iar cea mai mare parte a datelor dintr-un singur fi\u0219ier se repet\u0103.<br \/>\nExist\u0103 o alt\u0103 problem\u0103 \u2014 redundan\u021ba stoc\u0103rii distribuite a datelor. Folosim minio \u0219i datele sale sunt \u00een principiu redundante. Fie trebuia s\u0103 facem backup prin minio \u00eensu\u0219i \u2013 s\u0103-l solicit\u0103m \u0219i s\u0103 folosim toate intermedierele \u00eentre sistemul de fi\u0219iere, iar nu mai pu\u021bin important, exist\u0103 riscul de a uita o parte din bucket-uri \u0219i metainforma\u021bii. Fie s\u0103 utiliz\u0103m deduplicarea.<\/p>\n<p><\/p>\n<p>Exist\u0103 instrumente de backup cu deduplicare \u00een open source (au fost pe Habr\u0103 <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/southbridge\/blog\/449282\/\">recomand\u0103rile sunt aplicabile coronavirusurilor \u00een general \u0219i COVID-19 \u00een particular. A\u0219adar, recomand s\u0103 desc\u0103rca\u021bi \u0219i s\u0103 printa\u021bi articolul (pentru cei interesa\u021bi de acest subiect).<\/a><\/noindex> <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/flant\/blog\/420055\/\">pe aceast\u0103 tem\u0103<\/a><\/noindex>) \u0219i finali\u0219tii no\u0219tri au fost <noindex><a rel=\"nofollow\" href=\"https:\/\/borgbackup.readthedocs.io\/en\/stable\/\">Borg<\/a><\/noindex> \u0219i <noindex><a rel=\"nofollow\" href=\"https:\/\/restic.net\/\">Restic<\/a><\/noindex>. Despre compara\u021bia noastr\u0103 \u00eentre cele dou\u0103 aplica\u021bii mai jos, dar \u00eentre timp s\u0103 vorbim despre cum am organizat \u00eentreaga schem\u0103.<\/p>\n<p><\/p>\n<h2 id=\"upravlenie-sozdaniem-rezervnyh-kopiy\">Gestionarea cre\u0103rii copiilor de siguran\u021b\u0103<\/h2>\n<p><\/p>\n<p>Borg \u0219i Restic sunt bune, dar niciunul dintre aceste produse nu are un mecanism centralizat de gestionare. Pentru scopul gestion\u0103rii \u0219i controlului, am ales un instrument pe care deja \u00eel avem implementat, f\u0103r\u0103 de care nu ne putem imagina munca noastr\u0103, inclusiv \u00een ceea ce prive\u0219te automatizarea \u2013 este binecunoscutul CI\/CD \u2013 GitLab.<\/p>\n<p><\/p>\n<p>Ideea este urm\u0103toarea: pe fiecare nod care stocheaz\u0103 date Nextcloud se instaleaz\u0103 gitlab-runner. Runnerul ruleaz\u0103 conform unui program un script care monitorizeaz\u0103 procesul de backup, iar acesta lanseaz\u0103 Borg sau Restic.<\/p>\n<p><\/p>\n<p>Ce am ob\u021binut? Feedback din execu\u021bie, control convenabil asupra modific\u0103rilor, detalii \u00een caz de eroare.<\/p>\n<p><\/p>\n<p>Iat\u0103 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/molnia-com\/backup-by-runner\">aici pe GitHub<\/a><\/noindex> am publicat exemple de script pentru diferite sarcini, iar noi l-am integrat \u00een backupul nu doar pentru Nextcloud, ci \u0219i pentru multe alte servicii. Acolo se g\u0103se\u0219te \u0219i planificatorul, dac\u0103 nu vrei s\u0103-l configurezi manual (iar nou\u0103 nu ne place) \u0219i .gitlab-ci.yml<\/p>\n<p><\/p>\n<p>\u00cen API-ul GitLab-ului nu exist\u0103 \u00eenc\u0103 posibilitatea de a modifica timeout-ul CI\/CD, iar el este destul de mic. Trebuie s\u0103-l cre\u0219tem, de exemplu, la <code>1d<\/code>.<\/p>\n<p><\/p>\n<p>GitLab din fericire poate rula nu doar la commit, ci \u0219i conform unui program, exact ceea ce ne trebuie.<\/p>\n<p><\/p>\n<p>Acum despre scriptul-wrapper.<\/p>\n<p><\/p>\n<p>Am stabilit urm\u0103toarele condi\u021bii pentru acest script:<\/p>\n<p><\/p>\n<ul>\n<li>Trebuie s\u0103 fie rulat at\u00e2t de runner, c\u00e2t \u0219i manual din consol\u0103 cu aceea\u0219i func\u021bionalitate.<\/li>\n<li>Trebuie s\u0103 existe neap\u0103rat gestionari de erori:<\/li>\n<li>cod de returnare.<\/li>\n<li>c\u0103utarea unui \u0219ir \u00een log. De exemplu, pentru noi, o eroare poate fi un mesaj pe care programul nu-l consider\u0103 fatal.<\/li>\n<li>Timeout-ul de procesare. Timpul de execu\u021bie trebuie s\u0103 fie rezonabil.<\/li>\n<li>Avem nevoie de un jurnal detaliat. Dar doar \u00een cazul unei erori.<\/li>\n<li>De asemenea, se efectueaz\u0103 o serie de teste \u00eenainte de \u00eencepere.<\/li>\n<li>C\u00e2teva mici facilit\u0103\u021bi pentru confort, pe care le-am g\u0103sit utile \u00een procesul de suport:<\/li>\n<li>Pornirea \u0219i \u00eencetarea sunt \u00eenregistrate \u00een log-ul local al ma\u0219inii. Acest lucru ajut\u0103 la corelarea erorilor sistemului \u0219i a func\u021bion\u0103rii de backup.<\/li>\n<li>O parte din log-ul de erori, atunci c\u00e2nd exist\u0103, este afi\u0219at\u0103 \u00een stdout, \u00eentreaga \u00eenregistrare este scris\u0103 \u00eentr-un fi\u0219ier separat. Convenabil s\u0103 arunci o privire \u00een CI \u0219i s\u0103 evaluezi eroarea dac\u0103 este trivial\u0103.<\/li>\n<li>Moduri pentru depanare.<\/li>\n<\/ul>\n<p><\/p>\n<p>Log-ul complet este salvat ca un artefact \u00een GitLab; dac\u0103 nu exist\u0103 erori, log-ul este \u0219ters. Scriptul este scris \u00een bash.<\/p>\n<p><\/p>\n<p>Orice sugestii \u0219i observa\u021bii privind open-source-ul vor fi binevenite \u2014 welcome.<\/p>\n<p><\/p>\n<h2 id=\"kak-eto-rabotaet\">Cum func\u021bioneaz\u0103<\/h2>\n<p><\/p>\n<p>Pe nodul de backup se lanseaz\u0103 un runner cu executor bash. \u00cen planificator, \u00eentr-un repository special, se lanseaz\u0103 un job CI\/CD. Runner-ul ruleaz\u0103 un script wrapper universal pentru astfel de sarcini, \u00een care se efectueaz\u0103 verific\u0103ri de validitate pentru repository-ul de backup, punctele de montare \u0219i tot ce dorim, apoi se execut\u0103 backup-ul \u0219i cur\u0103\u021barea celor vechi. Backup-ul final preparat este trimis pe S3.<\/p>\n<p><\/p>\n<p>Folosim o schem\u0103 de acest fel \u2014 este un furnizor extern AWS sau un echivalent rusesc (este mai rapid \u0219i datele nu p\u0103r\u0103sesc RF). Sau, pentru client, instal\u0103m un cluster minio separat pe site-ul s\u0103u pentru aceste scopuri. De obicei, facem asta din motive de securitate, c\u00e2nd clientul nu dore\u0219te ca datele s\u0103 p\u0103r\u0103seasc\u0103 conturul s\u0103u.<\/p>\n<p><\/p>\n<p>Nu am utilizat func\u021bia de trimitere a backup-ului prin ssh. Asta nu adaug\u0103 securitate, iar capacit\u0103\u021bile de re\u021bea ale furnizorului S3 sunt mult mai mari dec\u00e2t ale unei singure ma\u0219ini ssh.<\/p>\n<p><\/p>\n<p>Pentru a ne proteja de un hacker pe ma\u0219ina local\u0103 \u2014 deoarece acesta poate \u0219terge datele de pe S3, trebuie s\u0103 activ\u0103m cu siguran\u021b\u0103 versiunea.<br \/>\nBackup-erul criptaz\u0103 \u00eentotdeauna backup-ul.<\/p>\n<p><\/p>\n<p>Borg are un mod f\u0103r\u0103 criptare. <code>none<\/code>, dar nu recomand\u0103m cu vehemen\u021b\u0103 activarea acestuia. \u00cen acest mod nu va exista nu doar criptare, ci nu se va calcula nici suma de control pentru ceea ce este scris, ceea ce \u00eenseamn\u0103 c\u0103 integritatea poate fi verificat\u0103 doar indirect, pe baza indexurilor.<\/p>\n<p><\/p>\n<p>Printr-un programator separat se efectueaz\u0103 verificarea backup-urilor pentru integritatea indexurilor \u0219i a con\u021binutului. Verificarea se desf\u0103\u0219oar\u0103 lent \u0219i \u00eendelungat, a\u0219a c\u0103 o lans\u0103m separat o dat\u0103 pe lun\u0103. Poate dura c\u00e2teva zile.<\/p>\n<p><\/p>\n<h2 id=\"ridmi-na-russkom\">Readme \u00een rom\u00e2n\u0103<\/h2>\n<p><\/p>\n<h3 id=\"osnovnye-funkcii\">Func\u021bii principale<\/h3>\n<p><\/p>\n<ul>\n<li><code>prepare<\/code> preg\u0103tire<\/li>\n<li><code>testcheck<\/code> verificare a disponibilit\u0103\u021bii<\/li>\n<li><code>maincommand<\/code> comanda principal\u0103<\/li>\n<li><code>forcepostscript<\/code> func\u021bia care se execut\u0103 la final sau \u00een caz de eroare. O folosim pentru a demonta parti\u021bia.<\/li>\n<\/ul>\n<p><\/p>\n<h3 id=\"service-functions\">Func\u021bii de serviciu<\/h3>\n<p><\/p>\n<ul>\n<li><code>cleanup<\/code> not\u0103m erorile sau \u0219tergem fi\u0219ierul de log.<\/li>\n<li><code>checklog<\/code> analiz\u0103m log-ul pentru a g\u0103si un \u0219ir cu o eroare.<\/li>\n<li><code>ret<\/code> handler de ie\u0219ire.<\/li>\n<li><code>checktimeout<\/code> verificarea timeout-ului.<\/li>\n<\/ul>\n<p><\/p>\n<h3 id=\"environment\">Mediu<\/h3>\n<p><\/p>\n<ul>\n<li><code>VERBOSE=1<\/code> afi\u0219\u0103m erorile pe ecran imediat (stdout).<\/li>\n<li><code>SAVELOGSONSUCCES=1<\/code> salv\u0103m log-ul \u00een cazul succesului.<\/li>\n<li><code>INIT_REPO_IF_NOT_EXIST=1<\/code> Cre\u0103m un depozit, dac\u0103 nu exista. \u00cen mod implicit, este dezactivat.<\/li>\n<li><code>TIMEOUT<\/code> timpul maxim pentru opera\u021biunea principal\u0103. \u00cel po\u021bi seta ca &#8216;m&#8217;, &#8216;h&#8217; sau &#8216;d&#8217; la final.<\/li>\n<\/ul>\n<p><\/p>\n<p>Modul de p\u0103strare a copiilor vechi. \u00cen mod implicit:<\/p>\n<p><\/p>\n<ul>\n<li><code>KEEP_DAILY=7<\/code><\/li>\n<li><code>KEEP_WEEKLY=4<\/code><\/li>\n<li><code>KEEP_MONTHLY=6<\/code><\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"peremennye-vnutri-skripta\">Variabile \u00een script<\/h2>\n<p><\/p>\n<ul>\n<li><code>ERROR_STRING<\/code> \u2014 \u0219ir pentru verificarea \u00een log pentru eroare.<\/li>\n<li><code>EXTRACT_ERROR_STRING<\/code> \u2014 expresie pentru a afi\u0219a \u0219irul \u00een cazul unei erori.<\/li>\n<li><code>KILL_TIMEOUT_SIGNAL<\/code> \u2014 semnal pentru a opri \u00een cazul timeout-ului.<\/li>\n<li><code>TAIL<\/code> \u2014 c\u00e2te \u0219iruri cu erori pe ecran.<\/li>\n<li><code>COLORMSG<\/code> \u2014 culoarea mesajului (default galben).<\/li>\n<\/ul>\n<p><\/p>\n<p>Scriptul numit WordPress este denumit a\u0219a, iar caracteristica sa este c\u0103 face backup la baza de date MySQL. Asta \u00eenseamn\u0103 c\u0103 poate fi utilizat pentru instal\u0103rile unice Nexcloud, unde po\u021bi de asemenea s\u0103 faci backup la baza de date. Avantajul nu este doar c\u0103 totul este \u00eentr-un singur loc, dar \u0219i con\u021binutul bazei de date este apropiat de con\u021binutul fi\u0219ierelor, deoarece diferen\u021ba de timp este minim\u0103.<\/p>\n<p><\/p>\n<h2 id=\"restic-vs-borg\">Restic vs Borg<\/h2>\n<p><\/p>\n<p>Compara\u021biile \u00eentre Borg \u0219i Restic exist\u0103, de asemenea, <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/southbridge\/blog\/454734\/\">aici pe Habr<\/a><\/noindex>, \u0219i nu ne-a fost scopul s\u0103 cre\u0103m doar \u00eenc\u0103 unul, ci al nostru. A fost important pentru noi cum va ar\u0103ta pe datele noastre, cu specificitatea noastr\u0103. Le aducem.<\/p>\n<p><\/p>\n<p>Criteriile noastre de selec\u021bie, pe l\u00e2ng\u0103 cele men\u021bionate anterior (deduplicare, recuperare rapid\u0103 etc.):<\/p>\n<p><\/p>\n<ul>\n<li>Rezilien\u021ba la activitatea nefinalizat\u0103. Verificarea pentru kill -9.<\/li>\n<li>Dimensiunea pe disc.<\/li>\n<li>Cerin\u021bele de resurse (CPU, memorie).<\/li>\n<li>Dimensiunea blob-urilor p\u0103strate.<\/li>\n<li>Lucrul cu S3.<\/li>\n<li>Verificarea integrit\u0103\u021bii.<\/li>\n<\/ul>\n<p><\/p>\n<p>Pentru testare, am luat un client cu date reale \u0219i o dimensiune total\u0103 de 1,6TB.<br \/>\nCondi\u021bii.<\/p>\n<p><\/p>\n<p>Borg nu poate lucra direct cu S3, iar noi am montat ca disc fuse, prin <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kahing\/goofys\">goofys<\/a><\/noindex>. Restic trimitea \u00een S3 el \u00eensu\u0219i.<\/p>\n<p><\/p>\n<p>Goofys func\u021bioneaz\u0103 foarte repede \u0219i bine, \u0219i are <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kahing\/catfs\">un modul de cache pe disc<\/a><\/noindex>, care accelereaz\u0103 \u0219i mai mult procesul. Este \u00een stadiu beta, \u0219i, s\u0103 recunosc, ne-a c\u0103zut cu pierderi de date la teste (alte). Dar avantajul este c\u0103 procedura de backup necesit\u0103 pu\u021bin citire, mai mult scriere, astfel \u00eenc\u00e2t cache-ul \u00eel folosim doar \u00een timpul verific\u0103rii integrit\u0103\u021bii.<\/p>\n<p><\/p>\n<p>Pentru a reduce influen\u021ba re\u021belei, am folosit un provider local - Yandex Cloud.<\/p>\n<p><\/p>\n<p>Rezultatele test\u0103rii comparativului.<\/p>\n<p><\/p>\n<ul>\n<li>Kill -9 cu o repornire ulterioar\u0103, ambele au avut succes.<\/li>\n<li>Dimensiunea pe disc. Borg poate comprima, a\u0219adar rezultatele sunt a\u0219teptate.<\/li>\n<\/ul>\n<p><\/p>\n<p>Backuper<br \/>\nDimensiune<\/p>\n<p>Borg<br \/>\n562Gb<\/p>\n<p>Restic<br \/>\n628Gb<\/p>\n<p><\/p>\n<ul>\n<li>Referitor la CPU<br \/>\nBorg consum\u0103 pu\u021bin \u00een sine, cu compresia implicit\u0103, dar trebuie evaluat \u00eempreun\u0103 cu procesul goofys. \u00cen total, sunt comparabile \u0219i utilizeaz\u0103 aproximativ 1,2 nuclee pe aceea\u0219i ma\u0219in\u0103 virtual\u0103 de test.<\/li>\n<li>Memorie. Restic aproximativ 0,5Gb, Borg aproximativ 200Mb. Dar acestea sunt neglijabile \u00een compara\u021bie cu memoria cache a sistemului. A\u0219adar, este de preferat s\u0103 se aloce mai mult\u0103 memorie.<\/li>\n<li>Diferen\u021ba \u00een dimensiunea bloburilor s-a dovedit a fi spectaculoas\u0103.<\/li>\n<\/ul>\n<p><\/p>\n<p>Backuper<br \/>\nDimensiune<\/p>\n<p>Borg<br \/>\naproximativ 500Mb<\/p>\n<p>Restic<br \/>\naproximativ 5Mb<\/p>\n<p><\/p>\n<ul>\n<li>Lucrul cu S3 de la Restic este excelent. Func\u021bionarea Borg prin goofys nu ridic\u0103 \u00eentreb\u0103ri, dar s-a observat c\u0103 este recomandabil s\u0103 se fac\u0103 umount la finalizarea backup-ului pentru a reseta complet cache-ul. O caracteristic\u0103 a func\u021bion\u0103rii S3 este c\u0103 buc\u0103\u021bile neterminate nu vor fi niciodat\u0103 trimise \u00een bucket, ceea ce \u00eenseamn\u0103 c\u0103 datele incomplete conduc la daune mari.<\/li>\n<li>Verificarea integrit\u0103\u021bii func\u021bioneaz\u0103 bine \u00een ambele cazuri, dar viteza difer\u0103 semnificativ.<br \/>\nRestic \u2013 <strong>3,5 ore<\/strong>.<br \/>\nBorg, cu un cache de fi\u0219iere de 100Gb SSD \u2013 <strong>5 ore<\/strong>. Un rezultat aproximativ similar \u00een vitez\u0103 dac\u0103 datele sunt stocate pe un disc local.<br \/>\nBorg cite\u0219te direct de la S3 f\u0103r\u0103 cache <strong>33 de ore<\/strong>. \u00cengrozitor de lung.<\/li>\n<\/ul>\n<p><\/p>\n<p>\u00cen concluzie, Borg poate comprima \u0219i are bloburi mai mari \u2014 ceea ce face stocarea \u0219i opera\u021biunile GET\/PUT \u00een S3 mai ieftine. Dar pentru aceasta se pl\u0103te\u0219te printr-o verificare mai complex\u0103 \u0219i mai lent\u0103. \u00cen ceea ce prive\u0219te viteza de recuperare, nu am observat diferen\u021be. Backup-urile ulterioare (dup\u0103 primul) restic dureaz\u0103 pu\u021bin mai mult, dar nu semnificativ.<\/p>\n<p><\/p>\n<p>Dimensiunea comunit\u0103\u021bii a jucat de asemenea un rol important \u00een alegere.<\/p>\n<p><\/p>\n<p>\u0218i am ales borg.<\/p>\n<p><\/p>\n<h2 id=\"paru-slov-o-szhatii\">C\u00e2teva cuvinte despre compresie<\/h2>\n<p><\/p>\n<p>Borg are \u00een arsenal un nou algoritm de compresie excelent \u2014 zstd. \u00cen ceea ce prive\u0219te calitatea compresiei, nu este mai r\u0103u dec\u00e2t gzip, dar semnificativ mai rapid. \u0218i comparabil \u00een vitez\u0103 cu lz4 implicit.<\/p>\n<p><\/p>\n<p>De exemplu, un dump al bazei de date MySQL se comprima de dou\u0103 ori mai bine dec\u00e2t lz4 la aceea\u0219i vitez\u0103. Totu\u0219i, experien\u021ba pe date reale arat\u0103 c\u0103 exist\u0103 o diferen\u021b\u0103 foarte mic\u0103 \u00een gradul de compresie al nodului Nextcloud.<\/p>\n<p><\/p>\n<p>\u00cen Borg exist\u0103 un mod de compresie bonus \u2014 dac\u0103 fi\u0219ierul are o entropie mare, compresia nu este aplicat\u0103 deloc, ceea ce cre\u0219te viteza de operare. Se activeaz\u0103 printr-o op\u021biune la crearea sa<br \/>\n<code>-C auto,zstd<\/code><br \/>\npentru algoritmul zstd<br \/>\nA\u0219adar, cu aceast\u0103 op\u021biune, comparativ cu compresia implicit\u0103, am ob\u021binut<br \/>\n560Gb \u0219i 562Gb, respectiv. Datele din exemplul de mai sus, reamintesc, f\u0103r\u0103 compresie, rezultatul este 628Gb. Rezultatul cu o diferen\u021b\u0103 de 2Gb ne-a surprins pu\u021bin, dar am decis s\u0103 alegem totu\u0219i <code>auto,zstd<\/code>.<\/p>\n<p><\/p>\n<h2 id=\"metodika-proverki-bekapa\">Metodologia de verificare a backup-ului<\/h2>\n<p><\/p>\n<p>Prin scheduler se lanseaz\u0103 o ma\u0219in\u0103 virtual\u0103 direct la furnizor sau la client, ceea ce reduce semnificativ \u00eenc\u0103rc\u0103tura re\u021belei. Cel pu\u021bin, este mai ieftin dec\u00e2t s\u0103 ridici la tine \u0219i s\u0103 generezi trafic.<\/p>\n<p><\/p>\n<pre><code class=\"plaintext\">goofys --cache &quot;--free:5%:\\\/mnt\\\/cache&quot; -o allow_other --endpoint https:\\\/\\\/storage.yandexcloud.net --file-mode=0666 --dir-mode=0777 xxxxxxx.com \\\/mnt\\\/goofys\nexport BORG_PASSCOMMAND=&quot;cat \\\/home\\\/borg\\\/.borg-passphrase&quot;\nborg list \\\/mnt\\\/goofys\\\/borg1\\\/\nborg check --debug -p --verify-data \\\/mnt\\\/goofys\\\/borg1\\\/<\/code><\/pre>\n<p><\/p>\n<p>Dup\u0103 aceea\u0219i schem\u0103, verific\u0103m fi\u0219ierele cu antivirusul (postfactum). Deoarece utilizatorii \u00eencarc\u0103 diverse \u00een Nextcloud \u0219i nu to\u021bi au antivirus. Verificarea \u00een momentul \u00eenc\u0103rc\u0103rii dureaz\u0103 prea mult timp \u0219i \u00eempiedic\u0103 afacerea.<\/p>\n<p><\/p>\n<p>Scalabilitatea este ob\u021binut\u0103 prin lansarea runner-ilor pe noduri diferite cu etichete diferite.<br \/>\n\u00cen monitorizarea noastr\u0103, statusurile backup-urilor sunt colectate prin API-ul GitLab \u00eentr-o singur\u0103 fereastr\u0103, iar problemele pot fi observate cu u\u0219urin\u021b\u0103, fiind de asemenea u\u0219or de localizat.<\/p>\n<p><\/p>\n<h2 id=\"zaklyuchenie\">Concluzie<\/h2>\n<p><\/p>\n<p>\u00cen concluzie, \u0219tim cu siguran\u021b\u0103 c\u0103 facem backup-uri, c\u0103 backup-urile noastre sunt valide, problemele care apar cu acestea dureaz\u0103 pu\u021bin timp \u0219i se rezolv\u0103 la nivel de administrator de serviciu. Backup-urile ocup\u0103 \u00eentr-adev\u0103r foarte pu\u021bin spa\u021biu \u00een compara\u021bie cu tar.gz sau Bacula.<\/p>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/508758\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0421\u0422\u041e \u0432 \u00ab\u041c\u043e\u043b\u043d\u0438\u044f \u0410\u041a\u00bb, \u0433\u0434\u0435 \u043c\u044b \u0437\u0430\u043d\u0438\u043c\u0430\u0435\u043c\u0441\u044f \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u043c \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435\u043c IT \u0441\u0438\u0441\u0442\u0435\u043c, \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0434\u0430\u043d\u043d\u044b\u0445 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0441\u044f Nextcloud. \u0412 \u0442\u043e\u043c \u0447\u0438\u0441\u043b\u0435, \u0441 \u0440\u0430\u0441\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u043d\u043e\u0439 \u0441\u0442\u0440\u0443\u043a\u0442\u0443\u0440\u043e\u0439, \u0441 \u0440\u0435\u0437\u0435\u0440\u0432\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c. \u041f\u0440\u043e\u0431\u043b\u0435\u043c\u044b \u0432\u044b\u0442\u0435\u043a\u0430\u044e\u0449\u0438\u0435 \u0438\u0437 \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u0435\u0439 \u0438\u043d\u0441\u0442\u0430\u043b\u043b\u044f\u0446\u0438\u0439 \u0432 \u0442\u043e\u043c, \u0447\u0442\u043e \u0434\u0430\u043d\u043d\u044b\u0445 \u043c\u043d\u043e\u0433\u043e. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":86905,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-86904","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a GitLab \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0431\u044d\u043a\u0430\u043f\u044b \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 NextCloud | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-01T05:42:30+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-01T05:42:30+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Cum ajut\u0103 GitLab la realizarea backup-urilor pentru mari stoc\u0103ri NextCloud | ProHoster","description":"Salut, Habr! Ast\u0103zi vreau s\u0103 v\u0103 povestesc despre experien\u021ba noastr\u0103 \u00een automatizarea backup-ului pentru stoc\u0103ri mari de date Nextcloud \u00een diferite configura\u021bii.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a GitLab \u043f\u043e\u043c\u043e\u0433\u0430\u0435\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0431\u044d\u043a\u0430\u043f\u044b \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 NextCloud | ProHoster","og:description":"\u041f\u0440\u0438\u0432\u0435\u0442, \u0425\u0430\u0431\u0440! \u0421\u0435\u0433\u043e\u0434\u043d\u044f \u044f \u0445\u043e\u0447\u0443 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u0430\u0442\u044c \u043e \u043d\u0430\u0448\u0435\u043c \u043e\u043f\u044b\u0442\u0435 \u0430\u0432\u0442\u043e\u043c\u0430\u0442\u0438\u0437\u0430\u0446\u0438\u0438 \u0440\u0435\u0437\u0435\u0440\u0432\u043d\u043e\u0433\u043e \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449 Nextcloud \u0432 \u0440\u0430\u0437\u043d\u044b\u0445 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u044f\u0445.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-gitlab-pomogaet-delat-bekapy-bolshih-hranilishh-nextcloud","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-01T05:42:30+00:00","article:modified_time":"2020-07-01T05:42:30+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"86904","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 14:03:54","updated":"2022-10-06 09:05:46","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/86904","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=86904"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/86904\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/86905"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=86904"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=86904"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=86904"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}