Yadda GitLab ke taimaka muku madadin manyan ma'ajiyar NextCloud

Hai Habr!

A yau ina so in yi magana game da kwarewarmu wajen sarrafa sarrafa manyan bayanai daga ma'ajiyar ta Nextcloud a cikin jeri daban-daban. Ina aiki a matsayin tashar sabis a Molniya AK, inda muke yin tsarin sarrafa tsarin IT; Nextcloud ana amfani dashi don adana bayanai. Ciki har da, tare da tsarin rarraba, tare da sakewa.

Matsalolin da ke fitowa daga fasalulluka na shigarwa shine cewa akwai bayanai da yawa. Sigar da Nextcloud ya samar, sakewa, dalilai na zahiri, da ƙari suna ƙirƙirar kwafi da yawa.

prehistory

Lokacin gudanar da Nextcloud, matsalar shirya ingantaccen madadin ta taso, wanda dole ne a rufaffen, tunda bayanan suna da mahimmanci.

Muna ba da zaɓuɓɓuka don adana madogara a wurinmu ko a abokin ciniki akan injuna daban daga Nextcloud, wanda ke buƙatar sassauƙan tsarin gudanarwa ta atomatik.

Akwai abokan ciniki da yawa, dukkansu suna da tsari daban-daban, kuma duk akan rukunin yanar gizon su kuma tare da halayensu. Wannan daidaitaccen dabara ne lokacin da duk rukunin yanar gizon ya zama naku, kuma ana yin ajiyar kuɗi daga rawanin; bai dace da kyau ba.

Da farko, bari mu duba bayanan shigarwa. Muna bukata:

• Scalability dangane da kumburi ɗaya ko da yawa. Don manyan shigarwa muna amfani da minio azaman ajiya.
• Nemo game da matsaloli tare da yin madadin.
• Kuna buƙatar kiyaye wariyar ajiya tare da abokan cinikin ku da/ko tare da mu.
• Magance matsaloli cikin sauri da sauƙi.
• Abokan ciniki da shigarwa sun bambanta da juna - ba za a iya samun daidaito ba.
• Gudun dawowa ya kamata ya zama kadan a cikin yanayi biyu: cikakken farfadowa (bala'i), babban fayil daya goge bisa kuskure.
• Ana buƙatar aikin ƙaddamarwa.

Don magance matsalar sarrafa madadin, mun shigar da GitLab. Karin bayani ta hanyar magance.

Tabbas, ba mu ne farkon da za mu magance irin wannan matsalar ba, amma da alama a gare mu cewa ƙwarewar da muka samu ta aiki tana da ban sha'awa kuma a shirye muke mu raba ta.

Tun da kamfaninmu yana da manufar buɗe tushen, muna neman mafita ta hanyar buɗe ido. Bi da bi, mu raba mu ci gaban da post su. Misali, akan GitHub akwai plugin ɗin mu na Nextcloud, wanda muke samarwa ga abokan ciniki, inganta tsaro na bayanai idan an yi kuskure ko sharewa da gangan.

Ajiyayyen kayan aikin

Mun fara nemo hanyoyin warwarewa ta hanyar zabar kayan aikin ƙirƙirar madadin.

Tar + gzip na yau da kullun baya aiki da kyau - an kwafi bayanan. Ƙaruwa sau da yawa ya ƙunshi ƴan canje-canje na gaske, kuma yawancin bayanai da ke cikin fayil guda ana maimaita su.
Akwai wata matsala - redundancy na rarraba bayanai ajiya. Muna amfani da minio kuma bayanan sa ba su da yawa. Ko kuma dole ne ku yi wariyar ajiya ta hanyar minio kanta - loda shi kuma yi amfani da duk masu sarari tsakanin tsarin fayil, kuma, ba ƙaramin mahimmanci ba, akwai haɗarin mantawa game da wasu buckets da bayanan meta. Ko amfani da cirewa.

Ana samun kayan aikin madadin tare da kwafi a buɗaɗɗen tushe (a kan Habré akwai labarai game da wannan batu) kuma 'yan wasan mu sun kasance Borg и Karkara. Kwatancenmu na aikace-aikacen biyu yana ƙasa, amma a yanzu za mu gaya muku yadda muka tsara tsarin gaba ɗaya.

Sarrafa madadin

Borg da Restic suna da kyau, amma ba samfurin da ke da tsarin sarrafawa ta tsakiya. Don manufar gudanarwa da sarrafawa, mun zaɓi kayan aiki da muka riga muka aiwatar, ba tare da wanda ba za mu iya tunanin aikinmu ba, gami da aiki da kai - wannan shine sanannen CI / CD - GitLab.

Manufar ita ce kamar haka: an shigar da gitlab-mai gudu akan kowane kumburi da ke adana bayanan Nextcloud. Mai gudu yana gudanar da rubutun akan jadawali wanda ke lura da tsarin ajiyar kuɗi, kuma yana ƙaddamar da Borg ko Restic.

Me muka samu? Sake amsawa daga aiwatarwa, ingantaccen iko akan canje-canje, cikakkun bayanai idan akwai kuskure.

a nan nan akan GitHub mun buga misalan rubutun don ayyuka daban-daban, kuma mun ƙare haɗa shi zuwa madadin ba kawai Nextcloud ba, har ma da sauran ayyuka da yawa. Hakanan akwai mai tsarawa a wurin idan ba kwa son saita shi da hannu (kuma ba ma so) da .gitlab-ci.yml

Babu wata hanya ta canza lokacin CI/CD a cikin Gitlab API tukuna, amma ƙarami ne. Yana buƙatar ƙarawa, in ji 1d.

GitLab, da sa'a, na iya ƙaddamar da ba kawai bisa ga alƙawarin ba, amma bisa ga jadawalin, wannan shine ainihin abin da muke buƙata.

Yanzu game da rubutun nannade.

Mun saita sharuɗɗa masu zuwa don wannan rubutun:

• Ya kamata a ƙaddamar da shi duka ta mai gudu da hannu daga na'ura wasan bidiyo tare da ayyuka iri ɗaya.
• Dole ne a sami masu sarrafa kuskure:
• dawo code.
• nemo kirtani a cikin log ɗin. Misali, a gare mu kuskure yana iya zama saƙon da shirin bai ɗauka mai mutuwa ba.
• Lokacin aiwatarwa. Lokacin jagora dole ne ya zama m.
• Muna buƙatar cikakken bayani. Amma kawai idan akwai kuskure.
• Hakanan ana yin gwaje-gwaje da yawa kafin farawa.
• Ƙananan kari don dacewa waɗanda muka sami amfani yayin aiwatar da tallafi:
• An rubuta farkon da ƙarshen a cikin syslog na injin gida. Wannan yana taimakawa wajen haɗa kurakuran tsarin da aikin madadin.
• Wani ɓangare na log ɗin kuskure, idan akwai, ana fitarwa zuwa stdout, ana rubuta duk log ɗin zuwa wani fayil daban. Yana da dacewa don duba CI nan da nan kuma kimanta kuskuren idan yana da mahimmanci.
• Hanyoyin gyara kurakurai.

An adana cikakken log ɗin azaman kayan tarihi a GitLab; idan babu kuskure, an share log ɗin. Muna rubuta rubutun a cikin bash.

Za mu yi farin cikin yin la'akari da kowane shawarwari da sharhi game da buɗaɗɗen tushe - maraba.

Ta yaya wannan aikin

An ƙaddamar da mai gudu tare da Bash executor akan kullin madadin. A cewar mai tsarawa, an ƙaddamar da aikin CI / CD a cikin juzu'i na musamman. Mai gudu yana ƙaddamar da rubutun kundi na duniya don irin waɗannan ayyuka, yana bincika ingancin ma'ajin ajiyar ajiya, wuraren hawan dutse da duk abin da muke so, sannan ya dawo da tsaftace tsohon. Ana aika madadin da aka gama da kansa zuwa S3.

Muna aiki bisa ga wannan makirci - mai ba da AWS na waje ne ko kuma daidai da Rasha (yana da sauri kuma bayanan baya barin Tarayyar Rasha). Ko kuma mu sanya wani gungu na minio daban don abokin ciniki akan rukunin yanar gizon sa don waɗannan dalilai. Yawancin lokaci muna yin haka don dalilai na tsaro, lokacin da abokin ciniki ba ya son bayanan su bar kewayen su kwata-kwata.

Ba mu yi amfani da fasalin aika madadin ta hanyar ssh ba. Wannan baya ƙara tsaro, kuma ƙarfin hanyar sadarwa na mai bada S3 ya fi na'urar ssh ɗin mu ɗaya girma.

Domin kare na'urar ku daga mai satar bayanai, tunda yana iya goge bayanai akan S3, dole ne ku kunna sigar.
Ajiyayyen ko da yaushe yana ɓoye maajiyar.

Borg yana da yanayin da ba a ɓoye ba none, amma ba mu ba da shawarar kunna shi ba. A cikin wannan yanayin, ba kawai za a sami ɓoyewa ba, amma ba a ƙididdige adadin abin da ake rubutawa ba, wanda ke nufin za a iya bincika amincin kawai a kaikaice, ta amfani da maƙasudi.

Mai tsara tsarawa daban yana bincika madadin don amincin fihirisa da abun ciki. Duban yana jinkiri kuma yana da tsayi, don haka muna gudanar da shi daban sau ɗaya a wata. Yana iya ɗaukar kwanaki da yawa.

Readme cikin Rashanci

Babban ayyuka

• prepare shiri
• testcheck duban shiri
• maincommand tawagar core
• forcepostscript aikin da aka aiwatar a ƙarshe ko bisa kuskure. Muna amfani da shi don cire bangare.

Ayyukan sabis

• cleanup Muna rikodin kurakurai ko goge fayil ɗin log ɗin.
• checklog bitar log ɗin don faruwar layi tare da kuskure.
• ret fita mai sarrafa.
• checktimeout duba lokacin fita.

muhalli

• VERBOSE=1 Muna nuna kurakurai akan allon nan da nan (stdout).
• SAVELOGSONSUCCES=1 ajiye log akan nasara.
• INIT_REPO_IF_NOT_EXIST=1 Ƙirƙiri wurin ajiya idan babu shi. An kashe ta tsohuwa.
• TIMEOUT matsakaicin lokaci don babban aiki. Kuna iya saita shi azaman 'm', 'h' ko 'd' a ƙarshen.

Yanayin ajiya don tsoffin kwafi. Na baya:

• KEEP_DAILY=7
• KEEP_WEEKLY=4
• KEEP_MONTHLY=6

Canje-canje a cikin rubutun

• ERROR_STRING - kirtani don rajistan shiga log don kuskure.
• EXTRACT_ERROR_STRING - magana don nunin kirtani idan kuskure.
• KILL_TIMEOUT_SIGNAL - sigina don kisa idan lokaci ya ƙare.
• TAIL - nawa igiyoyi masu kurakurai akan allo.
• COLORMSG - launi na saƙo (tsoho rawaya).

Wannan rubutun, wanda ake kira wordpress, yana da suna mai ka'ida, dabararsa ita ce ita ma tana adana bayanan sirri na mysql. Wannan yana nufin ana iya amfani da shi don shigarwar Nexcloud mai kumburi guda ɗaya, inda kuma zaku iya adana bayanan. Dacewar ba wai kawai cewa komai yana wuri ɗaya ba, amma har ma abubuwan da ke cikin bayanan suna kusa da abubuwan da ke cikin fayilolin, tunda bambancin lokaci kaɗan ne.

Restic vs Borg

Hakanan akwai kwatancen tsakanin Borg da Restic a nan Habre, kuma ba mu da aikin yin wani kawai, amma namu. Yana da mahimmanci a gare mu yadda zai kalli bayananmu, tare da takamaiman bayananmu. Mu kawo su.

Ma'auni na zaɓinmu, ban da waɗanda aka ambata (haɓaka, farfadowa da sauri, da sauransu):

• Juriya ga aikin da ba a gama ba. Duba kisa -9.
• Girma a kan faifai.
• Bukatar albarkatu (CPU, ƙwaƙwalwar ajiya).
• Girman ɓangarorin da aka adana.
• Yin aiki tare da S3.
• Tabbatar da mutunci.

Don gwaji, mun ɗauki abokin ciniki ɗaya tare da ainihin bayanai da jimlar girman 1,6 TB.
Sharuɗɗa.

Borg bai san yadda ake aiki kai tsaye tare da S3 ba, kuma mun hau faifan azaman fuse, ta hanyar gofis. Restic ya aika shi zuwa S3 kanta.

Goofys yana aiki da sauri kuma da kyau, kuma akwai faifai cache module, wanda ke kara hanzarta aikin. Yana cikin matakin beta, kuma, a zahiri, mun yi karo da asarar bayanai yayin gwaje-gwaje (wasu). Amma saukaka shi ne cewa tsarin ajiyar kanta baya buƙatar karatu mai yawa, amma yawanci rubuce-rubuce, don haka muna amfani da cache ne kawai a lokacin binciken gaskiya.

Don rage tasirin hanyar sadarwar, mun yi amfani da mai ba da gida - Yandex Cloud.

Sakamakon gwajin kwatance.

• Kill -9 tare da ƙarin sake kunnawa duka sun yi nasara.
• Girma a kan faifai. Borg na iya damfara, don haka sakamakon ya kasance kamar yadda ake sa ran.

Mai baya
size

Borg
562Gb

Karkara
628Gb

• Da CPU
  Borg kanta yana cinye kadan, tare da tsoho matsawa, amma dole ne a kimanta shi tare da tsarin goofys. Gabaɗaya, suna kwatankwacinsu kuma suna amfani da kusan nau'ikan 1,2 akan injin kama-da-wane na gwaji iri ɗaya.
• Ƙwaƙwalwar ajiya. Restic yana kusan 0,5GB, Borg yana kusan 200MB. Amma wannan duk ba shi da mahimmanci idan aka kwatanta da cache fayil ɗin tsarin. Don haka yana da kyau a ware ƙarin ƙwaƙwalwar ajiya.
• Bambance-bambancen girman blob ya kasance mai ban mamaki.

Mai baya
size

Borg
kusan 500MB

Karkara
kusan 5MB

• Kwarewar Restic ta S3 tana da kyau kwarai. Yin aiki tare da Borg ta hanyar goofys ba ya tayar da wasu tambayoyi, amma an lura cewa yana da kyau a yi umount bayan an gama ajiyar ajiyar don sake saita cache gaba daya. Bambance-bambancen S3 shine cewa ba za a taɓa aika guntun da ba za a iya turawa zuwa guga ba, wanda ke nufin cewa cikakkun bayanan da ba su cika ba suna haifar da babbar lalacewa.
• Tabbatar da amincin yana aiki da kyau a cikin duka biyun, amma saurin ya bambanta sosai.
  Restic 3,5 hours.
  Borg, tare da 100GB SSD cache fayil - 5 hours.Kimanin sakamakon saurin guda ɗaya idan bayanan suna kan diski na gida.
  Borg yana karantawa kai tsaye daga S3 ba tare da cache ba 33 hours. Babban tsayi.

Layin ƙasa shine Borg na iya damfara kuma yana da manyan ɓangarorin - wanda ke sanya ajiya da ayyukan GET/PUT a cikin S3 mai rahusa. Amma wannan ya zo a farashin ƙarin hadaddun da tabbatarwa a hankali. Amma game da saurin dawowa, ba mu lura da wani bambanci ba. Restic yana ɗaukar bayanan baya (bayan na farko) ɗan lokaci kaɗan, amma ba mahimmanci ba.

A ƙarshe amma ba ko kaɗan a cikin zaɓin shine girman al'umma.

Kuma mun zabi borg.

Kalmomi kaɗan game da matsawa

Borg yana da kyakkyawan sabon matsawa algorithm a cikin arsenal - zstd. Ingancin matsawa bai fi gzip muni ba, amma da sauri. Kuma kwatankwacinsa cikin sauri zuwa tsoho lz4.

Misali, juji na MySQL ana matsawa sau biyu fiye da lz4 a cikin gudu iri ɗaya. Duk da haka, ƙwarewa tare da ainihin bayanan yana nuna cewa akwai ɗan bambanci sosai a cikin ma'aunin matsawa na kullin Nextcloud.

Borg yana da yanayin matsawa na kyauta - idan fayil ɗin yana da babban entropy, to ba a amfani da matsawa kwata-kwata, wanda ke ƙara saurin gudu. An kunna ta zaɓi lokacin ƙirƙira
-C auto,zstd
don zstd algorithm
Don haka tare da wannan zaɓi, idan aka kwatanta da matsawar tsoho, mun samu
560Gb da 562Gb bi da bi. Bayanan daga misalin da ke sama, bari in tunatar da ku, ba tare da matsawa ba sakamakon shine 628Gb. Sakamakon bambancin 2GB ya ɗan ba mu mamaki, amma mun yi tunanin cewa za mu zaɓa bayan haka. auto,zstd.

Hanyar tabbatarwa ta Ajiyayyen

Bisa ga mai tsarawa, ana ƙaddamar da na'ura mai mahimmanci kai tsaye daga mai badawa ko daga abokin ciniki, wanda ke rage yawan nauyin cibiyar sadarwa. Aƙalla yana da arha fiye da haɓaka shi da kanka da tuƙin zirga-zirga.

goofys --cache "--free:5%:/mnt/cache" -o allow_other --endpoint https://storage.yandexcloud.net --file-mode=0666 --dir-mode=0777 xxxxxxx.com /mnt/goofys
export BORG_PASSCOMMAND="cat /home/borg/.borg-passphrase"
borg list /mnt/goofys/borg1/
borg check --debug -p --verify-data /mnt/goofys/borg1/

Yin amfani da wannan makirci, muna bincika fayiloli tare da riga-kafi (bayan gaskiya). Bayan haka, masu amfani suna loda abubuwa daban-daban zuwa Nextcloud kuma ba kowa yana da riga-kafi ba. Gudanar da dubawa a lokacin zuba jari yana ɗaukar lokaci mai yawa kuma yana tsoma baki cikin kasuwanci.

Ana samun scalability ta hanyar gudu masu gudu akan nodes daban-daban tare da alamomi daban-daban.
Sa ido kan mu yana tattara bayanan ajiya ta hanyar GitLab API a cikin taga ɗaya; idan ya cancanta, ana iya lura da matsaloli cikin sauƙi kuma kamar yadda a sauƙaƙe.

ƙarshe

A sakamakon haka, mun san tabbas cewa muna yin ajiyar kuɗi, cewa bayananmu suna da inganci, matsalolin da suka taso tare da su suna ɗaukar lokaci kaɗan kuma ana warware su a matakin mai kula da aiki. Ajiyayyen yana ɗaukar sarari kaɗan sosai idan aka kwatanta da tar.gz ko Bacula.

source: www.habr.com