Heino nyaya yakachinja zvachose maitiro angu ekuita basa re devops. Kudzoka munguva dzepre-Covid, nguva refu, pamberi pavo, ini nevakomana patakanga tichingoronga bhizinesi redu uye kusununguka pamaodha asina kujairika, imwe yekupihwa yakawira mungoro yangu.
Iyo kambani yakanyora iyi yaive data analytics kambani. Aigadzirisa zviuru zvezvikumbiro zuva nezuva. Ivo vakauya kwatiri nemazwi anoti: vakomana, isu tine ClickHouse uye isu tinoda kuita otomatiki kumisikidzwa kwayo uye kuisirwa. Tinoda Ansible, Terraform, Docker uye kuti zvese zvichengetwe muGit. Tinoda sumbu remanodhi mana ane replica mbiri imwe neimwe.
Icho chikumbiro chakajairwa, kune akawanda acho, uye iwe unoda yakaenzana yakanaka chiyero mhinduro. Takati "ok", uye mushure memavhiki e2-3 zvinhu zvose zvakagadzirira. Vakagamuchira basa uye vakatanga kuenda kune imwe Clickhouse cluster vachishandisa zvishandiso zvedu.
Hapana aida kana kuziva nzira yekutamba neClickhouse. Ipapo takafunga kuti iri ndiro dambudziko ravo guru, uye naizvozvo chiteshi chebasa chekambani chakangopa mvumo kuchikwata changu kuti chigadzirise basa racho nepose patinogona napo, kuti ndisazoendako ini pachangu.
Isu takaperekedza kufamba, mamwe mabasa akamuka - kuseta backups uye yekutarisa. Panguva imwecheteyo, chiteshi chebasa chekambani iyi chakabatana neimwe chirongwa, chichitisiya nemumwe wedu - Leonid - semutungamiriri. Lenya akanga asiri mukomana ane chipo. Mugadziri akareruka akangoerekana aiswa kutungamira Clickhouse. Zvinoita sokuti iri rakanga riri basa rake rokutanga rokutarisira chimwe chinhu, uye rukudzo rwakakura rwakaita kuti anzwe achinetseka.
Tose takatanga kugadzira ma backups. Ndakakurudzira kutsigira data rekutanga ipapo. Ingoitora, zvipi uye uikande zvine mutsindo mune imwe c3. Raw data igoridhe. Paive neimwe sarudzo - kutsigira matafura ivo pachavo muClickhouse, uchishandisa chando uye kukopa. Asi Lenya akauya nemhinduro yake.
Akazivisa kuti tinoda yechipiri Clickhouse cluster. Uye kubva zvino zvichienda mberi tichanyora data kumasumbu maviri - iyo huru uye yekuchengetedza. Ini ndinomuudza, Lenya, haizove backup, asi inoshanda replica. Uye kana data ikatanga kurasika mukugadzira, zvakafanana zvichaitika pane yako backup.
Asi Lenya akabata chidhiraivho zvakasimba akaramba kuteerera kupopotedzana kwangu. Takataura naye kwenguva yakareba mukutaura, asi pakanga pasina chekuita - Lenya aitungamirira chirongwa ichi, isu taingobhadharwa vana kubva mumugwagwa.
Isu takatarisa mamiriro echikwata uye tichibhadharisa chete basa revatariri. Yakachena Clickhouse manejimendi pasina kupinda mu data. Iyo cluster yaivepo, ma disks aive akanaka, ma node aive akanaka.
Hatina kuziva kuti takagamuchira odha iyi nekuda kwekusanzwisisana kunotyisa mukati mechikwata chavo
Maneja haana kufara kuti Clickhouse yainonoka uye data dzimwe nguva rairasika. Akagadza chiteshi chake chebasa basa rekufungidzira. Akazviona nepaaigona napo ndokugumisa kuti isu taingoda kuita otomatiki iyo Clickhouse - ndizvo chete. Asi sezvazvakazojeka, vaisada timu yemadevops zvachose.
Zvese izvi zvakaratidza kuva zvakanyanya, zvakanyanya kurwadza. Uye chakanyanya kugumbura ndechekuti raive zuva rangu rekuzvarwa.
Chishanu manheru. Ndakatsvaga pawaini yandaifarira ndikakoka vanamumba.
Anenge asati aenda, tinogashira basa rekugadzira alter, tinoipedza, zvese zvakanaka. Alter yapfuura, clickhouse yakasimbiswa. Isu tave kutoenda kubhawa, uye vanotinyorera kuti hapana data yakakwana. Takaverenga kuti zvese zvinoita kunge zvakakwana. Vakaenda kundopembera.
Muresitorendi maiva neruzha musi weChishanu. Takahodha zvinwiwa nechikafu, takagara pamasofa. Nguva yese iyi, shamhu yangu yaizara zvishoma nezvishoma nemashoko. Vakanyora chimwe chinhu pamusoro pekushaya data. Ndakafunga - mangwanani akachenjera kupfuura manheru. Kunyanya nhasi.
Kwave kuda kuma eleven vakabva vatanga kufona. Akanga ari mukuru wekambani ... "Zvimwe ndasarudza kundikorokotedza," ndakafunga ndichizengurira, uye ndakatora foni.
Uye ndakanzwa chimwe chinhu chakaita sekuti: "Wakanganisa data redu! Ndinokubhadhara, asi hapana chinoshanda! Iwe waive nebasa rekuchengetedza, uye hauna kuita chinhu chakaipa! Ngatigadzirise!" - chete kunyange ruder.
- Unoziva chii, bvisa iyo fuck! Nhasi izuva rangu rekuzvarwa, uye ikozvino ndichanwa, uye kwete kuita zvechikumi zvigadzirwa zvekumba kubva kune crap nematanda!
Ndizvo zvandisina kutaura. Ndakatora laptop yangu ndichibva ndatanga basa.
Aiwa, ndabhomba, ndabhomba segehena! Akadurura caustic "Ndakakuudza kudaro" muhurukuro - nokuti iyo backup, iyo yakanga isiri yekuchengetedza zvachose, - hongu, haina kuchengetedza chero chinhu.
Vakomana neni takafunga nzira yekumisa kurekodha nekutarisa zvese. Isu takaita chokwadi chekuti imwe data haina kunyorwa.
Takarega kurekodha uye takaverenga nhamba yezviitiko zvaivepo pazuva. Vakaisa mamwe data, iyo chete chetatu haina kunyorwa. Shard nhatu dzine 2 replicas imwe neimwe. Iwe unopinza 100.000 mitsara - 33.000 haina kunyorwa.
Paive nekuvhiringika chaiko. Wese munhu akaudza mumwe nemumwe kuti asiyane achichinjana: Lenya akaenda ikoko kutanga, achiteverwa neni uye muvambi wekambani. Chete chiteshi chesevhisi chakabatana chakaedza kutsausa mafoni edu ekushevedzera netsamba kutsvaga mhinduro kudambudziko.
Hapana ainzwisisa kuti chii chaizvo chaiitika
Ini nevakomana takangorohwa nemhepo patakaona kuti chetatu che data yese haina kunyorwa chete, yakarasika! Zvakazoitika kuti kurongeka mukambani kwaive sekutevera: mushure mekuiswa, iyo data yakabviswa zvisingaite, zviitiko zvakaraswa mumabheti. Ndaifungidzira kuti Sergei aizoshandura sei zvose izvi kuva rubles akarasika.
Birthday rangu rakakandwawo mumarara. Takagara pabhawa tichigadzira pfungwa, tichiedza kugadzirisa dambanemazwi rakanga rakandwa patiri. Chikonzero chekudonha kwaClickhouse chaive chisiri pachena. Pamwe ndiyo network, pamwe ndeye Linux marongero. Hongu, chero chaunoda, kwave nekufungidzira kwakakwana.
Ini handina kutora mhiko yemugadziri, asi kwaive kusatendeseka kusiya vakomana kune rumwe rutivi rwemutsara - kunyangwe vakatipa mhosva pane zvese. Ndakanga ndine chokwadi che 99% kuti dambudziko rakanga risiri muzvisarudzo zvedu, kwete kudivi redu. The 1% mukana wekuti takanga takanganisa waipisa nekufunganya. Asi zvisinei kuti dambudziko raiva divi ripi, raifanira kugadziriswa. Kusiya vatengi, zvisinei kuti ndivanaani, neiyo inotyisa data leak hutsinye hwakanyanya.
Takashanda patafura yekudyira kusvika kumaXNUMX:XNUMX mangwanani. Takawedzera zviitiko, isa sarudza, uye takaenda kunozadza mapeji. Kana iwe uchikwenya iyo data, aya ndiwo maitiro aunozviita: iwe unotora avhareji data yemazuva apfuura uye woiisa mune iwo akaputirwa.
Dzakakwana XNUMX mumangwanani, ini neshamwari yangu takaenda kumba kwangu ndokunohodha doro pamusika wedoro. Ndakanga ndakagara nelaptop neClickhouse matambudziko, shamwari yanga ichindiudza chimwe chinhu. Somugumisiro, mushure meawa akagumburwa kuti ndaishanda uye ndisingamwi doro naye, ndokubva aenda. Classic - Ndaive shamwari yeDevops.
Na6 am, ndakagadzira tafura zvakare, uye data yakatanga kufashama. Zvese zvakashanda pasina kana kurasikirwa.
Ipapo zvakanga zvakaoma. Vese vakapomerana mhosva yekurasikirwa nedata. Dai tsikidzi itsva yakaitika, ndine chokwadi chekuti pangadai pakave nekupfurana
Mukurwa uku, isu takazotanga kunzwisisa - kambani yaifunga kuti isu taive vakomana vanoshanda nedata uye vanoongorora chimiro chematafura. Vakavhiringa maadmin nema dealers. Uye vakauya kuzotibvunza zvakasiyana nemaadmin.
Chichemo chavo chikuru ndeche - chii gehena, iwe waive nemhosva yekuchengeta uye hauna kuzviita nemazvo, wakaramba uchiparadza data. Uye izvi zvese nemameti ekudzosera kumashure.
Ndaida kururamisira. Ndakachera tsamba uye ndakabatanidza zviratidziro zvemunhu wese, uko Leonid nesimba rake rose anomanikidza kuti vagadzire backup yakaitwa. Service station yavo yakatora rutivi rwedu mushure mekufona kwangu. Gare gare Lenya akabvuma mhosva yake.
Mukuru wekambani, pane kudaro, aisada kupa mhosva vanhu vake. Screenshots nemashoko akange asina basa paari. Aidavira kuti sezvo taiva nyanzvi pano, taifanira kupwisa munhu wose ndokuomerera pachisarudzo chedu. Sezviri pachena, basa redu rakanga riri rokudzidzisa Lenya uye, uyezve, kumupfuura, uyo akagadzwa semaneja wepurojekiti, kuti asvike kune chinhu chikuru uye isu pachedu kudurura kusava nechokwadi kwedu kose pamusoro peiyo pfungwa yekuchengetedza kwaari.
Kukurukurirana kwakazara neruvengo, hutsinye hwakavanzika uye husingavanzike. Ndakashaya kuti ndoita sei. Zvinhu zvese zvakamira. Uye vakabva vandirayira nzira iri nyore - kunyora meseji yemunhu kune maneja uye kuronga musangano naye. Vasya, vanhu muhupenyu chaihwo havakurumidze sezvavari mukutaura. Sabhuku akapindura kune yangu meseji: huya, hapana mubvunzo.
Wakanga uri musangano unotyisa zvikuru mubasa rangu. Shamwari yangu kubva kumutengi - STO - haina kuwana nguva. Ndakaenda kumeeting nabamukuru naLena.
Ndakadzokorora ndadzokororazve nhaurirano yedu inobvira mumusoro mangu. Ndakakwanisa kusvika nokukurumidza, pachine hafu yeawa. Ndakatanga kutya, ndakasvuta fodya 10. Ndakanzwisisa, ndizvozvo - ndiri kusvuta ndega. Handichakwanisa kuvanyengetedza. Akabva apinda muelevator.
Achiri kusimuka akarova lighter zvekuti akatsemuka.
Somugumisiro, Lenya akanga asiri pamusangano. Uye takava nekurukurirano huru pamusoro pezvose nasabhuku! Sergei akandiudza nezvekurwadziwa kwake. Akanga asingadi "kuitira Clickhouse" - aida "kuita kuti mibvunzo ishande."
Handina kuona mbudzi, asi mukomana akanaka, ainetseka nebhizimisi rake, akanyura mubasa 24/7. Kukurukurirana kunowanzotikwevera isu vanhu vakaipa, vanyengeri uye vanhu vakapusa. Asi muhupenyu ava vanhu vakangofanana newe.
Sergei aisada ma devops maviri ekubhadhara. Dambudziko ravaive naro rakazonyanya kukura.
Ndakati ndaigona kugadzirisa matambudziko ake - rinongova basa rakasiyana zvachose, uye ndine shamwari inorishandira. Dai takaziva kubva pakutanga kuti iri raive dhiri kwavari, tingadai takanzvenga zvakawanda. Kwapera nguva, asi takaona kuti dambudziko riri mukupenga kwedata management, kwete mukuvaka.
Takakwazisana, vakasimudza mubhadharo wedu kaviri nehafu, asi nemamiriro ezvinhu ekuti ini ndinotora zvachose kukanganisa nedata ravo uye Clickhouse yangu. Murifiti, ndakataurirana neiye DI mukomana Max ndikamubatanidza kuti ashande. Zvakanga zvakafanira kufoshora sumbu rose.
Paiva nemarara akawanda mupurojekiti yakagamuchirwa. Kutanga ne "backup" yakataurwa. Zvakazoitika kuti iyi imwechete "backup" cluster yakanga isiri yega. Vakaedza zvese pairi, dzimwe nguva vachizviisa mukugadzirwa.
Vagadziri vedu vari-mumba vakagadzira yavo yetsika data yekuisa. Akashanda seizvi: akabatanidza mafaera, akamhanyisa script uye akabatanidza iyo data mutafura. Asi dambudziko guru nderekuti huwandu hukuru hwe data hwakagamuchirwa kune chimwe chikumbiro chakareruka. Chikumbiro chakabatanidza data sekondi yega yega. Zvose nokuda kwenhamba imwe - mari pazuva.
Vagadziri vemukati vakashandisa iyo analytics chishandiso zvisiri izvo. Vakaenda kumagrafana ndokunyora chikumbiro chavo chehumambo. Akaisa data kwemavhiki maviri. Yakazova girafu rakanaka. Asi muchokwadi, chikumbiro chedata chaive masekonzi gumi ega ega. Zvese izvi zvaiunganidzirana mumutsara nekuti Clickhouse haina kungotora kugadzirisa. Apa ndipo pakavigwa chikonzero chikuru. Hapana chakashanda muGrafana, zvikumbiro zvakamira mumutsara, uye data yekare, isina basa yaigara ichisvika.
Isu takagadzirisa zvakare cluster, takaitazve kuisa. Vagadziri vemukati vakanyora zvakare "inseter" yavo, uye yakatanga kugovana data nemazvo.
Max akaita ongororo yakazara yezvivakwa. Akatsanangura hurongwa hwekushandura kune yakazara-yakazara backend. Asi izvi hazvina kukodzera kambani. Vaitarisira chakavanzika chemashiripiti kubva kuna Max chaizovabvumira kushanda nenzira yechinyakare, asi nemazvo chete. Lenya akanga achiri kutarisira basa racho, uye hapana chaakadzidza. Kubva pane zvese zvakapihwa, akasarudza zvakare imwe yake. Senguva dzose, iyi ndiyo yaive yakanyanya kusarudza ... sarudzo yakashinga. Lenya akatenda kuti kambani yake ine nzira inokosha. Minzwa uye izere nechando.
Chaizvoizvo, ndiko kwatakaparadzana - takaita zvataigona.
Tizere neruzivo nehuchenjeri kubva munhoroondo iyi, takavhura bhizinesi redu pachedu uye tikazvigadzirira misimboti yakati wandei. Hatife takatanga basa nenzira imwe cheteyo sezvataiita kare.
DJ Max akabatana nesu mushure mechirongwa ichi, uye tichiri kushanda zvakanaka pamwe chete. Nyaya yeClickhouse yakandidzidzisa maitiro ekuita ongororo yakakwana uye yakakwana yezvivakwa ndisati ndatanga basa. Tinonzwisisa kuti zvinhu zvose zvinoshanda sei uye chete ipapo tinobvuma mabasa. Uye kana pakutanga taizokurumidza kukurumidza kuchengetedza zvivakwa, zvino isu tinotanga kuita chirongwa chenguva imwe chete, icho chinotibatsira kunzwisisa kuti tingaiunza sei mumamiriro ekushanda.
Uye hongu, isu tinodzivirira mapurojekiti ane crappy infrastructure. Kunyange kana nokuda mari yakawanda, kunyange kana kunze ushamwari. Hazvibatsiri kuita zvirongwa zvinorwara. Kuziva izvi kwakatibatsira kukura. Ingave purojekiti yenguva imwe chete yekugadzira zvivakwa zvakarongeka uyezve chibvumirano chekugadzirisa, kana isu tinongobhururuka. Yakapfuura imwe iceberg.
PS Saka kana uine mibvunzo nezve zvivakwa zvako, .
Tine 2 emahara edhisheni pamwedzi, pamwe chirongwa chako chichava chimwe chazvo.
Source: www.habr.com