Unjiniyela we-SRE - oqeqeshwayo
Okokuqala, ake ngizethule. Nginguye , unjiniyela ophambili eqenjini I-GitLab. Ngesonto eledlule, ngibe nelungelo lokufunda nomunye wonjiniyela bethu be-SRE. Umgomo bekuwukubona ukuthi unjiniyela osocingweni uphendula kanjani ezehlakalweni nsuku zonke futhi athole ulwazi lomhlaba wangempela. Singathanda ukuthi onjiniyela bethu baqonde kangcono izidingo zabasebenzisi. Iqapha::Impilo.
Nganikezwa umsebenzi wokuthungatha unjiniyela we-SRE isonto lonke. Lokhu kusho ukuthi ngangikhona ekunikezeni, ngiqaphe iziteshi ezifanayo zokuxwayisa, futhi ngiphendula izehlakalo uma zenzekile futhi nini.
Izehlakalo
Kube nezigameko ezi-2 ngesonto.
1. I-Cryptominer
I-GitLab.com ibone i-spike isetshenziswa ngoLwesithathu. Kubikwe ukwephulwa komthetho okudalwe yimizamo yokusebenzisa amaminithi wokumba izimayini ze-cryptocurrency. Isigameko saxazululwa kusetshenziswa ithuluzi lethu lokunciphisa, elinqamula imisebenzi yomgijimi futhi lisuse iphrojekthi ehlobene ne-akhawunti.
Ukube lo mcimbi wawungazange ubonwe, ngabe ubanjwe ithuluzi elizenzakalelayo, kodwa kulokhu, unjiniyela we-SRE wabona ukwephulwa kuqala. Kwadalwa umsebenzi walesi sigameko, kodwa ulwazi mayelana naso lwavalwa.
2. Ukwehliswa kokusebenza kwezicelo ze-Canary kanye ne-Main
Isigameko sidalwe ukwehla nokwenyuka kwamazinga amaphutha ku-canary kanye nezinhlelo zokusebenza zewebhu eziyinhloko ku-Gitlab.com. Amanani amaningana e-Apdex aphuliwe.
Vula umsebenzi wesigameko:
Ukutholwa Okusemqoka
Nazi izinto ezimbalwa engizifundile phakathi nesonto ngisemsebenzini.
1. Izaziso ziwusizo kakhulu lapho zibona ukuchezuka kokujwayelekile.
Izaziso zingahlukaniswa zibe izinhlobo ezimbalwa:
- Izaziso ezisekelwe kumkhawulo othile, njengokuthi "amaphutha angu-10 5xx enzeka ngomzuzwana."
- Izexwayiso lapho i-threshold iyinani lephesenti "njengezinga lamaphutha angu-5xx ngo-10% wesamba sevolumu yesicelo ngesikhathi esithile."
- Izaziso ezisuselwe kusilinganiso somlando esifana "namaphutha angu-5xx kumaphesenti angama-90".
Ngokuvamile, Izinhlobo 2 kanye no-3 ziwusizo kakhulu kuma-SRE ocingweni ngoba ziveza ukuchezuka kokujwayelekile enqubweni.
2. Izaziso eziningi azikhuli zibe izehlakalo
Onjiniyela be-SR babhekana nezixwayiso eziqhubekayo, eziningi zazo ezingabalulekile ngempela.
Ngakho kungani ungagcini izexwayiso kulezo ezibaluleke ngempela kuphela? Kodwa-ke, le ndlela ingase iphuthelwe yizimpawu eziyisixwayiso zakuqala zokuthi yini engase ibe inkinga yangempela esongela umonakalo omkhulu.
Umsebenzi we-SRE osemsebenzini ukunquma ukuthi yiziphi izexwayiso ezibucayi ngempela nokuthi zidinga ukukhushulwa futhi ziphenywe. Ngisola ukuthi lokhu kuphinde kube ngenxa yokuqina kokuxwayisa: kungaba ngcono ukwethula amaleveli amaningi noma izindlela "ezihlakaniphile" zokumisa izexwayiso ngokuya ngesimo esichazwe ngenhla.
Isiphakamiso Sesici:
3. Ama-SRE ethu ocingo asebenzisa amathuluzi amaningi
Ngaphakathi:
- Iphrojekthi ye-infra ye-GitLab: ikhaya lama-runbook, ukuhanjiswa/ukunikezwa komsebenzi wamasonto onke, kanye nemisebenzi yokuphendula izigameko.
- Izinkinga ze-GitLab: uphenyo, ukubuyekezwa, nokunakekelwa nakho kulandelelwa emisebenzini.
- Amalebula e-GitLab: Imisebenzi ye-automation iqalwa amalebula athile, ama-bots awasebenzisayo ukulandelela umsebenzi.
Kwangaphandle:
- I-PagerDuty: Izaziso
- I-Slack: Yilapho imilayezo ye-PagerDuty/AlertManager ithunyelwa khona. Ukuhlanganiswa nemiyalo ye-slash kuvumela imisebenzi ehlukahlukene, njengokuchitha isexwayiso noma ukusidlulisela esenzakalweni.
- I-Grafana: Ukubukwa kwe-Metrics okugxile kumathrendi esikhathi eside.
- I-Kibana: Ihlinzeka ngokubonwa kwelogi/ukusesha, ikhono lokumba ujule emicimbini ethile.
- Sondeza: I-Zoom ine "igumbi lomhlangano weqembu" elisebenza unomphela. Lokhu kuvumela onjiniyela be-SRE ukuthi baxoxe ngokushesha ngemicimbi ngaphandle kokumosha isikhathi esiyigugu bedala igumbi futhi babelane ngesixhumanisi nabahlanganyeli.
Futhi okuningi, okuningi kakhulu.
4. Ukuqapha i-GitLab.com nge-GitLab kuyiphuzu elilodwa lokwehluleka
Uma i-GitLab.com ihlangabezana nokunqamuka kwesevisi okukhulu, ngeke sifune ukuthi kuthinte ikhono lethu lokuxazulula inkinga. Lokhu kungancishiswa ngokwethula isenzakalo sesibili se-GitLab ukuze uphathe i-GitLab.com. Eqinisweni, lokhu sekuvele kusisebenzela: .
5. Izici ezimbalwa ongacabanga ukuzingeza ku-GitLab
- , efana ne-Google Amadokhumenti. Lokhu kungaba usizo emisebenzini esekelwe ezehlakalweni phakathi nomcimbi, kanye nasekuxoxisaneni. Kuzo zombili izimo, abahlanganyeli abaningi bangase badinge ukungeza okuthile ngesikhathi sangempela.
- Ama-webhook amaningi emisebenzi. Ikhono lokucupha izinyathelo zokugeleza komsebenzi ze-GitLab ngaphakathi kuzosiza ukunciphisa ukuthembela ekuhlanganisweni kwe-Slack. Isibonelo, ikhono lokunika izaziso amandla ku-PagerDuty ngomyalo we-slash kumsebenzi we-GitLab.
isiphetho
Onjiniyela be-SRE babhekene nenqwaba yezinselelo. Kungaba kuhle ukubona imikhiqizo eminingi ye-GitLab ibhekana nalezi zinkinga. Sesivele sisebenza kwezinye izengezo zomkhiqizo ezizokwenza kube lula ukugeleza komsebenzi okukhulunywe ngenhla. Imininingwane iyatholakala ku .
Ngo-2020, sinweba ithimba lethu ukuze sihlanganise zonke lezi zici ezinhle. Uma ungathanda, sicela uhlole , futhi uzizwe ukhululekile ukuxhumana nanoma yiliphi iqembu lethu nganoma imiphi imibuzo.
Source: www.habr.com
