Indlela engalichitha ngayo isonto njengonjiniyela oqeqeshwayo we-SRE. Umsebenzi ngeso likanjiniyela wesoftware

Unjiniyela we-SRE - oqeqeshwayo

Okokuqala, ake ngizethule. mina - @tristan.read, unjiniyela ophambili eqenjini Iqapha::Impilo I-GitLab. Ngesonto eledlule ngibe nelungelo lokufunda nomunye wonjiniyela bethu be-SRE. Umgomo wawuwukubona ukuthi isikhulu esisemsebenzini sisabela kanjani ezenzakalweni nsuku zonke futhi sithole ulwazi lwempilo yangempela emsebenzini. Singathanda ukuthi onjiniyela bethu baqonde kangcono izidingo zabasebenzisi imisebenzi Iqapha::Impilo.

Kwadingeka ngilandele unjiniyela we-SRE yonke indawo isonto lonke. Okusho ukuthi, ngangikhona ekunikezeni impahla, ngaqapha iziteshi ezifanayo zokuxwayisa futhi ngasabela ezigamekweni uma zenzeka futhi nini.

Izehlakalo

Kube nezigameko ezi-2 phakathi nesonto.

1. Umvukuzi we-Crypto

I-GitLab.com ibone ukweqa ekusebenziseni ngoLwesithathu I-GitLab Runner'a, okubangelwa imizamo yokusebenzisa amaminithi omgijimi ukumba i-cryptocurrency. Isigameko kwasingathwa ngaso kusetshenziswa ithuluzi lethu lokwephula umthetho, elimisa imisebenzi yomgijimi futhi lisuse iphrojekthi ne-akhawunti ehlotshaniswa nayo.

Ukube lo mcimbi ubungaqashelwanga, ithuluzi elizenzakalelayo belizowubamba, kodwa kulokhu, unjiniyela we-SRE wabona ukwephulwa komthetho kuqala. Umsebenzi wesigameko wakhiwe, kodwa ulwazi ngawo luvaliwe.

2. Ukwehliswa kokusebenza kwezicelo ze-Canary kanye ne-Main

Isigameko sidalwe ukwehla nokwanda kwamaphutha ku-canary kanye nezinhlelo zokusebenza zewebhu eziyinhloko ku-Gitlab.com. Amanani amaningana e-Apdex aphuliwe.

Vula umsebenzi wesigameko: https://gitlab.com/gitlab-com/gl-infra/production/issues/1442

Ukutholwa Okusemqoka

Nazi izinto ezimbalwa engizifundile phakathi nesonto ngisemsebenzini.

1. Izaziso ziwusizo kakhulu lapho uthola ukuchezuka kokujwayelekile.

Izaziso zingahlukaniswa zibe izinhlobo eziningana:

• Izexwayiso ezisuselwe kunani elithile lomkhawulo, njengokuthi "amaphutha angu-10 5xx enzeka ngomzuzwana."
• Izexwayiso lapho i-threshold iyivelu yephesenti "njengemvamisa yamaphutha angu-5xx ngo-10% wesamba sevolumu yezicelo ngesikhathi esithile."
• Izexwayiso ezisuselwe kusilinganiso somlando "njengamaphutha angu-5xx ku-90th percentile".

Ngokuvamile, izinhlobo 2 kanye no-3 ziwusizo kakhulu kuma-SRE asemsebenzini, njengoba ziveza ukuchezuka kokujwayelekile kwinqubo.

2. Izaziso eziningi azifiki ezehlakalweni.

Onjiniyela be-SR babhekana nezixwayiso eziqhubekayo, eziningi zazo ezingabalulekile ngempela.

Ngakho kungani ungagcini izexwayiso zakho kwezibaluleke ngempela kuphela? Ngale ndlela, nokho, ungase ungaziboni izimpawu zokuqala zokuthi i-snowball izoba yinkinga yangempela esongela umonakalo omkhulu.

Umsebenzi we-SRE ocingweni uwukuthola ukuthi yiziphi izexwayiso ezibonisa okuthile okubucayi, nokuthi zidinga ukukhushulwa futhi kubhekwane nazo. Ngisola ukuthi lokhu futhi kungenxa yokungaguquguquki kwezixwayiso: bekungaba ngcono uma bekunamaleveli amaningana noma izindlela "ezihlakaniphile" zokumisa izexwayiso ngokuhambisana nesimo esichazwe ngenhla.

Isiphakamiso Sesici: https://gitlab.com/gitlab-org/gitlab/issues/42633

3. Ama-SRE ethu asemsebenzini asebenzisa amathuluzi amaningi.

Ngaphakathi:

• Iphrojekthi ye-infra ye-GitLab: ama-runbook ahlala lapha, izabelo zeshifu/zesonto, imisebenzi yokuphendula isigameko.
• Izinkinga ze-GitLab: Uphenyo, ukubuyekezwa, nokunakekelwa nakho kulandelelwa ezindabeni.
• Amalebula e-GitLab: Imisebenzi ye-automation yethulwa kusetshenziswa amalebula athile, ama-bots awasebenzisayo ukulandelela umsebenzi owenziwayo.

Kwangaphandle:

• I-PagerDuty: Izaziso
• I-Slack: Ukugeleza komlayezo we-PagerDuty/AlertManager kuya lapha. Ukuhlanganiswa nemiyalo ye-slash ukwenza imisebenzi ehlukahlukene, njengokuvala isexwayiso noma ukukhuphukela esenzakalweni.
• I-Grafana: ukuboniswa kwamamethrikhi okugxilwe kumathrendi esikhathi eside.
• I-Kibana: Inikeza ukubuka/ukusesha kwelogi, ikhono lokumba ujule emicimbini ethile.
• Sondeza: Kukhona “igumbi lokuphumula” elihlala lisebenza ku-Zoom. Lokhu kuvumela onjiniyela be-SRE ukuthi baxoxe ngokushesha ngemicimbi ngaphandle kokumosha isikhathi esibalulekile bedala igumbi futhi baxhumanise ababambiqhaza.

Futhi abaningi abanye abaningi.

4. Ukuqapha i-GitLab.com nge-GitLab kuyiphuzu elilodwa lokwehluleka

Uma i-GitLab.com ihlangabezana nokunqamuka kwesevisi okukhulu, asifuni ukuthi kuthinte ikhono lethu lokuxazulula inkinga. Ingamiswa ngokwethula isibonelo sesibili se-GitLab ukuphatha i-GitLab.com. Eqinisweni, lokhu vele kuyasisebenzela: https://ops.gitlab.net/.

5. Izici ezimbalwa ongacabanga ukuzingeza ku-GitLab

• Umsebenzi wokuhlela wabasebenzisi abaningi, efana ne-Google Amadokhumenti. Lokhu kuzosiza ngemisebenzi yezehlakalo ngesikhathi somcimbi, kanye nemisebenzi yokuxoxisana. Kuzo zombili izimo, ababambiqhaza abambalwa bangadinga ukungeza okuthile ngesikhathi sangempela.
• Ama-webhook amaningi emisebenzi. Ikhono lokusebenzisa izinyathelo zokugeleza komsebenzi ze-GitLab kusuka ngaphakathi kuzosiza ukunciphisa ukuthembela kwakho ekuhlanganisweni kwe-Slack. Isibonelo, ikhono lokuvumela isexwayiso ku-PagerDuty ngomyalo we-slash odabeni lwe-GitLab.
  isiphetho

Onjiniyela be-SRE banesikhathi esinzima ngezinto eziningi eziyinkimbinkimbi. Kungaba kuhle ukubona imikhiqizo eminingi ye-GitLab ikhuluma nalezi zinkinga. Sesivele sisebenzela ezinye izengezo emkhiqizweni ezizokwenza ukuhamba komsebenzi okukhulunywe ngenhla kube lula. Imininingwane itholakala ku Isigaba se-Ops Product Vision.

Sinweba iqembu ngo-2020 ukuze sihlanganise zonke lezi zici ezinhle. Uma unentshisekelo, sicela uhlole izikhala, futhi uzizwe ukhululekile ukuxhumana nanoma ubani eqenjini lethu nganoma imiphi imibuzo.

Source: www.habr.com