Indlela endiyichithe ngayo iveki njengenjineli ye-SRE intern. Umsebenzi ngeliso lenjineli yesoftware

Injineli ye-SRE - umqeqeshi

Okokuqala, mandizazise. Mna - @tristan.funda, injineli yangaphambili kwiqela Moni ::Impilo GitLab. Kule veki iphelileyo ndiye ndanewonga lokuziqhelanisa nomnye weenjineli zethu ze-SRE. Injongo yayikukujonga indlela igosa elisebenza ngalo elisabela ngayo kwiziganeko zemihla ngemihla kunye nokufumana amava obomi bokwenyani emsebenzini. Singathanda ukuba iinjineli zethu ziqonde ngcono iimfuno zabasebenzisi imisebenzi Moni ::Impilo.

Kwafuneka ndilandele injineli ye-SRE kuyo yonke indawo iveki. Oko kukuthi, bendikho ekunikezelweni, ndibeke iliso kwitshaneli zesilumkiso ezifanayo kwaye ndiphendule kwizehlo ukuba zenzeke kwaye nini.

Iziganeko

Kwakukho iziganeko ze-2 phakathi kweveki.

1. I-Cryptominer

I-GitLab.com ibone ukutsiba ekusebenziseni ngoLwesithathu Imbaleki yeGitLab'a, okubangelwa iinzame zokusebenzisa imizuzu imbaleki ukuba cryptocurrency yam. Esi siganeko sasingathwa ngokusetyenziswa kwesixhobo sethu sokwaphulwa kwe-neutralization, esimisa imisebenzi yembaleki kwaye sicime iprojekthi kunye neakhawunti ehambelana nayo.

Ukuba esi siganeko sasingazange siqatshelwe, isixhobo esizenzekelayo sasiyibambe, kodwa kulo mzekelo, injineli ye-SRE yaqaphela ukuphulwa kuqala. Umsebenzi wesiganeko wenziwa, kodwa ulwazi malunga nalo luvaliwe.

2. Ukuthotywa komgangatho wokusebenza kweCanary kunye nezicelo eziPhambili

Isiganeko sibangelwa ukucotha kunye nokunyuka rhoqo kweempazamo kwi-canary kunye nezicelo eziphambili zewebhu kwi-Gitlab.com. Amanani amaninzi e-Apdex aye aphulwa.

Vula umsebenzi wesiganeko: https://gitlab.com/gitlab-com/gl-infra/production/issues/1442

Iziphumo eziphambili

Nazi izinto ezimbalwa endizifundileyo kwiveki yam emsebenzini.

1. Izilumkiso ziluncedo kakhulu xa uqaphela ukutenxa kwisiqhelo.

Izilumkiso zinokwahlulwa zibe ziindidi ezininzi:

• Izivuseleli ezisekelwe kwixabiso elithile le-threshold, ezifana "neempazamo eziyi-10 5xx zenzekile ngomzuzwana."
• Izaziso apho umda ulixabiso lepesenti "ezifana neempazamo ze-5xx nge-10% yomthamo uwonke wezicelo ngexesha elithile."
• Izivuseleli ezisekelwe kumndilili wembali ezifana "neempazamo ze-5xx kwi-90th percentile".

Ngokubanzi, iindidi zesi-2 kunye no-3 ziluncedo ngakumbi kwii-SRE ezisemsebenzini, kuba zityhila ukutenxa kwisiqhelo kwinkqubo.

2. Izilumkiso ezininzi azinakuya kwiziganeko.

Iinjineli ze-SR zijongana nothotho lwezilumkiso, uninzi lwazo ezingabalulekanga ngokwenene.

Ngoko kutheni ungazibekeli umda izilumkiso zakho kuphela kwezo zibaluleke ngokwenene? Ngale ndlela, nangona kunjalo, awukwazi ukuqaphela iimpawu zokuqala ze-snowball ibe yingxaki yokwenyani esongela umonakalo omkhulu.

Umsebenzi we-SRE ekufowuneni kukuqinisekisa ukuba zeziphi izilumkiso ezibonisa into enzulu, kwaye nokuba kufuneka zinyuswe kwaye kujongwane nazo. Ndiyakrokrela ukuba oku kungenxa yokungaguquguquki kwezilumkiso: kuya kuba ngcono ukuba kukho amanqanaba amaninzi okanye iindlela "ezihlakaniphile" zokuqwalasela izilumkiso ngokuhambelana nemeko echazwe ngasentla.

Ingcebiso ngophawu: https://gitlab.com/gitlab-org/gitlab/issues/42633

3. Ii-SRE zethu emsebenzini zisebenzisa izixhobo ezininzi.

Ngaphakathi:

• Iprojekthi ye-infra ye-GitLab: iincwadi zokubaleka zihlala apha, izabelo zeshifti/zeveki, imisebenzi yokuphendula ngeziganeko.
• Imiba ye-GitLab: Uphando, uphononongo, kunye nokugcinwa kulandelwa kwimiba.
• Iileyibhile zeGitLab: imisebenzi ezenzekelayo iqaliswa ngokusekwe kwiilebhile ezithile, ezisetyenziswa ziibhothi ukulandelela umsebenzi.

Ngaphandle:

• PagerUmsebenzi: Izilumkiso
• Slack: Ukuhamba komyalezo wePagerDuty/AlertManager kuya apha. Ukudityaniswa nemiyalelo ye-slash ukwenza imisebenzi eyahlukeneyo, njengokuvala isilumkiso okanye ukunyuka kwisehlo.
• I-Grafana: umboniso weemetrikhi ezigxile kwiintsingiselo zexesha elide.
• I-Kibana: Inika umbono / ukukhangela kwelogi, ukukwazi ukumba nzulu kwiziganeko ezithile.
• Sondeza: Kukho “igumbi lokuphumla” elihlala liqhuba kwiZoom. Oku kuvumela iinjineli ze-SRE ukuba zixoxe ngokukhawuleza iziganeko ngaphandle kokuchitha ixesha elixabisekileyo lokudala igumbi kunye nokudibanisa abathathi-nxaxheba.

Kwaye abaninzi abaninzi.

4. Ukubeka iliso kwi-GitLab.com nge-GitLab yindawo enye yokusilela

Ukuba i-GitLab.com inengxaki enkulu yokucima kwenkonzo, asifuni ukuba ichaphazele amandla ethu okusombulula lo mba. Inokumiswa ngokuphehlelela umzekelo weGitLab wesibini ukulawula iGitLab.com. Enyanisweni, oku sele kusebenza kuthi: https://ops.gitlab.net/.

5. Iimpawu ezimbalwa omawuthathelwe ingqalelo ukongeza kwi-GitLab

• Ukuhlelwa komsebenzi wabasebenzisi abaninzi, iyafana ne Google Docs. Oku kuya kunceda kwimisebenzi yeziganeko ngexesha lomsitho, kunye nemisebenzi yokuxoxa. Kuzo zombini ezi meko, abathathi-nxaxheba abaninzi banokufuna ukongeza into ngexesha lokwenyani.
• Iiwebhook ezininzi zemisebenzi. Ukukwazi ukuqhuba amanyathelo ahlukeneyo okuhamba komsebenzi we-GitLab ngaphakathi kuya kunceda ukunciphisa ukuthembela kwakho ekudityanisweni kwe-Slack. Umzekelo, ukukwazi ukuvumela isilumkiso kwiPagerDuty ngomyalelo we-slash kumcimbi weGitLab.
  isiphelo

Iinjineli ze-SRE zinexesha elinzima kunye nobunzima obuninzi. Kuya kuba kuhle ukubona iimveliso ezininzi zeGitLab zilungisa le miba. Sele sisebenza kwezinye izongezo kwimveliso eya kwenza ukuba ukuhamba komsebenzi okukhankanywe ngasentla kube lula. Iinkcukacha ziyafumaneka kwi Icandelo le-Ops Product Vision.

Sandisa iqela ngo-2020 ukuhlanganisa zonke ezi zinto zibalaseleyo. Ukuba unomdla, nceda ujonge izithuba, kwaye uzive ukhululekile ukuqhagamshelana nabani na kwiqela lethu ngayo nayiphi na imibuzo.

umthombo: www.habr.com