Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com

Chikwata chedu chinoda zviedzo. Imwe neimwe Slurm haisi kudzokorora kwakamira kwezvakapfuura, asi ratidziro pane chiitiko uye shanduko kubva kune yakanaka kuenda kune nani. Asi pamwe Slurm SRE takasarudza kushandisa chimiro chitsva - kupa vatori vechikamu mamiriro ari padyo sezvinobvira "kurwa".

Kana tikadudza muchidimbu zvatakaita panguva yekosi yakakura: β€œTinovaka, tinoputsa, tinogadzirisa,
tiri kudzidza." SRE yakakosha zvishoma mudzidziso chete - kudzidzira chete, mhinduro chaidzo, matambudziko chaiwo.

Vatori vechikamu vakakamurwa kuita zvikwata kuitira kuti mweya wekukwikwidza wakasimba usabvumira chero munhu kurara kana kutanga "Shiri Dzakatsamwira" pa iPhone, achitevera muenzaniso waDmitry Anatolyevich.

Matambudziko, glitches, tsikidzi uye mabasa zvakapihwa kune vatori vechikamu nevarairidzi vana. Ivan Kruglov, Principal Developer at Booking.com (Netherlands). Ben Tyler, Mugadziri Mukuru paBooking.com (USA). Eduard Medvedev, CTO kuTungsten Labs (Germany). Evgeniy Varavva, general developer paGoogle (San Francisco).

Uyezve, vatori vechikamu vakakamurwa kuita zvikwata uye vanokwikwidzana. Inonakidza?

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com
Ivan, Ben, Eduard naEvgeniy vanotarisa varombo veSlurm SRE vatori vechikamu vane mutsa Leninist squints vasati vatanga makwikwi.

Saka basa:

Isu tiri vedu, tichavaka nyika itsva...

Pane webhusaiti yemabhaisikopo aggregator. Zviitiko zvinogadzirwa nevanopa mazano mune yakafanoshandwa mamiriro (kunyangwe pasina munhu asingasanganisi akanyanya kuomarara uye anonyengera kuvandudza), kuita kwesaiti kunotsanangurwa neakasiyana metrics. Matambudziko anogona kunge akasiyana zvakanyanya: matikiti eMoulin Rouge theatre haana kuiswa mudhatabhesi; mapepa emafirimu nemitambo anoiswa mudhatabhesi mumasekonzi anopfuura gumi; tsananguro yefirimu yega inoomesa; 10% yemirairo yakatochengetwa; Nguva nenguva iyo yekubhadhara yekugadzirisa system inoparara kweminiti kana maviri. Uye zvakawanda, zvizhinji, zvakawanda zvisingafadzi zvinhu zvinogona kuwira Slurm SRE anotora chikamu pabasa rake chairo.

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com
Takagadzirira kubata chero chinhu ... uye munhu wese.

Webhusaiti yedu yemoyo murefu ine akati wandei ma microservices. Basa rayo ndere kuunganidza dhata pazviratidziro, mitengo uye zvigaro zviripo kubva kumabhaisikopo ese; inoratidza zviziviso zvemufirimu, inobvumidza iwe kusarudza sinema, show, horo nenzvimbo, bhuku uye kubhadhara matikiti. Kazhinji, zvese izvo muoni anogona kungorota chete. Asi mushandisi haatombofungidzire kuti sei kurwira kusimba kwekugadzikana uye kuwanikwa kwesaiti kuri kuitika mukati.

Kune yakadzika saiti, isu takagadzira SLO, SLI, SLA zviratidzi, zvakagadziridzwa zvivakwa uye zvivakwa, takaisa saiti, kumisikidza kuongorora uye kunyevera. Uye toenda.

SLO, SLI, SLA

SLI - zviratidzo zvezinga rebasa. SLOs zvinangwa zvenhanho yebasa. SLA - zvibvumirano zvenhanho yebasa.

SLA ishoko reTIL methodology inoreva chibvumirano chepamutemo pakati pemutengi wesevhisi nemutengesi wayo, ine tsananguro yebasa, kodzero uye zvisungo zvemapato uye, zvinonyanya kukosha, chiyero chakabvumiranwa chemhando yekupihwa kweichi. service.

Iyo SLO ibasa renhanho chinangwa: kukosha kwainotarirwa kana huwandu hwehuwandu hwesevhisi nhanho inoyerwa neSLI. Yakajairika kukosha kweSLO "SLI ≀ Target" kana "Lower Limit ≀ SLI ≀ Upper Limit".

Iyo SLI iratidziro yezinga rebasa-yakanyatsotsanangurwa chiyero chechikamu chimwe chedanho rebasa rakapihwa. Kune akawanda masevhisi, kiyi SLI inoonekwa sekukumbira latency - zvinotora nguva yakareba sei kudzorera mhinduro kune chikumbiro. Mamwe maSLI akajairika anosanganisira mwero wekukanganisa, unowanzo kuratidzwa sechikamu chezvikumbiro zvese zvakagamuchirwa, uye system throughput, inowanzoyerwa mukukumbira pasekondi.

Chokutanga pane zvose, tichaputsa ndege, uye ipapo vasikana, uye ipapo vasikana ...

Zvinhu zvemukati nekunze zvakatanga "kushatisa" SLO kubva pamaminetsi ekutanga. Zvese zvakawira pamisoro yevatariri - zvikanganiso zvevagadziri, kutadza kwezvivakwa, kuwanda kwevashanyi, uye DDoS kurwiswa. Zvese zvinowedzera SLO.

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com
"- Vanodiwa vatori vechikamu, ndinokurumidza kukufadzai, chinhu chekutanga chaunokundikana ... zvese!"

Vari munzira, vatauri vakakurukura kugadzikana, bhajeti rekukanganisa, maitiro ekuyedza, manejimendi ekukanganisa uye mutoro wekushanda.

Isu hatisi stoker, kwete kuveza ...

Ipapo vatori vechikamu vakatanga kugadzirisa zvinhu - chinhu chikuru ndechokunzwisisa kuti chii chokutanga kubata.

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com
"- Ishe, handisati ndamboiona ichiputsika seizvi, muchimiro ichi uye munzvimbo yakadai!"

Saka, tsaona yakaitika. Basa rekugadzirisa mubhadharo rakaderera. Nzira yekuita sei kuti udzorere kushanda munguva shoma shoma inobvira?

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com
Nyanzvi, dzichitarisa nerudo kune vatori vechikamu, dziri kugadzirira imwe nzira.

Chikwata chega chega chinoronga basa reboka kuti ribvise njodzi - rinosanganisira vaunoshanda navo, rinozivisa vanofarira (vanobatanidzwa). Panguva imwe cheteyo, zvinokosha zvinoiswa. Nenzira iyi, vatori vechikamu vakadzidziswa kushanda pasi pekumanikidzwa pasi pemamiriro enguva shoma.

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com
"Kutyisa kwerudzii kwabuda?!"

Exhale ... uye kupedza basa racho

Pamwe chete nevatauri, mushure mekunge dambudziko rega rega ragadziriswa uye nzvimbo yacho yakagadziriswa kwekanguva, boka racho rakadzidza zviitiko kubva kune SRE maonero. Takaongorora matambudziko zvakadzama - zvikonzero zvekuitika, kufambira mberi kwekubvisa. Mushure meizvozvo, zvese zviri zviviri timu-ne-timu uye pamwe chete, takaita sarudzo dzekuti tingadzidzivirira sei: nzira yekuvandudza kutarisa, maitiro ehuchenjeri kushandura zvivakwa, maitiro ekugadzirisa nzira yekuvandudza nekushanda, nzira yekugadzirisa mitemo. Vatauri vakaratidza muitiro wekuita post-mortem.

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com
β€œNdianizve anoda kutambudzwa! -Ini!"

Kubudirira kwezvikwata kwakanyorwa zvakasimba uye zvakajeka pamagetsi emagetsi.

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com

Kune nzvimbo dzekutanga - bhonasi kubva kune vanobatana.

Slurm SRE. Kuedza kwakazara nenyanzvi kubva Booking.com uye Google.com

Source: www.habr.com

Voeg