I-webinar inomsindo ongemuhle, ngakho-ke senze okulotshiweyo.
Igama lami nginguMedvedev Eduard. Namuhla ngizokhuluma ngokuthi iyini i-SRE, ukuthi i-SRE yavela kanjani, yiziphi izindlela zokusebenza zonjiniyela be-SRE, okuncane mayelana nenqubo yokuthembeka, kancane mayelana nokuqapha kwayo. Sizodlulela phezulu, ngoba awukwazi ukusho okuningi ngehora, kodwa ngizokunikeza okokusebenza ukuze ubuyekeze kabanzi, futhi sonke sikulindile ku- . eMoscow ekupheleni kukaJanuwari.
Okokuqala, ake sikhulume ngokuthi iyini i-SRE - Site Reliability Engineering -. Futhi ukuthi ibonakale kanjani njengesikhundla esihlukile, njengesiqondiso esihlukile. Konke kwaqala ngokuthi emibuthanweni yokuthuthukiswa kwendabuko, i-Dev ne-Ops amaqembu amabili ahluke ngokuphelele, ngokuvamile anemigomo emibili ehluke ngokuphelele. Umgomo wethimba lokuthuthukisa ukukhipha izici ezintsha ukuhlangabezana nezidingo zebhizinisi. Umgomo weqembu le-Ops uwukuqinisekisa ukuthi yonke into iyasebenza futhi akukho okuphukayo. Ngokusobala, le migomo iphikisana ngokuqondile: ukuze yonke into isebenze futhi akukho lutho oluphukayo, kungcono ukukhipha izici ezintsha kancane ngangokunokwenzeka. Ngenxa yalokhu, izingxabano eziningi zangaphakathi ziyavela, leyo indlela manje ebizwa ngokuthi i-DevOps ezama ukuyixazulula.
Inkinga ukuthi asinayo incazelo ecacile ye-DevOps kanye nokusetshenziswa okucacile kwe-DevOps. Ngakhuluma engqungqutheleni eyayiseYekaterinburg eminyakeni emi-2 edlule, futhi kuze kube manje isigaba se-DevOps saqala ngombiko othi "Yini i-DevOps." Ngo-2017, ama-devops cishe aneminyaka engu-10 ubudala, kodwa sisaphikisana ngokuthi kuyini. Futhi lesi yisimo esiyinqaba kakhulu i-Google eyazama ukusixazulula eminyakeni embalwa edlule.
Ngo-2016, i-Google yakhipha ibhuku elibizwa ngokuthi “Site Reliability Engineering.” Futhi eqinisweni, kwakungale ncwadi lapho ukunyakaza kwe-SRE kwaqala khona. I-SRE inketho ethile yokusebenzisa i-paradigm ye-DevOps enkampanini ethile. Onjiniyela be-SRE bazibekela umgomo wokuqinisekisa ukusebenza okuthembekile kwezinhlelo. Ikakhulukazi ithathwe konjiniyela, kwesinye isikhathi kubaphathi abanesizinda esiqinile sokuthuthuka. Futhi benza lokho abaphathi besistimu ababevame ukukwenza, kodwa isizinda esiqinile ekuthuthukisweni kanye nolwazi lwesistimu kusukela ekubukeni kwekhodi kuholela eqinisweni lokuthi laba bantu abathambekele emsebenzini wokuphatha ojwayelekile, kodwa bathambekele ekuzenzeleni.
Kuvele ukuthi i-paradigm ye-DevOps emaqenjini e-SRE isetshenziswa yiqiniso lokuthi kukhona onjiniyela be-SRE abaxazulula izinkinga zesakhiwo. Nakhu, ukuxhumana okufanayo phakathi kwe-Dev ne-Ops abantu asebekhulume ngakho iminyaka engu-8. Indima ye-SRE ifana neyomakhi wezakhiwo ngokuthi abaqalayo ababi ama-SRE. Abantu ekuqaleni kwemisebenzi yabo abakabi naso isipiliyoni futhi abanalo ububanzi obudingekayo bolwazi. Ngoba i-SRE idinga ulwazi oluyinkimbinkimbi kakhulu lokuthi yini ngempela futhi nini okungahamba kahle. Ngakho-ke, uhlobo oluthile lokuhlangenwe nakho luyadingeka lapha, njengomthetho, ngaphakathi kwenkampani nangaphandle.
Babuza ukuthi ngabe umehluko phakathi kwe-SRE nama-devops uzochazwa yini. Usanda kuchazwa. Singakhuluma ngendawo ye-SRE enhlanganweni. Ngokungafani nendlela yakudala ye-DevOps, lapho i-Ops isewumnyango ohlukile, i-SRE iyingxenye yethimba lokuthuthukisa. Babambe iqhaza ekuthuthukisweni komkhiqizo. Kukhona ngisho nendlela lapho i-SRE iyindima edlula isuka konjiniyela iye komunye. Babamba iqhaza ekubuyekezweni kwekhodi ngendlela efanayo, njengesibonelo, abaklami be-UX, abathuthukisi ngokwabo, futhi ngezinye izikhathi abaphathi bemikhiqizo. Ama-SRE asebenza kuleli zinga elifanayo. Sidinga ukugunyazwa kwabo, sidinga ukubuyekezwa kwabo, ukuze ekusetshenzisweni ngakunye i-SRE ithi: “Kulungile, lokhu kuthunyelwa, lo mkhiqizo ngeke uthinte kabi ukuthembeka. Futhi uma kwenzeka, kuzoba ngaphakathi kwemingcele eyamukelekayo.” Sizokhuluma ngalokhu futhi.
Ngokufanelekile, i-SRE ine-veto ekushintsheni amakhodi. Futhi ngokuvamile, lokhu kuphinde kuholele ekungqubuzaneni okuncane uma i-SRE isetshenziswa ngokungalungile. Kuyo kanye leyo ncwadi emayelana Nobunjiniyela Bokuthembela Kwendawo, izingxenye eziningi, nangaphezulu kweyodwa, zisho ukuthi zingagwenywa kanjani lezi zingxabano.
Abantu babuza ukuthi i-SRE ihlobana kanjani nokuphepha kolwazi. I-SRE ayibandakanyeki ngokuqondile ekuvikelekeni kolwazi. Ikakhulukazi ezinkampanini ezinkulu, lokhu kwenziwa abantu ngabanye, abahloli, nabahlaziyi. Kodwa i-SRE iphinde ihlanganyele nabo ngomqondo wokuthi okunye ukusebenza, okunye okwenzayo, okunye ukuthunyelwa okuthinta ukuphepha nakho kungathinta ukutholakala komkhiqizo. Ngakho-ke, i-SRE ngokuvamile inokusebenzelana nanoma imaphi amaqembu, okuhlanganisa namathimba ezokuphepha, okuhlanganisa nabahlaziyi. Ngakho-ke, ama-SRE adingeka kakhulu uma kuzanywa ukusebenzisa i-DevOps, kodwa umthwalo kubathuthukisi uba mkhulu kakhulu. Okusho ukuthi, ithimba lokuthuthukiswa ngokwalo alisakwazi ukubhekana neqiniso lokuthi manje nabo badinga ukubhekana ne-Ops. Futhi kuvela indima ehlukile. Le ndima ihlelwe esabelweni sezimali. Ngezinye izikhathi le ndima yakhiwe ngobukhulu beqembu, umuntu ohlukile uvela, ngezinye izikhathi omunye wabathuthukisi uba nguye. Lena yindlela i-SRE yokuqala evela ngayo eqenjini.
Ubunzima besistimu obuthintwa yi-SRE, ubunkimbinkimbi obuthinta ukuthembeka kokusebenza, kungadingeka noma ngengozi. Ubunkimbinkimbi obudingekayo yilapho ubunkimbinkimbi bomkhiqizo bukhula ngezinga izici ezintsha zomkhiqizo ezidingekayo. Ubunkimbinkimbi obungahleliwe yilapho ubunkimbinkimbi besistimu bukhula, kodwa isici somkhiqizo nezimfuneko zebhizinisi akuthinti lokhu ngokuqondile. Kuvele ukuthi kungenzeka ukuthi umthuthukisi wenze iphutha ndawana thize, noma i-algorithm ayilungile, noma kwethulwa ezinye izintshisekelo ezingeziwe ezikhuphula ubunkimbinkimbi bomkhiqizo ngokungenasidingo. I-SRE enhle kufanele ihlale igwema lesi simo. Okusho ukuthi, noma yikuphi ukuzibophezela, noma yikuphi ukuthunyelwa, noma yisiphi isicelo sokudonsa esikhulisa ubunkimbinkimbi ngenxa yokwengezwa okungahleliwe kufanele kuvinjwe.
Umbuzo uthi kungani ungavele uqashe unjiniyela, umlawuli wesistimu onolwazi oluningi, ukujoyina iqembu. Unjiniyela odlala indima kanjiniyela, sitshelwa ukuthi, akasona isixazululo esingcono kakhulu sabasebenzi. Umthuthukisi obambe iqhaza likanjiniyela akasona njalo isixazululo esiphelele sabasebenzi, kodwa iphuzu lapha ukuthi umthuthukisi obambe iqhaza ku-Ops unesifiso esithe xaxa sokuzenzakalela, unolwazi oluthe xaxa nekhono elibekiwe ukuze enze lokhu. okuzenzakalelayo. Futhi ngokufanele, asinciphisi nje kuphela isikhathi semisebenzi ethile ethile, hhayi nje isimiso, kodwa futhi nemingcele yebhizinisi ebalulekile njenge-MTTR (Isikhathi Esiqondile Sokubuyisela, isikhathi sokubuyisela). Ngakho, futhi sizophinde sikhulume ngalokhu ngokuhamba kwesikhathi, silondoloza imali yenhlangano.
Manje ake sikhulume ngemibandela yokusebenza kwe-SRE. Futhi okokuqala mayelana nokwethembeka. Ezinkampanini ezincane kanye nokuqala, kuvame ukwenzeka ukuthi abantu bacabange ukuthi uma isevisi ibhaliwe kahle, uma umkhiqizo ubhalwe kahle futhi ngendlela efanele, uzosebenza, ngeke uphule. Yilokho, sibhala ikhodi enhle, ngakho-ke akukho lutho oluzophulwa. Ikhodi ilula kakhulu, akukho lutho oluzophulwa. Laba mayelana nabantu abafanayo abathi asidingi ukuhlolwa, ngoba, bheka, lezi zindlela ezintathu ze-VPI, kungani ukhathazeka?
Konke lokhu akulungile, kunjalo. Futhi laba bantu bavame ukulimala ngalolu hlobo lwekhodi ekusebenzeni, ngoba izinto ziyaphuka. Izinto ngezinye izikhathi ziphuka ngezindlela ezingalindelekile. Kwesinye isikhathi abantu bathi cha, ngeke kwenzeke. Futhi kusenzeka. Kwenzeka kaningi. Futhi yingakho kungekho muntu oke alwele ukutholakala okungu-100%, ngoba ukutholakala okungu-100% akwenzeki. Lokhu kujwayelekile. Futhi yingakho sihlala sikhuluma ngabayisishiyagalolunye uma sikhuluma ngokutholakala kwesevisi. 2 nine, 3 nine, 4 nine, 5 nine. Uma lokhu sihumushela esikhathini sokuphumula, isibonelo, i-5 nines ingaphezudlwana kwemizuzu emi-5 yesikhathi sokuphumula ngonyaka, ama-2 nines ayizinsuku ezi-3,5 zesikhathi sokuphumula.
Kodwa kusobala ukuthi ngesinye isikhathi kukhona ukwehla kwe-POI nokubuya ekutshalweni kwezimali. Ukusuka kumaphuzu ayisishiyagalolunye kuya kwezintathu kusho ukunciphisa isikhathi sokuphumula ngezinsuku ezingaphezu kwezingu-3. Ukusuka kwezine kuya kwezinhlanu kunciphisa isikhathi sokuphumula ngemizuzu engama-47 ngonyaka. Futhi kuvela ukuthi lokhu kungase kungabi bucayi ebhizinisini. Futhi ngokuvamile, ukwethembeka okudingekayo akuyona inkinga yezobuchwepheshe, okokuqala, kuyinkinga yebhizinisi, kuyinkinga yomkhiqizo. Yiliphi izinga lokuphumula elamukelekayo kubasebenzisi bomkhiqizo, yini abayilindele, ukuthi bakhokha malini, isibonelo, balahlekelwa yimalini, ingakanani imali elahlekelwa yisistimu.
Umbuzo obalulekile ukuthi kuyini ukwethembeka kwezingxenye ezisele. Ngoba umehluko phakathi kwe-4 ne-5 nines ngeke ubonakale ku-smartphone ene-2 enokwethenjelwa nines. Uma sikhuluma nje, uma okuthile kuphuka ku-smartphone enkonzweni yakho izikhathi ezingu-10 ngonyaka, cishe izikhathi eziyisi-8 ukuwohloka kwenzeka ngasohlangothini lwe-OS. Umsebenzisi ukujwayele lokhu, futhi ngeke akunake isikhathi esisodwa esengeziwe ngonyaka. Kuyadingeka ukuqhathanisa intengo yokwethenjelwa okwandayo kanye nenzuzo ekhulayo.
Encwadini nje ye-SRE kunesibonelo esihle sokukhuphuka sifinyelele ku-4 nines sisuka ku-3 nines. Kuvele ukuthi ukunyuka kokutholakala kungaphansi kancane kuka-0,1%. Futhi uma imali engenayo yesevisi ingu-$1 million ngonyaka, khona-ke ukukhuphuka kwemali engenayo kungu-$900. Uma ukwanda kokutholakala ngesishiyagalolunye kusibiza ngaphansi kuka-$900 ngonyaka, ukwenyuka kwenza umqondo wezezimali. Uma kubiza ngaphezu kwamaRandi angu-900 ngonyaka, akusenangqondo, ngoba ukwanda kwemali engenayo akunxeshezeli izindleko zabasebenzi kanye nezindleko zezinsiza. Futhi ama-3 ayisishiyagalolunye azokwanela.
Lokhu kuyisibonelo esenziwe lula lapho zonke izicelo zilingana. Futhi kusukela ku-3 nine kuya ku-4 nine kulula kakhulu ukuhamba, kodwa ngesikhathi esifanayo, isibonelo, ukusuka ku-2 nines kuya ku-3 kakade kuwukulondoloza amadola ayizinkulungwane ezingu-9, kungenza umqondo wezezimali. Ngokwemvelo, empeleni, ukwehluleka ukubhalisa isicelo kubi kunokwehluleka ukubonisa ikhasi; izicelo zinesisindo esihlukile. Bangase babe nenqubo ehluke ngokuphelele ekubukeni kwebhizinisi, kodwa noma kunjalo, njengomthetho, uma singakhulumi nganoma yiziphi izinsizakalo ezithile, lokhu ukulinganisa okuthembekile.
Sithole umbuzo wokuthi ngabe i-SRE ingomunye wabaxhumanisi lapho ikhetha isixazululo sezokwakha senkonzo. Lokhu kwamukelekile mayelana nokuhlanganiswa nengqalasizinda ekhona ukuze kungabikho ukulahlekelwa ekuzinzeni kwayo. Yebo, ama-SRE athonya izicelo zokudonsa, ukuzibophezela, ukukhishwa ngendlela efanayo; athonya ukwakhiwa, ukuqaliswa kwezinsizakalo ezintsha, ama-microservices, kanye nokuqaliswa kwezixazululo ezintsha. Kungani ngike ngasho ngaphambili ukuthi udinga isipiliyoni, udinga iziqu. Eqinisweni, i-SRE ingelinye lamazwi avimbayo kunoma yisiphi isixazululo sezakhiwo nesoftware. Ngakho-ke, i-SRE njengonjiniyela kufanele, okokuqala, ingaqondisisi kuphela, kodwa futhi iqonde ukuthi ezinye izinqumo eziqondile zizothinta kanjani ukwethembeka, ukuzinza, nokuqonda ukuthi lokhu kuhlobana kanjani nezidingo zebhizinisi, futhi kusukela ngamuphi umbono lokhu kungavunyelwa, futhi engekho ngayo.
Ngakho-ke, manje yisikhathi sokukhuluma ngemibandela yokwethembeka, okuthi ku-SRE ngokwesiko kuchazwe njenge-SLA (Isivumelwano Sezinga Lesevisi). Cishe igama elijwayelekile. I-SLI (Inkomba Yezinga Lesevisi). I-SLO (Injongo Yezinga Lesevisi). Isivumelwano Sezinga Lesevisi mhlawumbe yitemu elibalulekile, ikakhulukazi uma usebenze namanethiwekhi, abahlinzeki, kanye nokubamba. Lesi isivumelwano esijwayelekile esichaza ukusebenza kwayo yonke isevisi yakho, izinhlawulo, izinhlawulo ezithile zamaphutha, amamethrikhi, imibandela. Futhi i-SLI iyimethrikhi yokufinyeleleka ngokwayo. Okusho ukuthi, i-SLI ingaba yini: isikhathi sokuphendula kusuka kusevisi, inani lamaphutha njengephesenti. Lokhu kungaba umkhawulokudonsa uma sikhuluma ngohlobo oluthile lokusingathwa kwefayela. Uma sikhuluma ngama-algorithms wokuqashelwa, inkomba ingaba, isibonelo, ukunemba kwempendulo. I-SLO (Injongo Yezinga Lesevisi) ngokulandelana, inhlanganisela yenkomba ye-SLI, inani layo nesikhathi.
Ake sithi i-SLA ingaba kanje. Isevisi itholakala ngo-99,95% wesikhathi unyaka wonke. Noma amathikithi angu-99 okusekela abalulekile azovalwa phakathi namahora angu-3 ngekota. Noma u-85% wemibuzo uzophendulwa phakathi kwamasekhondi angu-1,5 njalo ngenyanga. Okungukuthi, kancane kancane siya siqonda ukuthi amaphutha nokwehluleka kuyinto evamile. Lesi isimo esamukelekile, siyasihlelela, sithembele ngisho nangokwezinga elithile. Okusho ukuthi, i-SRE yakha amasistimu angenza amaphutha, okumele aphendule ngokujwayelekile emaphutheni, futhi okufanele awacabangele. Futhi uma kungenzeka, kufanele baphathe amaphutha ngendlela yokuthi umsebenzisi angawaboni, noma awaqaphele, kodwa kukhona uhlobo oluthile lokulungisa ukuze yonke into ingabhidliki ngokuphelele.
Isibonelo, uma ulayisha ividiyo ku-YouTube, futhi i-YouTube ayikwazi ukuyiguqula ngokushesha, uma ividiyo inkulu kakhulu, uma ifomethi ingalungile, isicelo ngokwemvelo ngeke sehluleke ngokuphela kwesikhathi, i-YouTube ngeke ibonise i-502 iphutha, i-YouTube izothi: “Sidale yonke into, ividiyo yakho iyacutshungulwa. Izobe isilungile esikhathini esingangemizuzu eyi-10.” Lona umgomo wokucekelwa phansi okunomusa, okujwayelekile, isibonelo, kusukela ekuthuthukisweni okuphambili uma uke wakwenza lokhu.
Imigomo elandelayo esizokhuluma ngayo, ebaluleke kakhulu ekusebenzeni ngokwethembeka, ngamaphutha, ngokulindelwe, yi-MTBF ne-MTTR. I-MTBF yisikhathi esimaphakathi phakathi kokuhluleka. I-MTTR Isho Isikhathi Sokubuyisela, isikhathi esimaphakathi sokululama. Okusho ukuthi, singakanani isikhathi esidlulile kusukela ngesikhathi iphutha litholwa, kusukela ngesikhathi iphutha livela kuze kube yilapho isevisi ibuyiselwa ekusebenzeni okuvamile ngokuphelele. I-MTBF ilungiswa ikakhulukazi ngokusebenza ngekhwalithi yekhodi. Okusho ukuthi, iqiniso lokuthi ama-SRE angasho ukuthi "cha". Futhi ithimba lonke lidinga ukuqonda ukuthi uma i-SRE ithi "cha," ayisho ngoba iyingozi, hhayi ngoba imbi, kodwa ngoba uma kungenjalo wonke umuntu uzohlupheka.
Futhi, kunezihloko eziningi, izindlela eziningi, izindlela eziningi, ngisho nakuyo kanye incwadi engivame ukubhekisela kuyo, indlela yokuqinisekisa ukuthi abanye abathuthukisi abaqali ukuzonda i-SRE. I-MTTR, ngakolunye uhlangothi, imayelana nokusebenza ku-SLO yakho (Injongo Yezinga Lesevisi). Futhi lokhu ngokuvamile ku-automation. Ngoba, ngokwesibonelo, i-SLO yethu iyisikhathi sokuphumula se-4 nines ngekota. Lokhu kusho ukuthi ezinyangeni ezi-3 singavumela imizuzu engu-13 yesikhathi sokuphumula. Futhi kuvele ukuthi i-MTTR yethu ayikwazi ukuba ngaphezu kwemizuzu eyi-13. Uma sithatha amaminithi angu-13 ukuphendula okungenani isikhathi esi-1 sokuphumula, lokhu kusho ukuthi sesivele sisiqedile sonke isabelomali sekota. Sephula i-SLO. Imizuzu eyi-13 yokusabela nokulungisa ukwehluleka miningi emshinini, kodwa incane kakhulu kumuntu. Ngoba ngesikhathi umuntu ethola isexwayiso, ngesikhathi esabela ngaso, ngesikhathi ethola iphutha, sekuvele kuyimizuzu embalwa. Kuze kube yilapho umuntu eqonda indlela yokuyilungisa, yini ngempela okufanele ayilungise, yini okufanele ayenze, kuzothatha imizuzu embalwa eyengeziwe. Futhi eqinisweni, noma ngabe udinga nje ukuqalisa kabusha iseva, njengoba kuvela, noma ukuphakamisa indawo entsha, bese i-MTTR ithatha imizuzu eyi-7-8 ngesandla. Lapho kwenziwa inqubo ngokuzenzakalelayo, i-MTTR ivamise ukufika kwesekhondi, kwesinye isikhathi ama-millisecond. I-Google ijwayele ukukhuluma ngama-millisecond, kodwa empeleni, izinto azilungile kangako.
Ngokufanelekile, i-SRE kufanele icishe ishintshe ngokuphelele umsebenzi wayo, ngoba lokhu kuthinta ngokuqondile i-MTTR, amamethrikhi ayo, i-SLO yayo yonke isevisi, futhi, ngokufanelekile, inzuzo yebhizinisi. Uma isikhathi seqiwe, siyabuzwa ukuthi icala liku-SRE. Ngenhlanhla, icala alibekwa kunoma ubani. Futhi lesi isiko elihlukile, elibizwa nge-balmeless postmortem, esingeke sikhulume ngalo namuhla, kodwa sizolihlaziya ku-Slurm. Lesi yisihloko esithakazelisa kakhulu okungakhulunywa ngaso kakhulu. Uma sikhuluma nje, uma isikhathi ebesibekiwe ngekota sidlulile, khona-ke wonke umuntu kufanele asolwe kancane, okusho ukuthi ukusola wonke umuntu akukhiqizi, esikhundleni salokho, mhlawumbe, singasoli muntu, kodwa silungise isimo futhi sisebenze ngalokho esinakho. Ngokuhlangenwe nakho kwami, le ndlela ingaphandle kancane emaqenjini amaningi, ikakhulukazi eRussia, kodwa inengqondo futhi isebenza kahle kakhulu. Ngakho-ke, ekugcineni ngizoncoma izihloko nezincwadi ongazifunda ngalesi sihloko. Noma uze ku-Slurm SRE.
Ake ngichaze. Uma isikhathi se-SLO sekota sidluliwe, uma isikhathi sokuphumula besingeyona imizuzu eyi-13, kodwa eyi-15, ubani ongasolwa ngalokhu? Yiqiniso, i-SRE ingase ibe nephutha ngoba yenza ngokusobala ukuzibophezela okubi noma ukuthunyelwa. Umphathi wesikhungo sedatha angase asolwe ngalokhu, ngoba kungenzeka ukuthi wenze ukulungiswa okungahleliwe. Uma umlawuli wesikhungo sedatha enecala ngalokhu, khona-ke umuntu ovela kwa-Ops kufanele asolwe ngokungabali ukugcinwa lapho evumelana ne-SLO. Leli iphutha lomphathi, umqondisi wezobuchwepheshe, noma othile osayine inkontileka yesikhungo sedatha futhi akazange anake iqiniso lokuthi i-SLA yesikhungo sedatha ayakhelwe isikhathi sokuphumula esidingekayo. Ngakho-ke, wonke umuntu unecala elincane ngalesi simo. Futhi lokho kusho ukuthi asikho isidingo sokubeka icala kunoma ngubani ikakhulukazi ngalesi simo. Kodwa-ke idinga ukulungiswa. Yingakho kukhona ama-postmortems. Futhi uma ufunda, ngokwesibonelo, ama-postmortems e-GitHub, futhi lokhu kuhlale kuyindaba ethakazelisa kakhulu, encane futhi engalindelekile esimweni ngasinye, ungabuyisela ukuthi akekho owake washo ukuthi lo muntu wayenecala. Icala lihlala libekwe ezinqubweni ezithile ezintula.
Asiqhubekele embuzweni olandelayo. Okuzenzakalelayo. Ngokuvamile, uma ngikhuluma ngokuzenzakalelayo kwezinye izimo, ngivame ukubhekisela kuthebula elikhuluma ngokuthi ungasebenza isikhathi esingakanani ekwenzeni umsebenzi ngokuzenzakalelayo ukuze ungathathi isikhathi esiningi ukuwuwenza ngokuzenzakalelayo kunalokho ovame ukonga. Kukhona ukubamba. Okubanjiwe ukuthi uma ama-SRE enza umsebenzi ngokuzenzakalelayo, akongi isikhathi kuphela, kodwa alondoloza imali ngoba i-automation ithinta ngokuqondile i-MTTR. Basindisa, ngomqondo ongokomfanekiso, isimilo sabasebenzi nabathuthukisi, okubuye kube insiza engapheli. Banciphisa isimiso. Futhi konke lokhu kunomphumela omuhle emsebenzini futhi, ngenxa yalokho, ebhizinisini, ngisho noma kubonakala sengathi i-automation ayinangqondo ngokwezindleko zesikhathi.
Eqinisweni, cishe ihlala yenza njalo, futhi kunezimo ezimbalwa kakhulu lapho kungafaneleki khona ukwenza okuthile endimeni ye-SRE. Okulandelayo sizokhuluma ngalokho okubizwa ngokuthi isabelomali samaphutha, isabelomali samaphutha. Eqinisweni, kuvele ukuthi uma wenza kangcono kakhulu kune-SLO ozibekele yona, lokhu nakho akukuhle kakhulu. Lokhu kubi kunalokho, ngoba i-SLO ayisebenzi nje kuphela njengesibopho esiphansi, kodwa futhi njengesibopho esingaphezulu esilinganiselwe. Uma uzibekela i-SLO yokutholakala kwe-99%, futhi empeleni une-99,99%, kuvela ukuthi unendawo ethile yokuhlola, engeke ilimaze ibhizinisi nhlobo, ngoba wena ngokwakho unqume lokhu ndawonye, futhi nawe ube nalesi sikhala ungasisebenzisi. Unebhajethi yamaphutha, okuyinto esimweni sakho engasetshenziswanga.
Senzani ngayo? Siyisebenzisa ngokoqobo yonke into. Okokuhlola ezimeni zokukhiqiza, zokukhipha izici ezintsha ezingase zithinte ukusebenza, ukukhishwa, ukunakekelwa, izikhathi zokuphumula ezihleliwe. Umthetho ophambene nawo uyasebenza: uma isabelomali siphelile, asikwazi ukukhulula noma yini entsha, ngoba ngaphandle kwalokho sizodlula i-SLO. Isabelomali sesivele siphelile, sikhiphe okuthile, uma kuthinta kabi ukusebenza, okungukuthi, uma kungelona uhlobo oluthile lokulungisa lokho ngokwalo okwandisa i-SLO, bese sidlula isabelomali, futhi lesi isimo esibi. , kudinga ukuhlaziya , postmortem, futhi ngokunokwenzeka nokulungiswa okuthile kwenqubo.
Okusho ukuthi, kuvela ukuthi uma insizakalo ngokwayo ingasebenzi kahle, futhi i-SLO isetshenzisiwe futhi isabelomali asichithwa ekuhlolweni, hhayi kunoma yikuphi ukukhishwa, kodwa ngokwaso, bese esikhundleni sokulungiswa okuthakazelisayo, esikhundleni sokuthakazelisa. izici, esikhundleni sokukhishwa okuthakazelisayo. Esikhundleni sokwenza noma yimuphi umsebenzi wokudala, kuzodingeka wenze ukulungisa okuyizithulu ukuze ubuyisele isabelomali ngendlela efanele, noma uhlele i-SLO, futhi lokhu kuyinqubo okungafanele yenzeke kaningi.
Ngakho-ke, kuvela ukuthi esimweni lapho sinesabelomali esiningi samaphutha, wonke umuntu unentshisekelo: kokubili i-SRE nabathuthukisi. Konjiniyela, isabelomali esikhulu samaphutha sisho ukuthi bangabhekana nokukhishwa, ukuhlolwa, nokuhlolwa. Kuma-SRE, isabelomali samaphutha nokungena kulesi sabelomali kusho ukuthi empeleni benza umsebenzi omuhle. Futhi lokhu kuthinta isisusa sohlobo oluthile lomsebenzi ohlanganyelwe. Uma ulalela ama-SRE akho njengabathuthukisi, uzoba nendawo eyengeziwe yokwenza umsebenzi omuhle kanye nemisebenzi emincane kakhulu.
Kuvele ukuthi ukuhlolwa ekukhiqizeni kubaluleke kakhulu futhi kuyingxenye ebalulekile ye-SRE emaqenjini amakhulu. Futhi kuvame ukubizwa ngegama lobunjiniyela besiphithiphithi, elivela eqenjini leNetflix elikhiphe insiza ebizwa nge-Chaos Monkey.
I-Chaos Monkey ixhuma epayipini le-CI/CD futhi iphahlaze iseva lapho kukhiqizwa. Futhi, esakhiweni se-SRE sithi iseva ephahlazekile ayimbi ngokwayo, ilindelekile. Futhi uma ifakiwe kusabelomali, iyamukeleka futhi ayilimazi ibhizinisi. Yiqiniso, i-Netflix inamaseva anele angenalutho, ukuphindaphinda okwanele, ukuthi konke lokhu kungalungiswa ngaphandle komsebenzisi ewonke ngisho nokuqaphela, futhi ngokuqinisekile akekho oshiya iseva eyodwa kunoma yisiphi isabelomali.
I-Netflix ngesikhathi esisodwa yayinesethi yonke yezinsiza ezinjalo, enye yazo, i-Chaos Gorilla, ikhubaza ngokuphelele enye yezindawo ezitholakalayo e-Amazon. Futhi izinto ezinjalo zisiza kahle ukukhomba, okokuqala, ukuncika okufihliwe, lapho kungacaci ngokuphelele ukuthi yini ethonya lokho, yini encike kulokho. Futhi lokhu, uma usebenza ne-microservice futhi imibhalo ingaphelele ngokuphelele, lokhu kungenzeka ukuthi ujwayelene nawe. Futhi futhi, lokhu kusiza ukubamba amaphutha kukhodi ongeke ukwazi ukubamba ngesikhathi sesiteji, ngoba noma yisiphi isiteji asisona isifaniso esinembile, ngenxa yokuthi isikali somthwalo sihlukile, iphethini yomthwalo ihlukile, imishini nayo, iningi. kungenzeka, okunye. Imithwalo ephakeme kakhulu ingase ingalindelekile futhi ingabonakali. Futhi ukuhlola okunjalo, okungaphinde kudlule isabelomali, kusiza kahle kakhulu ukubamba amaphutha engqalasizinda lapho isiteji, ukuhlola okuzenzakalelayo, kanye namapayipi e-CI/CD angeke awabambe. Futhi inqobo nje uma lokhu konke kufakiwe kusabelomali sakho, akunandaba ukuthi isevisi yakho yehlile lapho, nakuba kungase kubonakale kusabisa kakhulu, iseva iphahlazekile, yeka iphupho elibi. Cha, lokho kuvamile, kuhle lokho, kusiza ukubamba amaphutha. Uma unesabelomali, ungasisebenzisa.
Umbuzo: yiziphi izincwadi engingazincoma? Uhlu lusekugcineni. Kukhona izincwadi eziningi, ngingancoma imibiko eminingana. Isebenza kanjani nokuthi i-SRE isebenza kanjani ezinkampanini ngaphandle komkhiqizo wazo wesoftware noma ngokuthuthuka okuncane. Isibonelo, ebhizinisini, lapho umsebenzi oyinhloko kungeyona isofthiwe. Ebhizinisini, lapho umsebenzi oyinhloko ungeyona isofthiwe, i-SRE isebenza ngokufana ncamashí nanoma iyiphi enye indawo, ngoba ebhizinisini udinga futhi ukusebenzisa, noma ungathuthukisi, imikhiqizo yesofthiwe, udinga ukukhipha izibuyekezo, wena. udinga ukushintsha ingqalasizinda, udinga ukukhula, udinga ukukala. Futhi ama-SRE asiza ukuhlonza nokubikezela izinkinga ezingaba khona kulezi zinqubo futhi azilawule ngemva kokuthi ukukhula okuthile kuqalile futhi nebhizinisi lidinga ushintsho. Ngoba akudingekile neze ukuzibandakanya ekuthuthukisweni kwesoftware ukuze ube ne-SRE, uma okungenani unamaseva amaningana futhi ulindele okungenani ukukhula okuthile.
Okufanayo kuya kumaphrojekthi amancane, izinhlangano ezincane, ngoba izinkampani ezinkulu zinesabelomali nendawo yokuhlola. Kepha ngasikhathi sinye, zonke lezi zithelo zokuhlola zingasetshenziswa noma kuphi, okungukuthi, ama-SRE, vele, avele kuGoogle, Netflix, nakuDropbox. Kodwa ngasikhathi sinye, izinkampani ezincane kanye neziqalo sezingakwazi ukufunda izinto ezifingqiwe, zifunde izincwadi, nemibiko yokubuka. Baqala ukuzwa ngalokhu kaningi, babheke izibonelo ezithile, ngicabanga, kulungile, lokhu kungaba usizo ngempela, siyakudinga nalokhu, kupholile.
Okusho ukuthi, wonke umsebenzi oyinhloko wokulinganisa lezi zinqubo usuvele wenzelwe wena. Okufanele ukwenze ukuchaza indima ye-SRE ikakhulukazi enkampanini yakho futhi uqale ukusebenzisa zonke lezi zinqubo, esezichaziwe futhi. Okungukuthi, kusukela ezimisweni eziwusizo zezinkampani ezincane, lokhu kuhlale kuyincazelo ye-SLA, SLI, SLO. Uma ungabandakanyi kusofthiwe, khona-ke lezi kuzoba ama-SLA angaphakathi nama-SLO angaphakathi, isabelomali sangaphakathi samaphutha. Lokhu cishe kuhlale kuholela ezingxoxweni ezithokozisayo ngaphakathi kweqembu nangaphakathi kwebhizinisi, ngoba kungase kuvele ukuthi usebenzisa imali engaphezu kwalokho edingekayo kungqalasizinda, ohlotsheni oluthile lwenhlangano yezinqubo ezifanele, ipayipi elifanele. Futhi lezi 4 nine onazo emnyangweni we-IT, awuzidingi ngempela manje. Kodwa ngesikhathi esifanayo, kwakungenzeka ukuchitha isikhathi, ukuchitha isabelomali ngamaphutha kwenye into.
Ngakho-ke, ukuqapha kanye nokuhleleka kokuqapha kuyasiza enkampanini yanoma yimuphi usayizi. Futhi ngokujwayelekile, le ndlela yokucabanga, lapho amaphutha eyinto eyamukelekayo, lapho kunesabelomali, lapho Izinjongo zikhona, iphinde ibe usizo enkampanini yanoma yimuphi usayizi, kusukela ekuqalisweni kwabantu abangu-3.
Okokugcina kwama-nuances ezobuchwepheshe esingakhuluma ngawo ukuqapha. Ngoba uma sikhuluma nge-SLA, i-SLI, i-SLO, asikwazi ukuqonda ngaphandle kokuqapha ukuthi sihambisana yini nesabelomali, ukuthi sihambisana yini neziNhloso zethu, nokuthi siyithonya kanjani i-SLA yokugcina. Ngiye ngabona izikhathi eziningi ukuthi ukuqapha kwenzeka ngale ndlela elandelayo: kukhona inani elithile, isibonelo, isikhathi sesicelo kuseva, isikhathi esimaphakathi noma inombolo yezicelo kusizindalwazi. Unezinga elinqunywe unjiniyela. Uma imethrikhi ichezuka kokujwayelekile, i-imeyili ithunyelwa. Konke lokhu akusizi ngalutho, njengomthetho, ngoba kuholela ekugcwalisweni kwezixwayiso, ukugcwala ngokweqile kwemiyalezo yokuqapha, lapho umuntu, okokuqala, kufanele azihumushe ngaso sonke isikhathi, okungukuthi, anqume ukuthi inani lemethrikhi lisho isidingo uhlobo oluthile lwesenzo. Okwesibili, umane ayeke ukuqaphela zonke lezi zixwayiso, kuyilapho ngokuyisisekelo kungekho senzo esidingekayo kuye. Okusho ukuthi, umthetho omuhle wokuqapha kanye nomthetho wokuqala lapho kusetshenziswa i-SRE ukuthi isaziso kufanele size kuphela uma kudingeka isenzo.
Esimeni esijwayelekile kunamazinga ama-3 emicimbi. Kunezixwayiso, kukhona amathikithi, kukhona izingodo. Izaziso ziyinoma yini edinga isinyathelo esisheshayo kuwe. Okusho ukuthi, yonke into iphukile, idinga ukulungiswa khona manje. Amathikithi ayinto edinga isenzo esisalindile. Yebo, udinga ukwenza okuthile, udinga ukwenza okuthile ngesandla, i-automation ihlulekile, kodwa akudingeki ukwenze emaminithini ambalwa alandelayo. Amalogi ayikho konke okungadingi senzo, futhi ngokuvamile, uma izinto zihamba kahle, akekho oyoke azifunde. Kuzodingeka ukufunda izingodo kuphela lapho, ngokubheka emuva, kuvela ukuthi kukhona okuphukile isikhathi esithile, sasingazi ngakho. Noma uhlobo oluthile lophenyo ludinga ukwenziwa. Kodwa ngokuvamile, yonke into engadingi noma yisiphi isenzo iya ezingodweni.
Njengomphumela ongaseceleni wakho konke lokhu, uma sihlonze ukuthi yiziphi izehlakalo ezidinga izenzo futhi sichaze kahle ukuthi lezo zenzo kufanele zibe yini, lokhu kusho ukuthi isenzo singazenzakalela. Okungukuthi, kwenzekani. Siphuma kusaziso. Asingene esenzweni. Ake siye encazelweni yalesi senzo. Bese siqhubekela ku-automation. Okusho ukuthi, noma iyiphi i-automation iqala ngokusabela kumcimbi.
Ukusuka ekuqapheni sidlulela egameni elithi Observability. Kuphinde kwaba khona i-hype ezungeze leli gama eminyakeni embalwa edlule. Futhi bambalwa abantu abaqonda ukuthi lokhu kusho ukuthini ngaphandle komongo. Kodwa iphuzu eliyinhloko ukuthi i-Observability i-metric of transparency system. Uma kukhona okungahambanga kahle, unganquma ngokushesha kangakanani ukuthi yini ngempela engahambanga kahle nokuthi sasiyini isimo sesistimu ngaleso sikhathi. Ngokombono wekhodi: yimuphi umsebenzi ohlulekile, iyiphi isevisi ehlulekile. Sasinjani isimo, isibonelo, okuguquguqukayo kwangaphakathi, ukucushwa. Ngokombono wengqalasizinda, yilapho ukwehluleka kwenzeka khona indawo yokutholakala, futhi uma unohlobo oluthile lwe-Kubernetes, khona-ke lapho ukwehluleka kwenzeke khona, sasinjani isimo se-pod. Futhi ngokufanelekile, i-Observability inobudlelwano obuqondile ne-MTTR. Ukuphakama kwe-Observability yesevisi, kuba lula ukubona iphutha, kulula ukulungisa iphutha, kulula ukwenza iphutha ngokuzenzakalelayo, i-MTTR iphansi.
Uma siqhubekela ezinkampanini ezincane futhi, bavame ukubuza, ngisho namanje, ukuthi benzeni ngobukhulu beqembu, nokuthi kudingekile yini ukuqasha i-SRE ehlukile eqenjini elincane. Sengike ngakhuluma ngalokhu ngaphambili kancane. Ezigabeni zokuqala zokuthuthukiswa kokuqalisa noma, isibonelo, iqembu, lokhu akudingekile nhlobo, ngoba i-SRE ingenziwa indima yesikhashana. Futhi lokhu kuzophilisa iqembu kancane, ngoba kukhona okungenani ukuhlukahluka okuthile. Futhi futhi kuzolungiselela abantu ukuthi njengoba bekhula, ngokuvamile, imithwalo yemfanelo ye-SRE izoshintsha kakhulu. Uma uqasha umuntu, khona-ke, yiqiniso, kukhona akulindele. Futhi lokhu okulindelekile ngeke kushintshe ngokuhamba kwesikhathi, kodwa izidingo zizoshintsha kakhulu. Ngakho-ke, ukuqasha i-SRE kunzima kakhulu ezigabeni zokuqala. Kulula kakhulu ukukhulisa owakho. Kodwa kufanelekile ukucabanga ngakho.
Okuwukuphela kokuhlukile, mhlawumbe, yilapho kunezidingo eziqinile kakhulu futhi ezichazwe kahle zokuphakama. Okusho ukuthi, esimweni sokuqala, lokhu kungase kube uhlobo oluthile lokucindezela kubatshalizimali, uhlobo oluthile lwesibikezelo sokukhula izikhathi eziningana ngesikhathi esisodwa. Khona-ke ukuqasha i-SRE ngokuvamile kuyafaneleka ngoba kungathethelelwa. Sinezidingo zokukhula, sidinga umuntu ozoba nomthwalo wemfanelo wokuqinisekisa ukuthi ngokukhula okungaka akukho okuphukayo.
Omunye umbuzo. Okufanele ukwenze lapho izikhathi eziningana abathuthukisi besika isici esidlula izivivinyo, kodwa siphula umkhiqizo, silayishe i-database, iphule ezinye izici, iyiphi inqubo okufanele isetshenziswe. Ngakho-ke, kulokhu, isabelomali samaphutha siyethulwa. Futhi ezinye izinsizakalo, ezinye izici zihlolwa ngokushesha ekukhiqizeni. Lokhu kungaba i-canary, lapho inani elincane kuphela labasebenzisi, kodwa kakade ekukhiqizweni, lisebenzisa isici, kodwa ngokulindela ukuthi uma okuthile kuphuka, ngokwesibonelo, ingxenye yephesenti yabo bonke abasebenzisi, sisazongena ngaphakathi isabelomali samaphutha. Ngakho-ke, yebo, kuzoba nephutha, kwabanye abasebenzisi konke kuzophuka, kodwa sesishilo kakade ukuthi lokhu kuvamile.
Kube khona umbuzo mayelana namathuluzi e-SRE. Okusho ukuthi, ingabe kukhona okuthile okuqondile ama-SRE angakusebenzisa okungeke kusetshenziswe wonke umuntu? Eqinisweni, kukhona izinsiza ezikhethekile kakhulu, kunesofthiwe ethile, isibonelo, elingisa imithwalo noma ukuhlola i-canary A/B. Kodwa ngokuyisisekelo, i-SRE tooling yilokho onjiniyela bakho asebevele bekusebenzisa. Ngoba i-SRE isebenzisana ngokuqondile nethimba lokuthuthukisa. Futhi uma unamathuluzi ahlukene, kuvela ukuthi kuthatha isikhathi ukuvumelanisa. Ikakhulukazi uma ama-SRE asebenza emaqenjini amakhulu, ezinkampanini ezinkulu lapho kungase kube khona amaqembu amaningana, ukulinganisa kwenkampani yonke kuzosiza kakhulu lapha, ngoba uma amaqembu angu-50 esebenzisa izinsiza ezahlukene ze-50, lokhu kusho ukuthi i-SRE kumele ibazi bonke. Futhi-ke lokhu akusoze kwenzeka. Futhi izinga lomsebenzi, izinga lokulawula okungenani amanye amaqembu lizokwehla kakhulu.
I-webinar yethu kancane kancane iya ekupheleni. Ngikwazile ukukutshela izinto eziyisisekelo. Vele, akukho lutho nge-SRE olungatshelwa futhi luqondwe ngehora. Kodwa ngithemba ukuthi ngikwazile ukuveza le ndlela yokucabanga, amaphuzu ayinhloko ayinhloko. Futhi-ke, uma unentshisekelo, ungangena ujule esihlokweni, uzifundele wedwa, futhi ubheke ukuthi senziwa kanjani ngabanye abantu, kwezinye izinkampani. Futhi ngokufanele, ekuqaleni kukaFebhuwari, woza kithi e-Slurm SRE.
I-Slurm SRE iyisifundo esijulile sezinsuku ezintathu esizohlanganisa cishe lokho engikhuluma ngakho manje, kodwa ngokujula okukhulu kakhulu, ngezimo zangempela, ngokuzijwayeza, konke okujulile kuhloswe emsebenzini ongokoqobo. Abantu bazohlukaniswa ngamaqembu. Nonke nizosebenza ngamacala angempela. Ngokunjalo, sinabafundisi abavela ku-Booking.com u-Ivan Kruglov no-Ben Tyler. Sino-Evgeniy Varabbas omuhle wase-Google, wase-San Francisco. Futhi ngizokutshela okuthile futhi. Ngakho qiniseka ukuthi uzosivakashela.
Ngakho, uhlu lwezinkomba. Kukhona izixhumanisi ku-SRE. kuleyo ncwadi efanayo, noma kunalokho ezincwadini ezi-2 ezimayelana ne-SRE, ezibhalwe i-Google. Omunye , lapho imigomo nokusetshenziswa kwayo kuchazwa ngokuningiliziwe okwengeziwe. Oku-3 okulandelayo kuyimibiko nge-SRE ezinkampanini ezahlukene. Okokuqala - , lena inkulumo eyisihluthulelo evela ku-Ben Trainer wakwa-Google. Okwesibili - . Eyesithathu futhi mayelana . Umbiko wesine kusuka , enabasebenzi abangu-5 ababalulekile be-SRE emazweni angu-190. Kuyathakazelisa kakhulu ukubheka konke lokhu, ngoba njengoba nje i-DevOps isho izinto ezihluke kakhulu ezinkampanini ezahlukene ngisho namaqembu ahlukene, i-SRE inemithwalo yemfanelo ehluke kakhulu, ngisho nasezinkampanini ezinobukhulu obufanayo.
Ezinye izixhumanisi ezi-2 ezimisweni zobunjiniyela besiphithiphithi: , . Futhi ekugcineni kukhona izinhlu ezi-3 ezivela ochungechungeni lwe-Awesome Lists mayelana , mayelana futhi mayelana . Uhlu oluku-SRE lukhulu ngendlela emangalisayo, akudingeki ukuthi udlule kukho konke, kunezihloko ezingaba ngu-200. Ngincoma kakhulu izindatshana ezimayelana nokuhlelwa kwamandla kanye ne-postmortem engenasici.
I-athikili ethokozisayo:
Ngiyabonga ngokungilalela sonke lesi sikhathi. Ngithemba ukuthi ufunde okuthile. Ngithemba ukuthi unezinsiza ezanele zokufunda okwengeziwe. Futhi sizokubona kamuva. Ngethemba ngoFebhuwari.
I-webinar yayiphethwe ngu-Eduard Medvedev.
I-PS: Kulabo abathanda ukufunda, u-Eduard unikeze uhlu lwezinkomba. Labo abakhetha ukuyiqonda ngokusebenza bamukelekile ku .
Source: www.habr.com