Qaphela. transl.: Siyajabula ukwabelana ngokuhunyushwa kwezinto ezinhle ezivela ku-Senior Technology Evangelist ovela kwa-AWS - Adrian Hornsby. Ngamagama alula, uchaza ukubaluleka kokuhlolwa ukuze kuncishiswe imiphumela yokwehluleka ezinhlelweni ze-IT. Cishe usuzwile kakade nge-Chaos Monkey (noma waze wasebenzisa izixazululo ezifanayo)? Namuhla, izindlela zokudala amathuluzi anjalo kanye nokusetshenziswa kwawo esimweni esibanzi zenziwa ngaphakathi kohlaka lomsebenzi obizwa ngokuthi ubunjiniyela besiphithiphithi. Funda kabanzi ngakho kulesi sihloko.
"Kodwa ngemuva kwabo bonke lobu buhle kukhona isiphithiphithi nobuhlanya." - Tanner Walling
Izicishamlilo. Laba basebenzi abaqeqeshwe kakhulu babeka ukuphila kwabo engozini nsuku zonke belwa nemililo. Ubuwazi ukuthi kufanele uchithe okungenani amahora angama-600 uziqeqesha ngaphambi kokuba yisicishamlilo? Futhi lokhu kuyisiqalo nje. Ngokwemibiko, izicishamlilo ziqeqesha kuze kufike ku-80% wesikhathi sabo sokusebenza.
Kungani?
Lapho isicishamlilo silwa nomlilo wangempela, udinga okufanele umuzwa. Ukuze uyithuthukise, kufanele uqeqeshe ihora nehora, usuku nosuku. Njengoba besho, ukuzijwayeza kwenza izimanga.
“Zibonakala zingena engqikithini yomlilo; ofana noDkt. Phil ngamalangabi.” -
Qaphela. transl.: U-Phillip Calvin "Phil" McGraw uyisazi sokusebenza kwengqondo saseMelika, umbhali, kanye nomsingathi wohlelo lwethelevishini oludumile uDkt. Phil, lapho umsingathi enikeza ababambiqhaza bakhe izixazululo ezinkingeni zabo.
Ngesinye isikhathi e-Seattle
Ekuqaleni kweminyaka yama-2000s , owayenesikhundla e-Amazon ngesihloko esisemthethweni Inkosi Yezinhlekelele, idale futhi yahola uhlelo lwe-GameDay. Kwakusekelwe kokuhlangenwe nakho kwakhe njengesicishamlilo. I-GameDay yayihloselwe ukuhlola, ukuqeqesha, nokulungiselela amasistimu ahlukahlukene e-Amazon, isofthiwe, kanye nabantu ngezimo ezingase zibe nzima.
Njengoba izicishamlilo zithuthukisa umuzwa wokulwa nemililo, u-Jesse wayefuna ukusiza iqembu lakhe lithuthukise umuzwa wokuzwana ukuze libhekane nezenzakalo ezinkulu eziyinhlekelele.
"Usuku Lomdlalo: Ukudala Ukuqina Ngokubhubhisa" - uJesse Robbins
yaklanyelwe ukukhulisa ukuzinza kwendawo yokudayisa yase-Amazon ngokwethula ngamabomu amaphutha ezinhlelweni ezibucayi.
I-GameDay iqale ngochungechunge lwezimemezelo kuyo yonke inkampani ukuthi kuhlelwe umshini wokuzivivinya - kwesinye isikhathi omkhulu, ngokwesibonelo, ukuvala yonke isikhungo sedatha. Kwanikezwa imininingwane emincane mayelana nokunqamuka okwakuhleliwe, futhi iqembu lanikezwa izinyanga ezimbalwa ukuba lizilungiselele. Inhloso enkulu yalo msebenzi bekuwukuhlola ukuthi abasebenzi bangakwazi yini ukubhekana nenkinga yendawo futhi baxazulule ngokushesha imiphumela yayo.
Phakathi nalezi zivivinyo, amathuluzi athile nezinqubo zasetshenziswa, njengokuqapha, izexwayiso nezingcingo eziphuthumayo, ukuhlaziya nokuhlonza amaphutha ezinqubweni zokuphendula isigameko. Njengoba kuvela, i-GameDay inhle ekuboneni izinkinga zezakhiwo zakudala. Kwesinye isikhathi bekungenzeka futhi ukuthi kutholwe lokho okubizwa ngokuthi “izinkinga ezicashile” - izinkinga ezibonakala ngenxa yemininingwane yesigameko. Isibonelo, amasistimu okulawula izigameko abalulekile enqubweni yokutakula ahlulekile ngenxa yemiphumela engemihle engalindelekile ebangelwa inkinga eyenziwe umuntu.
Njengoba inkampani ikhula, i-GameDay's theory blast radius yanda. Ekugcineni, lezi zivivinyo ziye zayekwa ngoba umonakalo ongase ube khona enkampanini uma izinto zingahambi ngokohlelo ube mkhulu kakhulu. Kusukela lapho, uhlelo seluphenduke lwaba uchungechunge lwezivivinyo ezihlukene, ezingezona ezebhizinisi zokuqeqesha abasebenzi ezimeni ezibucayi. Ngeke ngichaze kabanzi mayelana nokuhlola kulesi sihloko, kodwa ngizokwenza kanjalo esikhathini esizayo. Kulokhu ngifuna ukuxoxa ngombono obalulekile osekelwe ku-GameDay: ubunjiniyela bokwethembeka (unjiniyela wokuqina), eyaziwa nangokuthi ubunjiniyela besiphithiphithi ().
Ukuphakama Kwezinkawu
Cishe uzwile nge-Netflix, umhlinzeki wokuqukethwe kwamavidiyo aku-inthanethi. AbakwaNetflix baqale ukusuka esikhungweni sabo sedatha baya ku-AWS Cloud ngo-Agasti 2008. Lesi sinyathelo sigqugquzelwe ukukhohlakala kwedatha engathi sína okubambezele ukuthunyelwa kwe-DVD ngezinsuku ezintathu (yebo, iNetflix iqale ukuthumela amafilimu ngemeyili yomnenke). Ukuthuthela emafini kwaqhutshwa isidingo sokusingatha imithwalo yokusakaza ephezulu kakhulu, kanye nesifiso sokusuka kusakhiwo se-monolithic futhi siye kumasevisi amancane angakala kalula kuye ngenani labasebenzisi nosayizi wethimba lonjiniyela. Uhlangothi lwabathengi lwesevisi yokusakaza luthuthele ku-AWS kuqala, phakathi kuka-2010 no-2011, lwalandelwa yi-IT yebhizinisi kanye nazo zonke ezinye izakhiwo. Isikhungo sedatha seNetflix uqobo savalwa ngo-2016. Inkampani ikala ukutholakala njengesilinganiso senani lemizamo eyimpumelelo yokwethula ifilimu enani eliphelele, esikhundleni sokuqhathanisa kalula isikhathi nesikhathi sokuphumula, futhi ilwela ukuzuza isibalo esingu-0,9999 esifundeni ngasinye njalo ngekota (it ngokuvamile uyaphumelela). Izakhiwo zomhlaba wonke zeNetflix zithatha izifunda ezintathu ze-AWS. Ngakho-ke, uma izinkinga ziphakama kwesinye sezifunda, inkampani inamandla okuqondisa kabusha abasebenzisi kwabanye.
Ngizophinda enye yezingcaphuno zami eziyintandokazi:
“Ukuphazamiseka akunakugwenywa; ekugcineni noma yiluphi uhlelo luzowa ngokuhamba kwesikhathi.” - Werner Vogels
Eqinisweni, ukwehluleka ezinhlelweni ezisabalalisiwe, ikakhulukazi ezinkulu, akunakugwenywa, ngisho nasefwini. Kodwa-ke, ifu le-AWS kanye nezinto zayo zokuqala zokuphinda ziphinde ziphinde zisebenze—ikakhulukazi , eyakhiwe phezu kwayo, ivumela noma ubani ukuthi aklame izinsizakalo ezinokwethenjelwa kakhulu.
Ukusebenzisa izimiso zokuphindaphinda (ukungasebenzi) kanye nokwehla kancane kancane ekusebenzeni (ukwehliswa kwesithunzi), I-Netflix ngaphandle kokuthinta abasebenzisi bokugcina.
Kusukela ekuqaleni, i-Netflix ibambelele emigomeni yezokwakha eqinile. Esinye sezicelo zokuqala abazisebenzisa ku-AWS kwaba zabo — ukusekela ukukala okuzenzakalelayo kwama-microservices angenasimo. Ngamanye amazwi, noma yisiphi isenzakalo singamiswa futhi sishintshwe ngokuzenzakalelayo ngaphandle kokulahlekelwa yisimo. I-Chaos Monkey yenza isiqiniseko sokuthi akekho owephula lesi simiso.
Qaphela. transl.: Ngendlela, ku-Kubernetes kukhona i-analogue ebizwa , okubukeka sengathi kume ngoMashi walo nyaka.
I-Netflix inomunye umthetho osabalalisa isevisi ngayinye kuzo zonke izindawo ezintathu ezitholakalayo. Kufanele iqhubeke nokusebenza uma kutholakala amabili kuphela. Ukuqinisekisa ukuthi lo mthetho uyalandelwa, ikhubaza izindawo zokutholakala. Ngezinga lomhlaba wonke iyakwazi ukuvala isifunda sonke se-AWS ukuze iqinisekise ukuthi bonke abasebenzisi beNetflix bangahlinzekwa kunoma yiziphi izifunda ezintathu. Futhi benza lezi zivivinyo ezibanzi njalo ngemva kwamasonto ambalwa ekukhiqizeni ukuze baqiniseke ukuthi akukho lutho oluye lwadlula emifantwini.
Ekugcineni, iNetflix nayo ithuthuke igxile kakhulu ukusiza ukukhomba izinkinga ngama-microservices kanye nesakhiwo sesitoreji. Ungafunda kabanzi mayelana nalawa masu encwadini ethi Chaos Engineering, engiyincoma kunoma ubani onentshisekelo kulesi sihloko.
"Ngokwenza izivivinyo njalo ezilingisa ukuphuma kwesifunda, sikwazile ukuhlonza ukushiyeka kwesistimu okuhlukahlukene kusenesikhathi futhi sikulungise." -
Namuhla izimiso zobunjiniyela besiphithiphithi ; banikezwa incazelo elandelayo:
"Ubunjiniyela be-Chaos kuyindlela ehlanganisa ukwenza izivivinyo ohlelweni lokukhiqiza ukuze kuqinisekiswe amandla alo okumelana neziphazamiso ezihlukahlukene eziphakama ngesikhathi sokusebenza." -
Nokho, empilweni yakhe ozinikele kubunjiniyela besiphithiphithi, , umdali wangaphambili wezakhiwo zamafu zeNetflix osize inkampani ukuthi ithuthele kwingqalasizinda yamafu wonke, wethule enye incazelo yobunjiniyela besiphithiphithi. Ngokubona kwami, inembe kakhudlwana futhi isungulwe:
"Ubunjiniyela be-Chaos kuwukuhlola okuklanyelwe ukunciphisa imiphumela yokwehluleka."
Eqinisweni, siyazi ukuthi ukwehluleka kwenzeka ngaso sonke isikhathi. Uma ziphendulwe kahle, akufanele zithinte abasebenzisi bokugcina. Umgomo oyinhloko wobunjiniyela besiphithiphithi ukuthola izinkinga ezingabhekwana nazo ngendlela efanele.
Izimo ezidingekayo zokudala isiphithiphithi
Ngaphambi kokuthi uqale ubunjiniyela besiphithiphithi, qiniseka ukuthi wenze wonke umsebenzi odingekayo ukuze uqinisekise ukusimama kuwo wonke amazinga enhlangano. Ukudala amasistimu abekezelela amaphutha akukhona nje isoftware. Iqala ezingeni ingqalasizinda, isakazeka inethiwekhi nedatha, kuthinta isakhiwo izicelo, futhi ekugcineni ihlanganisa abantu namasiko. Ngibhale kabanzi esikhathini esidlule mayelana namamodeli okuqina nokwehluleka (, , и ) futhi ngeke ngigxile kulokhu manje, kodwa angikwazi ukwenza ngaphandle kwesikhumbuzi esincane.
Ezinye izinto ezidingekayo ngaphambi kokwethula isiphithiphithi ohlelweni (uhlu aluphelele)
Izigaba zobunjiniyela besiphithiphithi
Kubalulekile ukuqonda ukuthi ingqikithi yobunjiniyela besiphithiphithi CHA kuwukukhulula izinkawu endle futhi uzivumele zibhubhise yonke into, ngaphandle kwanoma iyiphi injongo. Iphuzu lalesi siqondiso ukuphazamisa izici ezithile zesistimu endaweni elawulwayo ngokusebenzisa izivivinyo eziklanywe kahle ukuze ubone ukuthi uhlelo lwakho lokusebenza lungakwazi yini ukumelana nezimo ezinesiyaluyalu.
Ukuze wenze lokhu, kufanele ulandele inqubo echazwe ngokucacile, esemthethweni echazwe esithombeni esingezansi. Ingakusiza ukuthi usuke ekuqondeni isimo sokuzinza sesistimu yakho uye ekwakhiweni kwe-hypothesis, ukuyihlola, futhi ekugcineni uhlaziye ulwazi olutholwe ekuhlolweni nasekuthuthukiseni ukuzinza kwesistimu ngokwayo.
Izigaba zobunjiniyela besiphithiphithi
1. Isimo esizinzile
Enye yezinto ezibaluleke kakhulu zobunjiniyela besiphithiphithi ukuqonda ukuziphatha kwesistimu ngaphansi kwezimo ezijwayelekile.
Kungani? Kulula: ngemva kokwethula ukwehluleka kokwenziwa, kufanele uqinisekise ukuthi isistimu ibuyele esimweni esizinzile esifundwe kahle futhi ukuhlola akusaphazamisi ukuziphatha kwayo okuvamile.
Okubalulekile lapha ukungagxili kuzibaluli zesistimu yangaphakathi (iphrosesa, inkumbulo, njll.) kodwa emiphumeleni elinganisekayo exhumanisa ukusebenza nolwazi lomsebenzisi. Ukuze le miphumela ibe sesimweni esizinzile, ukuziphatha okuqashelwe kwesistimu kufanele kube nephethini ebikezelwayo, kodwa kushintshe kakhulu uma kwenzeka ukwehluleka ohlelweni.
Ukukhumbula , ehlongozwe ngenhla ngu-Adrian Cockcroft, lesi simo esizinzile siyashintsha lapho iphutha elingalawuleki lidala inkinga engalindelekile futhi libonisa ukuthi ukuhlolwa kwesiphithiphithi kufanele kunqanyulwe.
Njengesibonelo sezifunda ezizinzile, ake sithathe isipiliyoni se-Amazon. Inkampani isebenzisa ivolumu yoku-oda njengenye yamamethrikhi ayo esimo esiqinile, futhi ngesizathu esihle. Ngo-2007, uGreg Linden, owayengowase-Amazon, wachaza indlela yokuhlola esebenzisa le ndlela Ngizamile ukwehlisa isikhathi sokulayisha amakhasi ewebhusayithi ngo-100 ms increments futhi ngathola ukuthi ngisho nokubambezeleka okuncane kuholela ekwehleni okukhulu kwemali engenayo. Ngokunyuka kwesikhathi sokulayisha ngo-100 ms, inani lama-oda (ngakho-ke ukuthengisa) lehle ngo-1%. Kungakho inombolo yama-oda iyikhandidethi elihle kakhulu lamamethrikhi esimo esizinzile.
I-Netflix isebenzisa i-metric yohlangothi lweseva ehlotshaniswa nokuqala kokudlala - inombolo yokuchofoza inkinobho ethi "dlala". Baqaphele iphethini ekuziphatheni kwenkomba ye-SPS (isiqalo-ngesekhondi) kanye nokuguquguquka kwayo okuphawulekayo lapho kwenzeka ukwehluleka kwesistimu. Imethrikhi ibizwa nge- “Netflix's Pulse” ().
Izinombolo ze-oda ze-Amazon kanye ne-Netflix's Pulse zingama-barometer amahle kakhulu ezimo ezizinzile ngoba zihlanganisa ulwazi lomsebenzisi namamethrikhi okusebenza zibe imethrikhi eyodwa, elinganisekayo, nebikezeleka kakhulu.
Kala, kala futhi ukale futhi
Kuyaqondakala ukuthi uma ungakwazi ukuthwebula kahle amamethrikhi esistimu, ngeke ukwazi ukuqapha (noma ngisho nokubona) izinguquko esimweni sokuzinza. Naka ngokukhethekile ukufunda yonke imingcele/izinkomba, kusukela kunethiwekhi, ihadiwe ukuya kuhlelo lokusebenza nakubantu. Dweba amagrafu alezi zilinganiso, noma zingashintshi ngokuhamba kwesikhathi. Uzomangala ukuthola izixhumanisi obungazi ukuthi zikhona.
"Yenza kube lula ngangokunokwenzeka konjiniyela ukufinyelela idatha abangayibala noma igrafu." -
2. Ukuqagela
Ngemva kokubhekana nesimo esizinzile, ungakwazi ukuqhubekela phambili ekwakheni i-hypothesis.
- Kuthiwani uma injini yokuncoma ima?
- Kuthiwani uma isilinganisi somthwalo sehla?
- Kuthiwani uma ukulondoloza isikhashana kwehluleka?
- Kuthiwani uma ukubambezeleka kukhuphuka ngo-300ms?
- Kuthiwani uma isizindalwazi esiyinhloko siphahlazeka?
Yiqiniso, kufanele ukhethe i-hypothesis eyodwa kuphela futhi akufanele uyixabanise kungenasidingo. Qala kancane. Ngithanda ukuqala nge-hypothesis yabasebenzi. Uke wezwa mayelana isici sebhasi ()? Isici sebhasi isilinganiso sobungozi obuhambisana nolwazi olusatshalaliswa ngokungalingani phakathi kwamalungu eqembu. Ikuvumela ukubala inombolo encane yabahlanganyeli bayo, ngemva kokulahlekelwa okungazelelwe lapho iphrojekthi izoyeka ngenxa yokuntula ulwazi noma isipiliyoni.
Izinkampani eziningi zinochwepheshe bezobuchwepheshe ukunyamalala kwabo kungazelelwe (“ukushayiswa ibhasi”) kungaba nomthelela omubi kukho kokubili iphrojekthi kanye neqembu. Khomba laba bantu futhi wenze izivivinyo zesiphithiphithi kubo: isibonelo, thatha amakhompuyutha abo uwathumele ekhaya ngosuku, bese ubheka imiphumela (evame ukuba nesiphithiphithi).
Yenza inkinga ijwayele wonke umuntu!
Khanga iqembu lonke ukuthuthukisa i-hypothesis. Vumela wonke umuntu abambe iqhaza ekudingideni: umnikazi womkhiqizo, umphathi wezobuchwepheshe, onjiniyela basemuva nabaphambili, abaklami, abaklami bezakhiwo, njll. Wonke umuntu ngendlela eyodwa noma enye oxhumene nomkhiqizo.
Okokuqala, cela wonke umuntu ukuthi abhale impendulo yakhe embuzweni othi “Kungenzekani uma...?” ephepheni. Uzobona ukuthi ezimweni eziningi wonke umuntu uzoba nempendulo ehlukile, futhi uzobona ukuthi abanye beqembu bebengakaze bacabange ngale nkinga kuze kube manje.
Yima kancane kuleli qophelo bese uxoxa ngokuthi kungani amalungu eqembu enemibono ehlukene mayelana nokuthi umkhiqizo uzoziphatha kanjani kokuthi “Kuthiwani uma...?” Buyela emuva ekucacisweni kwayo futhi uqiniseke ukuthi wonke umuntu unombono omuhle wokuthi kuzokwenzekani ngokulandelayo.
Thatha, ngokwesibonelo, indawo yokuthengisa eshiwo ngenhla ye-Amazon. Kuthiwani uma i-Shonga nge-Category iyeka ukulayisha ekhasini lasekhaya?
Ingabe kufanele ngibuyisele iphutha le-404? Ingabe kufanelekile ukulayisha ikhasi, ushiye isikhala esingenalutho njengasesithombeni-skrini esingezansi?
Ingabe kufanelekile ukudela ukusebenza okuthile futhi, isibonelo, ukuvumela ikhasi ukuthi landa futhi lifihle iphutha?
Futhi lokho kusohlangothini lwe-UI. Yini okufanele yenzeke ku-backend? Ingabe kufanele kuthunyelwe izexwayiso? Ingabe isevisi ehlulekayo kufanele iqhubeke nokuthola izicelo njalo lapho umsebenzisi elayisha ikhasi lasekhaya, noma kufanele i-backend inqamule ngokuphelele?
Futhi into eyodwa yokugcina. Sicela ungakhi inkolelo-mbono owaziyo kusengaphambili ukuthi izodala inkinga! Linga ngezingxenye zesistimu ocabanga ukuthi zizinzile—phela, iwona wonke iphuzu lokuhlola.
3. Dizayina futhi wenze ukuhlola
- Khetha i-hypothesis eyodwa;
- Chaza ububanzi bokuhlolwa;
- Khomba izinkomba ezihlobene ezizokalwa;
- Yazisa inhlangano.
Namuhla abantu abaningi, kanye nesayithi , khuthaza umbono wobunjiniyela besiphithiphithi ekukhiqizeni. Nakuba lokhu kufanele kube umgomo wokugcina, izinhlangano eziningi ziyesaba le ndlela, ngakho-ke akuyona indawo enhle yokuqala.
Kimina, ubunjiniyela besiphithiphithi abukona nje ukucekelwa phansi kwezinto ezihlukahlukene zezinhlelo zokukhiqiza. Lolu uhambo. Uhambo oluya emhlabeni wolwazi, oluxhunyaniswe ngokungenakuhlukaniswa nomsebenzi ofana nokucekelwa phansi kwezinhlelo endaweni elawulwayo - noma iyiphi indawo, kungaba imvelo yendawo ye-dev, i-beta, isiteji noma i-prod. Phakamisa ngokuhlolwa okuklanywe kahle ukwakha ukuzethemba kukhono lohlelo lwakho lokusebenza lokumelana nezimo ezinesiyaluyalu. "Ukwakha Ukuzethemba” iphuzu elibalulekile lapha ngoba liyisandulela sezinguquko zamasiko ezidingekayo ukuze kusetshenziswe ngempumelelo ubunjiniyela besiphithiphithi nokwethembeka enkampanini yakho.
Eqinisweni, amaqembu amaningi afunda okuningi ekuphuleni izinto, ngisho nasendaweni engakhiqizi. Vele uzame ukwenza docker stop database endaweni yangakini futhi ubone ukuthi ungakwazi yini ukusingatha le nkinga ngaphandle kwemiphumela. Maningi amathuba okuthi ngeke.
Ukumisa Isizindalwazi - Isibonelo
Qala kancane kancane futhi kancane kancane wakhe ukuzethemba eqenjini lakho kanye nenhlangano. Abantu bazokutshela ukuthi "ithrafikhi yokukhiqiza yangempela iyona ndlela kuphela yokuthwebula ukuziphatha kwesistimu ngokuthembekile." Lalela, mamatheka futhi uqhubeke nokwenza kancane kancane okwenzayo. Into embi kakhulu ongayenza ukusebenzisa ubunjiniyela besiphithiphithi ekukhiqizeni futhi wehluleke kabuhlungu. Ngemva kwalokhu, akekho oyokwethemba, futhi uzophoqeleka ukuba ukhohlwe "izinkawu ze-chaos" kuze kube phakade.
Okokuqala, zuza ukwethenjwa. Bonisa inhlangano nozakwenu ukuthi niyayazi into eniyenzayo. Iba isicishamlilo futhi ufunde okuningi ngangokunokwenzeka mayelana namalangabi ngaphambi kokuqhubekela phambili uphile ukuqeqeshwa komlilo. Zuza ukwethembeka kwakho. Khumbula indaba yofudu nonogwaja? Unesineke nonesineke uhlala ewina umjaho.
Elinye lamaphuzu abaluleke kakhulu phakathi nokuhlolwa ukuqonda amandla indawo yokulimala kusukela ekuhlulekeni okwethulayo kanye nokuncishiswa kwakho. Zibuze imibuzo elandelayo:
- Mangaki amaklayenti azothinteka kulokhu kuhlolwa?
- Yikuphi ukusebenza okuzothinteka?
- Yiziphi izindawo ezizothinteka?
Cabanga "ngenkinobho yokubulala" noma indlela yokuchitha ngokushesha isilingo bese ubuyela esimweni esizinzile ngokushesha okukhulu. Ngithanda ukwenza izivivinyo ngisebenzisa okuthiwa. Ukukhishwa kwe-"canary". Le nqubo inciphisa ingcuphe yokwehluleka lapho kwethulwa izinguqulo ezintsha zohlelo lokusebenza ekukhiqizweni ngokukhipha kancane kancane izinguquko kusethi encane yabasebenzisi bese zizisakaza kancane kancane kuyo yonke ingqalasizinda nakubo bonke abasebenzisi. Ngithanda ukukhishwa kwe-canary ngoba nje kugculisa umgomo , futhi ukuhlola ngokwako kulula kakhulu ukukumisa.
Isibonelo sokukhishwa kwe-canary okusekelwe ku-DNS kokuhlolwa kwesiphithiphithi
Qaphela ngokuhlolwa okushintsha isimo sohlelo lokusebenza (inqolobane noma isizindalwazi) noma lezo ezingakwazi ukuhlehliswa (kalula noma nhlobo).
Kuyathakazelisa ukuthi u-Adrian Cockcroft wangitshela ukuthi esinye sezizathu ezenza ukuthi iNetflix iqale ukusebenzisa imininingwane yolwazi ye-NoSQL ukuthi yayingenaso i-schema sezinguquko noma ukubuyisela emuva, ngakho-ke bekulula kakhulu ukuvuselela ngokuqhubekayo noma ukulungisa amarekhodi edatha ngayinye (okungukuthi babenobungane kakhulu kubunjiniyela besiphithiphithi) .
4. Qaphela futhi ufunde
Ukuze ufunde okuthile okusha futhi ugade ukuqhubeka kokuhlolwa, udinga ukwazi ukuqapha ukusebenza kwesistimu. Njengoba kushiwo ngaphambili, naka kakhulu zonke izinhlobo zamamethrikhi namapharamitha! Bese ulinganisa imiphumela futhi njalo - njalo! — qaphela isikhathi ngaphambi kokuba kuvele izimpawu zokuqala zenkinga. Sekwenzekile izikhathi eziningi emlandweni wami ukuthi izinhlelo zokuxwayisa ziye zahluleka futhi amakhasimende abhala ku-Twitter ngenkinga kuqala... ngithembe, awufuni ukuphelela kuleso simo, ngakho-ke sebenzisa izivivinyo ze-chaos ukuhlola izinhlelo zakho zokuqapha kanye nezixwayiso njenge kahle.
- Isikhathi kuze kutholakale?
- Isikhathi ngaphambi kwesaziso kanye nokuqala kwezenzo ezisebenzayo?
- Isikhathi kuze kube isaziso somphakathi?
- Isikhathi kuze kube ukulahlekelwa ingxenye yokusebenza?
- Sithatha isikhathi esingakanani ukuziphilisa?
- Isikhathi kuze kube yilapho ukululama okuphelele noma okuncane?
- Isikhathi kuze kube sekupheleni kwenhlekelele futhi sibuyele esimweni esizinzile?
Khumbula ukuthi ayikho imbangela eyodwa yokwehluleka. Izingozi ezinkulu zihlale ziwumphumela wokungaphumeleli okuncane okumbalwa okunqwabelanayo nokuholela enkingeni enkulu.
Yenza ukuhlaziya okuningiliziwe kwe-postmortem kokuhlolwa ngakunye!
Kwa-AWS, sibeka ukugcizelela okukhulu ekuhlaziyeni ukwehluleka okutholiwe kanye nokuqonda ukuthi yini okubangele ukuze sikwazi ukuvimbela izinkinga ezifanayo ngokuzayo. Zonke iziphetho kanye nemiphumela yokuhlolwa kufingqwa kudokhumenti ebizwa ngokuthi Ukulungiswa Kwamaphutha (COE). I-COE isivumela ukuthi sifunde emaphutheni ethu, kungaba amaphutha kwezobuchwepheshe, inqubo, noma inhlangano. Sisebenzisa le nqubo ukuze sisuse izimbangela eziyinhloko zokuqhekeka futhi siqhubeke sithuthukisa.
Isihluthulelo sempumelelo kule nqubo ukuvuleleka nokuphumela obala ngokuthi yini engahambanga kahle. Omunye wezimiso ezibaluleke kakhulu ekubhaleni i-COE enhle ukungakhethi futhi ugweme ukusho abantu abathile. Lokhu kuvame ukuba nzima endaweni edikibalisa ukuziphatha okunjalo futhi engavumeli ukuba nokwenzeka kokwehluleka. I-Amazon isebenzisa iqoqo "lezimiso zobuholi" () ukukhuthaza ukuziphatha okunjalo - isib. ukuzigxeka, indlela yokuhlaziya, ukuzibophezela emazingeni aphezulu kanye nesibopho ziyizingxenye ezibalulekile zenqubo ye-COE kanye nokusebenza kahle ngokujwayelekile.
Umbiko we-COE unezigaba ezinhlanu eziyinhloko:
- Kwenzekeni (ukulandelana kwezikhathi)?
- Waba yini umthelela kumakhasimende?
- Kungani kwenzeke iphutha? ()
- Yini esiyifundile?
- Ungakuvimbela kanjani lokhu esikhathini esizayo?
Le mibuzo kunzima kakhulu ukuyiphendula kunalokho ebonakala uma uqala ukuyibuka, njengoba udinga ukwenza isiqiniseko sokuthi wonke amaphuzu angacacile/angaziwa afundwa ngokucophelela.
Ukuze siguqule indlela ye-COE ibe yinqubo egcwele, sihlala senza ukubuyekezwa ngendlela yemihlangano yamasonto onke ngokuhlaziywa okuyisibopho kwamamethrikhi okusebenza. Ngaphezu kwalokho, abahola bezobuchwepheshe benza ukubuyekezwa kwamamethrikhi masonto onke nabo bonke abasebenzi be-AWS.
5. Lungisa futhi uthuthukise!
Isifundo esikhulu lapha lungisa izinkinga ezikhonjwe ngesikhathi sokuhlolwa kwesiphithiphithi kuqala, uzinikeze okubaluleke kakhulu kunokuthuthukisa izici ezintsha. Faka phakathi abaphathi abaphezulu kule nqubo futhi ufake kubo umqondo wokuthi ukulungisa izinkinga zamanje kubaluleke kakhulu kunokuthuthukisa ukusebenza okusha.
Ngake ngasiza iklayenti ukuthi libone izinkinga ezibucayi zokuzinza lisebenzisa ukuhlolwa kwesiphithiphithi, kodwa ngenxa yengcindezi evela eqenjini labadayisi, ukulungiswa kwenqatshelwe futhi yonke imizamo yayigxile ekwethuleni into entsha “eyayibaluleke kakhulu” kumakhasimende. Emasontweni amabili kamuva, isikhathi sokuphumula samahora ayi-16 saphoqa inkampani ukuthi ibhekane nezinkinga ezifanayo esizihlonze ngesikhathi sokuhlolwa kwesiphithiphithi. Ukulahlekelwa kuphela okuvele kuphezulu kakhulu.
Izinzuzo zobunjiniyela be-Chaos
Ziningi izinzuzo. Ngizogqamisa okubili, ngokombono wami, okubaluleke kakhulu:
Okokuqala, ubunjiniyela besiphithiphithi busiza ukudalula izinkinga ezingaziwa ohlelweni futhi bazilungise ngaphambi kokuthi zenze ukukhiqizwa kuphahlazeke, ake sithi, ngo-3 ekuseni ngeSonto. Okungukuthi, yena kwandisa ukumelana nokwehluleka futhi, empeleni, ikhwalithi yokulala.
Okwesibili, ukuhlolwa kwesiphithiphithi okwenziwe ngempumelelo kuhlala kubangela izinguquko ezibanzi (ikakhulukazi ezamasiko) kunalokho obekulindelwe. Mhlawumbe okubaluleke kakhulu kulokhu ukuziphendukela kwemvelo kuya "innocent" (ukungasoli) isiko, lapho umbuzo othi “Kungani wenze lokho?” kuba “Singakugwema kanjani lokhu esikhathini esizayo?” Umphumela uba iqembu elijabule, elisebenza kahle kakhulu, elizibandakanyayo neliphumelele kakhulu. Futhi lokho kuhle!
Lokhu kuphetha ingxenye yokuqala. Ngethemba ukuthi ukuthandile. Sicela ubhale ukubuyekezwa, wabelane ngemibono noma uvele ushaye izandla . Engxenyeni elandelayo, ngizobheka amathuluzi namasu okwethula amaphutha ezinhlelweni. Kuze kube!
Kulabo abalangazelela ukufunda ingxenye yesibili, nginikeza inkulumo yami ngesihloko se-chaos engineering e-NDC e-Oslo. Kuyo ngikhuluma ngamathuluzi amaningi engiwathandayo:
I-PS evela kumhumushi
Ingxenye yesibili ye-athikili ngesiNgisi futhi sizoyihumusha futhi uma sibona intshisekelo eyanele evela kubafundi be-Habr kule nto - ukuphawula okufanele ngesihloko kwamukelekile!
Funda futhi kubhulogi yethu:
- «";
- «";
- «".
Source: www.habr.com
