I-SRE (i-Site Reliability Engineering) yindlela yokuqinisekisa ukufumaneka kweeprojekthi zewebhu. Kuthathwa njengesakhelo se-DevOps kwaye sithetha malunga nendlela yokufumana impumelelo ekusebenziseni izenzo ze-DevOps. Inguqulelo kweli nqaku iincwadi ukusuka kuGoogle. Ndiyilungiselele ngokwam le nguqulelo kwaye ndithembele kumava am ekuqondeni iinkqubo zokubeka iliso. Kwitshaneli yetelegram ΠΈ Ndikwapapashe ikhonkco lokuguqulelwa kweSahluko sesi-4 sale ncwadi inye malunga neenjongo zomgangatho wenkonzo.
Inguqulelo yikati. Konwabele ukufunda!
Amaqela e-SRE kaGoogle anemigaqo esisiseko kunye nezona ndlela zilungileyo zokudala uhlolo oluyimpumelelo kunye neenkqubo zokwazisa. Esi sahluko sinika isikhokelo malunga nokuba zeziphi iingxaki umtyeleli wekhasi lewebhu anokuthi ahlangabezane nazo kunye nendlela yokusombulula iingxaki ezenza amaphepha ewebhu kube nzima ukubonisa.
Iinkcazo
Akukho sigama esinye esisetyenziselwa ukuxoxa ngezihloko ezinxulumene nokubeka iliso. KwanakwiGoogle, le migaqo ingezantsi ayisetyenziswanga ngokuqhelekileyo, kodwa siza kudwelisa ezona zitoliko ziqhelekileyo.
Ukubeka iliso
Ukuqokelela, ukucubungula, ukuhlanganiswa kunye nokuboniswa kwedatha yobuninzi ngexesha langempela malunga nenkqubo: inani lezicelo kunye neentlobo zezicelo, inani leempazamo kunye neentlobo zeempazamo, ixesha lokucubungula isicelo kunye nexesha lomncedisi.
Ukujongwa kwebhokisi emhlophe
Ukubeka iliso okusekwe kwiimetrics eziboniswe ngamalungu enkqubo yangaphakathi, kuquka nelog, iJava Virtual Machine profiling metrics, okanye iHTTP yokuphatha iimetrics eyenza ubalo lwangaphakathi.
Black-ibhokisi esweni
Ukuvavanya ukuziphatha kwesicelo ngokwembono yomsebenzisi.
Dashboard
I-interface (ngokuqhelekileyo iwebhu) enikezela isishwankathelo sezibonakaliso eziphambili zezempilo zeenkonzo. Ideshibhodi inokuba nezihluzi, ukukwazi ukukhetha izikhombisi ezibonisiweyo, njl njl. I-interface yenzelwe ukuchonga izikhombisi ezibaluleke kakhulu kubasebenzisi. Ideshibhodi inokuphinda ibonise ulwazi lwabasebenzi benkxaso yobugcisa: umgca wezicelo, uluhlu lweempazamo eziphambili, kunye nenjineli eyabelwe indawo ethile yoxanduva.
Isivuseleli (isaziso)
Izaziso ezijoliswe ukuba zifunyanwe ngumntu nge-imeyile okanye ngezinye iindlela, ezinokuthi zibangelwe ziimpazamo okanye ukwanda kumgca wesicelo. Izaziso zihlelwa njenge: amatikiti, izilumkiso ze-imeyile kunye nemiyalezo yemiyalezo ekhawulezileyo.
Unobangela weengcambu
Isiphene sesoftware okanye impazamo yomntu ethe, xa ilungisiwe, akufuneki iphinde yenzeke. Ingxaki inokuba nezizathu ezininzi eziphambili: ukungonelanga kwenkqubo yokuzenzekela, isiphene sesoftware, ukuchazwa okunganelanga kwengqiqo yesicelo. Nganye kwezi zinto inokuba ngunobangela oyintloko, yaye nganye kuzo kufuneka ipheliswe.
I-node kunye nomatshini (i-node kunye nomatshini)
Amagama atshintshisanayo abhekiselele kumzekelo omnye wesicelo esisebenzayo kumncedisi obonakalayo, umatshini wenyani, okanye isikhongozeli. Umatshini omnye unokusingatha iinkonzo ezininzi. Iinkonzo zinokuba:
- iqhagamshelwe enye kwenye: umzekelo, iseva ye-caching kunye neseva yewebhu;
- iinkonzo ezingazalaniyo kwisiqwenga esinye sehardware: umzekelo, indawo yokugcina ikhowudi kunye newizard yenkqubo yoqwalaselo, njenge okanye .
Dudula
Naluphi na utshintsho kubumbeko lwesoftware.
Kutheni kufuneka ukubeka iliso?
Kukho izizathu ezininzi zokuba kutheni izicelo kufuneka zibekwe esweni:
Uhlalutyo lweendlela zexesha elide
Ingakanani idatabase kwaye ikhula ngokukhawuleza kangakanani? Ingaba inani lemihla ngemihla labasebenzisi litshintsha njani?
Ukuthelekisa ukusebenza
Ngaba izicelo zikhawuleza kwi-Acme Bucket ye-Bytes 2.72 xa kuthelekiswa ne-Ajax DB 3.14? Kungcono kangakanani ukuba izicelo zigcinwe emva kokubonakala kwe-node eyongezelelweyo? Ngaba isiza sisebenza kancinci xa sithelekisa neveki ephelileyo?
Isilumkiso (izaziso)
Kukho into eyaphukileyo kwaye kufuneka umntu ayilungise. Okanye into iya kuphuka ngokukhawuleza kwaye umntu kufuneka ayijonge ngokukhawuleza.
Ukudala iideshibhodi
Iidashbhodi kufuneka ziphendule imibuzo esisiseko kwaye zibandakanye into esuka kuyo - ukulibaziseka (latency), i-traffic (i-traffic), iimpazamo (iimpazamo) kunye nobukhulu bomthwalo (ukugcwalisa).
Ukuqhuba uhlalutyo lokubuyela umva (debugging)
Ukulibaziseka kwesicelo kunyukile, kodwa yintoni enye eyenzekayo ngexesha elinye?
Iinkqubo zokubeka iliso ziluncedo njengomthombo wedatha kwiinkqubo zobuntlola zoshishino kunye nokuququzelela uhlalutyo lweziganeko zokhuseleko. Ngenxa yokuba le ncwadi igxile kwiinkalo zobunjineli apho ii-SRE zinobuchule, asiyi kuxoxa ngeendlela zokubeka iliso apha.
Ukubeka iliso kunye nezilumkiso zivumela inkqubo ukuba ikuxelele xa yophukile okanye iza kuphuka. Xa inkqubo ingakwazi ukuzilungisa ngokuzenzekelayo, sifuna ukuba umntu asihlalutye eso silumkiso, abone enoba ingxaki isasebenza, ayicombulule, aze afumanise oyena nobangela. Ukuba awuphicothi amacandelo enkqubo, awusoze ufumane isilumkiso ngenxa yokuba "into ibonakala ingaqhelekanga."
Ukuthwalisa umntu uxanduva ngezaziso kukusebenzisa ixesha lomqeshwa elixabisa kakhulu. Ukuba umqeshwa uyasebenza, isilumkiso siyayiphazamisa inkqubo yomsebenzi. Ukuba umqeshwa usekhaya, isilumkiso siphazamisa ixesha lomntu kwaye mhlawumbi silala. Xa izilumkiso zisenzeka rhoqo, abasebenzi bayaziphonononga kuzo, bazibeke ecaleni, okanye bangazihoyi izilumkiso ezingenayo. Amaxesha ngamaxesha bayasityeshela isilumkiso sokwenene, esigqunywa ziziganeko zengxolo. Ukuphazamiseka kwenkonzo kunokuhlala ixesha elide njengoko iziganeko zengxolo zithintela ingxaki ukuba ifunyanwe ngokukhawuleza kwaye ilungiswe. Iinkqubo ezisebenzayo zokulumkisa zinomlinganiselo omhle wesignali ukuya kwingxolo.
Ukumisela ulindelo olufanelekileyo lwenkqubo yokubeka iliso
Ukubeka iliso kwisicelo esinzima ngumsebenzi wobunjineli onzima ngokwawo. Nokuba kukho isiseko esibalulekileyo sokuqokelela, ukubonisa, kunye nezixhobo zokulumkisa, iqela le-Google SRE lamalungu ayi-10-12 ngokuqhelekileyo libandakanya umntu omnye okanye ababini abanjongo yabo iphambili kukwakha nokugcina iinkqubo zokubeka iliso. Eli nani liye lehla ngokuhamba kwexesha njengoko sidibanisa kwaye sibeka kwindawo enye iziseko zovavanyo, kodwa iqela ngalinye le-SRE lidla ngokuba nomntu omnye ozinikele ekubekeni iliso kuphela. Kuya kufuneka sitsho ukuba ngelixa iideshibhodi zenkqubo yokubeka esweni zinomdla kakhulu ukujonga, amaqela e-SRE aphephe ngononophelo iimeko ezifuna ukuba umntu ajonge isikrini ukujonga iingxaki.
Lilonke, uGoogle uye wafudukela kwiinkqubo ezilula nezikhawulezayo zokubeka iliso ezinezixhobo zokuhlalutya emva kwenyani. Siyazinqanda iinkqubo "zomlingo" ezizama ukuqikelela i-thresholds okanye ukufumanisa ngokuzenzekelayo oyena nobangela. Iinzwa ezibona umxholo ongajongwanga kwizicelo zabasebenzisi bokugqibela kuphela komzekelo; Lo gama nje ezi zivamvo zihlala zilula, ziyakwazi ukubona ngokukhawuleza oonobangela beengxaki ezinzulu. Ezinye iifomati zokusebenzisa idatha yokubeka iliso, njengokucwangcisa umthamo okanye uqikelelo lwetrafikhi, zinzima kakhulu. Uqwalaselo ngexesha elide kakhulu (iinyanga okanye iminyaka) ngesantya esisezantsi sesampulu (iiyure okanye iintsuku) kuya kubonisa indlela yexesha elide.
Iqela likaGoogle SRE libe nempumelelo exubeneyo nenkqubo yokuxhomekeka entsonkothileyo. Asifane sisebenzise imigaqo efana "ukuba ndifumanisa ukuba i-database iyacotha, ndifumana isilumkiso sokuba i-database iyacotha, kungenjalo ndifumana isilumkiso sokuba indawo iyacotha." Imithetho esekwe kuxhomekeko ikholisa ukubhekisa kwiindawo ezingaguqukiyo zenkqubo yethu, ezifana nenkqubo yokucoca itrafikhi yabasebenzisi kwiziko ledatha. Umzekelo, "ukuba ukuhluzwa kwetrafikhi kwiziko ledatha kuqwalaselwe, ungandilumkisi malunga nokulibaziseka ekuqhubeni izicelo zabasebenzisi" ngumgaqo oqhelekileyo wezilumkiso ezivela kwiziko ledatha. Amaqela ambalwa kuGoogle axhasa inkqubo yokuxhomekeka entsonkothileyo kuba isiseko sethu sinesantya esiqhubekayo sokuphinda kuhlaziywe.
Ezinye zeengcamango ezichazwe kwesi sahluko zisafanelekile: kukho rhoqo ithuba lokuhamba ngokukhawuleza ukusuka kwimpawu ukuya kwiingcambu, ngokukodwa kwiinkqubo eziguqukayo rhoqo. Ngoko ke, ngelixa esi sahluko sichaza ezinye iinjongo zeenkqubo zokubeka iliso kunye nendlela yokuphumeza ezo njongo, kubalulekile ukuba iinkqubo zokubeka esweni zilula kwaye ziyaqondakala kuye wonke umntu kwiqela.
Ngokunjalo, ukugcina amanqanaba engxolo ephantsi kunye namaqondo omqondiso aphezulu, iindlela zokuhlola ii-asethi ezilumkileyo kufuneka zibelula kakhulu kwaye zithembeke. Imithetho evelisa izilumkiso kubantu kufuneka kube lula ukuyiqonda kwaye iveze ingxaki ecacileyo.
Iimpawu ngokuchasene noonobangela
Inkqubo yakho yokubeka iliso kufuneka iphendule imibuzo emibini: "yintoni eyaphukileyo" kwaye "kutheni yaphuka."
"Yintoni eyaphukileyo" ithetha ngempawu, kwaye "kutheni yaphuka" ithetha ngesizathu. Itheyibhile engezantsi ibonisa imizekelo yoqhagamshelwano olunjalo.
Uphawu
Isizathu
Ukufumana imposiso ye-HTTP ye-500 okanye i-404
Iiseva zedatabase ziyala udibaniso
Iimpendulo zeseva ezicothayo
Ukusetyenziswa kwe-CPU ephezulu okanye intambo ye-Ethernet eyonakeleyo
Abasebenzisi base-Antarctica abafumani ii-GIF zekati
I-CDN yakho iyayicaphukela izazinzulu kunye neekati, ngoko ke ezinye iidilesi ze-IP zaphela zifakwe kuluhlu olumnyama
Umxholo wabucala uyafumaneka kuyo yonke indawo
Ukukhutshwa kokukhutshwa kwesoftware entsha kwenza i-firewall ilibale zonke ii-ACL kwaye yavumela wonke umntu ukuba angene
"Yintoni" kwaye "kutheni" zezinye zeebhloko zokwakha ezibalulekileyo ekudaleni inkqubo efanelekileyo yokubeka iliso enomqondiso ophezulu kunye nengxolo encinci.
Ibhokisi emnyama vs Ibhokisi emhlophe
Sidibanisa ujongo olubanzi lwe-White-box kunye nokuthozama kwebhokisi ye-Black-esweni kwiimetriki ezibalulekileyo. Eyona ndlela ilula yokuthelekisa i-Black-box ne-White-box kukuba i-Black-box igxile kwiimpawu kwaye iyasebenza endaweni yokubeka iliso: "inkqubo ayisebenzi kakuhle ngoku." Ibhokisi emhlophe ixhomekeke kubuchule bokuqinisekisa bangaphakathi beenkqubo: iilog zesiganeko okanye iiseva zewebhu. Ngaloo ndlela, i-White-box ikuvumela ukuba ubone iingxaki ezizayo, iimpazamo ezibonakala ngathi kukuhanjiswa kwakhona kwesicelo, njl.
Qaphela ukuba kwinkqubo enemigangatho emininzi, uphawu kwindawo enye yenjineli yoxanduva luphawu kwindawo enye injineli yoxanduva. Umzekelo, ukusebenza kwedatha yehlile. Ukufundwa kwedatha kancinane kuluphawu lwe-SRE yedatha ebabhaqayo. Nangona kunjalo, kwi-SRE yangaphambili ejonga iwebhusayithi ecothayo, unobangela wokufundwa kwedatha ecothayo efanayo yidatabase ecothayo. Ke ngoko, ukubeka iliso kwibhokisi emhlophe ngamanye amaxesha kugxilwe kwiimpawu kwaye ngamanye amaxesha kugxilwe kunobangela, kuxhomekeke kubungakanani bayo.
Xa uqokelela i-telemetry yokulungiswa kweempazamo, i-White-box monitoring iyafuneka. Ukuba iiseva zewebhu zicotha ukuphendula imibuzo yedatha, kufuneka wazi ukuba umncedisi wewebhu unxibelelana ngokukhawuleza kangakanani nesiseko sedatha kwaye uphendula ngokukhawuleza kangakanani. Kungenjalo, awuzukwazi ukwahlula phakathi kweseva ecothayo yedatabase kunye nengxaki yenethiwekhi phakathi komncedisi wewebhu kunye nesiseko sedatha.
Ukujongwa kwebhokisi elimnyama kunenzuzo ephambili xa uthumela izilumkiso: uqalisa isaziso kumamkeli xa ingxaki sele ibangele iimpawu zangempela. Kwelinye icala, ukubeka iliso akunamsebenzi kwingxaki ye-Black-box engekaveli kodwa isondele.
Iimpawu ezine zegolide
Iimpawu ezine zegolide zokubeka iliso yi-latency, i-traffic, iimpazamo, kunye ne-saturation. Ukuba unokulinganisa kuphela iimetriki ezine zesixokelelwano somsebenzisi, gxininisa kwezo zine.
Ukucothisa
Ixesha elifunekayo lokuqwalaselwa kwesicelo. Kubalulekile ukwahlula phakathi kokubambezeleka kwezicelo eziphumeleleyo nezingaphumeleliyo. Ngokomzekelo, impazamo ye-HTTP 500 ebangelwa ukulahleka koxhumo kwisiseko sedatha okanye enye i-backend inokufunyaniswa ngokukhawuleza, nangona kunjalo, impazamo ye-HTTP 500 ingabonisa isicelo esingaphumeleli. Ukumisela impembelelo yempazamo ye-500 kwi-latency iyonke kunokukhokelela kwizigqibo eziphosakeleyo. Kwelinye icala, impazamo ecothayo nokuba yimpazamo ekhawulezileyo! Ke ngoko, kubalulekile ukubeka iliso kwi-latency yempazamo kunokucoca nje iimpazamo.
yendlela
Inani lezicelo kwisixokelelwano sakho lilinganiselwa kwi-high-level system metrics. Kwinkonzo yewebhu, lo mlinganiselo udla ngokumela inani lezicelo zeHTTP ngomzuzwana, zahlulwe ngobume bezicelo (umzekelo, umxholo omileyo okanye oguqukayo). Kwinkqubo yokusasazwa komsindo, lo mlinganiso unokugxila kwisantya se-I / O yenethiwekhi okanye inani leeseshoni zangaxeshanye. Kwinkqubo yokugcina ixabiso eliphambili, lo mlinganiso unokuba yintengiselwano okanye iziphumo zokukhangela ngesekhondi.
I mpazamo
Eli lizinga lezicelo ezingaphumelelanga ezicacileyo (umz. HTTP 500), ezifihlakeleyo (umz. HTTP 200 kodwa idityaniswe nomxholo ongasebenziyo) okanye umgaqo-nkqubo (umz. "Ukuba uthe wabamba impendulo ngomzuzwana omnye, nayiphi na isekhondi enye yimpazamo"). Ukuba iikhowudi zokuphendula ze-HTTP azanelanga ukubonisa zonke iimeko zokungaphumeleli, iiprothokholi zesibini (zangaphakathi) zingafuneka ukuba zibone ukungaphumeleli kwenxalenye. Ukubeka esweni zonke ezo zicelo zinempazamo kusenokungabi nalwazi, kanti iimvavanyo zesistim zokuphela-to-ekupheleni ziya kunceda ukufumanisa ukuba ulungisa umxholo ongachanekanga.
Ukuzalisa
I-metric ibonisa ukuba inkonzo yakho isetyenziswa ngamandla kangakanani. Lo ngumlinganiselo wenkqubo yokubeka iliso echonga izibonelelo ezinyanzeliswa kakhulu (umzekelo, kwinkqubo yememori yememori, ibonisa imemori, kwi-I / O-constrained system, ibonisa inani le-I / Os). Qaphela ukuba iinkqubo ezininzi zithobisa ukusebenza phambi kokuba zifikelele kwi-100% yokusetyenziswa, ngoko ke ukuba nenjongo yokusetyenziswa kubalulekile.
Kwiinkqubo ezintsonkothileyo, i-saturation inokuncediswa yimilinganiselo yomthwalo okwinqanaba eliphezulu: ingaba inkonzo yakho ingaphatha ngokufanelekileyo itrafikhi ephindwe kabini, iphathe kuphela i-10% yetrafikhi engaphezulu, okanye iphathe itrafikhi engaphantsi kunaleyo iyenzayo ngoku? Kwiinkonzo ezilula ezingenazo iiparamitha ezitshintsha ukuntsonkotha kwesicelo (umzekelo, "Ungandiniki nto" okanye "Ndidinga inani elipheleleyo elilodwa le-monotonic"), ezingafane zitshintshe ubumbeko, ixabiso lovavanyo lomthwalo ongatshintshiyo linokwanela. Nangona kunjalo, njengoko kuxutyushwa kumhlathi wangaphambili, iinkonzo ezininzi kufuneka zisebenzise izibonakaliso ezingathanga ngqo, ezifana nokusetyenziswa kwe-CPU okanye i-bandwidth yenethiwekhi, eyaziwayo ephezulu. Ukonyuka kwe-latency kudla ngokuba luphawu oluphambili lokugcwala. Ukulinganisa ixesha le-99 yepesenti yokuphendula kwifestile encinci (umzekelo, umzuzu omnye) unokubonelela ngomqondiso wokuqala wokuzalisa.
Okokugqibela, i-saturation ikwanxulunyaniswa noqikelelo malunga nokuzaliseka okuzayo, umzekelo: "Kukhangeleka ngathi idatabase yakho iya kuzalisa i-hard drive yakho kwiiyure ze-4."
Ukuba ulinganisa zonke izibonakaliso ezine zegolide kwaye xa kukho ingxaki kwenye yeemetriki (okanye, kwimeko yokuzalisa, ingxaki esondeleyo), ulumkise umntu, inkonzo yakho iya kugutyungelwa ngakumbi okanye ngaphantsi ngokubeka iliso.
Ukukhathazeka malunga "nomsila" (okanye izixhobo kunye nokusebenza)
Xa udala inkqubo yokubeka iliso ukusuka ekuqaleni, kukho isilingo sokuphuhlisa inkqubo esekelwe kumaxabiso aphakathi: i-latency ephakathi, i-avareji yokusetyenziswa kwe-CPU ye-nodes, okanye i-avareji yedatha epheleleyo. Ingozi yemizekelo emibini yokugqibela icacile: iiprosesa kunye nedathabheyisi zilahlwa ngendlela engalindelekanga. Oku kufana nokulibaziseka. Ukuba usebenzisa inkonzo yewebhu nge-avareji latency ye-100ms kunye nezicelo ze-1000 ngesekhondi, i-1% yezicelo inokuthatha imizuzwana emi-5. Ukuba abasebenzisi baxhomekeke kwiinkonzo ezininzi zewebhu ezinjalo, ipesenti ye-99 ye-backend enye ingaba lula ixesha lokuphendula eliphakathi kwi-frontend.
Eyona ndlela ilula yokwahlula phakathi kwe-avareji ecothayo kunye nomsila ocothayo wezicelo kukuqokelela imilinganiselo yezicelo ezichazwe kwizibalo (isixhobo esilungileyo sokubonisa ii-histograms) kunexesha elililo: zingaphi izicelo ezinikezelweyo zenkonzo ezithathe phakathi kwe-0 ms. kunye ne-10 ms, phakathi kwe-10 kunye ne-30 ms, phakathi kwe-30 ms kunye ne-100 ms, phakathi kwe-100 ms kunye ne-300 ms, njl. njl. Ukwandisa imida ye-histogram malunga nokwandisa (ngomlinganiselo we-3) kaninzi kuyindlela elula yokubona usasazo. yezicelo.
Ukukhetha umgangatho ofanelekileyo weenkcukacha kwimilinganiselo
Izinto ezahlukeneyo zesistim kufuneka zilinganiswe kumanqanaba ahlukeneyo eenkcukacha. Umzekelo:
- Ukubeka iliso ekusetyenzisweni kwe-CPU ngexesha elithile akuyi kubonisa ii-spikes zexesha elide ezikhokelela kwi-latencies ephezulu.
- Ngakolunye uhlangothi, kwinkonzo yewebhu ekujoliswe kuyo akukho ngaphezu kweeyure ze-9 zexesha lokuphumla ngonyaka (i-99,9% ye-uptime yonyaka), ukujonga impendulo ye-HTTP ye-200 ngaphezu kweyodwa okanye kabini ngomzuzu kunokwenzeka ukuba kungabi rhoqo ngokungeyomfuneko.
- Ngokunjalo, ukujonga indawo ye-hard drive ye-99,9% ukufumaneka ngaphezulu kweyodwa yonke imizuzu eyi-1-2 mhlawumbi ayiyomfuneko.
Lumka kwindlela omisa ngayo i-granularity yemilinganiselo yakho. Ukuqokelela umthwalo we-CPU kanye ngesekhondi kunokubonelela ngedatha enomdla, kodwa imilinganiselo enjalo rhoqo inokubiza kakhulu ukuqokelela, ukugcina, kunye nokuhlalutya. Ukuba injongo yakho yokubeka iliso ifuna i-granularity ephezulu kwaye ayifuni ukuphendula okuphezulu, unokunciphisa ezi ndleko ngokumisela ukuqokelela kweemetrics kumncedisi kwaye umise inkqubo yangaphandle ukuqokelela kunye nokudibanisa ezo metrics. Unga:
- Ukulinganisa umthwalo we-CPU nganye yesibini.
- Nciphisa iinkcukacha zibe yi-5%.
- Aggregate metrics ngomzuzu ngamnye.
Esi sicwangciso siya kukuvumela ukuba uqokelele idatha kwi-granularity ephezulu ngaphandle kokwenza uhlalutyo oluphezulu kunye nokugcinwa kwezinto eziphezulu.
Ngokulula kangangoko kunokwenzeka, kodwa akukho lula
Ukwaleka kweemfuno ezahlukeneyo ngaphezulu komnye kunokubangela inkqubo yokubeka iliso entsonkothileyo. Umzekelo, isixokelelwano sakho sinokuba nezi zinto zilandelayo zinzima:
- Izilumkiso ngokwemida eyahlukeneyo yesicelo sokulinda ukulinda, kwiipesenti ezahlukeneyo, kuzo zonke iintlobo zezalathi ezahlukeneyo.
- Ukubhala ikhowudi eyongezelelweyo ukufumanisa nokuchonga izizathu ezinokwenzeka.
- Yenza iideshbhodi ezinxulumeneyo kwizizathu ezinokuthi zibekho zeengxaki.
Imithombo yengxaki enokwenzeka ayipheli. Njengazo zonke iinkqubo zesoftware, ukubeka iliso kunokuba nzima kangangokuba kuye kube buthathaka kwaye kube nzima ukutshintsha nokugcina.
Ke ngoko, yila inkqubo yakho yokubeka iliso ukuze uyenze lula kangangoko. Xa ukhetha into oza kuyilandelela, gcina oku kulandelayo engqondweni:
- Imithetho ehlala ibamba iziganeko zokwenyani kufuneka ibe lula, iqikeleleke kwaye ithembeke kangangoko.
- Ulungelelwaniso lokuqokelela idatha, ukuhlanganiswa, kunye nokwazisa okwenziwa rhoqo (umzekelo, ngaphantsi kwekota kumaqela athile e-SRE) kufuneka kususwe.
- Iimetriki eziqokelelweyo kodwa ezingaboniswanga nakweyiphi na ideshibhodi yokujonga kwangaphambili okanye isetyenziswe nasiphi na isivuseleli ngabaviwa ukuba kucinywe.
KuGoogle, ukuqokelelwa kweemetrics ezisisiseko kunye nokudityaniswa, kudityaniswe nezilumkiso kunye needashbhodi, zisebenza kakuhle njengenkqubo yokuma yodwa (inkqubo yokubeka iliso kaGoogle yahlulahlulwe yaziinkqubo ezisezantsi, kodwa abantu bayayazi yonke imiba yezi nkqubo). Kungazama ukudibanisa ukubeka iliso kunye nezinye iindlela zokuvavanya iinkqubo ezinzima: iprofayili yenkqubo eneenkcukacha, ukulungiswa kwenkqubo, ukulandelela iinkcukacha malunga nokungafani okanye ukungaphumeleli, ukuvavanywa komthwalo, ukuqokelelwa kwelogi kunye nohlalutyo, okanye ukuhlolwa kwetrafikhi. Ngelixa uninzi lwezi zinto zinezinto ezifanayo kunye nokubeka iliso okusisiseko, ukuxubana kuya kubangela iziphumo ezininzi kwaye kudala inkqubo entsonkothileyo kunye nebuthathaka. Njengeminye imiba emininzi yophuhliso lwesoftware, ukuxhasa iinkqubo ezahlukeneyo ezineengongoma ezicacileyo, ezilula, ezidityanisiweyo ngokukhululekileyo yeyona ndlela ilungileyo (umzekelo, ukusebenzisa i-API yewebhu ukubuyisela idatha ehlanganisiweyo kwifomathi enokuhlala ihambelana ixesha elide. ).
Ukuhlanganisa iMigaqo
Imigaqo exutyushwa kwesi sahluko inokudityaniswa ibe yifilosofi yokubeka iliso kunye nokulumkisa evunyiweyo kwaye ilandelwe ngamaqela e-Google SRE. Ukubambelela kule filosofi yokubeka iliso kuyanqweneleka, sisiqalo esilungileyo sokwenza okanye ukuhlaziya indlela yakho yokwazisa, kwaye kunokukunceda ukuba ubuze imibuzo efanelekileyo yomsebenzi wakho wokusebenza, nokuba ungakanani umbutho wakho okanye ukuntsonkotha kwenkonzo okanye inkqubo.
Xa udala imithetho yokubeka iliso kunye nesilumkiso, ukubuza le mibuzo ilandelayo kunokukunceda ukuba uphephe izinto ezingalunganga kunye nezilumkiso ezingafunekiyo:
- Ngaba lo mgaqo ubhaqa enye imeko engafumanekiyo yenkqubo engxamisekileyo, ibiza inyathelo, kwaye ichaphazela umsebenzisi ngokungenakuthintelwa?
- Ngaba ndingasihoya esi silumkiso ndisazi ukuba silungile? Kunini kwaye kutheni ndingasihoyi esi silumkiso kwaye ndingayiphepha njani le meko?
- Ngaba esi silumkiso sithetha ukuba abasebenzisi bachaphazeleka kakubi? Ngaba kukho iimeko apho abasebenzisi bangachaphazeleki kakubi, njengokucoca izithuthi okanye xa usebenzisa iinkqubo zovavanyo apho izilumkiso kufuneka zihluzwe?
- Ndingathatha inyathelo ukuphendula kwesi silumkiso? Ngaba la manyathelo angxamisekile okanye angalinda kude kube kusasa? Ngaba isenzo sinokuzisebenzela ngokukhuselekileyo? Ngaba esi senzo siya kuba sisisombululo sexesha elide okanye i-workaround yexesha elifutshane?
- Abanye abantu bafumana izilumkiso ezininzi ngalo mba, ngaba ikho indlela yokunciphisa inani lezilumkiso?
Le mibuzo ibonisa intanda-bulumko esisiseko kwizilumkiso kunye neenkqubo zokulumkisa:
- Ngalo lonke ixesha isilumkiso singena, kufuneka ndisabele ngoko nangoko. Ndiyakwazi ukusabela ngokungxamisekileyo izihlandlo eziliqela ngosuku ngaphambi kokuba ndidinwe.
- Isilumkiso ngasinye kufuneka sihambelane.
- Yonke impendulo kwisilumkiso kufuneka ifune ungenelelo lomntu. Ukuba isaziso sinokuqhutyelwa ngokuzenzekelayo, akufanele sifike.
- Izilumkiso kufuneka zibe malunga nengxaki entsha okanye isiganeko ebesingekho ngaphambili.
Le ndlela iphazamisa ulwahlulo oluthile: ukuba isilumkiso siyanelisa iimeko ezine zangaphambili, akukhathaliseki nokuba isilumkiso sithunyelwe sisuka kwi-White-box okanye kwi-Black-Box monitoring system. Le ndlela iphinda ibethelele iiyantlukwano ezithile: kungcono ukuchitha iinzame ezininzi ekuchongeni iimpawu kunezizathu; xa kuziwa kwizizathu, kufuneka ukhathazeke kuphela ngezizathu ezingenakuthintelwa.
Ukubeka iliso ixesha elide
Kwiimeko zemveliso yanamhlanje, iinkqubo zokubeka iliso zibeke iliso kwinkqubo yemveliso ehlala iguquguquka enokutshintsha kolwakhiwo lwesoftware, iimpawu zomthwalo wokusebenza, kunye neethagethi zokusebenza. Izilumkiso ekunzima ngoku ukuzenza ngokuzenzekelayo zisenokuba yinto eqhelekileyo, mhlawumbi zifanele ukulungiswa. Kweli nqanaba, umntu kufuneka afumane kwaye aphelise oonobangela bengxaki; ukuba eso sigqibo asinakwenzeka, impendulo kwisivuseleli ifuna i-automation epheleleyo.
Kubalulekile ukuba izigqibo zokubeka esweni zenziwe ngeenjongo zexesha elide engqondweni. Isilumkiso ngasinye esisebenza namhlanje siphazamisa umntu ekuphuculeni inkqubo ngomso, ngoko kudla ngokuncitshiswa kokufumaneka okanye ukusebenza kwenkqubo yokuvelisa ixesha elifunekayo ukuze kuphuculwe inkqubo yokubeka iliso kwixesha elide. Makhe sijonge imizekelo emibini ukubonisa lo mkhwa.
I-Bigtable SRE: Ibali leSilumkiso esiPheleleyo
Iziseko zangaphakathi zikaGoogle zibonelelwa kwaye zilinganiswe kumgangatho wenkonzo (SLO). Kwiminyaka emininzi eyadlulayo, inkonzo ye-Bigtable ye-SLO yayisekwe kumndilili wokusebenza kwentengiselwano eyenziweyo elinganisa umxhasi ophilayo. Ngenxa yemibandela kwiBigtable kunye namanqanaba asezantsi ovimba wogcino, ukusebenza komndilili bekuqhutywa ngumsila "omkhulu": eyona 5% yemibuzo ibisoloko icotha kakhulu kuneminye.
Izaziso ze-imeyile zithunyelwe njengoko i-SLO threshold isondele, kwaye izaziso zezithunywa zathunyelwa xa i-SLO igqithisiwe. Zombini ezi ntlobo zezaziso zazithunyelwa rhoqo, zitya izixa ezingamkelekanga zexesha lobunjineli: iqela lichithe ixesha elininzi lilungisa izilumkiso ukufumana ezimbalwa ezazifanelekile. Besihlala siphoswa ngumba ochaphazela abasebenzisi kuba kuphela ezinye zezilumkiso bezingumba othile. Uninzi lwezilumkiso azizange zingxamiseke ngenxa yeengxaki eziqondakalayo kwiziseko zophuhliso kwaye ziqhutywe ngendlela esemgangathweni, okanye azizange ziqhutywe ngokupheleleyo.
Ukulungisa le meko, iqela lathatha indlela emacala amathathu: Ngelixa sisebenza nzima ukuphucula ukusebenza kweBigtable, sibeka okwethutyana injongo yethu ye-SLO ukuba ibe yipesenti ye-75 yempendulo yempendulo yombuzo. Siphinde sacima izilumkiso ze-imeyile kuba bezininzi kangangokuba bekungenakwenzeka ukuba sichithe ixesha sizixilonga.
Esi sicwangciso sasivumela ukuba igumbi lokuphefumla liqalise ukulungisa imiba yexesha elide kwi-Bigtable kunye namanqanaba aphantsi ogcino lwe-stack, kunokuhlala silungisa imiba yobuchule. Iinjineli zinokuwenza umsebenzi wenziwe ngaphandle kokubethwa ngezilumkiso ngalo lonke ixesha. Ekugqibeleni, ukuhlehlisa okwethutyana ukusetyenzwa kwezivuseleli kusivumele ukuba siphucule umgangatho wenkonzo yethu.
I-Gmail: Ukuqikelelwa, Iimpendulo zoLuntu ze-Algorithmic
Ekuqalweni kwayo, i-Gmail yakhelwe kwinkqubo yolawulo lwenkqubo yoMsebenzi elungisiweyo eyenzelwe ukuhlanganisa amaqhekeza esalathiso sokukhangela. Uluhlu lomsebenzi lulungelelaniswe kwiinkqubo ezihlala ixesha elide kwaye emva koko lwasetyenziswa kwi-Gmail, kodwa ezinye iibhugi kwikhowudi yesicwangcisi esicacileyo kubonakale kunzima kakhulu ukuzilungisa.
Ngelo xesha, ukubekwa esweni kwe-Gmail kwacwangciswa ukuze izilumkiso zivuswe xa imisebenzi yomntu ngamnye irhoxisiwe kusetyenziswa i-Workqueue. Le ndlela yayingafanelekanga kuba nangelo xesha, i-Gmail yenza amawaka amaninzi emisebenzi, nganye inikezelwe kwiqhezu lepesenti yabasebenzisi bethu. Besikhathazeke kakhulu malunga nokubonelela abasebenzisi beGmail ngamava alungileyo omsebenzisi, kodwa ukuphatha izilumkiso ezininzi bekungafikeleleki.
Ukujongana nalo mba, i-Gmail SRE yenza isixhobo sokunceda ukulungisa ingxaki umcwangcisi kangangoko kunokwenzeka ukunciphisa impembelelo kubasebenzisi. Iqela libe neengxoxo malunga nokuba ngaba ukwenza ngokuzenzekelayo umjikelo wonke ukusuka ekufumaneni ingxaki ngokulungiswa de kufumaneke isisombululo sexesha elide, kodwa abanye babenexhala lokuba isisombululo esinjalo sinokulibazisa ngokwenene ukulungisa ingxaki.
Olu xinzelelo lwaluxhaphakile kwiqela kwaye kaninzi lubonakalisa ukungazithembi ekuziphatheni: ngelixa amanye amalungu eqela efuna ukuvumela ixesha lokulungiswa okuchanekileyo, abanye banenkxalabo yokuba ukulungiswa kokugqibela kuya kulityalwa kwaye ukulungiswa kwexeshana kuya kuthatha ngonaphakade. Lo mba ufuna ukuqwalaselwa kuba kulula kakhulu ukulungisa iingxaki okwexeshana endaweni yokwenza imeko ibe yesigxina. Abaphathi kunye nabasebenzi bezobugcisa badlala indima ephambili ekuphumezeni izilungiso zexesha elide, ukuxhasa kunye nokubeka phambili izilungiso ezinokuthi zibekho emva kokuba "intlungu" yokuqala iyancipha.
Izilumkiso eziqhelekileyo, eziphindaphindiweyo kunye neempendulo ze-algorithmic kufuneka zibe yiflegi ebomvu. Ukungafuni kweqela lakho ukuzenzela ezi zilumkiso kuthetha ukuba iqela alinaluthemba lokuba linokuzithemba ii-algorithms. Le yingxaki enzima ekufuneka iqwalaselwe.
Ixesha elide
Umxholo oqhelekileyo udibanisa imizekelo yeBigtable kunye ne-Gmail: ukhuphiswano phakathi kwexesha elifutshane kunye nokufumaneka kwexesha elide. Ngokuqhelekileyo, umzamo oqinileyo unokunceda inkqubo ebuthathaka ifikeleleke ekufumanekeni okuphezulu, kodwa le ndlela idla ngokuhlala ixesha elifutshane, igcwele ukutshiswa kweqela kunye nokuxhomekeka kwinani elincinci lamalungu eqela elifanayo leqhawe.
Ukulawulwa, ukunciphisa ixesha elifutshane ekufumanekeni kudla ngokubuhlungu, kodwa kubalulekile ngokucwangcisekileyo ukuzinza kwexesha elide lenkqubo. Kubalulekile ukuba ungakhange ujonge isilumkiso ngasinye ngokuzimeleyo, kodwa ukuqwalasela ukuba inqanaba elipheleleyo lomthamo wokulumkisa likhokelela kwinkqubo ephilileyo, efikeleleke ngokufanelekileyo kunye neqela elisebenzayo kunye ne-prognosis efanelekileyo. Sihlalutya iinkcukacha-manani zesilumkiso (ezidla ngokuchazwa njengezehlo kwishifti nganye, apho isiganeko sinokuquka iziganeko ezininzi ezinxulumeneyo) kwiingxelo zekota eziya kubaphathi, sivumela abenzi bezigqibo ukuba babe nombono oqhubekayo womthwalo wenkqubo yesilumkiso kunye nempilo yeqela lilonke.
isiphelo
Indlela eya ekubekweni iliso okunempilo kunye nokwazisa ilula kwaye icacile. Igxininisa kwiimpawu zengxaki ezibangela izilumkiso, kwaye ukubeka esweni unobangela kusebenza njengoncedo kwiingxaki zokulungisa. Iimpawu zokubeka iliso kulula xa uphezulu kwi-stack olawulayo, nangona umthwalo wokubeka iliso kunye nokusebenza kwesiseko sedatha kufuneka kwenziwe ngokuthe ngqo kwisiseko sedatha ngokwayo. Izaziso ze-imeyile zinoncedo olulinganiselweyo kwaye zithande ukuba lula ukuba yingxolo; endaweni yoko, kufuneka usebenzise ideshbhodi ebeka iliso kuyo yonke imiba yangoku eqalisa izilumkiso ze-imeyile. Ideshibhodi inokudityaniswa nelog yesiganeko ukuhlalutya unxulumano lwembali.
Ngexesha elide, kuyimfuneko ukufezekisa ukujikeleza okuyimpumelelo kwezilumkiso malunga neempawu kunye neengxaki zangempela ezisondeleyo, ukulungelelanisa iinjongo zokuqinisekisa ukuba ukubeka iliso kuxhasa ukuxilongwa ngokukhawuleza.
Enkosi ngokufunda inguqulelo kude kube sekupheleni. Bhalisela kumjelo wetelegram malunga nokubeka iliso ΠΈ .
umthombo: www.habr.com