SRE (Site Reliability Engineering) ndi njira yopangira mawebusayiti kuti apezeke. Imatengedwa ngati chimango cha DevOps ndikuwuza momwe mungapambanire kugwiritsa ntchito machitidwe a DevOps. Nkhaniyi ikumasulira mabuku kuchokera ku Google. Ndinakonza ndekha zomasulirazi ndikudalira luso langa pakumvetsetsa njira zowunikira. Mu njira ya telegraph ΠΈ Ndinatumizanso ulalo womasulira Chaputala 4 cha buku lomwelo pa Zolinga za Utumiki wa Utumiki.
Kumasulira kwa mphaka. Sangalalani kuwerenga!
Magulu a Google SRE ali ndi mfundo zoyambira ndi njira zabwino zopangira njira zowunikira komanso zidziwitso. Mutuwu umapereka malingaliro pazovuta zomwe mlendo patsamba angakumane nazo komanso momwe angathetsere zovuta zomwe zimapangitsa kuti zikhale zovuta kuwonetsa masamba.
Malingaliro
Palibe mawu amodzi omwe amagwiritsidwa ntchito pokambirana mitu yokhudzana ndi kuyang'anira. Ngakhale pa Google, mawu omwe ali pansipa sagwiritsidwa ntchito mofanana, koma tidzalemba matanthauzidwe ambiri.
Kuwunikira
Kusonkhanitsa, kukonza, kusonkhanitsa ndi kuwonetsa zenizeni zenizeni zenizeni zokhudzana ndi dongosolo: chiwerengero cha zopempha ndi mitundu ya zopempha, chiwerengero cha zolakwika ndi mitundu ya zolakwika, kupempha nthawi yokonza ndi nthawi ya seva.
White box monitoring
Kuyang'anira motengera ma metric omwe amawonetsedwa ndi omwe ali mkati mwadongosolo, kuphatikiza logi, ma JVM kapena ma metric owongolera ma HTTP omwe amapanga ziwerengero zamkati.
Kuwunika kwa bokosi lakuda
Kuyesa machitidwe a pulogalamuyo malinga ndi momwe wogwiritsa ntchito amawonera.
Dashboard (ma dashboards)
Mawonekedwe (kawirikawiri mawonekedwe a intaneti) omwe amapereka chithunzithunzi cha zizindikiro zazikulu za umoyo wa mautumiki. Dashboard ikhoza kukhala ndi zosefera, kuthekera kosankha ma metric oti muwonetse, ndi zina. Dashboard imathanso kuwonetsa zidziwitso za ogwira ntchito zaukadaulo: mzere wopempha, mndandanda wa zolakwika zofunika kwambiri, mainjiniya omwe adapatsidwa gawo laudindo.
Chidziwitso (chidziwitso)
Zidziwitso zomwe zimayenera kulandiridwa ndi munthu kudzera pa imelo kapena ayi, zomwe zingayambike chifukwa cha zolakwika kapena kuwonjezeka kwa mzere wopempha. Zidziwitso zili m'magulu monga: matikiti, zidziwitso za imelo ndi mauthenga a messenger.
Choyambitsa (choyambitsa)
Vuto la pulogalamu kapena zolakwika zamunthu zomwe, zikakonzedwa, siziyenera kuchitikanso. Vutoli likhoza kukhala ndi zifukwa zingapo zazikulu: kusakwanira makina opangira, vuto la mapulogalamu, kuphunzira kosakwanira kwa malingaliro ogwiritsira ntchito. Chilichonse mwazinthu izi chikhoza kukhala chomwe chimayambitsa, ndipo chilichonse chiyenera kuthetsedwa.
Node ndi makina (mfundo ndi makina)
Mawu osinthika otanthauza nthawi imodzi ya pulogalamu yomwe ikuyenda pa seva yeniyeni, makina enieni, kapena chidebe. Pakhoza kukhala mautumiki angapo pa makina amodzi. Services akhoza kukhala:
- zokhudzana wina ndi mzake: mwachitsanzo, seva ya cache ndi seva ya intaneti;
- mautumiki osagwirizana pa hardware yomweyo: mwachitsanzo, code repository ndi wizard ya kasinthidwe kachitidwe, monga kapena .
Kankhani
Kusintha kulikonse kwa kasinthidwe ka mapulogalamu.
Chifukwa chiyani kuyang'anira ndikofunikira
Pali zifukwa zingapo zomwe ziyenera kuyang'aniridwa:
Kusanthula kwamayendedwe anthawi yayitali
Kodi database ndi yayikulu bwanji ndipo ikukula mwachangu? Kodi chiwerengero cha tsiku ndi tsiku cha ogwiritsa ntchito chimasintha bwanji?
Kufananiza Magwiridwe
Kodi mafunso amafulumira pa Acme Bucket ya Bytes 2.72 kuposa Ajax DB 3.14? Kodi zopempha zimasungidwa bwino bwanji pambuyo pakuwonekera kwa node yowonjezera? Kodi tsambalo lachedwa kuposa sabata yatha?
Chenjezo (zidziwitso)
Chinachake chathyoka ndipo wina ayenera kuchikonza. Kapena chinachake chidzasweka posachedwapa ndipo wina ayenera kuyang'ana posachedwa.
Kupanga dashboards
Madashibodi akuyenera kuyankha mafunso ofunikira ndikuphatikiza china chake - kuchedwa (latency), traffic (magalimoto), zolakwika (zolakwika) ndi mtengo wa katundu (machulukitsidwe).
Kupanga kusanthula kwa retrospective (kuchotsa zolakwika)
Pemphani kuchedwa kwa ntchito kwawonjezeka, ndi chiyani chinanso chinachitika nthawi yomweyo?
Machitidwe owunikira ndi othandiza ngati gwero la data pamakina anzeru zamabizinesi ndikuwongolera kuwunikira zochitika zachitetezo. Popeza bukhuli likuyang'ana kwambiri zaumisiri momwe ma SRE ali ndi ukadaulo, sitikambirana za njira zowunikira pano.
Kuwunika ndi zidziwitso zimalola dongosolo kuti lidziwe pamene lasweka kapena latsala pang'ono kusweka. Dongosolo likalephera kudzikonza lokha, timafuna kuti munthu apende chenjezo, aone ngati vutolo lidakalipo, kulikonza, ndi kudziwa chifukwa chake. Ngati simuyang'ana zida zamakina, simudzakhala tcheru chifukwa "china chake chikuwoneka ngati chosamvetseka."
Kuyika zidziwitso za anthu ndikugwiritsa ntchito nthawi yodula kwambiri. Ngati wogwira ntchito akugwira ntchito, chenjezo limasokoneza kayendedwe ka ntchito. Ngati wogwira ntchitoyo ali kunyumba, chenjezoli limasokoneza nthawi yake ndipo mwina kugona. Zidziwitso zikachitika pafupipafupi, ogwira ntchito amathamangira, kuchedwa, kapena kunyalanyaza zidziwitso zomwe zikubwera. NthaΕ΅i ndi nthawi amanyalanyaza chenjezo lenileni, lomwe limaphimbidwa ndi zochitika zaphokoso. Zosokoneza zautumiki zitha kukhala kwa nthawi yayitali chifukwa zochitika zaphokoso zimalepheretsa kuzindikira ndi kuthetsa vuto mwachangu. Njira zoyankhulirana ndi anthu zogwira mtima zimakhala ndi chiΕ΅erengero chabwino cha ma signal-to-phokoso.
Kupeza zoyembekeza zomveka kuchokera ku dongosolo lowunika
Kukhazikitsa kuyang'anira ntchito yovuta ndi ntchito yovuta ya uinjiniya palokha. Ngakhale tili ndi zida zofunikira zosonkhanitsira, zowonetsera, ndi zochenjeza, gulu la Google SRE la mamembala 10-12 nthawi zambiri limaphatikizapo munthu m'modzi kapena awiri omwe cholinga chake chachikulu ndikumanga ndi kukonza njira zowunikira. Chiwerengerochi chatsika pakapita nthawi pamene tikukonza ndikuyika malo owunikira, koma gulu lililonse la SRE limakhala ndi wogwira ntchito mmodzi yekha. Ziyenera kunenedwa kuti ngakhale ndizosangalatsa kuwonera ma dashboards oyang'anira, magulu a SRE amapewa mosamala zinthu zomwe zimafuna kuti wina aziyang'ana pazenera kuti awone zovuta.
Nthawi zambiri, Google yasamukira kumayendedwe osavuta komanso othamanga omwe ali ndi zida zowunikira pambuyo pake. Timapewa machitidwe a "matsenga" omwe amayesa kulosera zomwe zidzachitike kapena kudziwiratu zomwe zimayambitsa. Zomverera zomwe zimazindikira zomwe sizikufunidwa pazofunsira kwa ogwiritsa ntchito ndiye chitsanzo chokhacho; malinga ngati masensa amenewa amakhalabe osavuta, amatha kuzindikira mwachangu zomwe zimayambitsa zovuta zazikulu. Mawonekedwe ena ogwiritsira ntchito deta yowunikira, monga kulinganiza mphamvu kapena kulosera zam'tsogolo, ndizovuta kwambiri. Kuwona kwa nthawi yayitali kwambiri (miyezi kapena zaka) pamiyeso yotsika (maola kapena masiku) kudzawonetsa zomwe zimachitika nthawi yayitali.
Gulu la Google SRE lagwira ntchito mopambana mosiyanasiyana ndi magulu ovuta kudalira. Sitimagwiritsa ntchito malamulo monga "ndikapeza kuti database yayamba pang'onopang'ono, ndimalandira chenjezo la kuchepa kwa database, apo ayi ndimakhala tcheru pang'onopang'ono." Malamulo otengera kudalira nthawi zambiri amatanthawuza magawo osasinthika a dongosolo lathu, monga dongosolo losefera kuchuluka kwa anthu omwe amapita ku data center. Mwachitsanzo, "ngati zosefera zapakati pa data zakonzedwa, osandichenjeza za kuchedwa pokonza zopempha za ogwiritsa ntchito" ndi lamulo limodzi lodziwika bwino pazidziwitso zapa data. Magulu owerengeka a Google amathandizira magulu odalirana chifukwa cha zomangamanga zathu zimakhala ndi kusinthasintha kosalekeza.
Mfundo zina zomwe zafotokozedwa m'mutu uno zikugwirabe ntchito: nthawi zonse pali njira yopitira mofulumira kuchoka ku chizindikiro kupita ku mizu, makamaka mu machitidwe omwe amasintha nthawi zonse. Choncho, ngakhale mutuwu ukufotokoza zolinga za kalondolondo ndi momwe angakwaniritsire zolingazo, nkofunika kuti kalondolondo akhale wosavuta komanso womveka kwa aliyense pagulu.
Momwemonso, kuti phokoso likhale lochepa komanso kuti chizindikiro chikhale chokwera, njira zowunikira zinthu zomwe zikuchenjezedwa ziyenera kukhala zosavuta komanso zodalirika. Malamulo omwe amapereka machenjezo kwa anthu ayenera kukhala osavuta kumva ndikuwonetsa vuto lomveka bwino.
Zizindikiro motsutsana ndi zomwe zimayambitsa
Dongosolo lanu lowunikira likuyenera kuyankha mafunso awiri: "chaphwanyidwa ndi chiyani" ndi "chasweka bwanji".
"Chomwe chinathyoka" chimatanthawuza chizindikiro, ndipo "chifukwa chiyani chinasweka" chikutanthauza chifukwa. Gome ili m'munsili likuwonetsa zitsanzo za maulalo otere.
Chizindikiro
Chifukwa
Kulandira HTTP Error 500 kapena 404
Ma seva a database akukana kulumikizana
Mayankho ochedwa pa seva
Kugwiritsa Ntchito Kwambiri kwa CPU kapena Chingwe Chowonongeka cha Ethernet
Ogwiritsa ntchito ku Antarctica sakulandira ma GIF amphaka
CDN yanu imadana ndi asayansi ndi nyama zakutchire, kotero ma IP ena amalembedwa
Zachinsinsi zimapezeka paliponse
Kutulutsa pulogalamu yatsopano kumapangitsa kuti firewall iyiwale ma ACL onse ndikulola aliyense kulowa
"Chiyani" ndi "chifukwa" ndi chimodzi mwazomangamanga zofunika kwambiri popanga dongosolo labwino loyang'anira ndi chizindikiro chachikulu komanso phokoso lochepa.
Bokosi lakuda vs. White-box
Timaphatikiza kuyang'anira kwakukulu kwa bokosi loyera ndi kuyang'anira pang'onopang'ono kwa bokosi lakuda kwa ma metric ovuta. Njira yosavuta yofananizira Black-box ndi White-box ndikuti Black-box imayang'ana pazizindikiro ndipo imagwira ntchito m'malo moyang'anitsitsa: "dongosolo silikuyenda bwino pakali pano." Bokosi loyera limatengera kuthekera kwamkati kwamakina: zipika za zochitika kapena maseva apaintaneti. Chifukwa chake, White-box imakupatsani mwayi wozindikira zovuta zomwe zikubwera, zosokoneza zomwe zimawoneka ngati kubwezanso kwa pempho, ndi zina zambiri.
Zindikirani kuti mumitundu yambiri, chizindikiro m'dera laudindo wa injiniya m'modzi ndi chizindikiro m'dera laudindo wa injiniya wina. Mwachitsanzo, ntchito ya database yatsika. Kuwerengera pang'onopang'ono kwa database ndi chizindikiro cha database ya SRE yomwe imawazindikira. Komabe, kwa SRE yakutsogolo kuwonera tsamba lapang'onopang'ono, chifukwa chowerengera pang'onopang'ono database ndikuti databaseyo imachedwa. Chifukwa chake, kuyang'anira bokosi loyera nthawi zina kumangoyang'ana pazizindikiro ndipo nthawi zina pazifukwa, kutengera kukula kwake.
Mukasonkhanitsa telemetry kuti muchotse zolakwika, kuwunika kwa White-box ndikofunikira. Ngati ma seva a intaneti akuchedwa kuyankha mafunso a database, muyenera kudziwa momwe seva yapaintaneti imalumikizirana mwachangu ndi database komanso momwe ikuyankhira. Apo ayi, simungathe kusiyanitsa pakati pa seva yapang'onopang'ono ya database ndi vuto la intaneti pakati pa seva ya intaneti ndi database.
Kuwunika kwa bokosi lakuda kuli ndi mwayi waukulu potumiza zidziwitso: mumayambitsa chidziwitso kwa wolandira pamene vuto lidayambitsa kale zizindikiro zenizeni. Kumbali inayi, chifukwa cha vuto la Black-box lomwe silinayambepo, koma lomwe likubwera, kuyang'anira sikuthandiza.
Zizindikiro zinayi zagolide
Zizindikiro zinayi zowunikira zagolide ndi latency, traffic, zolakwika, ndi machulukitsidwe. Ngati mutha kuyeza ma metrics anayi okha, yang'anani pa anayiwo.
Kuchedwa
Nthawi yofunikira kuti tichite pempho. Ndikofunika kusiyanitsa pakati pa latency ya zopempha zopambana ndi zosapambana. Mwachitsanzo, cholakwika cha HTTP 500 chifukwa cha kulumikizidwa kotayika ku database kapena backend ina imatha kupezeka mwachangu kwambiri, komabe, cholakwika cha HTTP 500 chingasonyeze pempho lolephera. Kupeza zotsatira za cholakwika cha 500 pa latency yonse kumatha kubweretsa malingaliro olakwika. Kumbali ina, kulakwitsa pang'onopang'ono ndi kulakwitsa kwachangu! Chifukwa chake, ndikofunikira kuyang'anira zolakwika za latency m'malo mongosefa zolakwika.
magalimoto
Chiwerengero cha zopempha ku chipangizo chanu, choyezedwa ndi ma metrics apamwamba kwambiri. Pa ntchito yapaintaneti, muyeso uwu umayimira kuchuluka kwa zopempha za HTTP pa sekondi iliyonse yogawidwa ndi mtundu wa zopemphazo (mwachitsanzo, zokhazikika kapena zosunthika). Pa makina omvera, muyeso uwu ukhoza kukhazikika pa mlingo wa I/O wa netiweki kapena kuchuluka kwa magawo omwe amachitika nthawi imodzi. Pamakina osungira amtengo wapatali, muyeso uwu ukhoza kukhala wochitika kapena kuyang'ana pa sekondi imodzi.
Zolakwika
Uwu ndiye kuchuluka kwa zopempha zomwe zidalephera, mwina momveka bwino (mwachitsanzo, HTTP 500), mosabisa (mwachitsanzo, HTTP 200 koma kuphatikiza ndi zoyipa), kapena ndi mfundo (mwachitsanzo, "Ngati mutenga yankho mphindi imodzi, chilichonse mphindi imodzi ndi cholakwika"). Ngati palibe zizindikiro zokwanira za HTTP zowonetsera kulephera konse, ma protocol achiwiri (zamkati) angafunike kuti azindikire kulephera pang'ono. Kuwunika zopempha zolakwika zotere kumatha kukhala kopanda chidziwitso, pomwe kuyesa komaliza mpaka kumapeto kungakuthandizeni kuzindikira kuti mukukonza zolakwika.
Machulukidwe
Metric ikuwonetsa momwe ntchito yanu ikugwiritsidwira ntchito kwambiri. Uwu ndi muyeso wowunikira dongosolo womwe umazindikiritsa zinthu zomwe ndizochepa kwambiri (mwachitsanzo, m'dongosolo lokhala ndi kukumbukira pang'ono, zikuwonetsa kukumbukira, pamakina omwe ali ndi I / O ochepa, amawonetsa kuchuluka kwa I / O). Dziwani kuti machitidwe ambiri amatsika asanafike 100% kugwiritsidwa ntchito, kotero kukhala ndi cholinga chogwiritsa ntchito ndikofunikira.
M'makina ovuta, machulukidwe amatha kuwonjezeredwa ndi kuyeza kwa katundu wapamwamba kwambiri: kodi ntchito yanu imatha kuyendetsa bwino magalimoto awiri, kungogwira 10% yochulukirapo, kapena kuyendetsa magalimoto ochepa kuposa momwe ingathere pano? Pazithandizo zosavuta zomwe zilibe magawo omwe amasintha zovuta za pempho (monga "Musandipatse kalikonse" kapena "Ndikufuna nambala imodzi yokha ya monotonic") zomwe sizisintha kaΕ΅irikaΕ΅iri makonzedwe, mtengo woyezera katundu wokhazikika ungakhale wokwanira. Komabe, monga momwe tafotokozera m'ndime yapitayi, mautumiki ambiri ayenera kugwiritsa ntchito zizindikiro zosalunjika monga kugwiritsa ntchito CPU kapena bandwidth yomwe ili ndi malire odziwika. Kuchuluka kwa latency nthawi zambiri kumakhala chizindikiro chachikulu cha kukhutitsidwa. Kuyeza 99th percentile kuyankha pa zenera laling'ono (monga miniti imodzi) kungapereke chizindikiro choyambirira kwambiri.
Pomaliza, machulukitsidwe amalumikizidwanso ndi maulosi a kuchuluka komwe kukubwera, monga: "Zikuwoneka ngati database yanu idzadzaza hard drive yanu mu maola 4."
Ngati muyeza zizindikiro zonse zinayi za golide ndipo pakakhala vuto ndi imodzi mwazitsulo (kapena, ngati machulukidwe, pafupifupi vuto), mumamudziwitsa munthuyo, ntchito yanu idzayang'aniridwa mocheperapo.
Kuda nkhawa ndi mchira (kapena zida ndi magwiridwe antchito)
Mukamanga kachipangizo koyang'anira kuyambira pachiyambi, ndikuyesa kupanga dongosolo lokhazikika: pafupifupi latency, ma CPU ogwiritsira ntchito ma node, kapena kuchuluka kwa database. Kuopsa kwa zitsanzo ziwiri zomaliza ndizodziwikiratu: mapurosesa ndi ma database amatayidwa m'njira yosadziwika bwino. Zomwezo zimagwiranso ntchito kuchedwa. Ngati mukugwiritsa ntchito intaneti yokhala ndi latency ya 100ms pa zopempha 1000 pamphindikati, 1% ya zopempha zimatha kutenga masekondi asanu. Ngati ogwiritsa ntchito amadalira mautumiki angapo amtunduwu, 5th percentile ya backend imodzi imatha kukhala nthawi yoyankha yapakatikati.
Njira yosavuta yosiyanitsira pakati pa kuchedwa kwapang'onopang'ono ndi mchira wodekha kwambiri wa zopempha ndikusonkhanitsa miyeso ya zopempha zomwe zafotokozedwa mu ziwerengero (ma histogram ndi chida choyenera kuwonetsera), osati kuchedwa kwenikweni: ndi zopempha zingati zomwe zinaperekedwa ndi ntchito yomwe idatenga. pakati pa 0 ms ndi 10ms, pakati pa 10ms ndi 30ms, pakati pa 30ms ndi 100ms, pakati pa 100ms ndi 300ms, ndi zina zotero. Kukulitsa malire a histogram pafupifupi exponentially (ndi chinthu cha pafupifupi 3) nthawi zambiri ndi njira yosavuta yowonera kugawidwa kwa zopempha.
Kusankha granularity yoyenera yoyezera
Zinthu zosiyanasiyana zadongosolo ziyenera kuyesedwa ndi tsatanetsatane watsatanetsatane. Mwachitsanzo:
- Kuwona kugwiritsidwa ntchito kwa CPU pakapita nthawi sikudzawonetsa ma spikes aatali omwe amabweretsa kuchedwa kwambiri.
- Kumbali ina, ntchito yapaintaneti yomwe ikuyang'ana osapitilira maola a 9 pachaka (99,9% uptime pachaka), kuyang'ana kuyankha kwa HTTP 200 kangapo kamodzi kapena kawiri pamphindi kungakhale kosafunikira.
- Mofananamo, kuyang'ana malo aulere pa hard drive kuti 99,9% ipezeke kangapo kamodzi pa mphindi 1-2 mwina sikofunikira.
Samalani momwe mumapangira kukula kwa miyeso. Kugwiritsa ntchito kwa CPU kwa 1 pa sekondi iliyonse kumatha kupereka zambiri zosangalatsa, koma kuyeza kotereku kumatha kukhala kokwera mtengo kwambiri kusonkhanitsa, kusunga, ndi kusanthula. Ngati cholinga chanu chowunikira chimafuna kuchuluka kwakukulu ndipo sichikufuna kulabadira kwambiri, mutha kuchepetsa ndalamazi pokhazikitsa zosonkhanitsira ma metrics pa seva ndikusintha makina akunja kuti asonkhanitse ndikuphatikiza ma metricswo. Mungathe ku:
- Yesani kugwiritsa ntchito CPU sekondi iliyonse.
- Chepetsani zambiri mpaka 5%.
- Aggregate metrics miniti iliyonse.
Njirayi ikuthandizani kuti musonkhane zambiri za granular popanda kukumana ndi mitu yayikulu kuti muwunike ndikusunga.
Zosavuta momwe zingathere, koma osati zosavuta
Kuyika zofunikira zosiyanasiyana pamwamba pa wina ndi mzake kungayambitse dongosolo lowunikira kwambiri. Mwachitsanzo, makina anu akhoza kukhala ndi zinthu zotsatirazi:
- Zidziwitso molingana ndi magawo osiyanasiyana ofunsira latency, mumaperesenti osiyanasiyana, pamitundu yonse yamitundu yosiyanasiyana.
- Kulemba khodi yowonjezera kuti mudziwe ndi kuzindikira zomwe zingatheke.
- Pangani ma dashboards okhudzana ndi chilichonse chomwe chingayambitse mavuto.
Zomwe zimayambitsa zovuta sizimatha. Monga machitidwe onse a mapulogalamu, kuyang'anira kungakhale kovuta kwambiri moti kumakhala kosavuta, kovuta kusintha ndi kusamalira.
Chifukwa chake, pangani dongosolo lanu lowunikira kuti muchepetse momwe mungathere. Posankha zomwe mukufuna kutsatira, kumbukirani izi:
- Malamulo omwe nthawi zambiri amagwira zochitika zenizeni ayenera kukhala osavuta, odziwikiratu, komanso odalirika momwe angathere.
- Kusintha kwa kusonkhanitsa deta, kusonkhanitsa, ndi kuchenjeza komwe kumachitika kawirikawiri (mwachitsanzo, kuchepera kotala kwamagulu ena a SRE) kuyenera kuchotsedwa.
- Ma metric omwe amasonkhanitsidwa koma osawonetsedwa pachiwonetsero chilichonse kapena ogwiritsidwa ntchito ndi zidziwitso zilizonse ndizomwe zikuyenera kufufutidwa.
Ku Google, kusonkhanitsa kofunikira ndikuphatikiza ma metrics, kuphatikiza zidziwitso ndi ma dashboards, zimagwira ntchito bwino ngati njira yodziyimira yokha (dongosolo loyang'anira la Google limasweka m'magawo angapo, koma nthawi zambiri anthu amadziwa mbali zonse za subsystems). Zitha kukhala zokopa kuphatikiza kuwunika ndi njira zina zoyesera makina ovuta: kufotokoza mwatsatanetsatane kachitidwe, kukonza zolakwika, kutsatiridwa kapena kulephera, kuyezetsa katundu, kusonkhanitsa zipika ndi kusanthula, kapena kuyang'anira magalimoto. Ngakhale kuti zambiri mwazinthuzi zimagawana zofanana ndi kuyang'anitsitsa koyambirira, kuzisakaniza kumabweretsa zotsatira zambiri ndikupanga dongosolo lovuta komanso losasunthika. Monga momwe zilili ndi mbali zina zambiri za chitukuko cha mapulogalamu, kuthandizira machitidwe osiyanasiyana okhala ndi mfundo zomveka bwino, zosavuta, zosakanikirana zosakanikirana ndi njira yabwino kwambiri (mwachitsanzo, kugwiritsa ntchito intaneti API kuti mutengere chidule cha deta mumtundu womwe ukhoza kukhala wokhazikika kwa nthawi yaitali. ).
Kugwirizanitsa mfundo pamodzi
Mfundo zomwe zafotokozedwa m'mutuwu zitha kuphatikizidwa kukhala filosofi yowunikira ndi kuchenjeza yomwe imavomerezedwa ndikutsatiridwa ndi magulu a Google SRE. Kutsatira mfundo yowunikirayi ndikofunika, ndi poyambira bwino popanga kapena kukonzanso njira yochenjeza, ndipo kungakuthandizeni kufunsa mafunso oyenerera pazantchito posatengera kukula kwa bungwe lanu kapena zovuta zantchito kapena dongosolo.
Popanga malamulo owunikira ndi kuchenjeza, kufunsa mafunso otsatirawa kungakuthandizeni kupewa zabwinoza zabodza ndi zidziwitso zosafunikira:
- Kodi lamuloli limazindikira dongosolo lomwe silingawonekere lomwe ndi lachangu, loyitanitsa kuchitapo kanthu, komanso lomwe limakhudza wogwiritsa ntchito?
- Kodi ndinganyalanyaze chenjezoli podziwa kuti ndi labwino? Ndi liti komanso chifukwa chiyani ndinganyalanyaze chenjezoli ndipo ndingapewe bwanji izi?
- Kodi chenjezoli likutanthauza kuti ogwiritsa ntchito akukhudzidwa moyipa? Kodi pali zochitika zomwe ogwiritsa ntchito samakhudzidwa, mwachitsanzo, chifukwa cha kusefa kwa magalimoto kapena akamagwiritsa ntchito zoyeserera, zidziwitso zomwe ziyenera kusefedwa?
- Kodi ndingachitepo kanthu potsatira chenjezoli? Kodi izi ndizofunika kuchitapo kanthu mwachangu kapena zingadikire mpaka m'mawa? Kodi ndi zotetezeka kusinthira zochitazo? Kodi izi zitha kukhala yankho lanthawi yayitali kapena ntchito kwakanthawi kochepa?
- Anthu ena amalandila machenjezo angapo pankhaniyi, ndiye ndizotheka kuchepetsa chiwerengerocho?
Mafunso awa akuwonetsa malingaliro ofunikira pazidziwitso ndi machenjezo:
- Nthawi zonse chenjezo likabwera, ndimayenera kuchitapo kanthu mwachangu. Ndikhoza kuthamanga kangapo patsiku ndisanatope.
- Chenjezo lililonse liyenera kukhala laposachedwa.
- Yankho lililonse ku chenjezo likufunika kulowererapo kwa anthu. Ngati zidziwitso zitha kukonzedwa zokha, siziyenera kubwera.
- Zidziwitso ziyenera kukhala zokhudzana ndi nkhani yatsopano kapena chochitika chomwe sichinachitikepo.
Njirayi imasokoneza kusiyanitsa kwina: ngati chenjezo likukwaniritsa zinthu zinayi zam'mbuyomu, zilibe kanthu ngati chenjezo litumizidwa kuchokera ku White-box monitoring system kapena Black-Box. Njira imeneyi imalimbitsanso kusiyana kwina: ndi bwino kugwiritsa ntchito khama lalikulu pozindikira zizindikiro kusiyana ndi zomwe zimayambitsa; zikafika pazifukwa, muyenera kungodandaula ndi zomwe sizingalephereke.
Kuwunika kwanthawi yayitali
M'malo opangira masiku ano, makina owunikira amawunikira njira yopangira yomwe ikusintha nthawi zonse ndikusintha kamangidwe ka mapulogalamu, mawonekedwe olemetsa, ndi zomwe akufuna kuchita. Zidziwitso, zomwe pakali pano ndizovuta kuzipanga zokha, zitha kukhala zofala, mwinanso zoyenera kuthandizidwa. Panthawiyi, wina ayenera kupeza ndi kukonza zomwe zimayambitsa vutoli; ngati kusamvana koteroko sikungatheke, kuyankhidwa kwa chenjezo kumafuna makina athunthu.
Ndikofunikira kuti zisankho zowunikira zipangidwe ndi zolinga zanthawi yayitali. Chenjezo lililonse lomwe likuyenda lero limatenga munthu kuti asamayende bwino mawa, kotero kuti nthawi zambiri pamakhala kuchepa kwa kupezeka kapena kugwira ntchito kwa njira yopangira nthawi yomwe imatengera kukonza njira yowunikira pakapita nthawi. Tiyeni tione zitsanzo ziwiri zosonyeza zimenezi.
Bigtable SRE: Nkhani yochenjeza kwambiri
Zomangamanga zamkati za Google nthawi zambiri zimaperekedwa ndikuyesedwa ndi Service Level (SLO). Zaka zapitazo, SLO ya ntchito ya Bigtable idakhazikitsidwa pakuchita pafupifupi kachitidwe kopanga kofanana ndi kasitomala wothamanga. Chifukwa cha zovuta za Bigtable komanso zotsika zosungirako, magwiridwe antchito ambiri adayendetsedwa ndi mchira "wamkulu": mafunso oyipa kwambiri 5% nthawi zambiri amachedwa kwambiri kuposa ena onse.
Zidziwitso za imelo zidatumizidwa pomwe malire a SLO adayandikira, ndipo zidziwitso za amithenga zidatumizidwa pomwe SLO idapitilira. Zidziwitso zamitundu yonse ziwirizi zidatumizidwa pafupipafupi, kuwononga nthawi yosavomerezeka yaukadaulo: gululo lidawononga nthawi yochulukirapo kusanthula zidziwitsozo kuti lipeze ochepa omwe anali ofunikira. Nthawi zambiri tinkaphonya nkhani yomwe imakhudza ogwiritsa ntchito chifukwa zidziwitso zochepa chabe zinali za vuto lomwelo. Zidziwitso zambiri sizinali zachangu chifukwa cha zovuta zomveka za zomangamanga ndipo zidasamalidwa mwanjira yokhazikika, kapena osasamalidwa konse.
Pofuna kuthetsa vutoli, gululi linagwiritsa ntchito njira ya mbali zitatu: pamene tikugwira ntchito mwakhama kuti Bigtable igwire bwino ntchito, tinakhazikitsa kwakanthawi 75th percentile kuti tichedwe kuyankha mafunso monga chandamale chathu cha SLO. Tidazimitsanso machenjezo a imelo, popeza analipo ambiri mwakuti zinali zosatheka kuwononga nthawi kuwazindikira.
Njira imeneyi inatithandiza kuti tiyambepo kukonza zinthu zanthawi yaitali mu Bigtable ndi zigawo zapansi za nkhokwe zosungirako, m'malo momangokhalira kukonza zovuta. Mainjiniya amatha kugwira ntchitoyo pomwe sanakhale ndi zidziwitso nthawi zonse. Pamapeto pake, kuchedwa kwakanthawi pakukonza zidziwitso kunatilola kupititsa patsogolo ntchito yabwino.
Gmail: Zoneneratu, Mayankho a Algorithmic Anthu
Pachiyambi, Gmail idamangidwa pamayendedwe osinthidwa a Workqueue process yomwe idapangidwa kuti igwirizane ndi magawo akusaka. Mndandanda wa ntchito udasinthidwa kuti ukhale wanthawi yayitali ndipo kenako udagwiritsidwa ntchito ku Gmail, koma zolakwika zina mu code opaque scheduler zidakhala zovuta kukonza.
Panthawiyo, kuyang'anira kwa Gmail kudapangidwa kuti zidziwitso ziwotche ntchito iliyonse ikathetsedwa pogwiritsa ntchito Workqueue. Njira imeneyi sinali yabwino, chifukwa ngakhale panthawiyo Gmail inkagwira ntchito masauzande ambiri, iliyonse yomwe inaperekedwa ku tizigawo ta peresenti ya ogwiritsa ntchito. Tidasamala kwambiri kuwonetsetsa kuti ogwiritsa ntchito a Gmail azitha kugwiritsa ntchito bwino, koma kugwiritsa ntchito zidziwitso zambiri sikunali kofunikira.
Kuti athane ndi vutoli, Gmail SRE idapanga chida chothandizira kuthetsa vutoli momwe mungathere kuti muchepetse kukhudzidwa kwa ogwiritsa ntchito. Gululi linali ndi zokambirana zingapo zokhuza kungosintha kuzungulira konseko kuti apeze vuto mpaka kulikonza mpaka yankho lanthawi yayitali litapezeka, koma ena anali ndi nkhawa kuti yankho lotere lingachedwetse kukonza kwenikweni kwa vutolo.
Kusamvana kotereku kunali kofala m'gulu la timu ndipo nthawi zambiri kunkawonetsa kusadzidalira: pamene mamembala ena a gulu akufuna kupereka nthawi yokonzekera bwino, ena amadandaula kuti kukonza komaliza kudzaiwalika ndipo kukonza kwakanthawi kumatenga nthawi zonse. Vutoli liyenera kuyang'aniridwa, chifukwa ndikosavuta kukonza kwakanthawi, m'malo mokonza zokhazikika. Oyang'anira ndi ogwira ntchito zaluso amagwira ntchito yofunika kwambiri pakukhazikitsa kokhazikika kwa nthawi yayitali pothandizira ndikuyika patsogolo zomwe zingakhale za nthawi yayitali ngakhale kupweteka koyambirira kutha.
Zidziwitso zobwerezabwereza ndi machitidwe a algorithmic ayenera kukhala mbendera yofiira. Kukayika kwa gulu lanu kupanga zidziwitso izi zikutanthauza kuti gululi lilibe chidaliro kuti lingakhulupirire ma algorithms. Ili ndi vuto lalikulu lomwe liyenera kuthetsedwa.
Nthawi yayitali
Mutu wamba umagwirizanitsa zitsanzo za Bigtable ndi Gmail: mpikisano pakati pa kupezeka kwakanthawi kochepa komanso kwakanthawi. Nthawi zambiri kuyesetsa mwamphamvu kungathandize kuti dongosolo losalimba lipezeke kwambiri, koma njira iyi nthawi zambiri imakhala yochepa, yodzala ndi kupsinjika kwamagulu ndi kudalira anthu ochepa a gulu lomwelo la ngwazi.
Kutsika kolamuliridwa, kwakanthawi kochepa komwe kumakhalapo nthawi zambiri kumakhala kowawa, koma ndikofunikira kwambiri kuti dongosololi likhale lokhazikika. Ndikofunika kuti tisamaganizire chenjezo lililonse payekhapayekha, koma kulingalira ngati kuchuluka kwa zidziwitso kumatsogolera ku dongosolo lathanzi, lofikirika bwino lomwe lili ndi gulu lokhazikika komanso chiyembekezo chabwino. Timasanthula ziwerengero za chenjezo (nthawi zambiri zimawonetsedwa mosinthana, pomwe chochitika chikhoza kukhala ndi zochitika zingapo zokhudzana ndi zochitika) m'malipoti a kotala ndi oyang'anira, zomwe zimalola opanga zisankho kuti azipereka mosalekeza kuchuluka kwa zidziwitso ndi thanzi la gulu lonse.
Pomaliza
Njira yoyang'anira thanzi ndi zidziwitso ndizosavuta komanso zolunjika. Imayang'ana kwambiri pazizindikiro za vuto lomwe machenjezo amapangidwira, ndipo kuyang'anira chomwe chimayambitsa kumathandizira kuthetsa mavuto. Kuyang'anira zizindikiro ndikosavuta mukakhala pamwamba pazomwe mukuwongolera, ngakhale kuwunika kwa database ndi kuyang'anira magwiridwe antchito kuyenera kuchitidwa mwachindunji pankhokwe. Zidziwitso zamaimelo ndizochepa kwambiri ndipo zimakonda kukwera mpaka phokoso mosavuta; m'malo mwake, muyenera kugwiritsa ntchito dashboard yomwe imayang'anira zonse zomwe zikuchitika zomwe zimadziwitsidwa ndi imelo. Dashboard ikhozanso kuphatikizidwa ndi chipika cha zochitika kuti mufufuze zokhudzana ndi mbiri yakale.
M'kupita kwa nthawi, kusinthana bwino pakati pa zidziwitso za zizindikiro ndi mavuto enieni omwe ayandikira kuyenera kukwaniritsidwa, ndipo zolinga zisinthidwe kuti zitsimikizidwe kuti kuwunika kumathandizira kuzindikira msanga.
Zikomo powerenga zomasulira mpaka kumapeto. Lembetsani ku njira yanga ya telegalamu yokhudza kuyang'anira ΠΈ .
Source: www.habr.com