SRE (Saiti Kuvimbika Injiniya) inzira yekuona kuwanikwa kwemapurojekiti ewebhu. Iyo inoonekwa sehurongwa hweDevOps uye inotaura nezve maitiro ekubudirira mukushandisa DevOps maitiro. Dudziro muchinyorwa chino mabhuku kubva kuGoogle. Ndakagadzirira shanduro iyi pachangu uye ndinovimba neruzivo rwangu pakunzwisisa maitiro ekuongorora. Muchiteshi cheteregiramu ΠΈ Ndakaburitsawo chinongedzo cheshanduro yeChitsauko 4 chebhuku rimwechete pamusoro pezvinangwa zvedanho rebasa.
Dudziro nekatsi. Nakidzwa nekuverenga!
Zvikwata zveGoogle zveSRE zvine misimboti uye maitiro akanakisa ekugadzira akabudirira ekutarisa uye ekuzivisa masisitimu. Ichi chitsauko chinopa gwara rekuti ndeapi matambudziko angasangana nemuenzi wepeji uye nekugadzirisa matambudziko anoita kuti mapeji ewebhu anetse kuratidza.
Tsanangudzo
Hapana izwi rimwe chete rinoshandiswa kukurukura nyaya dzine chekuita nekutarisa. Kunyangwe paGoogle, mazwi ari pasi apa haawanzo kushandiswa, asi isu tichanyora kududzira kwakajairika.
Kuongorora
Kuunganidza, kugadzirisa, kuunganidza uye kuratidzwa kwehuwandu hwe data munguva chaiyo pamusoro pehurongwa: nhamba yezvikumbiro uye marudzi ezvikumbiro, nhamba yezvikanganiso nemhando dzezvikanganiso, chikumbiro chekugadzirisa nguva uye server uptime.
White-box monitoring
Monitoring yakavakirwa pamametrics anoratidzwa nemukati masisitimu zvikamu, zvinosanganisira matanda, Java Virtual Machine profiling metrics, kana HTTP handler metrics inoburitsa manhamba emukati.
Black-box monitoring
Kuedza maitiro ekushandisa kubva pamaonero emushandisi.
Dashboard
Iyo interface (kazhinji webhu) inopa tarisiro yeakakosha ehutano zviratidzo zvemasevhisi. Dashboard inogona kunge iine mafirita, kukwanisa kusarudza zviratidzo zvinoratidzwa, nezvimwewo. Chiratidziro chakagadzirirwa kuziva zviratidzo zvinonyanya kukosha kune vashandisi. Iyo dashibhodhi inogona zvakare kuratidza ruzivo rwevashandi vekutsigira tekinoroji: mutsara wezvikumbiro, runyorwa rwemhosho dzepamusoro-soro, uye injinjiniya yakapihwa yenzvimbo yakapihwa basa.
Chenjedzo (chiziviso)
Zviziviso zvinotarisirwa kugamuchirwa nemunhu kuburikidza neemail kana imwe nzira, iyo inogona kukonzerwa nekukanganisa kana kuwedzera kwemutsara wekukumbira. Zviziviso zvinoiswa se: matikiti, email ziviso uye meseji meseji.
Root cause
Kukanganisa kwesoftware kana kukanganisa kwevanhu uko, kana kwagadziriswa, hakufanirwe kuitika zvakare. Dambudziko rinogona kunge riine zvakati wandei zvinokonzeresa: kusakwana maitiro otomatiki, kukanganisa kwesoftware, kusakwana kutsanangudza kwemaitiro ekushandisa. Chimwe nechimwe chezvikonzero izvi chinogona kunge chiri chikonzero, uye chimwe nechimwe chazvo chinofanira kubviswa.
Node uye muchina (node ββuye muchina)
Mazwi anochinjika kureva chiitiko chimwe chete chekushanda pasevha yemuviri, muchina chaiwo, kana mudziyo. Muchina mumwe unogona kubata masevhisi akati wandei. Masevhisi anogona kuva:
- yakabatana kune mumwe nemumwe: semuenzaniso, caching server uye web server;
- masevhisi asina hukama pane imwe chidimbu chehardware: semuenzaniso, kodhi repository uye wizard yegadziriro yekugadzirisa, senge. kana .
kusunda
Chero shanduko mukugadziriswa kwesoftware.
Nei kuongorora kuchidiwa?
Pane zvikonzero zvakati nei zvikumbiro zvichifanira kutariswa:
Kuongororwa kwemaitiro enguva refu
Iyo dhatabhesi yakakura sei uye inokurumidza kukura sei? Nhamba yezuva nezuva yevashandisi inoshanduka sei?
Kuenzanisa kwekuita
Zvikumbiro zvinokurumidza paAcme Bucket yeBytes 2.72 zvichienzaniswa neAjax DB 3.14? Zviri nani sei zvikumbiro zvakachengetwa mushure mekuonekwa kweimwe node? Iyo saiti irikumhanya zvishoma kana ichienzaniswa nesvondo rapfuura?
Yambiro (zviziviso)
Pane chatyoka uye munhu anoda kuchigadzirisa. Kana kuti chimwe chinhu chichaputsika munguva pfupi uye mumwe munhu anofanira kuchitarisa munguva pfupi.
Kugadzira dashboards
Dashboards inofanira kupindura mibvunzo yakakosha uye inosanganisira chimwe chinhu kubva - kunonoka (latency), traffic (traffic), zvikanganiso (zvikanganiso) uye saizi yemutoro (saturation).
Kuita ongororo yekuongorora (debugging)
Iko kunonoka kugadzirisa chikumbiro kwawedzera, asi chii chimwe chakaitika panguva imwe chete?
Monitoring masisitimu anobatsira sesosi yedata yebhizinesi rehungwaru masisitimu uye kufambisa kuongororwa kwezviitiko zvekuchengetedza. Nekuti bhuku rino rinotarisa munzvimbo dzeinjiniya umo maSRE ane hunyanzvi, isu hatizokurukura maitiro ekutarisa pano.
Kuongorora uye zviziviso zvinobvumira sisitimu kuti ikuudze kana yaparara kana kuti yava kuda kuparara. Kana sisitimu isingakwanise kuzvigadzirisa pachayo, tinoda kuti munhu aongorore yambiro, aone kana dambudziko richiri kushanda, rigadzirise, uye kuona zvikonzero. Kana ukasaongorora masisitimu ezvikamu, haufe wakagashira yambiro nekuti "chimwe chinhu chinoita sechinoshamisa."
Kuremedza munhu nezviziviso kushandiswa kwakadhura kwenguva yemushandi. Kana mushandi ari kushanda, nyevero inokanganisa mashandiro ebasa. Kana mushandi ari kumba, yambiro yacho inovhiringa nguva yake uye pamwe nekurara. Kana chenjedzo dzichiitika kakawanda, vashandi vanodziongorora, kudziisa kure, kana kufuratira chenjedzo dzinouya. Nguva nenguva vanofuratira nyevero chaiyoiyo, iyo inofukidzwa nezvinoitika zvoruzha. Kuvhiringwa kwesevhisi kunogona kugara kwenguva yakareba sezvo zviitiko zveruzha zvinodzivirira dambudziko kubva nekukurumidza kuongororwa uye kugadziriswa. Masisitimu ekunyevera anobudirira ane chiyero chakanaka chechiratidzo-ku-ruzha.
Kuisa zvinotarisirwa zvinotarisirwa kune iyo monitoring system
Kumisikidza yekutarisa kune yakaoma application ibasa rakaoma reinjiniya pacharo. Kunyangwe iine hurongwa hwakakosha hwekuunganidza, kuratidza, uye maturusi ekuzivisa, timu yeGoogle SRE yenhengo gumi nembiri inowanzo sanganisira munhu mumwechete kana vaviri vane chinangwa chikuru kuvaka nekuchengetedza masisitimu ekutarisa. Huwandu uhu hwadzikira nekufamba kwenguva sezvo isu tichibatanidza nekuisa pakati pezvivakwa zvekutarisa, asi yega yega SRE timu kazhinji ine munhu mumwe akazvipira chete kutarisa. Isu tinofanirwa kutaura kuti nepo monitoring system dashboards inonakidza kutarisa, SRE zvikwata zvakanyatso dzivirira mamiriro anoda kuti mumwe munhu atarise skrini kuti atarise matambudziko.
Pakazere, Google yatamira kune akareruka uye nekukurumidza ekutarisa masisitimu ane akakwana emashure-iyo-chokwadi ekuongorora maturusi. Isu tinodzivirira "mashiripiti" masisitimu anoedza kufanotaura zvikumbaridzo kana kuona otomatiki chikonzero. Masensor anoona zvisingatarisirwi muzvikumbiro zvemushandisi wekupedzisira ndiyo chete inopikisa; Chero bedzi ma sensor aya achiramba ari nyore, anogona kukurumidza kuona zvikonzero zvekukanganisa kwakakomba. Mamwe mafomati ekushandisa data rekutarisa, senge kuronga kugona kana kufanotaura kwetraffic, zvakanyanya kuoma. Kucherekedza kwenguva yakareba kwazvo (mwedzi kana makore) pamwero wakaderera wesampling (maawa kana mazuva) kucharatidza maitiro enguva refu.
Chikwata cheGoogle SRE chave nebudiriro yakasanganiswa neyakaomarara yekutsamira hierarchies. Isu hatiwanzo shandisa mitemo senge "kana ndikaona kuti dhatabhesi iri kunonoka, ndinowana yambiro yekuti dhatabhesi iri kunonoka, zvikasadaro ndinowana yambiro yekuti saiti iri kunonoka." Mitemo yekutsamira-yakavakirwa kazhinji inoreva zvikamu zvisingachinji zvehurongwa hwedu, senge sisitimu yekusefa traffic yevashandisi kunzvimbo yedata. Semuyenzaniso, "kana traffic kusefa kuenda kunzvimbo yedata yakagadziriswa, usandichenjedza nezve kunonoka kugadzirisa zvikumbiro zvevashandisi" ndiwo mutemo umwechete wekuzivisa kubva kunzvimbo yedata. Zvikwata zvishoma paGoogle zvinotsigira hurongwa hwekutsamira nekuti zvivakwa zvedu zvine mwero wekugara uchigadzirisa.
Mamwe emafungiro anotsanangurwa muchitsauko chino achiri akakosha: pane nguva dzose mukana wekufamba nekukurumidza kubva pachiratidzo kuenda kune midzi, kunyanya mukuchinja nguva dzose masisitimu. Naizvozvo, nepo chitsauko chino chichidonongodza zvimwe zvinangwa zvehurongwa hwekutarisa uye nzira dzekuzadzisa zvinangwa izvozvo, zvakakosha kuti masisitimu ekutarisa ari nyore uye anonzwisisika kumunhu wese muchikwata.
Saizvozvovo, kuchengetedza huwandu hweruzha hwakaderera uye mazinga echiratidzo akakwira, nzira dzekutarisa zvinhu zvakaziviswa dzinofanira kunge dziri nyore uye dzakavimbika. Mitemo inoburitsa yambiro kuvanhu inofanirwa kuve nyore kunzwisisa uye kupa dambudziko rakajeka.
Zviratidzo vs zvikonzero
Yako yekutarisa sisitimu inofanirwa kupindura mibvunzo miviri: "chii chakaputsika" uye "nei chaputsika."
"Chii chakaputsika" chinotaura nezvechiratidzo, uye "nei chakaputsika" inotaura nezvechikonzero. Tafura iri pasi apa inoratidza mienzaniso yekubatana kwakadaro.
Symptom
Chikonzero
Kuwana HTTP Kanganiso 500 kana 404
Database servers anoramba kubatana
Slow server mhinduro
Yakakwira CPU kushandiswa kana yakakuvadzwa Ethernet tambo
Vashandisi muAntarctica havasi kugamuchira katsi maGIF
CDN yako inovenga masayendisiti nekatsi, saka dzimwe IP kero dzakaguma dzave kunyorwa
Zvakavanzika zvave kuwanikwa kubva kwese kwese
Kuburitswa kwesoftware kuburitswa kwakaita kuti firewall ikanganwe ese maACL uye regedza munhu wese apinde
"Chii" uye "nei" ndezvimwe zvezvivharo zvakakosha zvekugadzira yakanaka yekutarisisa sisitimu ine chiratidzo chepamusoro uye ruzha rushoma.
Black-bhokisi vs White-bhokisi
Isu tinosanganisa yakawedzera White-bhokisi yekutarisa ine mwero yeBlack-bhokisi yekutarisa kune akakosha metrics. Nzira iri nyore yekuenzanisa Black-bhokisi neWhite-bhokisi ndeyokuti Bhokisi-dema rinotarisa-chiratidzo uye rinoita basa kwete kutarisisa: "sisitimu yacho haisi kushanda nemazvo izvozvi." White-bhokisi zvinoenderana nemukati wekusimbisa masimba emasisitimu: matanda ezviitiko kana maseva ewebhu. Saka, White-box inokubvumira kuti uone matambudziko ari kuuya, zvikanganiso zvinoratidzika sekutumirazve chikumbiro, nezvimwe.
Ziva kuti mune akawanda-layer system, chiratidzo munzvimbo yeinjiniya yebasa chiratidzo mune imwe nzvimbo yeinjiniya yebasa. Semuenzaniso, kuita kwedatabase kwakadzikira. Inononoka dhatabhesi inoverengwa chiratidzo cheiyo SRE dhatabhesi inovaona. Nekudaro, kune yekumberi-yekupedzisira SRE ichicherekedza inononoka webhusaiti, chikonzero cheiyo inononoka dhatabhesi kuverenga ndeye inononoka dhatabhesi. Naizvozvo, chena-bhokisi rekutarisa dzimwe nguva chiratidzo-chakatarisana uye dzimwe nguva chinokonzera-chakatarisana, zvichienderana nekuti yakakura sei.
Paunenge uchiunganidza telemetry yekugadzirisa, White-box yekutarisa inodiwa. Kana maseva ewebhu achinonoka kupindura kumibvunzo yedatabase, unofanirwa kuziva kuti inokurumidza sei webhu server inotaurirana nedatabase uye kuti inokurumidza kupindura sei. Zvikasadaro, haugone kusiyanisa pakati peinononoka dhatabhesi server uye netiweki dambudziko pakati pewebhu server uye dhatabhesi.
Black-bhokisi yekutarisa ine yakakosha mukana kana uchitumira chenjedzo: iwe unokonzeresa ziviso kune anogamuchira kana dambudziko ratove nezviratidzo zvechokwadi. Kune rumwe rutivi, kutarisa hakuna maturo kune dambudziko reBlack-box iro risati ramuka asi rava pedyo.
Zviratidzo zvina zvegoridhe
Iwo mana masaini egoridhe ekutarisa ari latency, traffic, kukanganisa, uye saturation. Kana iwe uchingokwanisa kuyera mana mushandisi system metrics, tarisa pane iwo mana.
Kunonoka
Inguva inodiwa kugadzirisa chikumbiro. Zvakakosha kusiyanisa pakati pe latency yezvikumbiro zvakabudirira uye zvisingabudiriri. Semuenzaniso, kukanganisa kweHTTP 500 kunokonzerwa nekurasikirwa kwekubatanidza kune database kana imwe backend inogona kuonekwa nokukurumidza, zvisinei, kukanganisa kweHTTP 500 kunogona kuratidza chikumbiro chakakundikana. Kuona kukanganisa kwe500 kukanganisa pane yakazara latency kunogona kutungamirira kumhedzisiro isiriyo. Kune rumwe rutivi, kukanganisa kunononoka kunyange kukanganisa kwekukurumidza! Naizvozvo, zvakakosha kutarisa kukanganisa latency pane kungosefa kunze zvikanganiso.
Traffic
Huwandu hwezvikumbiro kuhurongwa hwako hunoyerwa nepamusoro-level system metrics. Kusevhisi yewebhu, kuyerwa uku kunowanzo miririra nhamba yezvikumbiro zveHTTP pasekondi imwe neimwe, yakakamurwa nemhando yezvikumbiro (semuenzaniso, static kana dynamic content). Kune yekuteerera yekufambisa sisitimu, chiyero ichi chinogona kutarisa kunetiweki I/O kumhanya kana huwandu hwezvikamu zvenguva imwe chete. Kune kiyi-value yekuchengetedza sisitimu, kuyerwa uku kunogona kuve kwekutengesa kana mibairo yekutsvaga pasekondi.
Kukanganisa
Uyu ndiwo mwero wezvikumbiro zvakundikana zviri pachena (eg HTTP 500), zviri pachena (eg HTTP 200 asi zvakasanganiswa nezvisirizvo) kana mutemo (semuenzaniso "Kana watora mhinduro musekondi imwe, chero sekondi imwe inokanganisa"). Kana macode emhinduro eHTTP asina kukwana kuratidza mamiriro ese ekutadza, sekondari (yemukati) maprotocol anogona kudikanwa kuti aone kutadza. Kutarisisa zvikumbiro zvisizvo zvakadaro zvinogona kunge zvisiri ruzivo, nepo kupera-kusvika-kumagumo system bvunzo kuchabatsira kuona kuti uri kugadzirisa zvisiri izvo.
Saturation
Iyo metric inoratidza kuti sevhisi yako inoshandiswa zvakanyanya sei. Ichi chiyero chekutarisa system chinoratidza zviwanikwa zvinonyanya kumanikidzwa (somuenzaniso, pane memory-constrained system, inoratidza chiyeuchidzo, pane I/O-constrained system, inoratidza nhamba yeI/Os). Ziva kuti masisitimu mazhinji anodzikisira mashandiro asati asvika 100% kushandiswa, saka kuve nechinangwa chekushandisa kwakakosha.
Mune masisitimu akaomarara, saturation inogona kuwedzeredzwa neyepamusoro-nhanho kuyerwa kuyerwa: sevhisi yako ingaite nemazvo kubata kaviri traffic, kubata chete 10% yakawanda traffic, kana kubata kunyange kushoma traffic pane yazvinoita? Kune masevhisi akareruka asina maparamendi anoshandura kuoma kwechikumbiro (semuenzaniso, "Musandipe chero chinhu" kana "Ini ndinoda yakasarudzika monotonic integer"), isingawanzo shandura gadziriso, static mutoro bvunzo kukosha kunogona kukwana. Nekudaro, sezvakakurukurwa mundima yapfuura, masevhisi mazhinji anofanirwa kushandisa masaini asina kunanga, akadai seCPU kushandiswa kana network bandwidth, ine inozivikanwa yepamusoro yakasungwa. Kuwedzera latency kazhinji chiratidzo chinotungamira chekuzara. Kuyera iyo 99th percentile yekupindura nguva muhwindo diki (semuenzaniso, miniti imwe) inogona kupa chiratidzo chekutanga chekuzara.
Chekupedzisira, saturation inobatanidzwawo nekufungidzira nezvekuuya kwekuzara, semuenzaniso: "Zvinoita sekunge dhatabhesi rako richazadza hard drive yako mumaawa mana."
Kana iwe ukayera ese mana masaini egoridhe uye kana paine dambudziko neimwe yemetrics (kana, kana iri saturation, dambudziko riri pedyo), iwe unonyevera munhu, sevhisi yako ichave yakanyanya kuvharwa nekutarisa.
Kunetseka nezve "muswe" (kana chiridzwa uye kuita)
Paunenge uchigadzira yekutarisa sisitimu kubva kutanga, pane muedzo wekugadzira sisitimu yakavakirwa paavhareji kukosha: avhareji latency, avhareji CPU kushandiswa kwemanodhi, kana avhareji kuzara kwedhatabhesi. Ngozi yemienzaniso miviri yekupedzisira iri pachena: processors uye dhatabhesi zvinoraswa nenzira isingatarisike. Izvi zvinoshandawo pakunonoka. Kana iwe uchimhanyisa sevhisi yewebhu ine avhareji latency ye100ms nezvikumbiro 1000 pasekondi, 1% yezvikumbiro inogona kutora masekonzi mashanu. Kana vashandisi vachivimba neakawanda akadai masevhisi ewebhu, iyo 5th percentile yeimwe backend inogona nyore kuve iyo yepakati mhinduro nguva yekumberi.
Nzira iri nyore yekusiyanisa pakati peavhareji inononoka uye inononoka muswe wezvikumbiro ndeyekuunganidza zviyero zvezvikumbiro zvinoratidzwa muhuwandu (chishandiso chakanaka chekuratidza histograms) pane kunonoka chaiko: zvingani zvikumbiro zvakashandiswa sevhisi zvakatora pakati pe0 ms. uye 10 ms, pakati pe 10 ms ne 30 ms, pakati pe 30 ms ne 100 ms, pakati pe 100 ms ne 300 ms, zvichingodaro. Kuwedzera miganhu yehistogram inenge exponentially (ne fungidziro ye 3) kazhinji inzira iri nyore yekuona kugoverwa. yezvikumbiro.
Kusarudza nhanho yakakodzera yeruzivo rwezviyero
Zvinhu zvakasiyana zvehurongwa zvinofanirwa kuyerwa pamatanho akasiyana eruzivo. Semuyenzaniso:
- Kuongorora kushandiswa kweCPU kwenguva yakati hakuzoratidze-refu-refu spikes inotungamira kune yakakwira latencies.
- Kune rimwe divi, yewebhu sevhisi inonangana isingapfuure 9 maawa ekudzika pasi pagore (99,9% gore negore uptime), kutarisa mhinduro yeHTTP 200 kanopfuura kamwe kana kaviri paminiti ingangove isingaite.
- Saizvozvo, kutarisa hard drive nzvimbo ye99,9% kuwanikwa kanopfuura kamwe chete maminetsi 1-2 pamwe hazvidiwi.
Chengetedza magadzirirwo aunoita granularity yezviyero zvako. Kuunganidza CPU mutoro kamwechete pasekondi kunogona kupa inonakidza data, asi zvakadaro zviyero zvinowanzoitika zvinogona kudhura zvakanyanya kuunganidza, kuchengeta, uye kuongorora. Kana chinangwa chako chekutarisa chichida granularity yakakwira uye isingade kupindurwa kwepamusoro, unogona kudzikisa mitengo iyi nekumisikidza metrics kuunganidzwa paseva uye wozomisa sisitimu yekunze yekuunganidza nekuunganidza iwo metrics. Unogona here:
- Kuyera CPU mutoro sekondi yega yega.
- Deredza ruzivo kusvika pa5%.
- Aggregate metrics miniti yega yega.
Iri zano rinokutendera kuti utore data pane yakakwira granularity pasina kuunza kuongororwa kwepamusoro uye kuchengetedza pamusoro.
Sezviri nyore sezvinobvira, asi kwete nyore
Kufukidzwa kwezvinodiwa zvakasiyana pamusoro peumwe neumwe kunogona kukonzera yakaoma kwazvo yekutarisa system. Semuenzaniso, system yako inogona kunge iine zvinotevera zvinonetsa zvinhu:
- Yambiro zvinoenderana neakasiyana maburi ekukumbira kugadzirisa latency, mune akasiyana percentiles, kune ese marudzi ezviratidzo zvakasiyana.
- Kunyora kodhi yekuwedzera kuona uye kuona zvikonzero zvinogona kuitika.
- Gadzira madhibhodhi ane hukama kune imwe neimwe yezvinokonzeresa zvematambudziko.
Mabviro ezvinokonzeresa kunetsa haaperi. Kufanana nese masoftware masisitimu, kutarisa kunogona kuve kwakaomarara zvekuti kunove kusina kusimba uye kunetsa kuchinja nekuchengetedza.
Naizvozvo, gadzira yako yekutarisa sisitimu kuti iite nyore sezvinobvira. Paunenge uchisarudza zvekutevera, chengeta zvinotevera mupfungwa:
- Mitemo inowanzobata zviitiko chaizvo inofanira kunge iri nyore, inofanotaurwa uye yakavimbika sezvinobvira.
- Magadzirirwo ekuunganidza data, kuunganidza, uye yambiro iyo isingawanzoitwa (semuenzaniso, isingasviki kota kune zvimwe zvikwata zveSRE) inofanira kubviswa.
- Mametrics anounganidzwa asi asina kuratidzwa mune chero preview dashboard kana kushandiswa nechero yambiro ndevamiriri kuti vadzime.
PaGoogle, mametrics ekutanga kuunganidzwa uye kuunganidzwa, akasanganiswa nekuzivisa uye dashboards, inoshanda nemazvo seyakamira-yega sisitimu (Google's monitoring system yakakamurwa kuita akati wandei masystem, asi vanhu vanowanzo ziva nezvese maficha eaya ma subsystems). Zvinogona kunge zvichiyedza kusanganisa kutarisa nemamwe matekiniki ekuongorora akaomarara masisitimu: yakadzama sisitimu profiling, maitiro debugging, yekutevera ruzivo nezve kunze kana kutadza, kuyedzwa kwemutoro, kuunganidza uye kuongorora, kana kuongorora traffic. Nepo zvizhinji zvezvinhu izvi zvine zvakaenzana nekutarisisa kwekutanga, kuzvisanganisa kunozokonzera mhedzisiro yakawandisa uye kugadzira yakaoma uye isina kusimba sisitimu. Sezvimwe nezvimwe zvakawanda zvekuvandudzwa kwesoftware, kutsigira masisitimu akasiyana ane akajeka, akapusa, akasununguka akabatanidza mapoinzi ndiyo yakanakisa zano (semuenzaniso, kushandisa web API kutora yakaunganidzwa data mufomati inogona kuramba ichienderana kwenguva yakareba. )
Kubatanidza Nheyo
Misimboti inokurukurwa muchitsauko chino inogona kusanganiswa kuita huzivi hwekutarisa uye yekuzivisa iyo inotsigirwa nekuteverwa nezvikwata zveGoogle SRE. Kutevedzera uzivi hwekutarisa uhu hunodiwa, inzvimbo yakanaka yekutanga kugadzira kana kudzokorora nzira yako yekuzivisa, uye inogona kukubatsira kubvunza mibvunzo chaiyo yebasa rako rekuita, zvisinei nehukuru hwesangano rako kana kuoma kwesevhisi kana sisitimu.
Paunenge uchigadzira mitemo yekutarisa uye yekunyevera, kubvunza mibvunzo inotevera kunogona kukubatsira kudzivirira manyepo uye chenjedzo dzisina basa:
- Mutemo uyu unoona here imwe neimwe nzira isingaonekwe yehurongwa iyo yekukurumidzira, inodana kuchiito, uye inokanganisa mushandisi?
- Ndinogona here kufuratira yambiro iyi ndichiziva kuti haina kunaka? Ndirini uye nei ndinogona kufuratira yambiro iyi uye ndingadzivirira sei mamiriro aya?
- Yambiro iyi inoreva here kuti vashandisi vari kukanganiswa? Pane here mamiriro ezvinhu apo vashandisi vasingakanganisirwe zvakashata, senge kuburikidza nekusefa kwetraffic kana pakushandisa ma test system ayo chenjedzo dzinofanirwa kusefa?
- Ndinogona here kuita chimwe chinhu mukupindura yambiro iyi? Matanho aya ndeekukurumidza here kana kuti anogona kumirira kusvika mangwanani? Chiitiko chinogona kuitwa otomatiki zvakachengeteka? Chiito ichi chichava mhinduro yenguva refu here kana yenguva pfupi yekushanda?
- Vamwe vanhu vari kuwana chenjedzo dzakawanda nezvenyaya iyi, saka pane nzira yekudzikisa huwandu hwechenjedzo?
Iyi mibvunzo inoratidza huzivi hwakakosha pakuzivisa uye masisitimu yambiro:
- Pese panouya nyevero, ndinofanira kuita pakarepo. Ndinogona kuita chimwe chinhu nokukurumidza kakawanda pazuva ndisati ndaneta.
- Nyevero imwe neimwe inofanirwa kuve yakakosha.
- Yese mhinduro kune yambiro inofanirwa kuda kupindira kwevanhu. Kana iyo chiziviso ichigona kugadziriswa otomatiki, haifanirwe kusvika.
- Yambiro inofanira kunge iri pamusoro pedambudziko idzva kana chiitiko chakanga chisipo kare.
Iyi nzira inodzima mimwe misiyano: kana yambiro ichizadzisa mamiriro mana apfuura, hazvina basa kuti yambiro inotumirwa kubva kuWhite-box kana Black-Bhokisi yekutarisa system. Iyi nzira inosimbisawo mimwe misiyano: zviri nani kushandisa simba rakawanda pakuziva zviratidzo pane pazvikonzero; kana zvasvika kune zvikonzero, iwe unongoda kunetseka pamusoro pezvikonzero zvisingadzivisiki.
Kutarisa kwenguva refu
Munzvimbo dzanhasi dzechigadzirwa, masisitimu ekutarisa anotarisisa inogara ichishanduka yekugadzira sisitimu ine shanduko yekugadzira software, hunhu hwebasa, uye zvinangwa zvekuita. Nyevero dzakaoma parizvino kuita otomatiki dzinogona kuve dzakajairika, pamwe dzinotofanira kugadziriswa. Panguva ino, mumwe munhu anofanira kuwana uye kubvisa zvikonzero zvechinetso; kana kugadzirisa kwakadaro kusingagoneki, mhinduro kune yambiro inoda yakazara otomatiki.
Zvakakosha kuti sarudzo dzekutarisa dziitwe nezvinangwa zvenguva refu mupfungwa. Yese yambiro inomhanya nhasi inovhiringidza munhu kubva pakuvandudza sisitimu mangwana, saka kazhinji pane kudzikiswa kwekuwanikwa kana kuita kwegadziriro inobereka nguva inodiwa kuvandudza hurongwa hwekutarisa munguva refu. Ngatitarisei mienzaniso miviri kuenzanisira chiitiko ichi.
Bigtable SRE: Ngano Yekupfuura-Yambiro
Zvivakwa zvemukati zveGoogle zvinowanzo kupihwa uye kuyerwa zvichienderana nedanho rebasa (SLO). Makore mazhinji apfuura, Bigtable's sevhisi SLO yaive yakavakirwa paavhareji mashandiro ekugadzirwa kwekutengesa kutevedzera mutengi mupenyu. Nekuda kwenyaya muBigtable uye yakaderera mazinga enzvimbo yekuchengetera, avhareji kuita kwaifambiswa ne "muhombe" muswe: yakaipisisa 5% yemibvunzo yaiwanzo nonoka pane yese.
E-mail zviziviso zvakatumirwa sezvo chikumbaridzo cheSLO chakasvika, uye nyevero dzevatumwa dzakatumirwa apo SLO yakapfuudzwa. Mhando mbiri dzechenjedzo dzakatumirwa kazhinji, dzichipedza huwandu husingagamuchirwe hwenguva yeinjiniya: timu yakapedza nguva yakati wandei ichironga kuburikidza neziviso kuti iwane mashoma ainyatsokodzera. Isu taigara tichipotsa nyaya yainyatsobata vashandisi nekuti zvimwe chete zvechenjedzo zvaive zveiyo nyaya chaiyo. Mazhinji echenjedzo akange asina kukurumidza nekuda kwematambudziko anonzwisisika muzvivakwa uye akagadziriswa nenzira yakajairwa, kana kuti haana kugadziridzwa zvachose.
Kugadzirisa mamiriro ezvinhu, timu yakatora nzira ine-mativi matatu: Tichiri kushanda nesimba kuti tivandudze mashandiro eBigtable, takambomira chinangwa chedu cheSLO kuve 75th percentile yekunonoka kupindura kwemubvunzo. Takadzimawo chenjedzo dzemaemail nekuti dzaive dzakawanda zvekuti takatadza kupedza nguva tichidziongorora.
Iri zano rakaita kuti isu imba yekufema itange kugadzirisa nyaya dzenguva refu muBigtable uye yakaderera mazinga enzvimbo yekuchengetera, pane kugara tichigadzirisa nyaya dzehungwaru. Mainjiniya vaigona kuita kuti basa riitwe pasina kubhombwa nenyevero nguva dzese. Pakupedzisira, kumbomira kugadziridza chenjedzo kwakatibvumira kuvandudza kunaka kwesevhisi yedu.
Gmail: Inofanotaurwa, Algorithmic Mhinduro dzevanhu
Pakuvambwa kwayo, Gmail yakavakirwa pane yakagadziridzwa Workqueue process management system iyo yakagadzirirwa batch process chunks ye index yekutsvaga. Mutsara webasa wakachinjirwa kumaitiro ekugara kwenguva refu uye wozoshandiswa kuGmail, asi mamwe mabhugi muopaque scheduler kodhi akaratidza kuoma kwazvo kugadzirisa.
Panguva iyoyo, kutariswa kweGmail kwakagadzirwa kuitira kuti chenjedzo dzitange kana mabasa ega ega akadzimwa uchishandisa Workqueue. Iyi nzira yakanga isina kunaka, nokuti kunyange panguva iyoyo, Gmail yakaita zviuru zvemabasa, rimwe nerimwe rakapiwa chikamu chezana chevashandi vedu. Isu tainetsekana zvikuru nekupa vashandisi veGmail ruzivo rwakanaka rwemushandisi, asi kubata chenjedzo dzakawanda kwaisasvika.
Kugadzirisa nyaya iyi, Gmail SRE yakagadzira chishandiso kubatsira kugadzirisa marongero zvakanyanya sezvinobvira kuderedza kukanganiswa kwevashandisi. Chikwata chaive nenhaurirano dzekuti chingo shandura kutenderera kwese kubva mukuwanikwa kwedambudziko kuburikidza nekugadzirisa kusvika mhinduro yenguva refu yawanikwa, asi vamwe vainetsekana kuti mhinduro yakadai yaizononoka kugadzirisa dambudziko.
Kukakavadzana uku kwaiwanzoitika muchikwata uye kazhinji kwairatidza kusavimbika mukuzvidzora: nepo dzimwe nhengo dzechikwata dzichida kubvumira nguva yekugadzirisa kwakaringana, vamwe vanonetsekana kuti kugadzirisa kwekupedzisira kuchakanganwika uye kugadzirisa kwechinguvana kuchatora zvachose. Nyaya iyi inofanirwa kutariswa nekuti zviri nyore kugadzirisa matambudziko kwekanguva pane kuita kuti mamiriro acho ave zvachose. Mamaneja nevashandi vehunyanzvi vanoita basa rakakosha pakuita zvigadziriso zvenguva refu, kutsigira uye kuisa pamberi zvingangove zvenguva refu zvigadziriso kunyangwe mushure mekunge "marwadzo" ekutanga apera.
Nguva dzose, kudzokorora chenjedzo uye algorithmic mhinduro dzinofanira kunge dziri mureza mutsvuku. Kusada kwechikwata chako kugadzirisa zviziviso izvi zvinoreva kuti timu haina chivimbo chekuti inogona kuvimba nema algorithms. Iri idambudziko rakakura rinoda kugadziriswa.
Nguva refu
Dingindira rakajairika rinobatanidza mienzaniso yeBigtable neGmail: makwikwi pakati pekuwanikwa kwenguva pfupi uye kwenguva refu. Kazhinji, kuedza kwakasimba kunogona kubatsira hurongwa husina kusimba kuwana kuwanikwa kwepamusoro, asi nzira iyi inowanzogara kwenguva pfupi, yakazara nekuneta kwechikwata uye kuvimba nenhamba shoma yenhengo dzechikwata chimwechete chegamba.
Kudzorwa, kuderedzwa kwenguva pfupi kwekuwanikwa kunowanzorwadza, asi zvine hungwaru zvakakosha pakugadzikana kwenguva refu kwehurongwa. Zvakakosha kuti usatarise yambiro yega yega uri wega, asi kufunga kuti huwandu hwese hwevhoriyamu yekuchenjerera huchitungamira kune ine hutano, inosvikika system ine timu inoshanda uye fungidziro yakanaka. Isu tinoongorora huwandu hweyambiro yenguva (inowanzo kuratidzwa sezviitiko pakuchinja, apo chiitiko chinogona kunge chine zviitiko zvakawanda zvine hukama) mumishumo yekota kune maneja, zvichibvumira vanoita sarudzo kuti vave nekuona kuenderera mberi kweyambiro system mutoro uye hutano hwechikwata chese.
mhedziso
Nzira yekutarisa kune hutano uye yambiro iri nyore uye yakajeka. Inotarisa pane zviratidzo zvedambudziko rinokonzera chenjedzo, uye kutarisa chikonzero kunoshanda seyamuro yekugadzirisa matambudziko. Kuongorora zviratidzo zviri nyore kana iwe uri mustack iwe waunodzora, kunyangwe kutarisisa mutoro uye kuita kwedhatabhesi kunofanirwa kuitwa zvakananga pane dhatabhesi pachayo. Email zviziviso zvine mashoma anobatsira uye anowanzoita ruzha nyore; pachinzvimbo, iwe unofanirwa kushandisa dashboard inotarisisa zvese zvazvino izvo zvinokonzeresa email chenjedzo. Dashboard inogonawo kupetwa nerogi yezviitiko kuti uongorore nhoroondo inoenderana.
Nekufamba kwenguva, zvinodikanwa kuti uwane kutenderera kwakabudirira kwekuzivisa nezvezviratidzo uye matambudziko chaiwo ari pedyo, kugadzirisa zvinangwa kuve nechokwadi chekuti kuongorora kunotsigira kukurumidza kuongororwa.
Ndinokutendai nekuverenga shanduro kusvika kumagumo. Nyorera kune yangu telegraph chiteshi nezve kutarisa ΠΈ .
Source: www.habr.com