Monitoring yakafa here? - Kugara kwenguva refu kutarisa

Kubva 2008, kambani yedu yanga ichinyanya kuita manejimendi ezvivakwa uye kutenderera-the-wachi tsigiro yehunyanzvi yemapurojekiti ewebhu: isu tine vanopfuura mazana mana vatengi, inova 400% yeRussia e-commerce. Saizvozvowo, chivakwa chakasiyana-siyana chinotsigirwa. Kana chimwe chinhu chikawa, tinosungirwa kuchigadzirisa mukati memaminitsi gumi nemashanu. Asi kuti unzwisise kuti njodzi yakaitika, unofanirwa kutarisa purojekiti uye kupindura kune zviitiko. Kuita sei izvi?

Ndinotenda kuti pane dambudziko mukuronga hurongwa hwekutarisa hwakakodzera. Dai pakange pasina dambudziko, kutaura kwangu kwaizove neimwe thesis: "Ndokumbira isa Prometheus + Grafana nemaplugins 1, 2, 3." Zvinosuruvarisa, hazvichashandi nenzira iyoyo zvakare. Uye dambudziko guru nderekuti munhu wese anoramba achitenda mune chimwe chinhu chaivepo muna 2008, maererano nezvikamu zvesoftware.

Nezvekurongwa kwehurongwa hwekutarisa, ndingada kutaura kuti ... mapurojekiti ane hunyanzvi hwekutarisa haapo. Uye mamiriro acho akashata zvekuti kana chimwe chinhu chikadonha, pane njodzi yekuti hachizoonekwa - mushure mezvose, munhu wese ane chokwadi chekuti "zvese zvinotariswa."
Zvichida zvinhu zvose zviri kuongororwa. Asi sei?

Tese takasangana nenyaya yakaita seinotevera: imwe devops, mumwe admin ari kushanda, timu yebudiriro inouya kwavari ndokuti - "tasunungurwa, ikozvino tarisa." Monitor chii? Inoshanda sei?

OK. Isu tinotarisisa nzira yechinyakare. Uye iri kutochinja, uye zvinozoitika kuti iwe wakatarisisa sevhisi A, iyo yakava sevhisi B, iyo inopindirana nebasa C. Asi boka rekusimudzira rinokuudza: "Isa software, inofanirwa kutarisa zvese!"

Saka chii chachinja? - Zvinhu zvese zvachinja!

2008 Zvese zvakanaka

Kune vaviri vanogadzira, imwe sevha, imwe database server. Zvose zvinobva pano. Tine rumwe ruzivo, tinoisa zabbix, Nagios, cacti. Uye tobva taisa chenjedzo dzakajeka paCPU, pane dhisiki kushanda, uye pane dhisiki nzvimbo. Isu tinoitawo macheki akati wandei kuti tive nechokwadi chekuti saiti yapindura uye kuti maodha arikusvika mudhatabhesi. Uye ndizvozvo - isu takachengetedzwa zvakanyanya kana kushoma.

Kana tikaenzanisa huwandu hwebasa rakaitwa nemutungamiriri panguva iyoyo kuti ape kutarisa, ipapo 98% yaro yaingove otomatiki: munhu anotarisisa anofanirwa kunzwisisa nzira yekuisa Zabbix, maitiro ekugadzirisa nekugadzirisa zviziviso. Uye 2% - yekuongorora kwekunze: kuti saiti inopindura uye inoita chikumbiro kune database, kuti mirairo mitsva yasvika.

2010 Mutoro uri kukura

Tiri kutanga kuyera webhu, tichiwedzera injini yekutsvaga. Isu tinoda kuve nechokwadi chekuti chigadzirwa catalog chine zvese zvigadzirwa. Uye iyo yekutsvaga chigadzirwa inoshanda. Kuti dhatabhesi iri kushanda, kuti maodha ari kuitwa, kuti saiti inopindura kunze uye inopindura kubva kumaseva maviri uye mushandisi haana kudzingwa kunze kwesaiti ichidzoreredzwa kune imwe sevha, nezvimwe. Kune mamwe masangano.

Zvakare, mubatanidzwa wakabatana nezvivakwa zvinoramba uri mukuru mumusoro wemaneja. Mumusoro mangu muchine imwe pfungwa yekuti munhu arikuita monitoring ndiye achaisa zabbix agokwanisa kuigadzira.

Asi panguva imwecheteyo, basa rinoonekwa pakuita cheki dzekunze, pakugadzira seti yekutsvaga indexer query scripts, seti yezvinyorwa kutarisa kuti tsvakiridzo inoshanduka panguva ye indexing process, seti yezvinyorwa zvinotarisa kuti zvinhu zvinoendeswa kune delivery service, etc. zvichingoenda zvakadaro.

Cherechedza: Ndakanyora "seti yezvinyorwa" 3 nguva. Kureva kuti munhu ane basa rekutarisa haasisiri iye anongoisa zabbix. Uyu munhu anotanga kukodha. Asi hapana chakachinja mupfungwa dzechikwata ichi.

Asi nyika iri kuchinja, ichiwedzera kuoma. A virtualization layer uye akati wandei masystem anowedzerwa. Vanotanga kudyidzana. Ndiani akati "kunhuwa kunge microservices?" Asi sevhisi yega yega ichiri kutaridzika sewebhusaiti yega. Tinogona kutendeukira kwairi uye kunzwisisa kuti inopa ruzivo rwakakosha uye inoshanda pachayo. Uye kana iwe uri maneja achigara achibatanidzwa mupurojekiti yave ichikura kwemakore 5-7-10, ruzivo urwu runounganidza: nhanho nyowani inooneka - wakazviona, imwe nhanho inooneka - wakazviona ...

Asi kashoma kuti chero munhu aperekedze chirongwa kwemakore gumi.

Monitoringman's resume

Ngatiti iwe wakasvika kune imwe nyowani yekutanga iyo yakabva yahaya vanogadzira makumi maviri, vakanyora gumi neshanu microservices, uye iwe uri admin anoudzwa kuti: “Vaka CI/CD. Ndapota." Iwe wakavaka CI / CD uye kamwe kamwe unonzwa: "Zvakaoma kwatiri kuti tishande nekugadzirwa mu "cube", tisinganzwisisi kuti chikumbiro chichashanda sei mairi. Tigadzirire bhokisi rejecha mu "cube" imwechete.
Iwe unogadzira bhokisi rejecha mune iyi cube. Ivo vanobva vangokuudza kuti: "Tinoda dhatabhesi yedanho rinogadziridzwa zuva rega rega kubva mukugadzirwa, kuitira kuti tinzwisise kuti inoshanda pane dhatabhesi, asi panguva imwechete isingaparadze dhatabhesi rekugadzira."

Unogara mune zvese izvi. Kune mavhiki e2 asara asati asunungurwa, vanokuudza kuti: "Zvino ngatitarisei izvi zvose ..." Ndizvo. tarisa iyo cluster zvivakwa, tarisa iyo microservice architecture, tarisa basa nemasevhisi ekunze ...

Uye vandinoshanda navo vanotora chirongwa chenguva dzose kubva mumisoro yavo ndokuti: "Zvakanaka, zvese zviri pachena pano! Isa chirongwa chinoongorora zvese izvi. ” Hongu, hongu: Prometheus + Grafana + plugins.
Uye vanowedzera: "Une mavhiki maviri, ita shuwa kuti zvese zvakachengeteka."

Mumapurojekiti mazhinji atinoona, munhu mumwechete anopihwa ongororo. Fungidzira kuti tinoda kuhaya munhu kuti aite monitoring kwe2 weeks, tomunyorera resume. Munhu uyu anofanirwa kuve nehunyanzvi hupi, maererano nezvose zvatataura kusvika pari zvino?

• Anofanirwa kunzwisisa kutariswa uye zvakanangana nekushanda kwesimbi simbi.
• Anofanirwa kunzwisisa izvo zvekutarisa Kubernetes (uye munhu wese anoda kuenda ku "cube", nekuti iwe unogona kubvisa kubva pane zvese, kuhwanda, nekuti iyo admin ichabata nezvimwe) - pachayo, zvivakwa zvayo, uye kunzwisisa maitiro ekutarisa zvikumbiro. mukati.
• Anofanira kunzwisisa kuti masevhisi anotaurirana neimwe neimwe nzira dzakakosha, uye kuziva izvo chaizvo zvekuti masevhisi anodyidzana sei. Zvinokwanisika kuona purojekiti apo mamwe masevhisi anotaurirana synchronously, nekuti hapana imwe nzira. Semuenzaniso, iyo backend inoenda kuburikidza neREST, kuburikidza ne gRPC kune iyo catalog sevhisi, inogamuchira runyorwa rwezvigadzirwa uye inodzosera iyo kumashure. Iwe haugone kumirira pano. Uye nemamwe masevhisi anoshanda asynchronously. Chinja odha kune yekuendesa sevhisi, tumira tsamba, nezvimwe.
  Iwe zvichida wakatotuhwina kubva pane zvese izvi? Uye admin, anoda kutarisa izvi, akawedzera kuvhiringika.
• Anofanira kukwanisa kuronga uye kuronga nemazvo - sezvo basa richiwedzera.
• Saka anofanirwa kugadzira zano kubva kune rakagadzirwa sevhisi kuti anzwisise nzira yekunyatso tarisa. Anoda kunzwisiswa kwezvivakwa zvepurojekiti uye kuvandudzwa kwayo + kunzwisisa kwetekinoroji inoshandiswa mukusimudzira.

Ngatiyeuke nyaya yakajairika: mamwe masevhisi ari muPHP, mamwe masevhisi ari muGo, mamwe masevhisi ari muJS. Ivo neimwe nzira vanoshanda pamwe chete. Apa ndipo panobva izwi rekuti "microservice": kune akawanda masisitimu ega ega zvekuti vanogadzira havagone kunzwisisa chirongwa chose. Chimwe chikamu chechikwata chinonyora masevhisi muJS anoshanda ega uye asingazive kuti imwe yese system inoshanda sei. Chimwe chikamu chinonyora masevhisi muPython uye hachikanganise mashandiro anoita mamwe masevhisi; ivo vakaparadzaniswa munzvimbo yavo. Yechitatu ndeyekunyora masevhisi muPHP kana chimwe chinhu.
Vese ava vanhu makumi maviri vakakamurwa kuita masevhisi gumi nemashanu, uye pane admin mumwechete anofanira kunzwisisa zvese izvi. Stop! isu takangopatsanura sisitimu kuita gumi neshanu microservices nekuti vanhu makumi maviri havagone kunzwisisa iyo system yese.

Asi inoda kutariswa neimwe nzira ...

Chii chinoguma? Nekuda kweizvozvo, pane munhu mumwechete anouya nezvese izvo timu yese yevagadziri isinganzwisise, uye panguva imwecheteyo anofanirwa kuziva uye nekugona kuita zvatakaratidza pamusoro - hardware zvivakwa, Kubernetes zvivakwa, nezvimwe.

Chii chandingati ... Houston, tine matambudziko.

Kuongorora chirongwa chemazuva ano software ipurojekiti yesoftware pachayo

Kubva pakutenda kwenhema kuti kutarisa isoftware, tinokudziridza kutenda muzvishamiso. Asi zvishamiso, maiwe, hazviitike. Iwe haugone kuisa zabbix uye kutarisira kuti zvese zvishande. Iko hakuna chikonzero chekuisa Grafana uye uchitarisira kuti zvese zvichanaka. Yakawanda yenguva ichashandiswa pakuronga macheki ekushanda kwesevhisi uye kudyidzana kwavo nemumwe, kutarisa kuti masisitimu ekunze anoshanda sei. Muchokwadi, 90% yenguva ichashandiswa kwete pakunyora zvinyorwa, asi pakugadzira software. Uye inofanira kubatwa neboka rinonzwisisa basa reprojekti.
Kana mumamiriro ezvinhu aya munhu mumwe akakandwa mukutarisa, ipapo njodzi ichaitika. Zvinova zviri kuitika kwese kwese.

Semuenzaniso, kune akati wandei masevhisi anotaurirana nemumwe kuburikidza neKafka. Kurongeka kwakasvika, takatumira meseji nezve kurongeka kuKafka. Pane sevhisi inoteerera ruzivo nezve kurongeka uye nengarava zvinhu. Pane sevhisi inoteerera ruzivo nezve kurongeka uye inotumira tsamba kumushandisi. Zvino mamwe masevhisi anooneka, totanga kuvhiringika.

Uye kana iwe ukapawo izvi kune admin nevagadziri padanho kana paine nguva pfupi yasara isati yaburitswa, munhu wacho achada kunzwisisa iyi protocol yese. Avo. Chirongwa chechiyero ichi chinotora nguva yakakura, uye izvi zvinofanirwa kuverengerwa mukuvandudza sisitimu.
Asi kazhinji, kunyanya mukutanga, tinoona kuti kutarisa kunomiswa sei kusvika gare gare. “Zvino tichagadzira Proof of Concept, tichaparura nayo, ngaiwire pasi – tagadzirira kubaira. Uye isu tichazviongorora zvese. " Kana (kana kana) purojekiti yatanga kuita mari, bhizinesi rinoda kuwedzera zvimwe zvinhu - nekuti ratanga kushanda, saka inoda kufambiswa mberi! Uye iwe uri panguva yaunotanga kuda kutarisa zvese zvakapfuura, izvo zvinotora kwete 1% yenguva, asi zvimwe zvakawanda. Uye nenzira, vagadziri vanozodikanwa pakutarisa, uye zviri nyore kuvarega vashande pazvinhu zvitsva. Nekuda kweizvozvo, maficha matsva anonyorwa, zvese zvinokandwa kumusoro, uye iwe uri mukusagadzikana kusingaperi.

Saka sei kutarisa purojekiti kubva pakutanga, uye chii chaunofanira kuita kana iwe ukawana chirongwa chinoda kuongororwa, asi iwe usingazivi kuti unotanga kupi?

Kutanga, unofanira kuronga.

Lyrical digression: kazhinji kazhinji ivo vanotanga nekutarisa zvivakwa. Semuenzaniso, isu tine Kubernetes. Ngatitangei nekuisa Prometheus neGrafana, nekuisa plugins yekutarisa iyo "cube". Kwete chete vanogadzira, asiwo vatungamiriri vane tsika isina kunaka yekuti: "Tichaisa iyi plugin, asi plugin inogona kunge inoziva maitirwo." Vanhu vanofarira kutanga nezviri nyore uye zvakananga, pane nezviito zvakakosha. Uye kuongorora kwezvivakwa kuri nyore.

Kutanga, sarudza kuti chii uye sei chaunoda kutarisa, uye wozosarudza chishandiso, nekuti vamwe vanhu havagone kukufungira. Uye vanofanira here? Vamwe vanhu vakazvifunga ivo pachavo, nezve universal system - kana kusafunga zvachose pakanyorwa plugin iyi. Uye nekuda kwekuti iyi plugin ine 5 zviuru vashandisi hazvireve kuti ndeye chero kushandiswa. Pamwe iwe uchave wechi5001st nekuti kwaitova nevanhu zviuru mazana mashanu ipapo.

Kana iwe ukatanga kutarisa zvivakwa uye kuseri kwechishandiso chako kunomira kupindura, vese vashandisi vanorasikirwa nekubatana nenharembozha. Pane chikanganiso chichaonekwa. Vachauya kwauri vachiti "Chikumbiro hachisi kushanda, uri kuitei pano?" - "Tiri kutarisa." - "Unotarisisa sei kana usingaone kuti application haisi kushanda?!"

1. Ini ndinotenda kuti iwe unofanirwa kutanga kutarisa chaizvo kubva pakupinda kwemushandisi. Kana mushandisi asingaone kuti application iri kushanda, ndizvozvo, iko kukundikana. Uye iyo yekutarisa system inofanirwa kunyevera nezve izvi kutanga.
2. Uye chete ipapo tinogona kutarisa zvivakwa. Kana kuti zviite zvakafanana. Zviri nyore nezvivakwa - pano isu tinogona pakupedzisira kungoisa zabbix.
3. Uye ikozvino iwe unofanirwa kuenda kumidzi yekushandisa kuti unzwisise uko zvinhu zvisiri kushanda.

Pfungwa yangu huru ndeyekuti kuongorora kunofanirwa kufambirana nemaitiro ebudiriro. Kana iwe ukakanganisa timu yekutarisa kune mamwe mabasa (kugadzira CI / CD, sandboxing, kurongekazve kwezvivakwa), kutarisa kunotanga kuderera uye haungambofa wakabata nebudiriro (kana nekukurumidza kana gare gare unozoimisa).

Zvese nemalevel

Aya ndiwo maonero andinoona kurongeka kwehurongwa hwekutarisa.

1) Chishandiso level:

• kutarisa kushandiswa kwebhizimisi logic;
• kuongorora hutano hwemasevhisi;
• kusanganisa kutarisa.

2) Infrastructure level:

• orchestration level monitoring;
• system software monitoring;
• iron level monitoring.

3) Zvakare iyo nhanho yekushandisa - asi sechigadzirwa cheinjiniya:

• kuunganidza uye kutarisa matanda ekushandisa;
• APM;
• kutsvaga.

4) Kuzivisa:

• kurongeka kwehurongwa hwekunyevera;
• kurongeka kwehurongwa hwebasa;
• sangano re "ruzivo base" uye mafambiro ebasa rekugadzirisa zviitiko.

kosha: tinosvika pakuchenjerera kwete mushure, asi pakarepo! Iko hakuna chikonzero chekutangisa kutarisa uye "neimwe neimwe nzira gare gare" kuona kuti ndiani achagamuchira yambiro. Mushure mezvose, ibasa rei rekutarisa: kunzwisisa kuti pane chimwe chinhu chiri kushanda zvisina kunaka muhurongwa, uye kuita kuti vanhu vakarurama vazive nezvazvo. Kana iwe ukasiya izvi kusvika kumagumo, ipapo vanhu vakarurama vachaziva kuti chimwe chinhu chiri kutadza chete nekudaidza kuti "hapana chiri kutiitira."

Application Layer - Bhizinesi Logic Monitoring

Pano tiri kutaura nezve kutarisa chokwadi chekuti application inoshanda kumushandisi.

Iyi nhanho inofanira kuitwa panguva yekukura. Semuenzaniso, isu tine mamiriro Prometheus: inoenda kune sevha inoita macheki, inodhonza iyo yekupedzisira, uye iyo yekupedzisira inoenda uye inotarisa iyo API.

Kana inowanzokumbirwa kutarisa peji reimba kuti ive nechokwadi chekuti saiti iri kushanda, vanogadzira mapurogiramu vanopa mubato unogona kudhonzwa pese pavanenge vachida kuve nechokwadi chekuti API iri kushanda. Uye vanogadzira panguva ino vachiri kutora nekunyora /api/test/helloworld
Nzira chete yekuita kuti zvese zvishande? - Aihwa!

• Kugadzira macheki akadaro ibasa revagadziri. Miedzo yeyuniti inofanirwa kunyorwa nevagadziri vanonyora kodhi. Nekuti kana iwe ukaidonhedza kune admin, "Dude, heino rondedzero yeAPI protocol kune ese makumi maviri nemabasa, ndapota tarisa zvese!" - hapana chichabuda.
• Kana iwe ukadhinda "mhoro nyika", hapana anozomboziva kuti API inofanirwa uye inoshanda. Yese API shanduko inofanirwa kutungamira kune shanduko mumacheki.
• Kana iwe uchitova nedambudziko rakadaro, misa maficha uye ugovere vagadziri vanozonyora macheki aya, kana kugamuchira kurasikirwa, bvuma kuti hapana chinotariswa uye chinotadza.

Mazano ehunyanzvi:

• Iva nechokwadi chekuronga sevha yekunze kuronga cheki - unofanirwa kuve nechokwadi chekuti chirongwa chako chinowanikwa kune kunze kwenyika.
• Ronga cheki mukati meiyo API protocol, kwete chete ega ekupedzisira.
• Gadzira prometheus-endpoint ine mhinduro dzebvunzo.

Application layer - hutano metrics yekutarisa

Iye zvino tiri kutaura nezvekunze kwehutano metrics emasevhisi.

Isu takasarudza kuti titarise ese "mabato" echishandiso tichishandisa cheki dzekunze, dzatinodaidza kubva kune yekunze yekutarisa system. Asi aya ndiwo "mabato" ayo mushandisi "anoona". Tinoda kuve nechokwadi chekuti masevhisi edu pachawo anoshanda. Pane nyaya iri nani pano: K8s ine cheki yehutano, kuitira kuti kanenge "cube" pachayo inogona kugutsikana kuti sevhisi iri kushanda. Asi hafu yemacheki andaona akafanana akadhinda “hello world”. Avo. Saka anodhonza kamwe mushure mekutumirwa, akapindura kuti zvese zvakanaka - ndizvo chete. Uye sevhisi, kana ichipa yayo API, ine nhamba hombe yekupinda kweiyo API imwe chete, iyo inodawo kuongororwa, nekuti isu tinoda kuziva kuti inoshanda. Uye isu tiri kutozvitarisa mukati.

Maitiro ekuita izvi nemazvo tekinoroji: sevhisi yega yega inoburitsa magumo nezvekuita kwayo kwazvino, uye mumagirafu eGrafana (kana chero imwe application) tinoona mamiriro emasevhisi ese.

• Yese API shanduko inofanirwa kutungamira kune shanduko mumacheki.
• Gadzira sevhisi nyowani ipapo uine hutano huyemu.
• Admin anogona kuuya kune vanogadzira obvunza "ndiwedzerewo akati wandei maficha kuti ndinzwisise zvese uye nekuwedzera ruzivo nezve izvi kune yangu yekutarisa system." Asi vanogadzira vanowanzopindura, "Hatisi kuwedzera chero chinhu mavhiki maviri isati yaburitswa."
  Vedevelopment managers ngavazive kuti pachave nekurasika kwakadaro, zivisaiwo manejimendi evedevelopment managers. Nekuti kana zvese zvadonha, mumwe munhu acharamba achifona uye anoda kutarisa "sevhisi inogara ichidonha" (c)
• Nenzira, govera vagadziri kuti vanyore plugins yeGrafana - ichi chichava rubatsiro rwakanaka kune admins.

Application Layer - Integration Monitoring

Kubatanidzwa kwekutarisa kunotarisana nekutarisa kutaurirana pakati pebhizinesi-yakakosha masisitimu.

Semuenzaniso, kune masevhisi gumi nemashanu anotaurirana. Aya haasisiri masaiti akaparadzana. Avo. isu hatigone kudhonza sevhisi pachayo, tora / helloworld uye unzwisise kuti sevhisi iri kushanda. Nekuti iyo yekuraira webhu sevhisi inofanirwa kutumira ruzivo nezve odha kubhazi - kubva mubhazi, sevhisi yekuchengetera zvinhu inofanirwa kugamuchira iyi meseji uye kushanda nayo mberi. Uye iyo email yekugovera sevhisi inofanirwa kugadzirisa izvi neimwe nzira mberi, nezvimwe.

Saizvozvo, isu hatigone kunzwisisa, tichitarisa pasevhisi yega yega, kuti zvese zvinoshanda. Nekuti isu tine imwe bhazi iyo kuburikidza nayo zvese zvinotaurirana nekudyidzana.
Naizvozvo, nhanho iyi inofanirwa kumaka nhanho yekuyedza masevhisi ekudyidzana nemamwe masevhisi. Hazvibviri kuronga kutariswa kwekutaurirana nekutarisa meseji broker. Kana paine sevhisi inoburitsa data uye sevhisi inoigamuchira, kana tichitarisa bhuroka tinongoona data rinobhururuka kubva kune rumwe rutivi. Kunyangwe isu neimwe nzira takakwanisa kutarisa kupindirana kweiyi data mukati - kuti mumwe mugadziri anotumira iyo data, mumwe munhu anoiverenga, kuyerera uku kunoramba kuchienda kuKafka - izvi hazvisati zvichizotipa ruzivo kana imwe sevhisi yakatumira meseji mune imwe vhezheni. , asi imwe sevhisi haina kutarisira iyi vhezheni uye yakadarika. Isu hatizoziva nezve izvi, sezvo masevhisi achatiudza kuti zvese zviri kushanda.

Zvandinokurudzira kuita:

• Yekuwirirana kutaurirana: iyo yekupedzisira inoita zvikumbiro kune inoenderana masevhisi. Avo. isu tinotora iyi yekupedzisira, dhonza script mukati mesevhisi, inoenda kune ese mapoinzi ndokuti "Ndinogona kudhonza ipapo, uye kudhonza ipapo, ndinogona kudhonza ipapo ..."
• Nekutaurirana kweasynchronous: mameseji anouya - iyo yekupedzisira inotarisa bhazi yebvunzo mameseji uye inoratidza mamiriro ekugadzirisa.
• Kune asynchronous kutaurirana: inobuda mameseji - iyo yekupedzisira inotumira bvunzo mameseji kubhazi.

Sezvinowanzoitika: isu tine sevhisi inokanda data mubhazi. Isu tinouya kubasa iri uye tinokukumbira kuti utiudze nezvehutano hwayo hwekubatanidza. Uye kana iyo sevhisi ichida kuburitsa meseji kumwe kumberi (WebApp), ipapo inoburitsa iyi bvunzo meseji. Uye kana isu tichimhanyisa sevhisi kudivi reOrderProcessing, rinotanga kutumira zvarinogona kutumira zvakazvimirira, uye kana paine zvimwe zvinhu zvinotsamira, zvino inoverenga seti yebvunzo mameseji kubva mubhazi, inonzwisisa kuti inogona kuzvigadzirisa, itaure uye , kana zvichidiwa, zvitumire mberi, uye pamusoro peizvi anoti - zvose zvakanaka, ndiri mupenyu.

Kazhinji tinonzwa mubvunzo wekuti "tingaedza sei izvi pane data rekurwa?" Semuenzaniso, isu tiri kutaura nezve imwechete yekurongeka sevhisi. Kurongeka kunotumira mameseji kunzvimbo yekuchengetera zvinhu kunonyorerwa zvinhu: isu hatigone kuyedza izvi pane data rekurwa, nekuti "zvinhu zvangu zvichadzimwa!" Solution: Ronga iyi bvunzo yese pakutanga. Iwe zvakare une mayunitsi bvunzo anoita madhiri. Saka, zviitei padanho rakadzama apo iwe une nzira yekutaurirana isingakuvadze kushanda kwebhizinesi.

Infrastructure level

Infrastructure monitoring ndechimwe chinhu chagara chichionekwa sechinozvitarisa.

• Kutariswa kwezvivakwa kunogona uye kunofanirwa kutangwa senzira yakaparadzana.
• Iwe haufanirwe kutanga nekutariswa kwezvivakwa pane purojekiti inomhanya, kunyangwe iwe uchida chaizvo. Uku kurwadza kwema devops ese. "Kutanga ndichatarisa sumbu, ini ndichaongorora zvivakwa" - i.e. Kutanga, inotarisa zviri pazasi, asi haipinde mukushandisa. Nekuti iyo application chinhu chisinga nzwisisike kune devops. Yakaburitswa kwaari, uye haanzwisise kuti inoshanda sei. Asi anonzwisisa zvivakwa uye anotanga nazvo. Asi kwete - iwe unogara uchida kutarisa application kutanga.
• Usapfuurire nenhamba yezviziviso. Tichifunga nezvekuoma kwemaitiro emazuva ano, chenjedzo dzinogara dzichibhururuka, uye iwe unofanirwa kurarama neboka iri rekuzivisa. Uye munhu wekufona, akatarisa zana rezviyedzo zvinotevera, achasarudza "Handidi kufunga nezvazvo." Nyevero dzinofanira kungozivisa nezvezvinhu zvakakosha.

Chikumbiro chenhanho sechikamu chebhizinesi

Pfungwa dzakakosha:

• ELK. Iyi ndiyo chiyero cheindasitiri. Kana nekuda kwechimwe chikonzero usiri kuunganidza matanda, tanga kuzviita nekukurumidza.
• APM. Ekunze APMs senzira yekukurumidza kuvhara application yekutarisa (NewRelic, BlackFire, Datadog). Unogona kuisa chinhu ichi kwenguva pfupi kuti unzwisise zviri kuitika newe.
• Tracing. Mune akawanda e microservices, iwe unofanirwa kuteedzera zvese, nekuti chikumbiro hachichararama chega. Izvo zvakanyanya kuoma kuwedzera gare gare, saka zviri nani kukurumidza kuronga kutsvaga mukusimudzira - iri ibasa uye kushandiswa kwevagadziri. Kana usati wazvishandisa, zvishandise! Ona Jaeger/Zipkin

Alerting

• Sangano reziviso system: mumamiriro ezvinhu ekutarisa boka rezvinhu, panofanirwa kuve nehurongwa hwakabatana hwekutumira zviziviso. Unogona kuGrafana. Kumadokero, munhu wese anoshandisa PagerDuty. Nyevero dzinofanirwa kuve dzakajeka (semuenzaniso kwadzakabva ...). Uye zvinokurudzirwa kudzora kuti zviziviso zvinogamuchirwa zvachose
• Sangano rehurongwa hwebasa: chenjedzo hadzifanirwe kutumirwa kumunhu wese (chero munhu wese achaita muboka revanhu, kana hapana achaita). Vagadziri vanofanirwawo kufona: ive shuwa yekutsanangura nzvimbo dzemutoro, ita mirairo yakajeka uye nyora mairi kuti ndiani chaizvo wekufona neMuvhuro neChitatu, uye ndiani wekufona neChipiri neChishanu (zvikasadaro havafonere chero munhu chiitiko chedambudziko guru - vanotya kukumutsa kana kukanganisa : vanhu kazhinji havadi kufona nekumutsa vamwe vanhu, kunyanya husiku). Uye tsanangura kuti kukumbira rubatsiro hakusi chiratidzo chekusakwanisa ("Ndinokumbira rubatsiro, zvinoreva kuti ndiri mushandi akaipa"), kurudzira zvikumbiro zvekubatsira.
• Sangano re "ruzivo base" uye kufambiswa kwebasa rekugadzirisa chiitiko: pane chimwe nechimwe chakakomba chiitiko, post-mortem inofanira kurongwa, uye seyero yenguva pfupi, zviito zvinogadzirisa chiitiko zvinofanira kunyorwa. Uye ita muitiro kuti yambiro inodzokororwa chivi; dzinoda kugadziriswa mukodhi kana basa rezvivakwa.

Technology stack

Ngatimbofungidzira kuti stack yedu yakaita seinotevera:

• kuunganidza data - Prometheus + Grafana;
• log analysis - ELK;
• ye APM kana Tracing - Jaeger (Zipkin).

Sarudzo yezvasarudzo haina kukosha. Nekuti kana pakutanga iwe wakanzwisisa maitiro ekutarisa sisitimu uye kunyora pasi chirongwa, ipapo iwe unotanga kusarudza maturusi kuti aenderane nezvaunoda. Mubvunzo ndewawakasarudza kutarisa pakutanga. Nekuti pamwe chishandiso chawakasarudza pakutanga hachienderane nezvaunoda zvachose.

Mapoinzi mashoma ehunyanzvi andinoona kwese nguva pfupi yadarika:

Prometheus ari kusundirwa mukati meKubernetes - ndiani akauya neizvi?! Kana cluster yako ikaparara, uchaitei? Kana iwe uine sumbu rakaoma mukati, saka panofanirwa kuve nemhando yekutarisa system mukati mesumbu, uye imwe kunze, iyo inounganidza data kubva mukati mesumbu.

Mukati mechikwata tinounganidza matanda nezvimwe zvese. Asi iyo monitoring system inofanirwa kunge iri kunze. Kazhinji, musumbu umo mune Promtheus yakaiswa mukati, kune zvakare masisitimu anoita ekunze macheki ekushanda kwesaiti. Ko kana kubatana kwako kune yekunze kwadonha uye application ikasashanda? Zvinoitika kuti zvese zvakanaka mukati, asi hazviiti kuti zvinhu zvive nyore kune vashandisi.

zvakawanikwa

• Monitoring budiriro hakusi kuiswa kwezvishandiso, asi kuvandudza kwechigadzirwa chesoftware. 98% yekutarisisa kwanhasi iri kukodha. Kukodha mumasevhisi, kukodha ekunze cheki, kutarisa ekunze masevhisi, uye ndizvo zvese.
• Usatambise nguva yevagadziri vako pakutarisa: zvinogona kutora kusvika makumi matatu muzana ebasa ravo, asi zvakakosha.
• Devops, usanetseka kuti haugone kutarisa chimwe chinhu, nekuti zvimwe zvinhu inzira yekufunga yakasiyana zvachose. Iwe wanga usiri mugadziri, uye basa rekutarisa ndiro chairo basa ravo.
• Kana iyo purojekiti yave kutoshanda uye isingatariswe (uye iwe uri maneja), govera zviwanikwa zvekutarisa.
• Kana chigadzirwa chacho chatove mukugadzirwa, uye iwe uri devops akaudzwa kuti "misa yekutarisa" - edza kutsanangurira manejimendi izvo zvandakanyora nezvese izvi.

Iyi ishanduro yakawedzerwa yemushumo pamusangano weSaint Highload ++.

Kana iwe uchifarira mazano angu uye pfungwa pazviri uye zvine hukama misoro, saka pano iwe unogona verenga chiteshi 🙂

Source: www.habr.com