Namuhla, ngaphezu kwekhodi ye-monolithic, iphrojekthi yethu ihlanganisa inqwaba yama-microservices. Ngamunye wabo udinga ukugadwa. Ukwenza lokhu ngesilinganiso esinjalo usebenzisa onjiniyela be-DevOps kuyinkinga. Senze isistimu yokuqapha esebenza njengesevisi yonjiniyela. Bangakwazi ukubhala ngokuzimela amamethrikhi ohlelweni lokuqapha, bawasebenzise, ββbakhe amadeshibhodi ngokusekelwe kuwo, futhi banamathisele izexwayiso kuzo ezizocushwa lapho amanani e-threshold efinyelelwa. Konjiniyela be-DevOps, ingqalasizinda kanye nemibhalo kuphela.
Lokhu okuthunyelwe kuwumbhalo wenkulumo yami newethu e-RIT++. Abantu abaningi basicele ukuthi senze izinguqulo zemibiko yemibhalo esuka lapho. Uma ubusenkomfeni noma ubukele ividiyo, ngeke uthole lutho olusha. Futhi wonke umuntu - wamukelekile ekati. Ngizokutshela ukuthi sifike kanjani ohlelweni olunjalo, ukuthi lusebenza kanjani nokuthi sihlela ukulubuyekeza kanjani.
Okwedlule: izikimu nezinhlelo
Sifike kanjani ohlelweni lwamanje lokuqapha? Ukuze uphendule lo mbuzo, udinga ukuya ku-2015. Nansi indlela eyayibukeka ngayo ngaleso sikhathi:
Sibe nama-node angaba ngu-24 ayenomsebenzi wokuqapha. Kukhona iphakethe eliphelele lemiqhele ehlukene, imibhalo, amademoni aqapha okuthile ngandlela thile, athumele imilayezo, futhi enze imisebenzi. Sasicabanga ukuthi lapho siqhubeka, uhlelo olunjalo luzoba lula. Asikho isidingo sokuyithuthukisa: inzima kakhulu.
Sinqume ukukhetha lezo zakhi zokuqapha esizozigcina futhi sizithuthukise, kanye nalezo esizozishiya. Kwakukhona angu-19. Kusele amagraphite, ama-aggregator kanye ne-Grafana njengedeshibhodi kuphela. Kodwa sizobukeka kanjani isimiso esisha? Kanje:
Sinesitoreji samamethrikhi: lawa amagraphite, azosuselwa kumadrayivu e-SSD asheshayo, lezi izihlanganisi ezithile zamamethrikhi. Okulandelayo - i-Grafana yokubonisa amadeshibhodi kanye ne-Moira yokuxwayisa. Besifuna nokwakha isistimu yokusesha okudidayo.
Okujwayelekile: Ukuqapha 2.0
Yile ndlela izinhlelo ezazibukeka ngayo ngo-2015. Kodwa kwakudingeka silungise hhayi kuphela ingqalasizinda kanye nenkonzo ngokwayo, kodwa futhi nemibhalo yayo. Sizenzele izinga lebhizinisi, esilibiza nge-monitoring 2.0. Zaziyini izidingo zesistimu?
- ukutholakala njalo;
- isikhawu sokugcinwa kwamamethrikhi = imizuzwana eyi-10;
- isitoreji esihlelekile samamethrikhi namadeshibhodi;
- I-SLA > 99,99%
- ukuqoqwa kwamamethrikhi omcimbi nge-UDP (!).
Besidinga i-UDP ngoba sinokugeleza okukhulu kwethrafikhi nemicimbi ekhiqiza amamethrikhi. Uma uwabhala wonke ku-graphite ngesikhathi esisodwa, isitoreji sizowa. Siphinde sakhetha iziqalo zeleveli yokuqala zawo wonke amamethrikhi.
Isiqalo ngasinye sinempahla ethile. Kukhona amamethrikhi amaseva, amanethiwekhi, iziqukathi, izinsiza, izinhlelo zokusebenza, nokunye. Ukuhlunga okucacile, okuqinile, okuthayiphiwe kusetshenzisiwe, lapho samukela khona ama-metrics eleveli yokuqala futhi simane silahle amanye. Lena yindlela esihlele ngayo lolu hlelo ngo-2015. Yini okwamanje?
Okwamanje: umdwebo wokusebenzelana kwezingxenye zokuqapha
Okokuqala, siqapha izinhlelo zokusebenza: ikhodi yethu ye-PHP, izinhlelo zokusebenza nama-microservices - ngamafuphi, yonke into ebhalwa abathuthukisi bethu. Zonke izinhlelo zokusebenza zithumela amamethrikhi nge-UDP kusihlanganisi se-Brubeck (izibalo, ezibhalwe kabusha ngo-C). Kwatholakala ukuthi iyona eshesha kakhulu ekuhlolweni kokwenziwa. Futhi ithumela amamethrikhi asevele ahlanganisiwe ku-Graphite nge-TCP.
Inohlobo lwamamethrikhi abizwa ngokuthi izibali sikhathi. Lokhu kuyinto elula kakhulu. Isibonelo, ekuxhumekeni komsebenzisi ngamunye kusevisi, uthumela imethrikhi enesikhathi sokuphendula ku-Brubeck. Kungene izimpendulo eziyisigidi, kodwa isihlanganisi sibuyise amamethrikhi ayi-10 kuphela. Unenombolo yabantu abezile, isikhathi esiphezulu, esincane nesimaphakathi sokuphendula, i-median namaphesenti angu-4. Bese idatha idluliselwa ku-Graphite futhi siyibona yonke bukhoma.
Siphinde sibe nokuhlanganisa amamethrikhi ku-hardware, isofthiwe, amamethrikhi esistimu kanye nesistimu yethu yokuqapha ye-Munin endala (yasisebenzela kwaze kwaba ngu-2015). Siqoqa konke lokhu nge-C daemon CollectD (inenqwaba yama-plugin ahlukene akhiwe kuyo, ingakwazi ukuvota zonke izinsiza zesistimu yokusingatha efakwe kuyo, vele ucacise ekucushweni ukuthi ungayibhala kuphi idatha) futhi bhala idatha ku-Graphite ngayo. Iphinde isekele ama-plugin we-python kanye nemibhalo yegobolondo, ukuze ukwazi ukubhala izixazululo zakho zangokwezifiso: I-CollectD izoqoqa le datha kumsingathi wendawo noma oqhelile (ithatha i-Curl) futhi iyithumele ku-Graphite.
Bese sithumela wonke amamethrikhi esiwaqoqile ku-Carbon-c-relay. Lesi yisixazululo se-Carbon Relay esisuka ku-Graphite, esilungiswe ngo-C. Lena irutha eqoqa wonke amamethrikhi esiwathumela kusukela kubahlanganisi bethu bese iwahambisa kumanodi. Futhi esigabeni somzila, ihlola ukufaneleka kwamamethrikhi. Okokuqala, kufanele zihambisane nohlelo lwesiqalo engilubonise ngaphambili futhi, okwesibili, lusebenza ku-graphite. Kungenjalo bazokwehla.
I-Carbon-c-relay bese ithumela ama-metrics kuqoqo le-Graphite. Sisebenzisa i-Carbon-cache, ebhalwe kabusha kokuthi Go, njengesitoreji esiyinhloko samamethrikhi. I-Go-carbon, ngenxa yokufundwa kwayo okuningi, idlula kude i-Carbon-cache. Ithola idatha futhi iyibhale kumadiski isebenzisa iphakheji yokuhleba (okujwayelekile, okubhalwe nge-python). Ukuze sifunde idatha kusitoreji sethu, sisebenzisa i-Graphite API. Ishesha kakhulu kune-graphite WEB ejwayelekile. Kwenzekani kudatha ngokulandelayo?
Baya eGrafana. Sisebenzisa amaqoqo ethu e-graphite njengomthombo oyinhloko wedatha, futhi sine-Grafana njengesixhumi esibonakalayo sewebhu sokubonisa amamethrikhi namadeshibhodi okwakha. Kusevisi yabo ngayinye, onjiniyela badala ideshibhodi yabo. Bese bakha amagrafu ngokusekelwe kuwo, abonisa amamethrikhi abawabhalayo kusukela kuzinhlelo zabo zokusebenza. Ngaphandle kweGrafana, siphinde sibe neSLAM. Leli idemoni le-python elibala i-SLA ngokusekelwe kudatha evela ku-graphite. Njengoba ngike ngasho, sineshumi nambili microservices, ngayinye enezidingo zayo. Sisebenzisa i-SLAM, siya emibhalweni futhi siyiqhathanise nalokho okuku-Graphite futhi siqhathanise ukuthi izidingo zifana kanjani nokutholakala kwezinsizakalo zethu.
Asiqhubeke: ukuxwayisa. Ihlelwe kusetshenziswa uhlelo oluqinile - i-Moira. Izimele ngoba ine-Graphite yayo ngaphansi kwe-hood. Kuthuthukiswe abafana abavela ku-SKB "Kontur", ebhalwe nge-python ne-Go, umthombo ovulekile ngokuphelele. U-Moira uthola ukugeleza okufanayo okungena kumagraphite. Uma ngesizathu esithile isitoreji sakho sifa, isexwayiso sakho sisazosebenza.
Sikhiphe i-Moira e-Kubernetes; isebenzisa iqoqo lamaseva e-Redis njengesizindalwazi esikhulu. Umphumela waba uhlelo lokubekezelela amaphutha. Iqhathanisa ukusakaza kwamamethrikhi nohlu lwezibangeli: uma kungekho okukhulunyiwe kukho, bese yehlisa imethrikhi. Ngakho iyakwazi ukugaya ama-gigabytes wamamethrikhi ngomzuzu.
Siphinde sanamathisela i-LDAP yebhizinisi kuyo, ngosizo lapho umsebenzisi ngamunye wesistimu yebhizinisi angakwazi ukuzidalela izaziso ngokusekelwe kuzibangeli ezikhona (noma ezisanda kwakhiwa). Njengoba i-Moira iqukethe i-Graphite, isekela zonke izici zayo. Ngakho-ke uqala ngokuthatha umugqa uwukopishele eGrafana. Bona ukuthi idatha iboniswa kanjani kumagrafu. Bese uthatha umugqa ofanayo futhi uwukopishe ku-Moira. Uyilenga ngemikhawulo futhi uthole isexwayiso ekuphumeni. Ukuze wenze konke lokhu, awudingi ulwazi oluthile. I-Moira ingaxwayisa nge-SMS, i-imeyili, iJira, i-Slack... Iphinde isekele ukukhishwa kwemibhalo yangokwezifiso. Uma i-trigger iyenzeka kuye, futhi ubhaliselwe kusikripthi sangokwezifiso noma kanambambili, uyasisebenzisa futhi athumele i-JSON ku-stdin yalokhu kanambambili. Ngokufanelekile, uhlelo lwakho kufanele luluhlaziye. Ukuthi uzokwenzani ngale JSON kukuwe. Uma ufuna, yithumele kuTelegram, uma ufuna, vula imisebenzi eJira, yenza noma yini.
Siphinde sisebenzise ukuthuthukiswa kwethu ukuxwayisa - Imagotag. Siguqule iphaneli, evame ukusetshenziselwa amathegi entengo kagesi ezitolo, ukuze ivumelane nezidingo zethu. Silethe ama-triggers avela e-Moira kuwo. Ikhombisa ukuthi zikusiphi isimo nokuthi zenzeka nini. Abanye babafana abathuthukayo bashiye izaziso ku-Slack kanye ne-imeyili bavuna leli phaneli.
Nokho, njengoba siyinkampani ethuthukayo, siphinde saqapha i-Kubernetes kulolu hlelo. Siyifake kusistimu sisebenzisa i-Heapster, esiyifake kuqoqo, iqoqa idatha bese iyithumela ku-Graphite. Ngenxa yalokho, umdwebo ubukeka kanjena:
Izingxenye Zokuqapha
Nalu uhlu lwezixhumanisi zezingxenye esizisebenzisele lo msebenzi. Zonke ziwumthombo ovulekile.
I-graphite:
- i-go-carbon:
- hleba:
- i-graphite-api:
I-Carbon-c-relay:
Brubeck:
Kuqoqiwe:
I-Moira:
Grafana:
I-Heapster:
Izibalo
Futhi nazi izinombolo ezithile mayelana nendlela uhlelo olusebenza ngayo kithi.
I-Aggregator (brubeck)
Inombolo yamamethrikhi: ~300/isekhondi
Isikhawu sokuthumela amamethrikhi ku-Graphite: 30 sec
Ukusetshenziswa kwensiza yeseva: ~ 6% CPU (sikhuluma ngamaseva agcwele); ~ 1Gb RAM; ~3 Mbps LAN
I-graphite (i-go-carbon)
Inombolo yamamethrikhi: ~ 1 / min
Isikhawu sokubuyekeza amamethrikhi: 30 sec
Isikimu sesitoreji samamethrikhi: 30sec 35d, 5min 90d, 10min 365d (sikunikeza ukuqonda ukuthi kwenzekani kusevisi esikhathini eside)
Ukusetshenziswa kwensiza yeseva: ~10% CPU; ~ 20Gb RAM; ~30 Mbps LAN
Ukuvumelana nezimo
Thina kwa-Avito sikwazisa ngempela ukuguquguquka enkonzweni yethu yokuqapha. Kungani ngempela aphenduka kanje? Okokuqala, izingxenye zayo ziyashintshashintsha: kokubili izingxenye ngokwazo kanye nezinguqulo zazo. Okwesibili, ukusekela. Njengoba yonke iphrojekthi ingumthombo ovulekile, ungazihlela ngokwakho ikhodi, wenze izinguquko, futhi usebenzise imisebenzi engatholakali ngaphandle kwebhokisi. Kusetshenziswa izitaki ezijwayelekile, ikakhulukazi i-Go nePython, ngakho-ke lokhu kwenziwa kalula.
Nasi isibonelo senkinga yangempela. Imethrikhi eku-Graphite yifayela. Inegama. Igama lefayela = igama lemethrikhi. Futhi kukhona indlela yokufika lapho. Amagama wamafayela ku-Linux anomkhawulo wezinhlamvu ezingama-255. Futhi sinabafana (βnjengamakhasimende angaphakathiβ) abavela emnyangweni wedathabheyisi. Bayasitshela: βSifuna ukuqapha imibuzo yethu ye-SQL. Futhi akuzona izinhlamvu ezingu-255, kodwa 8 MB ngayinye. Sifuna ukuwabonisa e-Grafana, sibone imingcele yalesi sicelo, futhi okungcono nakakhulu, sifuna ukubona phezulu kwezicelo ezinjalo. Kuyoba kuhle uma kuboniswa ngesikhathi sangempela. Kungaba kuhle kakhulu ukubafaka esixwayisweni.β
Umbuzo we-SQL oyisibonelo uthathwa njengesibonelo ku
Setha iseva ye-Redis futhi sisebenzisa ama-plugin ethu Aqoqiwe, aya ku-Postgres futhi athathe yonke idatha lapho, sithumela amamethrikhi ku-Graphite. Kodwa sishintsha igama lemethrikhi ngama-hashes. Ngesikhathi esifanayo sithumela i-hashi efanayo ku-Redis njengokhiye, nawo wonke umbuzo we-SQL njengenani. Okufanele sikwenze wukuqinisekisa ukuthi uGrafana uyakwazi ukuya eRedis ayothatha lolu lwazi. Sivula i-Graphite API ngoba... lesi yisixhumi esibonakalayo esiyinhloko sokusebenzisana kwazo zonke izingxenye zokuqapha nge-graphite, futhi sifaka umsebenzi omusha lapho obizwa ngokuthi i-aliasByHash() - kusuka eGrafana sithola igama lemethrikhi, futhi siyisebenzise ekuceleni i-Redis njengokhiye, impendulo sithola inani lokhiye, okuwumbuzo wethu "SQL" " Ngakho-ke, sibonise ku-Grafana umbukiso wombuzo we-SQL, lowo ngokombono owawungenakwenzeka ukuwubonisa lapho, kanye nezibalo kuwo (izingcingo, imigqa, inani_lesikhathi, ...).
Imiphumela
Ukutholakala Isevisi yethu yokuqapha iyatholakala 24/7 kunoma yiluphi uhlelo lokusebenza nanoma iyiphi ikhodi. Uma ukwazi ukufinyelela ezindaweni zokugcina, ungabhala idatha kusevisi. Ulimi alubalulekile, izinqumo azibalulekile. Udinga kuphela ukwazi ukuthi uvule kanjani isokhethi, ubeke imethrikhi lapho bese uvala isokhethi.
Ukuthembeka Zonke izingxenye ziyawabekezelela amaphutha futhi zisingatha imithwalo yethu kahle.
Isithiyo esiphansi sokungena. Ukuze usebenzise lolu hlelo, awudingi ukufunda izilimi zokuhlela nemibuzo ngesi-Grafana. Vele uvule isicelo sakho, ufake isokhethi kuyo ezothumela amamethrikhi ku-Graphite, uyivale, uvule i-Grafana, udale amadeshibhodi lapho futhi ubheke ukuziphatha kwamamethrikhi akho, ukuthola izaziso nge-Moira.
Ukuzimela. Ungakwenza konke lokhu ngokwakho, ngaphandle kosizo lonjiniyela be-DevOps. Futhi lokhu kuyinzuzo, ngoba ungakwazi ukuqapha iphrojekthi yakho njengamanje, asikho isidingo sokuthi ucele noma ubani - ukuqala umsebenzi noma ukwenza izinguquko.
Sihloseni?
Konke okubalwe ngezansi akuyona nje imicabango engabonakali, kodwa okuthile okuthathwe ngakho izinyathelo zokuqala.
- I-Anomaly detector. Sifuna ukudala isevisi ezoya ku-Graphite storages yethu futhi sihlole imethrikhi ngayinye sisebenzisa ama-algorithms ahlukahlukene. Sekuvele kukhona ama-algorithms esifuna ukuwabuka, kunedatha, siyazi ukuthi singasebenza kanjani ngayo.
- Imethadatha. Sinezinsiza eziningi, ziyashintsha ngokuhamba kwesikhathi, njengabantu abasebenza nazo. Ukugcina amadokhumenti mathupha njalo akuyona inketho. Kungakho manje sesishumeka imethadatha kumasevisi ethu amancane. Isho ukuthi ithuthukiswe ngubani, izilimi esebenzisana nayo, izidingo ze-SLA, ukuthi izaziso kufanele zithunyelwe kuphi futhi kubani. Lapho usebenzisa isevisi, yonke idatha yebhizinisi idalwa ngokuzimela. Ngenxa yalokho, uthola izixhumanisi ezimbili - eyodwa eya kuma-trigger, enye eya kumadeshibhodi e-Grafana.
- Ukuqapha kuwo wonke amakhaya. Sikholelwa ukuthi bonke onjiniyela kufanele basebenzise isistimu enjalo. Kulesi simo, uhlala uqonda ukuthi i-traffic yakho ikuphi, kwenzekani kuyo, lapho iwela khona, lapho ubuthakathaka bayo bukhona. Uma, ngokwesibonelo, okuthile kuza futhi kuphazamise isevisi yakho, khona-ke uzofunda ngakho hhayi ngesikhathi socingo oluvela kumphathi, kodwa kusukela kusaziso, futhi ungakwazi ngokushesha ukuvula izingodo zakamuva futhi ubone ukuthi kwenzekeni lapho.
- Ukusebenza okuphezulu. Iphrojekthi yethu ikhula njalo, futhi namuhla isebenza cishe amanani we-metric ayi-2 ngomzuzu. Ngonyaka odlule, lesi sibalo sasiyi-000 000. Futhi ukukhula kuyaqhubeka, futhi lokhu kusho ukuthi ngemva kwesikhathi esithile i-Graphite (ihlebeza) izoqala ukulayisha kakhulu i-disk subsystem. Njengoba ngike ngasho, lolu hlelo lokuqapha lukhona emhlabeni wonke ngenxa yokushintshashintsha kwezingxenye. Othile unakekela futhi ehlala enweba ingqalasizinda yakhe ngokuqondile ku-Graphite, kodwa sinqume ukuhamba umzila ohlukile: sebenzisa njengendawo yokugcina amamethrikhi ethu. Lolu shintsho seluzophela, futhi maduze nje ngizokutshela ngokuningiliziwe ukuthi lokhu kwenziwa kanjani: ukuthi ibuphi ubunzima obebukhona nokuthi banqotshwa kanjani, ukuthi inqubo yokufuduka yahamba kanjani, ngizochaza izakhi ezikhethiwe njengokubopha kanye nokucushwa kwazo.
Ngiyabonga ukulalela kwenu! Buza imibuzo yakho ngesihloko, ngizozama ukuphendula lapha noma kokuthunyelwe okulandelayo. Mhlawumbe othile unolwazi lokwakha uhlelo olufanayo lokuqapha noma ukushintshela ku-Clickhouse esimweni esifanayo - yabelana ngakho kumazwana.
Source: www.habr.com