Namhlanje, ukongeza kwikhowudi ye-monolithic, iprojekthi yethu ibandakanya uninzi lwee-microservices. Ngamnye kubo ufuna ukujongwa. Ukwenza oku kwinqanaba elinjalo usebenzisa iinjineli zeDevOps kuyingxaki. Siphuhlise inkqubo yokubeka iliso esebenza njengenkonzo yabaphuhlisi. Banokubhala ngokuzimeleyo iimethrikhi kwinkqubo yokubeka iliso, bazisebenzise, ββbakhe iideshibhodi ezisekwe kuzo, kwaye bancamathele izilumkiso kubo eziya kuthi zivuswe xa kufikwa amaxabiso e-threshold. Kwiinjineli ze-DevOps, kuphela izakhiwo kunye namaxwebhu.
Esi sithuba sisicatshulwa sentetho yam kunye neyethu e-RIT++. Abantu abaninzi basicele ukuba senze iinguqulelo ezibhaliweyo zeengxelo ukusuka apho. Ukuba ubukwinkomfa okanye ubukele ividiyo, awuzukufumana nto intsha. Kwaye wonke umntu - wamkelekile kwikati. Ndiza kukuxelela ukuba sifike njani kwinkqubo enjalo, isebenza njani kunye nendlela esiceba ukuyihlaziya ngayo.
Ixesha elidlulileyo: izicwangciso kunye nezicwangciso
Sifike njani kwinkqubo yangoku yokubeka iliso? Ukuze uphendule lo mbuzo, kufuneka uye ku-2015. Nantsi indlela eyayikhangeleka ngayo ngoko:
Sasineendawo ezimalunga nama-24 ezazinoxanduva lokubeka iliso. Kukho ipakethe epheleleyo yezithsaba ezahlukeneyo, izikripthi, iidaemoni ezibeka iliso ngandlela thile into, zithumele imiyalezo, kwaye zenze imisebenzi. Sasicinga ukuba okukhona sisiya phambili, kokukhona iya kuncipha ukusebenza kakuhle inkqubo enjalo. Akukho sizathu sokuyiphuhlisa: inzima kakhulu.
Sigqibe kwelokuba sikhethe ezo mpawu zokubekwa kweliso esiya kuzigcina kwaye siziphuhlise, kunye nezo siya kuzishiya. Bezili-19 kuphela iigraphite, ii-aggregators kunye neGrafana njengedeshibhodi eseleyo. Kodwa iya kuba njani inkqubo entsha? Ndiyayithanda lento:
Sinokugcinwa kweemetriki: ezi ziigrafu, eziza kusekelwa kwiidrive ze-SSD ezikhawulezayo, ezi zi-aggregators ezithile zeemetriki. Okulandelayo - iGrafana yokubonisa iideshibhodi kunye neMoira yokulumkisa. Sikwafuna ukuphuhlisa inkqubo yokukhangela izinto ezingaqhelekanga.
Umgangatho: Ukubeka iliso 2.0
Yile nto yayikhangeleka ngayo izicwangciso ngo-2015. Kodwa kwakudingeka silungiselele kuphela iziseko zophuhliso kunye nenkonzo ngokwayo, kodwa kunye namaxwebhu ayo. Sizenzele ngokwethu umgangatho wequmrhu, esiwubiza ngokuba yi-monitoring 2.0. Zaziziphi iimfuneko zenkqubo?
- ukufumaneka rhoqo;
- isithuba sokugcinwa kweemetrics = imizuzwana eyi-10;
- ukugcinwa okucwangcisiweyo kweemetriki kunye needeshibhodi;
- SLA > 99,99%
- ingqokelela yeemetriki zesiganeko nge-UDP (!).
Sasifuna i-UDP kuba sinokuhamba okukhulu kwetrafikhi kunye neziganeko ezivelisa iimethrikhi. Ukuba uzibhala zonke kwigraphite ngaxeshanye, ugcino luya kuwa. Siphinde sakhetha izimaphambili zenqanaba lokuqala kuzo zonke iimetrics.
Isimaphambili ngasinye sinendawo ethile. Kukho iimetrics zeeseva, iinethiwekhi, izikhongozeli, izixhobo, usetyenziso, njalo njalo. Ukuhluza okucacileyo, okungqongqo, okuchwetheziweyo kuye kwaphunyezwa, apho samkela imilinganiselo yomgangatho wokuqala kwaye silahle ezinye. Le yindlela ebesiyicwangcise ngayo le nkqubo ngo-2015. Yintoni ngoku?
Ngoku: Idayagram yentsebenziswano yamacandelo okubeka iliso
Okokuqala, sibeka iliso kwizicelo: ikhowudi yethu ye-PHP, usetyenziso kunye nee-microservices-ngamafutshane, yonke into ebhalwa ngabaphuhlisi bethu. Zonke izicelo zithumela i-metrics nge-UDP kwi-aggregator ye-Brubeck (i-statsd, ibhalwe ngokutsha kwi-C). Kwavela ukuba lolona vavanyo lukhawulezayo kwiimvavanyo zokwenziwa. Kwaye ithumela iimetrics esele zihlanganisiwe kwiGraphite ngeTCP.
Inodidi lweemetriki ezibizwa ngokuba zizibali-xesha. Le yinto elula kakhulu. Umzekelo, kuqhagamshelo lomsebenzisi ngamnye kwinkonzo, uthumela i-metric kunye nexesha lokuphendula kuBrubeck. Iimpendulo ezisisigidi zangena, kodwa i-aggregator ibuyise iimethrikhi ezili-10 kuphela. Unenani labantu abeza, ubuninzi, ubuncinci kunye nexesha lokuphendula eliphakathi, i-median kunye ne-4 yeepesenti. Emva koko idatha idluliselwa kwiGraphite kwaye siyibona yonke iphila.
Sikwanayo nokudityaniswa kweemetrics kwi-hardware, isoftware, i-system metrics kunye nenkqubo yethu endala yokubeka iliso ye-Munin (yasisebenzela kude kube ngu-2015). Siqokelela konke oku ngeC daemon CollectD (ineqela elipheleleyo leeplagi ezahlukeneyo ezakhelwe kuyo, inokuvota zonke izibonelelo zenkqubo yomkhosi efakwe kuyo, cacisa nje kuqwalaselo apho ubhala khona idatha) kwaye bhala idatha kwiGraphite ngayo. Ikwaxhasa iiplagi ze-python kunye nezikripthi ze-shell, ngoko ungabhala izisombululo zakho zesiko: I-CollectD iya kuqokelela le datha kwi-host host yendawo okanye ekude (ithatha i-Curl) kwaye uyithumele kwi-Graphite.
Emva koko sithumela zonke iimetrics esiziqokeleleyo kwiCarbon-c-relay. Esi sisisombululo seCarbon Relay ukusuka kwiGraphite, elungiswe kwi-C. Le yi-router eqokelela zonke iimethrikhi esizithumela kwii-aggregators zethu kwaye sizihambise kwii-nodes. Kwakhona kwinqanaba lomzila, ijonga ukunyaniseka kweemethrikhi. Okokuqala, kufuneka zihambelane neskimu sesimaphambili endisibonise ngaphambili kwaye, okwesibini, ziyasebenza kwigraphite. Kungenjalo baya kuwa.
ICarbon-c-relay emva koko ithumela iimetrics kwiqela leGraphite. Sisebenzisa iCarbon-cache, ebhalwe ngokutsha kwiGo, njengolona gcino lweemetrics. I-Go-carbon, ngenxa ye-multithreading yayo, idlula kude iCarbon-cache. Ifumana idatha kwaye iyibhale kwiidiski isebenzisa iphakheji yokusebeza (eqhelekileyo, ebhalwe ngepython). Ukuze sifunde idatha kwiisitoreji zethu, sisebenzisa i-graphite API. Ikhawuleza kakhulu kune-WEB yeGraphite eqhelekileyo. Kwenzeka ntoni kwidatha elandelayo?
Baya eGrafana. Sisebenzisa amaqela ethu egraphite njengowona mthombo wedatha, kwaye sineGrafana njengojongano lwewebhu lokubonisa iimetriki kunye nokwakha iidashbhodi. Kwinkonzo nganye yabo, abaphuhlisi benza eyabo ideshibhodi. Emva koko bakha iigrafu ezisekelwe kuzo, ezibonisa iimethrikhi abazibhalayo kwizicelo zabo. Ukongeza kwiGrafana, sikwanayo neSLAM. Le yidemon yepython ebala i-SLA ngokusekelwe kwidatha evela kwigraphite. Njengoko besenditshilo, sineshumi elinesibini le-microservices, nganye ineemfuno zayo. Ukusebenzisa i-SLAM, siya kumaxwebhu kwaye sithelekise noko kukwiGraphite kwaye sithelekise indlela iimfuno ezihambelana ngayo nokufumaneka kweenkonzo zethu.
Masiqhubele phambili: ukwazisa. Iququzelelwe kusetyenziswa inkqubo eyomeleleyo - iMoira. Izimele kuba ineGraphite yayo phantsi kwehood. Iphuhliswe ngabafana abavela kwi-SKB "Kontur", ebhalwe kwi-python kunye ne-Go, umthombo ovulekileyo ngokupheleleyo. UMoira ufumana umqukuqelo ofanayo ongena kwigraphite. Ukuba ngesizathu esithile ugcino lwakho luyafa, isilumkiso sakho sisasebenza.
Sasebenzisa i-Moira e-Kubernetes isebenzisa iqela leeseva ze-Redis njengesiseko sedatha engundoqo. Umphumo waba yinkqubo yokunyamezela iimpazamo. Ithelekisa umlambo weemethrikhi kunye noluhlu lwezinto ezibangela: ukuba akukho zikhankanyiweyo kuyo, ngoko iwisa i-metric. Ngoko ke iyakwazi ukwetyisa iigigabhayithi zeemethrikhi ngomzuzu.
Sikwaqhoboshele i-LDAP yequmrhu kuyo, ngoncedo apho umsebenzisi ngamnye wenkqubo yequmrhu anokuzidalela izaziso ngokusekelwe kwizinto ezikhoyo (okanye ezisandula ukuyilwa). Ekubeni iMoira iqulethe iGraphite, ixhasa zonke iimpawu zayo. Ngoko ke uqala ngokuthatha umgca uwukhuphele eGrafana. Jonga indlela idatha eboniswa ngayo kwiigrafu. Kwaye ke uthatha umgca ofanayo kwaye uwukhuphele kwi-Moira. Uyixhoma ngemida kwaye ufumane isilumkiso kwisiphumo. Ukwenza konke oku, awudingi naluphi na ulwazi oluthile. UMoira unokulumkisa ngeSMS, i-imeyile, iJira, iSlack ... Ikwaxhasa ukuphunyezwa kwezikripthi zesiko. Xa i-trigger isenzeka kuye, kwaye ubhaliselwe kwiscript esiqhelekileyo okanye yokubini, uyayiqhuba kwaye athumele i-JSON kwi-stdin yale yokubini. Ngokufanelekileyo, inkqubo yakho kufuneka yahlule. Yintoni oya kuyenza ngale JSON ixhomekeke kuwe. Ukuba uyafuna, yithumele kwiTelegram, ukuba uyafuna, vula imisebenzi eJira, yenza nantoni na.
Sikwasebenzisa uphuhliso lwethu lokulumkisa-Imagotag. Siye sahlengahlengisa ipaneli, edla ngokusetyenziselwa iithegi zamaxabiso e-elektroniki ezivenkileni, ukuze zihambelane neemfuno zethu. Sazisa iitriggers ukusuka eMoira ukuya kuyo. Ibonisa ukuba bakweyiphi na imeko kunye nokuba zenzeke nini. Abanye babafana bophuhliso bazishiyile izaziso kwi-Slack kunye ne-imeyile bexhasa le njongo.
Ewe, kuba siyinkampani enenkqubela phambili, siye sabeka iliso kuKubernetes kule nkqubo. Siyifake kwinkqubo usebenzisa i-Heapster, esiyifake kwi-cluster, iqokelela idatha kwaye iyithumele kwi-Graphite. Ngenxa yoko, umzobo ujongeka ngolu hlobo:
Amacandelo okubeka iliso
Nalu uluhlu lwamakhonkco kumacandelo esiwasebenzisileyo kulo msebenzi. Zonke ziyimithombo evulekileyo.
Igraphite:
- hamba-carbon:
- sebeza:
- igraphite-api:
I-Carbon-c-relay:
Brubeck:
Iqokelelwe:
Moira:
Grafana:
Heapster:
Izibalo
Kwaye nanga amanye amanani malunga nendlela esisebenza ngayo inkqubo.
I-Aggregator (brubeck)
Inani leemetriki: ~300/umzuzwana
Ithuba lokuthumela iimetrics kwiGraphite: 30 sec
Ukusetyenziswa komthombo womncedisi: ~ 6% CPU (sithetha malunga neeseva ezigcwele ngokupheleleyo); ~ 1Gb RAM; ~3 Mbps LAN
Igraphite (go-carbon)
Inani leemetriki: ~ 1 / min
Isithuba sohlaziyo lweMetrics: 30 sec
Iskimu sokugcina iMetrics: 30sec 35d, 5min 90d, 10min 365d (ikunika ukuqonda okwenzekayo kwinkonzo ngexesha elide)
Ukusetyenziswa komthombo womncedisi: ~ 10% CPU; ~ 20Gb RAM; ~30 Mbps LAN
Bhetyebhetye
Thina kwa-Avito sikuxabisa ngokwenene ukuguquguquka kwinkonzo yethu yokubeka iliso. Kwakutheni ukuze avele ngolu hlobo? Okokuqala, amacandelo ayo ayatshintsheka: omabini amacandelo ngokwawo kunye neenguqulelo zawo. Okwesibini, inkxaso. Ekubeni yonke iprojekthi ingumthombo ovulekileyo, unokuyihlela ikhowudi ngokwakho, wenze utshintsho, kwaye uphumeze imisebenzi engafumanekiyo ngaphandle kwebhokisi. Izitaki eziqhelekileyo ziyasetyenziswa, ikakhulu yiGo kunye nePython, ke oku kwenziwa ngokulula.
Nanku umzekelo wengxaki yokwenyani. Imetric kwiGraphite yifayile. Inegama. Igama lefayile = igama lemetric. Kwaye kukho indlela yokufika apho. Amagama eefayile kwiLinux anqunyelwe kwiimpawu ezingama-255. Kwaye sinabo (njengabathengi bangaphakathi") abafana abavela kwisebe ledatha. Basixelela: βSifuna ukujonga imibuzo yethu yeSQL. Kwaye ayingoonobumba abangama-255, kodwa i-8 MB nganye. Sifuna ukuzibonisa eGrafana, sibone iiparamitha zesi sicelo, kwaye nangcono, sifuna ukubona phezulu kwezo zicelo. Kuya kuba kuhle ukuba iboniswe ngexesha langempela. Kuya kuba kuhle kakhulu ukubabeka kwisilumkiso. "
Umzekelo umbuzo weSQL uthathwa njengomzekelo ukusuka
Siseta iseva yeRedis kwaye sisebenzise iiplagi zethu eziQokelelweyo, eziya kwi-Postgres kwaye zithathe yonke idatha ukusuka apho, ukuthumela i-metrics kwiGraphite. Kodwa sithatha indawo yegama lemetric ngeehashi. Kwangaxeshanye sithumela i-hash efanayo kwiRedis njengesitshixo, kunye nombuzo wonke weSQL njengexabiso. Into ekufuneka siyenzile kukuqinisekisa ukuba uGrafana angaya eRedis ayokuthatha olu lwazi. Sivula iGraphite API kuba... olu lujongano oluphambili lonxibelelwano lwawo onke amacandelo okubeka iliso ngegraphite, kwaye singenisa umsebenzi omtsha apho obizwa ngokuba yi-aliasByHash() - ukusuka eGrafana sifumana igama le metric, kwaye siyisebenzise kwisicelo kwiRedis njengesitshixo, kwi impendulo sifumana ixabiso lesitshixo, leyo yethu "umbuzo weSQL" " Ngaloo ndlela, sibonise kwiGrafana umboniso wombuzo we-SQL, onokuthi kwithiyori kwakungenakwenzeka ukubonisa apho, kunye nezibalo kuyo (iifowuni, imigqa, i-total_time, ...).
Iziphumo
Ukufumaneka Inkonzo yethu yokubeka iliso iyafumaneka 24/7 kuyo nayiphi na isicelo kunye nayiphi na ikhowudi. Ukuba unokufikelela kwiindawo zokugcina, ungabhala idatha kwinkonzo. Ulwimi alubalulekanga, izigqibo azibalulekanga. Kufuneka wazi kuphela indlela yokuvula isokhethi, ubeke i-metric apho kwaye uvale isokhethi.
Ukuthembeka Onke amacandelo ayakwazi ukunyamezela iimpazamo kwaye ayiphatha kakuhle imithwalo yethu.
Umqobo ophantsi wokungena. Ukuze usebenzise le nkqubo, awudingi ukufunda iilwimi zenkqubo kunye nemibuzo eGrafana. Vula nje isicelo sakho, faka i-socket kuyo eya kuthumela i-metrics kwi-Graphite, ivale, ivule i-Grafana, yenza ii-dashboards apho kwaye ujonge ukuziphatha kweemetrics zakho, ukufumana izaziso nge-Moira.
Ukuzimela. Unokwenza konke oku ngokwakho, ngaphandle koncedo lweenjineli zeDevOps. Kwaye oku kuyinzuzo, kuba unako ukubeka iliso kwiprojekthi yakho ngoku, akufuneki ubuze nabani na-nokuba uqale umsebenzi okanye wenze utshintsho.
Sijonge ntoni?
Yonke into edweliswe apha ngezantsi ayizongcinga nje ezingabonakaliyo, kodwa yinto ekuye kwathatyathwa kuyo amanyathelo okuqala.
- Isibonisi esingaqhelekanga. Sifuna ukwenza inkonzo eya kuthi kwiGraphite storages kwaye sijonge i-metric nganye sisebenzisa ii-algorithms ezahlukeneyo. Sele kukho i-algorithms esifuna ukuyijonga, kukho idatha, siyazi indlela yokusebenza nayo.
- Imetadata. Sineenkonzo ezininzi, ziyatshintsha ngokuhamba kwexesha, njengabantu abasebenza nabo. Ukugcina uxwebhu rhoqo ngesandla asiyondlela yokukhetha. Yiyo loo nto ngoku sifaka imethadatha kwiinkonzo zethu ezincinci. Ichaza ukuba ngubani owayiphuhlisayo, iilwimi enxibelelana nazo, iimfuno ze-SLA, apho kwaye izaziso kufuneka zithunyelwe kubani. Xa uhambisa inkonzo, yonke idatha yequmrhu yenziwe ngokuzimeleyo. Ngenxa yoko, ufumana amakhonkco amabini - enye kwi-triggers, enye kwiideshibhodi eGrafana.
- Ukubeka iliso kwikhaya ngalinye. Sikholelwa ukuba bonke abaphuhlisi kufuneka basebenzise inkqubo enjalo. Kule meko, uhlala uqonda apho i-traffic yakho ikhona, kwenzekani kuyo, apho iwela khona, apho ubuthathaka bayo. Ukuba, umzekelo, into ethile iza kwaye iphazamise inkonzo yakho, ngoko uya kufunda malunga nayo kungekhona ngexesha lomnxeba ovela kumphathi, kodwa ukusuka kwisilumkiso, kwaye unokuvula ngokukhawuleza iilogi zamva nje kwaye ubone oko kwenzekayo apho.
- Ukwenza okuphezulu. Iprojekthi yethu ikhula rhoqo, kwaye namhlanje iqhuba malunga ne-2 yamaxabiso e-metric ngomzuzu. Kunyaka odlulileyo, eli nani lalingama-000 kwaye ukukhula kuyaqhubeka, kwaye oku kuthetha ukuba emva kwexesha elithile i-Graphite (i-whisper) iya kuqala ukulayisha kakhulu i-disk subsystem. Njengoko besenditshilo, le nkqubo yokubeka iliso ikwindawo yonke ngenxa yokutshintshiselana kwamacandelo. Umntu ugcina kwaye esoloko ekhulisa iziseko zabo zeGraphite, kodwa sagqiba ekubeni sihambe ngendlela eyahlukileyo: sebenzisa njengendawo yokugcina iimethrikhi zethu. Olu tshintsho sele lugqityiwe, kwaye kungekudala ndiza kukuxelela ngeenkcukacha ezithe kratya ukuba kwenziwa njani oku: bubuphi ubunzima obukhoyo kunye nokuba boyiswa njani, inkqubo yokufuduka ihambe njani, ndiya kuchaza amacandelo akhethiweyo njengezibophelelo kunye nolungelelwaniso lwazo.
Enkosi ngosinaka kwakho! Buza imibuzo yakho ngesihloko, ndiya kuzama ukuphendula apha okanye kwizithuba ezilandelayo. Mhlawumbi umntu unamava okwakha inkqubo efanayo yokubeka iliso okanye ukutshintshela kwiClickhouse kwimeko efanayo - yabelana ngayo kwizimvo.
umthombo: www.habr.com