I-VictoriaMetrics yi-DBMS ekhawulezayo kunye neyoyikisayo yokugcina kunye nokucubungula idatha ngendlela yochungechunge lwexesha (irekhodi liquka ixesha kunye neseti yamaxabiso ahambelana neli xesha, umzekelo, efunyenwe ngokuvotelwa ngamaxesha athile ubume beenzwa okanye ingqokelela yeemetrikhi).
Igama lam nguKolobaev Pavel. I-DevOps, i-SRE, i-LeroyMerlin, yonke into ifana nekhowudi - konke ngathi: malunga nam kunye nabanye abasebenzi baseLeroyMerlin.
Kukho ilifu elisekelwe kwi-OpenStack. Kukho ikhonkco elincinci kwi-radar yobugcisa.
Yakhelwe kwi-hardware ye-Kubernetes, kunye nakuzo zonke iinkonzo ezinxulumene ne-OpenStack kunye nokungena.
Le yinkqubo ebesinayo kuphuhliso. Xa sasiphuhlisa konke oku, sasinomqhubi wePrometheus ogcina idatha ngaphakathi kweqela le-K8s ngokwayo. Ufumana ngokuzenzekelayo oko kufuneka kuhlanjululwe aze akubeke phantsi kweenyawo zakhe, ukuthetha ngokukhawuleza.
Kuya kufuneka sihambise yonke idatha ngaphandle kweqela le-Kubernetes, kuba ukuba kukho into eyenzekayo, kufuneka siqonde ukuba yintoni kwaye phi.
Isisombululo sokuqala kukuba sisebenzisa i-federation xa sinomntu wesithathu u-Prometheus, xa sisiya kwiqela le-Kubernetes ngokusebenzisa i-federation mechanism.
Kodwa kukho iingxaki ezincinci apha. Kwimeko yethu, iingxaki zaqala xa sineemitha ezingama-250, kwaye xa kukho i-000 yeemitha, saqonda ukuba asikwazi ukusebenza ngolo hlobo. Sandise i-scrape_timeout ukuya kwimizuzwana engama-400.
Kwakutheni ukuze siyenze le nto? U-Prometheus uqala ukubala ixesha lokuvala ukususela ekuqaleni kocingo. Ayinamsebenzi ukuba idatha isahamba. Ukuba ngeli xesha lichaziweyo idatha ayidityaniswanga kwaye iseshoni ayivalwanga nge-http, ke iseshoni ithathwa njengehlulekile kwaye idatha ayingeni kwi-Prometheus ngokwayo.
Wonke umntu uqhelene neegrafu esizifumanayo xa enye idatha ilahlekile. Iishedyuli zikrazukile kwaye asonwabanga yile nto.
Inketho elandelayo kukwahlulwa ngokusekwe kwiPrometheus ezimbini ezahlukeneyo ngendlela enye yomanyano.
Umzekelo, zithathe nje kwaye uzahlule ngamagama. Oku kunokusetyenziswa, kodwa sagqiba ekubeni siqhubele phambili.
Kuya kufuneka ngoku siwacubungule ezi shards ngandlela ithile. Ungathatha i-promxy, eya kwindawo ye-shard kwaye iphindaphinda idatha. Isebenza ngeeshadi ezimbini njengendawo yokungena enye. Oku kunokuphunyezwa ngepromxy, kodwa kusenzima kakhulu.
Inketho yokuqala kukuba sifuna ukuyishiya inkqubo yomanyano kuba icotha kakhulu.
Abaphuhlisi bePrometheus bathetha ngokucacileyo, "Guys, sebenzisa i-TimescaleDB eyahlukileyo kuba asiyi kuxhasa ukugcinwa kwexesha elide kweemetriki." Ayingomsebenzi wabo lo.
Sibhala phantsi ephepheni esisafuna ukulothula ngaphandle, ukuze singagcini yonke into kwindawo enye.
I-drawback yesibini kukusetyenziswa kwememori. Ewe, ndiyaqonda ukuba abaninzi baya kuthi ngo-2020 iigigabytes ezimbalwa zememori zixabisa ipeni, kodwa kunjalo.
Ngoku sinendawo ye-dev kunye ne-prod. Kwi-dev imalunga ne-9 gigabytes ye-350 yeemethrikhi. Kwiprod yi-000 gigabytes kunye ne-14 yeemetrics. Ngexesha elifanayo, ixesha lethu lokugcinwa yimizuzu engama-780 kuphela. Oku kubi. Kwaye ngoku ndiza kuchaza ukuba kutheni.
Senza ukubala, oko kukuthi, ngeemitha zesigidi kunye nesiqingatha, kwaye sele sisondele kubo, kwinqanaba lokuyila sifumana i-35-37 gigabytes yememori. Kodwa sele i-4 yezigidi zeemetrics zifuna malunga ne-90 gigabytes yememori. Oko kukuthi, kubalwa kusetyenziswa ifomula enikezelwe ngabaphuhlisi bePrometheus. Sijonge ulungelelwaniso kwaye saqonda ukuba asifuni kuhlawula isibini sezigidi kwiseva nje ngokubeka iliso.
Asiyi kwandisa kuphela inani loomatshini, sikwabeka iliso koomatshini benyani ngokwabo. Ke ngoko, okukhona oomatshini ababonakalayo, kokukhona iimetrics ezininzi zeentlobo ezahlukeneyo, njl.njl. Siya kuba nokukhula okukhethekileyo kweqela lethu ngokweemetriki.
Ngendawo yediski, ayizizo zonke izinto ezimbi apha, kodwa ndingathanda ukuyiphucula. Sifumene inani elipheleleyo le-gigabytes ye-15 kwiintsuku ze-120, apho i-100 i-data compressed data, i-20 yidatha enganyanzeliswanga, kodwa sihlala sifuna ngaphantsi.
Ngokufanelekileyo, sibhala enye inqaku elingakumbi - oku kukusetyenziswa okukhulu kwezixhobo, esisafuna ukuyigcina, kuba asifuni ukuba iqoqo lethu lokubeka iliso lidle izibonelelo ezininzi kuneqela lethu, elilawula i-OpenStack.
Kukho enye i-drawback ye-Prometheus, esizichonge yona, olu luhlobo oluthile lokunciphisa imemori. Nge-Prometheus, yonke into imbi kakhulu apha, kuba ayinayo i-twist enjalo konke konke. Ukusebenzisa umda kwi-docker nayo ayikhethi. Ukuba ngokukhawuleza i-RAF yakho yawa kwaye kukho i-20-30 gigabytes, ngoko kuya kuthatha ixesha elide kakhulu ukunyuka.
Esi sesinye isizathu sokuba i-Prometheus ingasifanelanga, oko kukuthi, asikwazi ukunciphisa ukusetyenziswa kwememori.
Kunokwenzeka ukuba kuze necebo elinjalo. Sifuna esi sikimu ukuze siququzelele iqela le-HA. Sifuna ukuba iimetriki zethu zifumaneke rhoqo naphi na, nokuba iseva egcina ezi metrics iphazamisekile. Kwaye ngoko kuya kufuneka sakhe iskimu esinjalo.
Esi sikimu sithi siya kuba nokuphinda-phindwa kweeshadi, kwaye, ngokufanelekileyo, ukuphinda-phindwa kweendleko zemithombo esetyenzisiweyo. Inokukalwa phantse ngokuthe tye, kodwa nangona kunjalo ukusetyenziswa kobutyebi kuya kuba sisihogo.
Izinto ezingeloncedo ngendlela esizibhalele ngayo:
- Ifuna ukulayisha iimetrikhi ngaphandle.
- Ukusetyenziswa kwezixhobo eziphezulu.
- Akukho ndlela yokunciphisa ukusetyenziswa kwememori.
- Ufezekiso oluntsonkothileyo nolusebenzisa ubutyebi beHA.
Thina ngokwethu, sagqiba ekubeni sisuka kude nePrometheus njengendawo yokugcina.
Siye sachonga iimfuneko ezongezelelekileyo kuthi ezifunekayo. Oku:
- Le yinkxaso ye-promql, kuba izinto ezininzi sele zibhaliwe kwi-Prometheus: imibuzo, izilumkiso.
- Kwaye ke sineGrafana, esele ibhalwe ngendlela efanayo nePrometheus njenge-backend. Andifuni kuphinda ndibhale iideshibhodi.
- Sifuna ukwakha ulwakhiwo lwe-HA oluqhelekileyo.
- Sifuna ukunciphisa ukusetyenziswa kwazo naziphi na izibonelelo.
- Kukho enye i-nuance encinci. Asikwazi ukusebenzisa iintlobo ezahlukeneyo zeenkqubo zokuqokelelwa kweemetrics zamafu. Asazi ukuba yintoni eza kuwela kwezi metrics okwangoku. Yaye ekubeni nantoni na inokubhabha apho, kufuneka sizibekele umda kwindawo yasekuhlaleni.
Kwakungekho ukukhetha. Siqokelele yonke into ebesinamava ngayo. Sijonge kwiphepha le-Prometheus kwicandelo lokudibanisa, safunda amanqaku amaninzi, kwaye sabona into ekhoyo. Kwaye ngokwethu, sakhetha iVictoriaMetrics njengokutshintsha iPrometheus.
Ngoba? Ngenxa yokuba:
- Uyayazi ipromql.
- Kukho ulwakhiwo lweemodyuli.
- Ayifuni tshintsho eGrafana.
- Kwaye okona kubaluleke kakhulu, siya kubonelela ngogcino lweemetrics ngaphakathi kwenkampani yethu njengenkonzo, ke sijonge kwangaphambili kwizithintelo zeentlobo ezahlukeneyo ukuze abasebenzisi bakwazi ukusebenzisa zonke izixhobo zeqela ngendlela ethile elinganiselwe, kuba kukho ithuba. ukuba iya kuphindaphinda.
Masenze uthelekiso lokuqala. Sithatha iPrometheus efanayo ngaphakathi kweqela, iPrometheus yangaphandle iya kuyo. Yongeza nge remoteBhala iVictoriaMetrics.
Ngokukhawuleza ndiza kwenza ugcino ukuba apha sibambe ukunyuka okuncinci kokusetyenziswa kwe-CPU ukusuka kwi-VictoriaMetrics. IWiki yeVictoriaMetrics ikuxelela ukuba zeziphi iiparamitha ezilungileyo. Sazijonga. Baye banciphisa ukusetyenziswa kwe-CPU kakuhle kakhulu.
Kwimeko yethu, ukusetyenziswa kwememori yePrometheus, efumaneka kwiqela le-Kubernetes, ayizange inyuke kakhulu.
Sithelekisa imithombo emibini yedatha yedatha efanayo. Kwi-Prometheus sibona idatha efanayo engekhoyo. Yonke into ilungile kwiVictoriaMetrics.
Iziphumo zovavanyo lwesithuba seDiski. Thina kwa-Prometheus sifumene i-120 gigabytes iyonke. Kwi-VictoriaMetrics sele sifumana iigigabytes ezi-4 ngosuku. Kukho indlela eyahlukileyo kancinane kunento siqhele ukuyibona kwiPrometheus. Oko kukuthi, idatha sele icinezelwe kakuhle ngosuku, kwisiqingatha seyure. Sele zivunwe kakuhle ngosuku, kwisiqingatha seyure, nangona idatha isaya kulahleka kamva. Ngenxa yoko, sigcine kwindawo yedisk.
Sikwagcina ekusetyenzisweni komthombo wememori. Ngexesha lokuvavanya, sasino-Prometheus esetyenziswe kumatshini obonakalayo - i-8 cores, i-24 gigabytes. U-Prometheus udla phantse yonke into. Wawa kwi-OOM Killer. Kwangaxeshanye, ngama-900 kuphela iimethrikhi ezisebenzayo zagalelwa kuyo. Oku kumalunga ne-000-25 yeemethrikhi ngesekhondi.
Siye saqhuba i-VictoriaMetrics kumatshini we-double-core virtual kunye ne-8 gigabytes ye-RAM. Sikwazile ukuba iVictoriaMetrics isebenze kakuhle ngokuzulazula ngezinto ezimbalwa kumatshini we-8GB. Ekugqibeleni, siyigcine kwi-7 gigabytes. Ngelo xesha, isantya sokuhanjiswa komxholo, oko kukuthi i-metrics, yayiphezulu nangaphezulu kune-Prometheus.
I-CPU ibe ngcono kakhulu xa ithelekiswa nePrometheus. Apha i-Prometheus idla i-2,5 cores, kwaye i-VictoriaMetrics idla kuphela i-0,25 cores. Ekuqaleni - 0,5 cores. Njengoko idibanisa, ifikelela kumbindi omnye, kodwa oku kunqabile, kunqabile kakhulu.
Kwimeko yethu, ukhetho lwawela kwiVictoriaMetrics ngezizathu ezicacileyo; besifuna ukonga imali kwaye senzile.
Masiwele amanqaku amabini kwangoko - ukulayishwa kweemetriki kunye nokusetyenziswa okuphezulu kwezixhobo. Kwaye kufuneka sithathe isigqibo ngamanqaku amabini esisazishiyele wona.
Apha ndiza kwenza ugcino ngoko nangoko, sithathela ingqalelo iVictoriaMetrics njengogcino lweemetrics. Kodwa kuba siza kubonelela ngeVictoriaMetrics njengendawo yokugcina kuyo yonke iLeroy, kufuneka sinciphise abo baya kusebenzisa eli qela ukuze bangasiniki.
Kukho iparameter emangalisayo ekuvumela ukuba unciphise ixesha, ngevolumu yedatha kunye nexesha lokuphumeza.
Kukwakho nokhetho olubalaseleyo olusivumela ukuba sinciphise ukusetyenziswa kwememori, ngaloo ndlela sinokufumana ibhalansi eya kusivumela ukuba sifumane isantya esiqhelekileyo sokusebenza kunye nokusetyenziswa kobutyebi okwaneleyo.
Susa enye inqaku, o.k.t. ukuwela inqaku - awukwazi ukunciphisa ukusetyenziswa kwememori.
Kuphindaphindo lokuqala, siye savavanya iVictoriaMetrics iNode enye. Okulandelayo siqhubela phambili kwi-VictoriaMetrics Cluster Version.
Apha sinesandla sasimahla sokwahlula iinkonzo ezahlukeneyo kwiVictoriaMetrics kuxhomekeke kwinto abaya kuqhuba kuyo kwaye zeziphi izixhobo abaza kuzidla. Esi sisisombululo esiguquguqukayo kakhulu kwaye esifanelekileyo. Sasebenzisa oku kuthi.
Amacandelo aphambili e-VictoriaMetrics Cluster Version yi-vmstsorage. Kunokubakho inani elingu-N labo. Kwimeko yethu kukho i-2 kubo ngoku.
Kwaye kukho vminsert. Le yiseva engummeli osivumela ukuba: silungiselele ukwahlula phakathi kwazo zonke izinto ezigciniweyo esizixelele ngazo, kwaye ikwavumela ukukopishwa, o.k.t. uya kuba nokwahlulwa kunye nekopi.
I-Vminsert ixhasa i-OpenTSDB, iGraphite, i-InfluxDB kunye ne-remoteWrite protocols ezivela kwi-Prometheus.
Kukho kwakhona vmselect. Umsebenzi wayo ophambili kukuya kwi-vmstorage, ufumane idatha kubo, ukhuphe le datha kwaye uyinike umxhasi.
Kukho into emangalisayo ebizwa ngokuba yi-vmagent. Simthanda ngokwenene. Ikuvumela ukuba uqwalasele kanye njengePrometheus kwaye wenze yonke into kanye njengePrometheus. Oko kukuthi, iqokelela iimetrics ezivela kumaziko ahlukeneyo kunye neenkonzo kwaye ithumele kwi-vminsert. Emva koko yonke into ixhomekeke kuwe.
Enye inkonzo enkulu i-vmalert, ekuvumela ukuba usebenzise i-VictoriaMetrics njenge-backend, ufumane idatha eqhutywe kwi-vminsert kwaye uyithumele kwi-vmselect. Iqhuba izilumkiso ngokwazo, kunye nemithetho. Kwimeko yezivuseleli, sifumana isilumkiso ngomphathi we-alert.
Kukho icandelo le-wmauth. Sinako okanye asinako (asikagqibanga koku okwangoku) siyisebenzise njengenkqubo yogunyaziso yoguqulelo lwamaqela amaninzi. Ixhasa i-remoteBhala ye-Prometheus kwaye inokugunyazisa ngokusekwe kwi-url, okanye endaweni yoko inxalenye yesibini yayo, apho unako okanye ungenako ukubhala.
Kukho kwakhona vmbackup, vmrestore. Oku, ngokwenene, ukubuyisela kunye nokugcinwa kwedatha yazo zonke iinkcukacha. Unokwenza i-S3, i-GCS, ifayile.
Uphinda-phindo lokuqala lweqela lethu lwenziwa ngexesha lokuvalelwa. Ngelo xesha, kwakungekho replica, ngoko ke ukuphinda-phinda kwethu kwakubandakanya amaqela amabini ahlukeneyo nazimeleyo apho safumana khona idatha nge-remoteWrite.
Apha ndiza kwenza ugcino ukuba xa sitshintshela kwiVictoriaMetrics Single Node ukuya kwiVictoriaMetrics Cluster Version, sisahleli nezixhobo ezisetyenzisiweyo ezifanayo, okt eyona iphambili yinkumbulo. Oku kumalunga nendlela idatha yethu, o.k.t. ukusetyenziswa kwemithombo, eyasasazwa ngayo.
Umfuziselo sele wongezwe apha. Konke oku sikudibanise kwiqela elinye elikhulu. Yonke idatha yethu yabiwe kwaye iphindwe kabini.
Yonke i-cluster ineendawo zokungena ze-N, oku kuthetha ukuba i-Prometheus inokongeza idatha nge-HAPROXY. Apha sinale ndawo yokungena. Kwaye ngale ndawo yokungena ungangena usuka eGrafana.
Kwimeko yethu, i-HAPROXY kuphela kwezibuko ezikhethwa yi-proxies, zifake kunye nezinye iinkonzo ngaphakathi kweli qela. Kwimeko yethu, bekungenakwenzeka ukwenza idilesi enye; kuye kwafuneka senze iindawo ezininzi zokungena, kuba oomatshini abakhoyo ngokwabo apho iqela leVictoriaMetrics liqhuba libekwe kwiindawo ezahlukeneyo zomboneleli welifu omnye, okt hayi ngaphakathi kwilifu lethu, kodwa ngaphandle. .
Sinesilumkiso. Siyayisebenzisa. Sisebenzisa i-alertmanager evela kwi-Prometheus. Sisebenzisa i-Opsgenie kunye neTelegram njengejelo lokuhambisa isilumkiso. KwiTelegram bagalela kwi-dev, mhlawumbi into evela kwi-prod, kodwa ubukhulu becala into yamanani, efunwa ziinjineli. Kwaye i-Opsgenie ibalulekile. Ezi ziifowuni, ulawulo lwezehlo.
Umbuzo ongunaphakade: "Ngubani obeka iliso?" Kwimeko yethu, ukubeka iliso esweni ngokwayo, kuba sisebenzisa vmagent kwindawo nganye. Kwaye ekubeni ii-node zethu zihanjiswa kumaziko ahlukeneyo edatha yomnikezeli ofanayo, isikhungo ngasinye sedatha sineshaneli yaso, sizimele, kwaye nangona ubuchopho obuhlukeneyo bufika, siya kufumana izilumkiso. Ewe, kuya kuba ninzi kubo, kodwa kungcono ukufumana izilumkiso ezininzi kunokungabikho.
Sigqiba uluhlu lwethu ngokuphunyezwa kwe-HA.
Kwaye ngakumbi ndingathanda ukuqaphela amava okunxibelelana noluntu lweVictoriaMetrics. Kwabonakala kukuhle kakhulu. Abafana bayasabela. Bazama ukuphanda kuzo zonke iimeko ezinikezelwayo.
Ndiqale imiba kwiGitHub. Zaconjululwa ngokukhawuleza. Kukho imiba embalwa engakumbi engavalwanga ngokupheleleyo, kodwa sele ndiyibona kwikhowudi esebenza kweli cala iqhubekayo.
Intlungu ephambili kum ngexesha lokuphindaphinda kukuba ukuba ndivale i-node, ngoko imizuzwana ye-30 yokuqala i-vminsert ayikwazanga ukuqonda ukuba akukho mva. Oku kugqitywe ngoku. Kwaye ngokoqobo okwesibini okanye ezimbini, idatha ithathwa kuzo zonke iindawo eziseleyo, kwaye isicelo siyayeka ukulinda loo node engekho.
Ngexesha elithile sasifuna ukuba iVictoriaMetrics ibe ngumqhubi weVictoriaMetrics. Siye samlinda. Ngoku ngokukhutheleyo sakha isakhelo kumqhubi we-VictoriaMetrics ukuba athathe yonke imithetho yangaphambili yokubala, njl.
Kukho izindululo zokuphucula ukuphunyezwa kweqela. Ndizichaze ngasentla.
Kwaye ndifuna ngokwenene ukuthoba isampulu. Kwimeko yethu, i-downsampling iyafuneka kuphela kwiindlela zokujonga. Xa sithetha nje, imetric enye yanele kum emini. Ezi ndlela zifunekayo unyaka, emithathu, emihlanu, iminyaka elishumi. Kwaye ixabiso elinye lemetric lanele.
- Siyazi iintlungu, njengabanye oogxa bethu, xa usebenzisa iPrometheus.
- Sizikhethele iVictoriaMetrics.
- Ilinganisa kakuhle ngokuthe nkqo nangokuthe tye.
- Singakwazi ukusabalalisa amacandelo ahlukeneyo kumanani ahlukeneyo ee-nodes kwiqela, ukukhawulela ngememori, ukongeza imemori, njl.
Siza kusebenzisa iVictoriaMetrics ekhaya kuba siyithande kakhulu. Yile nto yayiyiyo kwaye iye yaba yintoni na.
Iikhowudi ezimbalwa ze-QR zencoko yeVictoriaMetrics, abafowunelwa bam, iLeroyMerlin radar yobuchwephesha.
umthombo: www.habr.com