VictoriaMetrics ndeye DBMS inokurumidza uye inotyisa yekuchengetedza uye kugadzirisa data muchimiro chenguva yakatevedzana (rekodhi ine nguva uye seti yehukoshero inoenderana nenguva ino, semuenzaniso, inowanikwa kuburikidza nekuvhota nguva nenguva kwechimiro che sensors kana kuunganidzwa kwemetrics).
Zita rangu ndiKolobaev Pavel. DevOps, SRE, LeroyMerlin, zvese zvakaita sekodhi - zvese nezvedu: nezvangu uye nezve vamwe vashandi veLeroyMerlin.
Kune gore rakavakirwa paOpenStack. Pane diki link kune technical radar.
Iyo yakavakirwa paKubernetes Hardware, uye pane ese ane hukama masevhisi eOpenStack uye matanda.
Ichi ndicho chirongwa chataive nacho mubudiriro. Patakanga tichigadzira zvese izvi, taive nePrometheus opareta yaichengeta data mukati meK8s cluster pachayo. Anobva angowana chinoda kukweshwa obva ochiisa pasi petsoka dzake.
Tichada kufambisa data rese kunze kweKubernetes cluster, nekuti kana chimwe chinhu chikaitika, isu tinofanirwa kunzwisisa kuti chii uye kupi.
Mhinduro yekutanga ndeyekuti isu tinoshandisa mubatanidzwa kana isu tine wechitatu-bato Prometheus, patinoenda kuKubernetes cluster kuburikidza nemubatanidzwa maitiro.
Asi pane mamwe matambudziko madiki pano. Muchiitiko chedu, matambudziko akatanga patakanga tine 250 metrics, uye paive ne000 metrics, takaona kuti hatingagoni kushanda saizvozvo. Takawedzera scrape_timeout kusvika 400 seconds.
Nei taifanira kuita izvi? Prometheus anotanga kuverenga nguva yekubuda kubva pakutanga kwefenzi. Izvo hazvina basa kuti data ichiri kuyerera. Kana panguva iyi yakatarwa data haina kubatanidzwa uye chikamu chisina kuvharwa kuburikidza ne http, saka chikamu chinoonekwa sechakundikana uye iyo data haipinde muPrometheus pachayo.
Wese munhu anoziva magirafu atinowana kana imwe data isipo. Zvirongwa zvakabvaruka uye isu hatisi kufara nazvo.
Iyo inotevera sarudzo ndeye sharding yakavakirwa maviri akasiyana Prometheus kuburikidza neiyo mubatanidzwa maitiro.
Semuenzaniso, ingozvitora wozvigovanisa nemazita. Izvi zvinogonawo kushandiswa, asi takasarudza kuenderera mberi.
Iye zvino tichafanirwa kugadzirisa shards idzi neimwe nzira. Iwe unogona kutora promxy, inoenda kunzvimbo yeshard uye inowedzera data. Inoshanda nemashadhi maviri senzvimbo imwe yekupinda. Izvi zvinogona kuitwa kuburikidza nepromxy, asi zvakanyanya kuoma.
Sarudzo yekutanga ndeyekuti tinoda kusiya mashandiro emubatanidzwa nekuti ari kunonoka.
Vagadziri vePrometheus vari kutaura zvakajeka, "Varume, shandisai TimescaleDB yakasiyana nekuti isu hatitsigire kuchengetwa kwenguva refu kwema metrics." Iri harisi basa ravo.
Tinonyora pasi pabepa ratichiri kuda kuburitsa panze, kuitira kuti tisachengete zvinhu zvose panzvimbo imwe chete.
Yechipiri drawback ndeyekuyeuka kushandiswa. Ehe, ndinonzwisisa kuti vazhinji vachataura kuti muna 2020 akati wandei magigabytes ekurangarira anodhura peni, asi zvakadaro.
Iye zvino tine dev uye prod nharaunda. Mu dev inenge 9 gigabytes ye350 metrics. Muprod ine 000 gigabytes uye zvishoma pamusoro pe14 metrics. Panguva imwecheteyo, nguva yedu yekuchengetedza inongova maminitsi makumi matatu. Izvi zvakaipa. Uye zvino ndichatsanangura chikonzero nei.
Isu tinoita kuverenga, ndiko kuti, nemiriyoni imwe nehafu metrics, uye isu tatova pedyo navo, padanho rekugadzira tinowana 35-37 gigabytes yekuyeuka. Asi atova mamiriyoni mana metrics anoda anenge 4 gigabytes yendangariro. Ndiko kuti, yakaverengerwa uchishandisa fomula yakapihwa nevagadziri vePrometheus. Takatarisa kuwirirana tikaona kuti isu taisada kubhadhara mamirioni akati wandei sevhavha yekungotarisa.
Isu hatizongowedzera huwandu hwemakina, isu tiri kutarisa iwo chaiwo machina iwo. Naizvozvo, iwo akanyanya machina michina, anowedzera metrics emhando dzakasiyana, nezvimwe. Tichava nekukura kwakakosha kwesumbu redu maererano nemametrics.
Ne diski nzvimbo, hazvisi zvese zvakashata pano, asi ndinoda kuivandudza. Takagamuchira huwandu hwe15 gigabytes mumazuva gumi nemashanu, ayo zana ari akadzvanywa data, makumi maviri ari uncompressed data, asi isu tinogara tichida zvishoma.
Saizvozvo, isu tinonyora pasi imwezve pfungwa - uku ndiko kushandiswa kukuru kwezviwanikwa, izvo zvatichiri kuda kuchengetedza, nekuti hatidi yedu yekutarisa cluster kuti idye zvimwe zviwanikwa kupfuura cluster yedu, iyo inotarisira OpenStack.
Pane imwezve dhizaini yePrometheus, yatakazvionera pachedu, iyi ingangoita imwe mhando yekumisa ndangariro. NaPrometheus, zvese zvakanyanya kuipa pano, nekuti haina mamonyoro akadaro zvachose. Kushandisa muganho mu docker zvakare haisi sarudzo. Kana kamwe kamwe RAF yako yakawa uye kune 20-30 gigabytes, zvino zvichatora nguva yakareba kwazvo kusimuka.
Ichi ndicho chimwe chikonzero nei Prometheus isina kukodzera isu, i.e. hatigone kudzikamisa ndangariro kushandiswa.
Zvingabvira kuuya nechirongwa chakadaro. Tinoda chirongwa ichi kuitira kuronga HA sumbu. Tinoda kuti mametrics edu avepo nguva dzese uye kwese kwese, kunyangwe sevha inochengeta metrics iyi ikaparara. Uye nekudaro tichafanira kuvaka chirongwa chakadaro.
Ichi chirongwa chinoti tichave nekudzokororwa kweshadhi, uye, maererano, kudzokorora kwemitengo yezvinhu zvakadyiwa. Inogona kuyerwa zvakadarikidza zvakadzika, asi zvakadaro mashandisirwo ezviwanikwa achave gehena.
Zvakashata mukurongeka muchimiro chatakazvinyora pasi isu pachedu:
- Inoda kurodha metrics kunze.
- High resource kushandiswa.
- Iko hakuna nzira yekudzikamisa ndangariro kushandiswa.
- Yakaomarara uye zviwanikwa-yakanyanya kuita kweHA.
Isu pachedu, takasarudza kuti taisimuka kubva kuPrometheus senzvimbo yekuchengetedza.
Taona zvimwe zvinodiwa pachedu zvatinoda. Izvi:
- Uku kutsigirwa kwepromql, nekuti zvinhu zvakawanda zvakatonyorerwa Prometheus: mibvunzo, chenjedzo.
- Uye zvino tine Grafana, iyo yakatonyorwa nenzira imwecheteyo yePrometheus semusana. Handidi kunyora patsva madhibhodhi.
- Tinoda kuvaka yakajairwa HA architecture.
- Tinoda kuderedza kushandiswa kwezvimwe zviwanikwa.
- Pane imwe nuance diki. Hatigone kushandisa akasiyana marudzi egore metrics yekuunganidza masisitimu. Hatizivi kuti chii chichawira mumametric aya parizvino. Uye sezvo chinhu chipi nechipi chichigona kubhururuka ikoko, tinofanira kuzviganhurira timene pakuiswa kwomunzvimbomo.
Pakanga pasina sarudzo. Takaunganidza zvese zvataive tasangana nazvo. Takatarisa peji rePrometheus muchikamu chekubatanidza, takaverenga boka rezvinyorwa, uye takaona zvaive kunze uko. Uye isu pachedu, takasarudza VictoriaMetrics sechinzvimbo chePrometheus.
Sei? Nokuti:
- Anoziva promql.
- Kune modular architecture.
- Hazvidi shanduko kuGrafana.
- Uye zvinonyanya kukosha, isu tichazopa metrics ekuchengetedza mukati mekambani yedu sevhisi, saka isu tiri kutarisa mberi kune zvirambidzo zvemarudzi akasiyana-siyana kuitira kuti vashandisi vagone kushandisa zvese zviwanikwa zve cluster neimwe nzira shoma, nekuti pane mukana. kuti zvichaita multitenancy.
Ngatiite kuenzanisa kwekutanga. Isu tinotora yakafanana Prometheus mukati mesumbu, yekunze Prometheus inoenda kwairi. Wedzera kuburikidza nekureWrite VictoriaMetrics.
Ini ndichakurumidza kugadzirisa kuti pano takabata kuwedzera kudiki kweCPU kushandiswa kubva kuVictoriaMetrics. Iyo VictoriaMetrics wiki inokuudza kuti ndeapi ma paramita ari nani. Takavatarisa. Ivo vakaderedza CPU kushandiswa zvakanyanya.
Muchiitiko chedu, kushandiswa kwekuyeuka kwePrometheus, iyo iri muKubernetes cluster, haina kuwedzera zvakanyanya.
Isu tinofananidza maviri masosi data data imwechete. MuPrometheus tinoona yakafanana data isipo. Zvese zvakanaka paVictoriaMetrics.
Disk space test results. Isu paPrometheus takagamuchira 120 gigabytes pamwe chete. KuVictoriaMetrics tinotogamuchira 4 gigabytes pazuva. Pane nzira yakati siyanei pane yatakajaira kuona muPrometheus. Ndiko kuti, iyo data yatove yakamanikidzwa chaizvo muzuva, muhafu yeawa. Vakatove vakakohwa zvakanaka muzuva, muhafu yeawa, zvisinei nekuti iyo data icharamba yakarasika gare gare. Nekuda kweizvozvo, takachengeta pane dhisiki nzvimbo.
Isu tinochengetedzawo pane memory resource yekushandisa. Panguva yekuyedzwa, takanga tine Prometheus akaiswa pamushini chaiwo - 8 cores, 24 gigabytes. Prometheus anodya zvinenge zvese. Akawira paOOM Killer. Panguva imwecheteyo, 900 chete metrics inoshanda yakadururwa mairi. Izvi zvingangoita 000-25 metrics pasekondi.
Takamhanyisa VictoriaMetrics pamuchina weviri-musimboti chaiwo une 8 gigabytes ye RAM. Isu takakwanisa kuita kuti VictoriaMetrics ishande nemazvo nekutamba-tamba nezvinhu zvishoma pamushini we8GB. Pakupedzisira, takaichengeta kune 7 gigabytes. Panguva imwecheteyo, kumhanya kwekutumira zvemukati, i.e. metrics, yaive yakatokwira kupfuura iyo yePrometheus.
Iyo CPU yave nani zvakanyanya kana ichienzaniswa nePrometheus. Pano Prometheus inoshandisa 2,5 cores, uye VictoriaMetrics inongoshandisa 0,25 cores. Pakutanga - 0,5 cores. Sezvo ichibatanidza, inosvika pane imwe core, asi izvi zvakanyanya, zvisingawanzo.
Kwatiri, sarudzo yakawira paVictoriaMetrics nezvikonzero zviri pachena; isu taida kuchengetedza mari uye takaita.
Ngatibvise mapoinzi maviri ipapo - kurodha metrics uye kushandiswa kwakanyanya kwezviwanikwa. Uye isu tinongofanira kusarudza zvibodzwa zviviri zvatichiri kuzvisiira.
Pano ini ndichaita kubhuka ipapo ipapo, isu tinofunga VictoriaMetrics sekuchengetedza metrics. Asi sezvo isu tichizonyanya kupa VictoriaMetrics sechengetedzo kune ese Leroy, isu tinofanirwa kudzikamisa avo vachashandisa cluster iyi kuti varege kutipa.
Iko kune inoshamisa paramende iyo inokutendera iwe kudzikamisa nenguva, nehuwandu hwe data uye nenguva yekuuraya.
Kune zvakare yakanakisa sarudzo inotibvumira kudzikamisa ndangariro kushandiswa, nekudaro tinogona kuwana iyo yakaenzana iyo inotitendera kuwana yakajairika yekushanda kumhanya uye yakakwana yekushandisa zviwanikwa.
Minus imwe poindi, kureva kuyambuka poindi - haugone kudzikamisa ndangariro kushandiswa.
Mune yekutanga iterations, takaedza VictoriaMetrics Single Node. Tevere tinoenda kuVictoriaMetrics Cluster Version.
Pano isu tine ruoko rwemahara rwekuparadzanisa masevhisi akasiyana muVictoriaMetrics zvichienderana nekuti vachamhanya nei uye kuti vachashandisa zviwanikwa zvipi. Iyi imhinduro inoshanduka uye iri nyore. Takashandisa izvi pachedu.
Izvo zvakakosha zveVictoriaMetrics Cluster Version ndeye vmstsorage. Panogona kuva N nhamba yavo. Kwedu kune 2 vavo kusvika ikozvino.
Uye kune vminsert. Iyi iproxy server inotibvumira ku: kuronga sharding pakati pezvinhu zvese zvatakazviudza nezvazvo, uye zvakare inobvumira replica, i.e. iwe uchave uine zvese sharding uye replica.
Vminsert inotsigira OpenTSDB, Graphite, InfluxDB uye kureWrite protocol kubva kuPrometheus.
Kune zvakare vmselect. Basa rayo guru nderekuenda kuvmstorage, kugamuchira data kubva kwavari, kubvisa iyi data uye kupa kune mutengi.
Pane chinhu chinoshamisa chinonzi vmagent. Tinomuda chaizvo. Iyo inokutendera kuti ugadzirise chaizvo sePrometheus uye uchiri kuita zvese chaizvo sePrometheus. Ndiko kuti, inounganidza metrics kubva kune akasiyana masangano uye masevhisi uye inotumira kune vminsert. Zvadaro zvinhu zvose zvinoenderana newe.
Imwe sevhisi huru ndeye vmalert, iyo inokutendera iwe kushandisa VictoriaMetrics sebackend, gamuchira yakagadziridzwa data kubva kuvminsert uye utumire kune vmselect. Inogadzirisa zviziviso pachavo, pamwe chete nemitemo. Panyaya yezviziviso, tinogashira yambiro kuburikidza nealertmanager.
Pane wmauth chikamu. Isu tinogona kana kusadaro (hatisati tasarudza pane izvi parizvino) kuishandisa semvumo system yeiyo multitenancy vhezheni yemasumbu. Inotsigira kureWrite yePrometheus uye inogona kubvumidza zvichibva pane url, kana kuti chikamu chechipiri chayo, kwaunokwanisa kana kusanyora.
Kune zvakare vmbackup, vmrestore. Uku ndiko, muchidimbu, kudzoreredza uye kuchengetedza data rese. Inogona kuita S3, GCS, faira.
Yekutanga iteration yecluster yedu yakaitwa panguva yekuvharirwa. Panguva iyoyo, pakanga pasina replica, saka iteration yedu yaive nemapoka maviri akasiyana uye akazvimirira matakagamuchira data kuburikidza neremoteWrite.
Pano ini ndichachengetedza kuti patakachinja kubva kuVictoriaMetrics Single Node kuenda kuVictoriaMetrics Cluster Version, isu takaramba tiine zviwanikwa zvakadyiwa, kureva kuti chikuru ndangariro. Aya ndiwo mafambisirwo akaitwa data redu, kureva kushandisa zviwanikwa.
Replica yakatowedzerwa pano. Takabatanidza zvese izvi kuita sumbu rimwe rakakura. Yese data yedu yakagovaniswa uye yakadzokororwa.
Iyo yese cluster ine N ekupinda mapoinzi, i.e. Prometheus inogona kuwedzera data kuburikidza neHAPROXY. Pano tine ino yekupinda poindi. Uye kuburikidza neiyi poindi yekupinda iwe unogona kupinda kubva kuGrafana.
Kwatiri, HAPROXY ndiyo chete chiteshi chinosarudzwa nemaproxies, kuisa uye mamwe masevhisi mukati meboka iri. Kwatiri, zvanga zvisingaite kugadzira kero imwe chete; taifanira kugadzira akati wandei mapoinzi ekupinda, nekuti iwo machina chaiwo ayo eVictoriaMetrics cluster anomhanya ari munzvimbo dzakasiyana dzemupi wegore mumwechete, kureva kwete mukati megore redu, asi kunze. .
Tine yambiro. Isu tinoishandisa. Isu tinoshandisa alertmanager kubva kuPrometheus. Isu tinoshandisa Opsgenie uye Telegraph seyambiro yekuendesa chiteshi. MuTeregiramu vanodurura kubva ku dev, pamwe chimwe chinhu kubva kuprod, asi kazhinji chimwe chinhu chenhamba, chinodiwa nevainjiniya. Uye Opsgenie yakakosha. Aya ndiwo mafoni, manejimendi ezviitiko.
Mubvunzo usingaperi: "Ndiani anotarisa kutarisa?" Mune yedu kesi, monitors monitors inozvitarisa pachayo, nekuti isu tinoshandisa vmagent pane imwe neimwe node. Uye sezvo node dzedu dzichigoverwa munzvimbo dzakasiyana dze data dzemupi mumwe chete, nzvimbo yega yega yedata ine chiteshi chayo, ivo vakazvimiririra, uye kunyangwe kana huropi hwakapatsanuka hwasvika, isu tichangogamuchira ziviso. Hongu, pachava nevakawanda vavo, asi zviri nani kugamuchira chenjedzo dzakawanda pane kusagamuchira.
Isu tinopedzisa runyorwa rwedu neHA kuita.
Uye zvakare ndinoda kucherechedza chiitiko chekutaurirana nenharaunda yeVictoriaMetrics. Zvakava zvakanaka kwazvo. Vakomana vanopindura. Vanoedza kuongorora nyaya dzese dzinopihwa.
Ndakatanga nyaya paGitHub. Dzakagadziriswa nokukurumidza. Pane dzimwe nyaya dzinoverengeka dzisina kuvharwa zvachose, asi ini ndiri kutoona kubva kune kodhi inoshanda mune ino nzira iri kuitika.
Kurwadza kukuru kwandiri panguva yekudzokorora kwaiva kuti kana ndikavhara node, ipapo kwekutanga 30 seconds vminsert haigoni kunzwisisa kuti kwakanga kusina backend. Izvi zvino zvasarudzwa. Uye chaizvo musekondi kana maviri, iyo data inotorwa kubva kune ese asara node, uye chikumbiro chinomira kumirira icho chisipo node.
Pane imwe nguva taida kuti VictoriaMetrics ive VictoriaMetrics opareta. Takamumirira. Isu tiri kushingaira kuvaka hurongwa hweVictoriaMetrics opareta kuti atore ese pre-calculating mitemo, etc. Prometheus, nekuti isu tiri chaizvo kushingaira kushandisa mitemo inouya nePrometheus operator.
Pane zvikumbiro zvekuvandudza macluster kuita. Ndakavarondedzera pamusoro.
Uye ini ndinoda chaizvo kuderedza sampuli. Kwatiri isu, kudzikisa pasi kunodiwa chete pakuona maitiro. Kutaura zvazviri, metric imwe inondikwanira mukati mezuva. Aya maitiro anodiwa kwegore, matatu, mashanu, makore gumi. Uye imwe metric kukosha yakakwana.
- Isu takaziva kurwadziwa, sezvakaita vamwe vatinoshanda navo, pakushandisa Prometheus.
- Isu takasarudza VictoriaMetrics isu pachedu.
- Inoyera zvakanaka zvese zvakadzika uye zvakachinjika.
- Tinogona kugovera zvikamu zvakasiyana kune nhamba dzakasiyana dzenodhi musumbu, kudzimisa nendangariro, kuwedzera ndangariro, nezvimwe.
Tichashandisa VictoriaMetrics kumba nekuti takaifarira chaizvo. Izvi ndizvo zvaive uye zvavepo.
Mamwe maQR makodhi eVictoriaMetrics chat, vandinosangana navo, LeroyMerlin technical radar.
Source: www.habr.com