VictoriaMetrics ndi DBMS yachangu komanso yowopsa yosungira ndikusintha zidziwitso mumndandanda wanthawi (mbiri imakhala ndi nthawi ndi zinthu zomwe zikugwirizana ndi nthawi ino, mwachitsanzo, zopezedwa kudzera pakuvotera kwakanthawi kwa sensa kapena kusonkhanitsa ma metrics).
Dzina langa ndine Kolobaev Pavel. DevOps, SRE, LeroyMerlin, chirichonse chiri ngati code - zonse za ife: za ine ndi antchito ena a LeroyMerlin.
Pali mtambo wozikidwa pa OpenStack. Pali ulalo wocheperako ku radar yaukadaulo.
Zimamangidwa pa hardware ya Kubernetes, komanso pa ntchito zonse zokhudzana ndi OpenStack ndi kudula mitengo.
Iyi ndi ndondomeko yomwe tinali nayo pachitukuko. Pamene tinali kupanga zonsezi, tinali ndi woyendetsa Prometheus yemwe amasunga deta mkati mwa gulu la K8s lokha. Amangopeza zomwe zikufunika kuchapa ndikuziika pansi pa mapazi ake, kunena mosapita m'mbali.
Tidzafunika kusuntha deta yonse kunja kwa gulu la Kubernetes, chifukwa ngati chinachake chikuchitika, tiyenera kumvetsetsa kuti ndi chiyani.
Yankho loyamba ndiloti timagwiritsa ntchito federation tikakhala ndi Prometheus wachitatu, tikapita ku gulu la Kubernetes kudzera mu federal.
Koma pali mavuto ang'onoang'ono apa. Kwa ife, mavuto adayamba pamene tinali ndi ma metric 250, ndipo pamene panali ma metric 000, tinazindikira kuti sitingathe kugwira ntchito monga choncho. Tinachulukitsa scrape_timeout kufika masekondi 400.
N’cifukwa ciani tiyenela kucita zimenezi? Prometheus akuyamba kuwerengera nthawi yopuma kuyambira pachiyambi cha mpanda. Zilibe kanthu kuti deta ikuyendabe. Ngati panthawiyi deta sichikuphatikizidwa ndipo gawolo silinatsekedwe kudzera pa http, ndiye kuti gawoli likuonedwa kuti lalephera ndipo deta siilowa mu Prometheus yokha.
Aliyense amadziwa ma graph omwe timapeza pomwe zina zikusowa. Madongosolo adang'ambika ndipo sitikukondwera ndi izi.
Njira yotsatira ndikugawana kutengera ma Prometheus awiri osiyanasiyana kudzera mumgwirizano womwewo.
Mwachitsanzo, ingowatenga ndi kuwagawa ndi mayina. Izi zitha kugwiritsidwanso ntchito, koma tinaganiza zopita patsogolo.
Tsopano tiyenera kukonza shards izi mwanjira ina. Mutha kutenga promxy, yomwe imapita kudera la shard ndikuchulukitsa deta. Zimagwira ntchito ndi shards ziwiri ngati malo amodzi olowera. Izi zitha kukhazikitsidwa kudzera pa promxy, komabe ndizovuta kwambiri.
Njira yoyamba ndiyoti tikufuna kusiya machitidwe a federal chifukwa ndi ochedwa kwambiri.
Madivelopa a Prometheus akunena momveka bwino, "Anyamata, gwiritsani ntchito TimecaleDB yosiyana chifukwa sitidzathandizira kusungirako kwa nthawi yaitali kwazitsulo." Iyi si ntchito yawo.
Timalemba papepala lomwe timafunikirabe kutsitsa panja, kuti tisasunge chilichonse pamalo amodzi.
Yachiwiri drawback ndi kukumbukira kukumbukira. Inde, ndikumvetsa kuti ambiri adzanena kuti mu 2020 ma gigabytes angapo okumbukira amawononga khobiri, komabe.
Tsopano tili ndi chilengedwe cha dev ndi prod. Mu dev ndi pafupifupi 9 gigabytes kwa 350 metrics. Mu prod ndi 000 gigabytes ndi ma metric opitilira 14. Nthawi yomweyo, nthawi yathu yosungira ndi mphindi 780 zokha. Izi ndi zoipa. Ndipo tsopano ndifotokoza chifukwa chake.
Timawerengera, ndiko kuti, ndi ma metrics miliyoni imodzi ndi theka, ndipo takhala pafupi nawo, pamapangidwe timapeza 35-37 gigabytes ya kukumbukira. Koma kale ma metric 4 miliyoni amafunikira kukumbukira kwa 90 gigabytes. Ndiye kuti, idawerengedwa pogwiritsa ntchito njira yoperekedwa ndi opanga Prometheus. Tidayang'ana kulumikizana ndikuzindikira kuti sitikufuna kulipira mamiliyoni angapo pa seva kuti tingoyang'anira.
Sitidzangowonjezera kuchuluka kwa makina, tikuwunikanso makina enieniwo. Chifukwa chake, makina ochulukirachulukira, ma metric amitundu yosiyanasiyana, ndi zina zambiri. Tidzakhala ndi kukula kwapadera kwa gulu lathu potengera ma metric.
Ndi disk space, sikuti zonse ndizoyipa kwambiri pano, koma ndikufuna kuzikonza. Tinalandira ma gigabytes okwana 15 m'masiku 120, omwe 100 ndi deta yoponderezedwa, 20 ndi deta yosakanizidwa, koma nthawi zonse timafuna zochepa.
Chifukwa chake, timalembanso mfundo imodzi - iyi ndikugwiritsa ntchito zinthu zambiri, zomwe tikufunabe kupulumutsa, chifukwa sitikufuna kuti gulu lathu lowunikira liwononge zinthu zambiri kuposa gulu lathu, lomwe limayang'anira OpenStack.
Pali vuto linanso la Prometheus, lomwe tadzipezera tokha, uwu ndi mtundu wina wa kukumbukira kukumbukira. Ndi Prometheus, zonse ndizoyipa kwambiri pano, chifukwa zilibe zokhotakhota. Kugwiritsa ntchito malire mu docker sikungakhalenso mwayi. Ngati mwadzidzidzi RAF yanu idagwa ndipo pali 20-30 gigabytes, ndiye kuti idzatenga nthawi yayitali kuti iwuke.
Ichi ndi chifukwa china chomwe Prometheus sichiri choyenera kwa ife, mwachitsanzo, sitingathe kuchepetsa kukumbukira kukumbukira.
Zingakhale zotheka kubwera ndi ndondomeko yoteroyo. Tikufuna dongosolo ili kuti tikonze gulu la HA. Tikufuna kuti ma metric athu azipezeka nthawi zonse komanso paliponse, ngakhale seva yomwe imasunga ma metrics awa itagwa. Ndipo potero tidzayenera kupanga chiwembu choterocho.
Dongosololi likunena kuti tidzakhala ndi kubwereza kwa shards, ndipo, molingana, kubwereza kwa ndalama zomwe zimagwiritsidwa ntchito. Itha kuwongoleredwa pafupifupi mopingasa, koma kugwiritsa ntchito zinthu kumakhala koyipa.
Zoyipa mwa dongosolo lomwe tidazilembera tokha:
- Pamafunika kukweza ma metrics kunja.
- Kugwiritsa ntchito zinthu zambiri.
- Palibe njira yochepetsera kukumbukira kukumbukira.
- Kukhazikitsa kosavuta komanso kogwiritsa ntchito kwambiri kwa HA.
Kwa ife tokha, tinaganiza kuti tikuchoka ku Prometheus monga malo osungiramo zinthu.
Tapeza zofunika zinanso kwa ife tokha zimene timafunikira. Izi:
- Ichi ndi chithandizo cha promql, chifukwa zinthu zambiri zalembedwa kale kwa Prometheus: mafunso, zidziwitso.
- Ndiyeno tili ndi Grafana, yomwe yalembedwa kale chimodzimodzi kwa Prometheus monga backend. Sindikufuna kulembanso ma dashboards.
- Tikufuna kupanga zomanga za HA zabwinobwino.
- Tikufuna kuchepetsa kugwiritsa ntchito zinthu zilizonse.
- Palinso kagawo kakang'ono kakang'ono. Sitingathe kugwiritsa ntchito mitundu yosiyanasiyana yosonkhanitsira ma metrics mumtambo. Sitikudziwa zomwe zidzagwere muzitsulozi. Ndipo popeza kuti chilichonse chikhoza kuwuluka kumeneko, tiyenera kungokhala ndi malo akumaloko.
Panalibe chosankha. Tinasonkhanitsa zonse zomwe tinali nazo. Tidayang'ana patsamba la Prometheus mugawo lophatikizira, tidawerenga zolemba zambiri, ndikuwona zomwe zidalipo. Ndipo kwa ife tokha, tinasankha VictoriaMetrics m'malo mwa Prometheus.
Chifukwa chiyani? Chifukwa:
- Mutha kuchita promql.
- Pali zomanga modular.
- Sichifuna kusintha kwa Grafana.
- Ndipo chofunika kwambiri, tidzapereka ma metrics yosungirako mkati mwa kampani yathu ngati ntchito, kotero tikuyang'ana pasadakhale zoletsa zamitundu yosiyanasiyana kuti ogwiritsa ntchito agwiritse ntchito zonse zomwe zili mgululi mwa njira yochepa, chifukwa pali mwayi. kuti adzakhala multitenancy.
Tiyeni tifanizire koyamba. Timatenga Prometheus yemweyo mkati mwa tsango, Prometheus wakunja amapitako. Onjezani kudzera pa remoteWrite VictoriaMetrics.
Ndisungitsa nthawi yomweyo kuti pano tidakwera pang'ono kugwiritsa ntchito CPU kuchokera ku VictoriaMetrics. VictoriaMetrics wiki imakuwuzani magawo omwe ali abwino kwambiri. Tinawafufuza. Achepetsa kugwiritsa ntchito CPU bwino kwambiri.
Kwa ife, kugwiritsa ntchito kukumbukira kwa Prometheus, komwe kuli m'gulu la Kubernetes, sikunachuluke kwambiri.
Timayerekeza magwero awiri a data omwe ali ofanana. Ku Prometheus tikuwona zomwezo zomwe zikusowa. Zonse zili bwino ku VictoriaMetrics.
Zotsatira za mayeso a danga la Disk. Ife ku Prometheus tinalandira gigabytes 120 pamodzi. Ku VictoriaMetrics timalandira kale ma gigabytes 4 patsiku. Pali njira yosiyana pang'ono kuposa yomwe timazolowera ku Prometheus. Ndiko kuti, deta kale wothinikizidwa bwino ndithu tsiku, mu theka la ola. Iwo akolola kale bwino mu tsiku, mu theka la ola, ngakhale kuti deta idzatayikabe pambuyo pake. Zotsatira zake, tidasunga malo a disk.
Timasunganso pakugwiritsa ntchito kukumbukira. Pa nthawi yoyesedwa, tinali ndi Prometheus atayikidwa pamakina enieni - 8 cores, 24 gigabytes. Prometheus amadya pafupifupi chilichonse. Anagwa pa OOM Killer. Nthawi yomweyo, ma metrics 900 okha adatsanuliridwa momwemo. Izi ndi pafupifupi ma metric 000-25 pamphindikati.
Tidayendetsa VictoriaMetrics pamakina apawiri-core omwe ali ndi 8 gigabytes a RAM. Tidakwanitsa kuti VictoriaMetrics igwire ntchito bwino polimbana ndi zinthu zingapo pamakina a 8GB. Pamapeto pake, tinasunga ku 7 gigabytes. Nthawi yomweyo, liwiro la kutumiza zomwe zili, mwachitsanzo, ma metrics, anali apamwamba kuposa a Prometheus.
CPU yakhala yabwinoko poyerekeza ndi Prometheus. Apa Prometheus amadya ma cores 2,5, ndipo VictoriaMetrics amangodya ma cores 0,25. Poyamba - 0,5 cores. Pamene ikuphatikizana, imafika pachimake chimodzi, koma izi ndizosowa kwambiri.
Kwa ife, chisankho chinagwera pa VictoriaMetrics pazifukwa zomveka; tinkafuna kusunga ndalama ndipo tinatero.
Tiyeni tidutse mfundo ziwiri nthawi yomweyo - kukweza ma metric ndi kugwiritsa ntchito kwambiri zinthu. Ndipo tingosankha mfundo ziwiri zomwe tidadzisiyira tokha.
Apa ndikusungitsa nthawi yomweyo, timawona VictoriaMetrics ngati malo osungira ma metric. Koma popeza titha kupereka VictoriaMetrics ngati malo osungira a Leroy onse, tikuyenera kuchepetsa omwe adzagwiritse ntchito gululi kuti asatipatse.
Pali gawo labwino kwambiri lomwe limakupatsani mwayi wochepetsera nthawi, kuchuluka kwa data komanso nthawi yochitira.
Palinso njira yabwino kwambiri yomwe imatilola kuti tichepetse kukumbukira kukumbukira, potero titha kupeza malire omwe angatilole kuti tipeze kuthamanga kwanthawi zonse komanso kugwiritsa ntchito zida zokwanira.
Chotsani mfundo imodzi, mwachitsanzo, tulutsani mfundoyo - simungathe kuchepetsa kugwiritsa ntchito kukumbukira.
M'mawu oyamba, tidayesa VictoriaMetrics Single Node. Kenako timapita ku VictoriaMetrics Cluster Version.
Pano tili ndi dzanja laulere lolekanitsa mautumiki osiyanasiyana mu VictoriaMetrics kutengera zomwe azigwiritsa ntchito komanso zomwe azigwiritsa ntchito. Iyi ndi njira yosinthika komanso yosavuta. Tinagwiritsa ntchito izi patokha.
Zigawo zazikulu za VictoriaMetrics Cluster Version ndi vmstsorage. Pakhoza kukhala N nambala yawo. Kwa ife pali 2 mwa iwo mpaka pano.
Ndipo pali vminsert. Iyi ndi seva ya proxy yomwe imatilola: kukonza sharding pakati pa zosungira zonse zomwe tidazifotokozera, komanso imalola chofanizira, mwachitsanzo, mudzakhala ndi sharding ndi chofananira.
Vminsert imathandizira OpenTSDB, Graphite, InfluxDB ndi ma protocol a remoteWrite ochokera ku Prometheus.
Palinso vmselect. Ntchito yake yayikulu ndikupita ku vmstorage, kulandira deta kuchokera kwa iwo, kugawa deta iyi ndikuipereka kwa kasitomala.
Pali chinthu chodabwitsa chotchedwa vmagent. Timamukonda kwambiri. Zimakulolani kuti musinthe chimodzimodzi monga Prometheus ndikuchitabe zonse chimodzimodzi monga Prometheus. Ndiye kuti, imasonkhanitsa ma metric kuchokera ku mabungwe ndi mautumiki osiyanasiyana ndikutumiza ku vminsert. Ndiye zonse zimadalira inu.
Ntchito ina yabwino ndi vmalert, yomwe imakulolani kuti mugwiritse ntchito VictoriaMetrics ngati kumbuyo, kulandira deta yosinthidwa kuchokera ku vminsert ndikutumiza ku vmselect. Imakonza machenjezo okha, komanso malamulo. Pankhani ya zidziwitso, timalandira chenjezo kudzera mwa alertmanager.
Pali gawo la wmauth. Titha kapena ayi (sitinasankhepo izi) tigwiritse ntchito ngati njira yololeza gulu lamagulu ambiri. Imathandizira RemoteWrite for Prometheus ndipo imatha kuvomereza kutengera ulalo, kapena m'malo mwake gawo lachiwiri, pomwe mungathe kulemba kapena simungathe kulemba.
Palinso vmbackup, vmrestore. Uku ndiko, kwenikweni, kubwezeretsa ndi kusunga deta yonse. Mutha kuchita S3, GCS, fayilo.
Kubwereza koyamba kwa gulu lathu kudapangidwa panthawi yotsekeredwa. Panthawiyo, kunalibe chofananira, kotero kubwereza kwathu kunali magulu awiri osiyana komanso odziyimira pawokha momwe tidalandilamo data kudzera pa remoteWrite.
Apa ndisungitsa kuti titasintha kuchoka ku VictoriaMetrics Single Node kupita ku VictoriaMetrics Cluster Version, tidakhalabe ndi zinthu zomwe zimagwiritsidwa ntchito, mwachitsanzo, chachikulu ndikukumbukira. Umu ndi momwe deta yathu, mwachitsanzo, kugwiritsa ntchito zinthu, idagawidwira.
Chofananira chawonjezedwa kale pano. Tinaphatikiza zonsezi kukhala gulu limodzi lalikulu. Zambiri zathu zonse zimagawidwa komanso kubwerezedwa.
Gulu lonse lili ndi malo olowera N, kutanthauza kuti Prometheus akhoza kuwonjezera deta kudzera HAPROXY. Apa tili ndi polowera. Ndipo kudzera mu malo olowerawa mutha kulowa kuchokera ku Grafana.
Kwa ife, HAPROXY ndiye doko lokhalo lomwe ma proxies amasankha, kuyika ndi ntchito zina mkati mwa gululi. Kwa ife, sikunali kotheka kupanga adilesi imodzi; tidayenera kupanga malo angapo olowera, chifukwa makina enieni omwe gulu la VictoriaMetrics limayendera amakhala m'malo osiyanasiyana operekera mtambo womwewo, mwachitsanzo, osati mkati mwa mtambo wathu, koma kunja. .
Tili ndi chenjezo. Timachigwiritsa ntchito. Timagwiritsa ntchito alertmanager kuchokera ku Prometheus. Timagwiritsa ntchito Opsgenie ndi Telegraph ngati njira yotumizira chenjezo. Mu Telegalamu amatsanulira kuchokera ku dev, mwina china chake kuchokera ku prod, koma makamaka chiwerengero, chofunikira ndi mainjiniya. Ndipo Opsgenie ndi wovuta. Awa ndi mafoni, kasamalidwe ka zochitika.
Funso losatha: "Ndani amayang'anira kuwunika?" Kwa ife, kuyang'anira kuyang'anira kumadziyang'anira okha, chifukwa timagwiritsa ntchito vmagent pa mfundo iliyonse. Ndipo popeza ma node athu amagawidwa m'malo osiyanasiyana a data a wothandizira yemweyo, malo aliwonse a deta ali ndi njira yake, iwo ndi odziimira okha, ndipo ngakhale ubongo wogawanika ukafika, tidzalandirabe machenjezo. Inde, padzakhala ochuluka a iwo, koma ndi bwino kulandira machenjezo ambiri kusiyana ndi kusakhalapo.
Timamaliza mndandanda wathu ndi kukhazikitsa kwa HA.
Ndipo kupitilira apo ndikufuna kudziwa zomwe zachitika polumikizana ndi gulu la VictoriaMetrics. Zinakhala zabwino kwambiri. Anyamata amayankha. Amayesa kufufuza nkhani zonse zomwe zaperekedwa.
Ndinayambitsa zovuta pa GitHub. Anathetsedwa mwamsanga kwambiri. Pali zina zingapo zomwe sizinatsekedwe kwathunthu, koma ndikutha kuwona kuchokera pama code omwe akugwira ntchito motere.
Chowawa chachikulu kwa ine panthawi yobwerezabwereza chinali chakuti ngati nditseka mfundo, ndiye kwa masekondi 30 oyambirira vminsert sakanakhoza kumvetsa kuti panalibe backend. Izi tsopano zagamulidwa. Ndipo kwenikweni mu sekondi imodzi kapena ziwiri, deta imatengedwa kuchokera ku mfundo zonse zotsalira, ndipo pempho limasiya kuyembekezera mfundo yosowa.
Panthawi ina tinkafuna kuti VictoriaMetrics ikhale opareshoni ya VictoriaMetrics. Tinamuyembekezera. Tsopano tikumanga mwakhama dongosolo la VictoriaMetrics kuti atenge malamulo onse owerengera, ndi zina zotero. Prometheus, chifukwa tikugwiritsa ntchito mwakhama malamulo omwe amabwera ndi Prometheus operator.
Pali malingaliro owongolera kukhazikitsidwa kwamagulu. Ndinazifotokoza pamwambapa.
Ndipo ine ndikufuna kwenikweni kuchepetsa. Kwa ife, kutsitsa kumafunika kokha kuti muwonere zomwe mumakonda. Kunena zowona, metric imodzi ndiyokwanira kwa ine masana. Izi zimafunika kwa chaka chimodzi, zitatu, zisanu, zaka khumi. Ndipo mtengo wa metric umodzi ndiwokwanira.
- Tadziwa zowawa, monganso ena mwa anzathu, akamagwiritsa ntchito Prometheus.
- Tinadzisankhira tokha VictoriaMetrics.
- Imakula bwino molunjika komanso mopingasa.
- Titha kugawa magawo osiyanasiyana kumagulu osiyanasiyana amagulu mumagulu, kuwachepetsa ndi kukumbukira, kuwonjezera kukumbukira, ndi zina.
Tidzagwiritsa ntchito VictoriaMetrics kunyumba chifukwa tidakonda kwambiri. Izi ndi zomwe zinali ndi zomwe zakhala.
Ma code angapo a QR a VictoriaMetrics chat, olumikizana nawo, LeroyMerlin radar yaukadaulo.
Source: www.habr.com