I-VictoriaMetrics iyi-DBMS esheshayo futhi eyingozi yokugcina nokucubungula idatha ngendlela yochungechunge lwesikhathi (irekhodi liqukethe isikhathi nesethi yamanani ahambisana nalesi sikhathi, isibonelo, atholwe ngokuvota ngezikhathi ezithile kwesimo sezinzwa noma ukuqoqwa kwamamethrikhi).
Igama lami nginguKolobaev Pavel. I-DevOps, SRE, LeroyMerlin, yonke into ifana nekhodi - imayelana nathi: ngami kanye nabanye abasebenzi bakwa-LeroyMerlin.
Kukhona ifu elisekelwe ku-OpenStack. Kukhona isixhumanisi esincane se-radar yobuchwepheshe.
Yakhelwe ku-hardware ye-Kubernetes, nakuzo zonke izinsiza ezihlobene ze-OpenStack nokungena.
Lolu wuhlelo ebesinalo ekuthuthukisweni. Ngenkathi sithuthukisa konke lokhu, sasino-opharetha we-Prometheus owayegcina idatha ngaphakathi kweqoqo le-K8s ngokwalo. Uthola ngokuzenzakalelayo lokho okudinga ukukhuhlwa futhi akufake ngaphansi kwezinyawo zakhe, uma ekhuluma nje.
Kuzodingeka sihambise yonke idatha ngaphandle kweqoqo le-Kubernetes, ngoba uma kwenzeka okuthile, sidinga ukuqonda ukuthi yini futhi kuphi.
Isixazululo sokuqala ukuthi sisebenzisa umfelandawonye uma sinenkampani yangaphandle i-Prometheus, lapho siya kuqoqo le-Kubernetes ngokusebenzisa indlela yomfelandawonye.
Kodwa kunezinkinga ezincane lapha. Esimweni sethu, izinkinga ziqale lapho sinama-metrics angu-250, futhi lapho kukhona ama-metrics angu-000, sabona ukuthi asikwazi ukusebenza kanjalo. Sengeze i-scrape_timeout yaba amasekhondi angu-400.
Kungani kwadingeka senze lokhu? U-Prometheus uqala ukubala isikhathi sokuvala kusukela ekuqaleni kocingo. Akunandaba ukuthi idatha isageleza. Uma ngalesi sikhathi esishiwo idatha ingahlanganiswa futhi iseshini ivaliwe nge-http, khona-ke iseshini ithathwa njengehlulekile futhi idatha ayingeni ku-Prometheus ngokwayo.
Wonke umuntu ujwayelene namagrafu esiwatholayo lapho enye idatha ingekho. Amashejuli adabukile futhi aseneme ngalokhu.
Inketho elandelayo i-sharding ngokusekelwe ku-Prometheus emibili ehlukene ngokusebenzisa indlela efanayo yomfelandawonye.
Isibonelo, vele uwathathe futhi uwahlukanise ngamagama. Lokhu kungasetshenziswa futhi, kodwa sinqume ukuqhubeka.
Manje kuzomele sicubungule la mashadi ngandlela thize. Ungathatha i-promxy, eya endaweni ye-shard futhi iphindaphinde idatha. Isebenza ngamashadi amabili njengendawo eyodwa yokungena. Lokhu kungenziwa nge-promxy, kodwa kusenzima kakhulu.
Inketho yokuqala wukuthi sifuna ukuyiyeka indlela yomfelandawonye ngoba ihamba kancane.
Onjiniyela be-Prometheus basho ngokucacile, "Bafo, sebenzisani i-TimescaleDB ehlukile ngoba ngeke sisekele ukugcinwa kwesikhathi eside kwamamethrikhi." Lokhu akuwona umsebenzi wabo.
Sibhala phansi ephepheni esisadinga ukuyethula ngaphandle, ukuze singagcini yonke into endaweni eyodwa.
I-drawback yesibili ukusetshenziswa kwememori. Yebo, ngiyaqonda ukuthi abaningi bazothi ngo-2020 amagigabhayithi ambalwa enkumbulo abiza ipeni, kodwa noma kunjalo.
Manje sinendawo ye-dev ne-prod. Ku-dev cishe i-9 gigabytes kumamethrikhi angu-350. Ku-prod ingu-000 gigabytes kanye namamethrikhi angaphezu kuka-14. Ngesikhathi esifanayo, isikhathi sethu sokugcina imizuzu engama-780 kuphela. Kubi lokhu. Futhi manje ngizochaza ukuthi kungani.
Senza ukubala, okungukuthi, ngamamethrikhi ayisigidi nesigamu, futhi sesivele siseduze nabo, esiteji sokuklama sithola inkumbulo engu-35-37 gigabytes. Kodwa kakade izigidi ezi-4 zamamethrikhi zidinga cishe amagigabhayithi angama-90 enkumbulo. Okusho ukuthi, yabalwa kusetshenziswa ifomula enikezwe abathuthukisi be-Prometheus. Sibheke ukuhlobana futhi sabona ukuthi asifuni ukukhokha izigidi ezimbalwa zeseva ngokugadwa.
Ngeke nje sinyuse isibalo semishini, siphinde sibheke nemishini uqobo lwayo. Ngakho-ke, uma imishini ebonakalayo eyengeziwe, amamethrikhi engeziwe ezinhlobo ezahlukahlukene, njll. Sizoba nokukhula okukhethekile kweqoqo lethu ngokuya ngamamethrikhi.
Ngesikhala sediski, akuyona yonke into embi kakhulu lapha, kodwa ngingathanda ukuyithuthukisa. Sithole ingqikithi yamagigabhayithi angu-15 ngezinsuku ezingu-120, okuyi-100 kuwo okuyidatha ecindezelwe, angu-20 ayidatha engacindezelwe, kodwa sihlale sifuna okuncane.
Ngakho-ke, sibhala iphuzu elilodwa futhi - lokhu ukusetshenziswa okukhulu kwezinsiza, esisafuna ukukugcina, ngoba asifuni iqoqo lethu lokuqapha lidle izinsiza eziningi kuneqoqo lethu, elilawula i-OpenStack.
Kukhona enye enye inhlawulo ka-Prometheus, esizitholele yona, lokhu okungenani uhlobo oluthile lokulinganiselwa kwenkumbulo. Nge-Prometheus, konke kubi kakhulu lapha, ngoba ayinakho nhlobo ukusonteka okunjalo. Ukusebenzisa umkhawulo ku-docker nakho akuyona inketho. Uma kungazelelwe i-RAF yakho iwa futhi kukhona amagigabhayithi angu-20-30, kuzothatha isikhathi eside kakhulu ukuphakama.
Lesi esinye isizathu esenza ukuthi i-Prometheus ingasifanele, okungukuthi asikwazi ukukhawulela ukusetshenziswa kwememori.
Bekungenzeka ukuqhamuka nohlelo olunjalo. Sidinga lolu hlelo ukuze sihlele iqoqo le-HA. Sifuna amamethrikhi ethu atholakale ngaso sonke isikhathi futhi yonke indawo, ngisho noma iseva egcina lawa mamethrikhi iphahlazeka. Futhi ngaleyo ndlela kuzodingeka sakhe uhlelo olunjalo.
Lolu hlelo luthi sizophinde sibe nokuphindaphinda ama-shards, futhi, ngokufanele, ukuphindwaphindwa kwezindleko zezinsiza ezisetshenzisiwe. Ingalinganiswa cishe ngokuvundlile, kodwa nokho ukusetshenziswa kwensiza kuzoba isihogo.
Ukungalungi ngendlela esizibhalele ngayo:
- Idinga ukulayisha amamethrikhi ngaphandle.
- Ukusetshenziswa okuphezulu kwezinsiza.
- Ayikho indlela yokukhawulela ukusetshenziswa kwememori.
- Ukuqaliswa okuyinkimbinkimbi kanye nezinsizakusebenza kwe-HA.
Ngokwethu, sanquma ukuthi sisuka kude ne-Prometheus njengendawo yokugcina izinto.
Sizihlonze izimfuneko ezengeziwe esizidingayo. Lokhu:
- Lokhu ukusekelwa kwe-promql, ngoba izinto eziningi sezibhalelwe i-Prometheus: imibuzo, izexwayiso.
- Futhi-ke sineGrafana, esevele ibhalwe ngendlela efanayo ku-Prometheus njenge-backend. Angifuni ukubhala kabusha amadeshibhodi.
- Sifuna ukwakha isakhiwo esijwayelekile se-HA.
- Sifuna ukunciphisa ukusetshenziswa kwanoma yiziphi izinsiza.
- Kukhona enye i-nuance encane. Asikwazi ukusebenzisa izinhlobo ezahlukahlukene zamasistimu wokuqoqwa kwamamethrikhi wamafu. Asazi ukuthi yini ezowela kulawa mamethrikhi okwamanje. Futhi njengoba noma yini ingandizela lapho, kufanele sizibekele umkhawulo ekubekweni kwendawo.
Kwakungekho ukukhetha. Siqoqe yonke into ebesinolwazi ngayo. Sibheke ikhasi le-Prometheus engxenyeni yokuhlanganisa, safunda inqwaba yezindatshana, futhi sabona ukuthi yini eyayilapho. Futhi ngokwethu, sikhethe i-VictoriaMetrics esikhundleni se-Prometheus.
Kungani? Ngoba:
- Ingenza i-promql.
- Kukhona i-architecture ye-modular.
- Ayidingi izinguquko ku-Grafana.
- Futhi okubaluleke kakhulu, cishe sizohlinzeka ngesitoreji samamethrikhi ngaphakathi kwenkampani yethu njengesevisi, ngakho-ke sibheke kusengaphambili imikhawulo yezinhlobo ezahlukahlukene ukuze abasebenzisi bakwazi ukusebenzisa zonke izinsiza zeqoqo ngandlela thize, ngoba kukhona ithuba. ukuthi kuzoba multitenancy.
Asenze isiqhathaniso sokuqala. Sithatha i-Prometheus efanayo ngaphakathi kweqoqo, i-Prometheus yangaphandle iya kuyo. Engeza nge-remoteBhala i-VictoriaMetrics.
Ngokushesha ngizokwenza ukubhuka ukuthi lapha sibambe ukwenyuka okuncane kokusetshenziswa kwe-CPU kusuka ku-VictoriaMetrics. I-VictoriaMetrics wiki ikutshela ukuthi yimaphi amapharamitha angcono kakhulu. Sawahlola. Banciphise ukusetshenziswa kwe-CPU kahle kakhulu.
Esimweni sethu, ukusetshenziswa kwenkumbulo ye-Prometheus, etholakala kuqoqo le-Kubernetes, akuzange kukhule kakhulu.
Siqhathanisa imithombo yedatha emibili yedatha efanayo. Ku-Prometheus sibona idatha efanayo engekho. Konke kuhamba kahle kwaVictoriaMetrics.
Imiphumela yokuhlolwa kwesikhala sediski. Thina kwa-Prometheus sithole amagigabhayithi angu-120 esewonke. Kwa-VictoriaMetrics sesivele sithola amagigabhayithi ama-4 ngosuku. Kukhona indlela ehluke kancane kunaleyo esijwayele ukuyibona ku-Prometheus. Okusho ukuthi, idatha isivele icindezelwe kahle ngosuku, engxenyeni yehora. Sebevele bavunwa kahle ngosuku, engxenyeni yehora, naphezu kokuthi idatha isazolahleka kamuva. Ngenxa yalokho, silondoloze isikhala sediski.
Siphinde songa ekusetshenzisweni kwensiza yememori. Ngesikhathi sokuhlolwa, i-Prometheus yafakwa emshinini obonakalayo - ama-cores angu-8, ama-gigabytes angu-24. U-Prometheus udla cishe yonke into. Wawela ku-OOM Killer. Ngesikhathi esifanayo, amamethrikhi asebenzayo angu-900 kuphela athululelwa kuwo. Lokhu cishe amamethrikhi angu-000-25 ngomzuzwana.
Sisebenzise i-VictoriaMetrics emshinini we-virtual-core onamagigabhayithi angu-8 we-RAM. Sikwazile ukwenza i-VictoriaMetrics isebenze kahle ngokuzulazula ngezinto ezimbalwa emshinini ongu-8GB. Ekugcineni, siyigcine ku-7 gigabytes. Ngesikhathi esifanayo, isivinini sokulethwa kokuqukethwe, okungukuthi amamethrikhi, sasiphezulu kakhulu kunesika-Prometheus.
I-CPU isibe ngcono kakhulu uma iqhathaniswa ne-Prometheus. Lapha i-Prometheus idla ama-cores angu-2,5, kanti i-VictoriaMetrics idla kuphela ama-cores angu-0,25. Ekuqaleni - 0,5 cores. Njengoba ihlangana, ifinyelela umongo owodwa, kodwa lokhu kuyivelakancane ngokwedlulele.
Esimweni sethu, ukukhetha kwawela ku-VictoriaMetrics ngezizathu ezisobala; besifuna ukonga imali futhi senza.
Ake sikhiphe amaphuzu amabili ngokushesha - ukulayishwa kwamamethrikhi nokusetshenziswa okuphezulu kwezisetshenziswa. Futhi kufanele sinqume amaphuzu amabili esisazishiyele wona.
Lapha ngizobhukha khona manjalo, sibheka i-VictoriaMetrics njengendawo yokugcina yamamethrikhi. Kodwa njengoba cishe sizohlinzeka nge-VictoriaMetrics njengesitoreji sayo yonke i-Leroy, sidinga ukukhawulela labo abazosebenzisa leli qoqo ukuze bangasiniki lona.
Kukhona ipharamitha emangalisayo ekuvumela ukuthi ukhawule ngesikhathi, ngevolumu yedatha nangesikhathi sokwenza.
Kukhona futhi inketho enhle kakhulu esivumela ukuthi sinciphise ukusetshenziswa kwememori, ngaleyo ndlela singathola ibhalansi ezosivumela ukuthi sithole isivinini esivamile sokusebenza kanye nokusetshenziswa kwezinsiza ezanele.
Susa iphuzu elilodwa ngaphezulu, i.e. weqa iphuzu - awukwazi ukukhawulela ukusetshenziswa kwememori.
Ekuphindaphindweni kokuqala, sihlole i-VictoriaMetrics Single Node. Okulandelayo sidlulela ku-VictoriaMetrics Cluster Version.
Lapha sinesandla samahhala sokuhlukanisa izinsizakalo ezahlukahlukene ku-VictoriaMetrics kuye ngokuthi bazosebenzisa ini nokuthi bazosebenzisa ziphi izinsiza. Lesi yisixazululo esivumelana nezimo kakhulu futhi esikahle. Sasebenzisa lokhu kithi ngokwethu.
Izingxenye eziyinhloko ze-VictoriaMetrics Cluster Version yi-vmstsorage. Kungaba no-N inombolo yazo. Esimeni sethu kukhona ezi-2 zazo kuze kube manje.
Futhi kukhona vminsert. Lena iseva elibamba esivumela ukuthi: sihlele ukwaba phakathi kwazo zonke izitoreji esizitshele ngazo, futhi ivumela ukukopishwa, okungukuthi uzoba nakho kokubili u-sharding kanye nesifaniso.
I-Vminsert isekela i-OpenTSDB, Graphite, InfluxDB kanye ne-remoteWrite protocols kusuka ku-Prometheus.
Kukhona futhi vmselect. Umsebenzi wayo oyinhloko ukuya ku-vmstorage, uthole idatha kubo, ukhiphe le datha bese uyinikeza iklayenti.
Kunento emangalisayo ebizwa nge-vmagent. Simthanda ngempela. Ikuvumela ukuthi ulungiselele ncamashi njenge-Prometheus futhi wenze yonke into njenge-Prometheus. Okusho ukuthi, iqoqa amamethrikhi avela ezinhlanganweni namasevisi ahlukene iwathumele ku-vminsert. Khona-ke konke kuncike kuwe.
Enye isevisi enhle i-vmalert, ekuvumela ukuthi usebenzise i-VictoriaMetrics njenge-backend, uthole idatha ecutshunguliwe ku-vminsert bese uyithumela ku-vmselect. Icubungula izexwayiso ngokwazo, kanye nemithetho. Endabeni yezaziso, sithola isaziso ngomphathi we-alert.
Kukhona ingxenye ye-wmauth. Singase noma singase (asikakanqumi ngalokhu) siyisebenzise njengohlelo lokugunyaza lwenguqulo ye-multitenancy yamaqoqo. Isekela i-remoteWrite ye-Prometheus futhi ingagunyaza ngokusekelwe ku-url, noma kunalokho ingxenye yesibili yayo, lapho ungakwazi khona ukubhala noma ongakwazi ukubhala.
Kukhona futhi i-vmbackup, i-vmrestore. Lokhu, empeleni, ukubuyiselwa nokwenza ikhophi yasenqolobaneni yayo yonke idatha. Ingenza i-S3, i-GCS, ifayela.
Ukuphindaphinda kokuqala kweqoqo lethu kwenziwa ngesikhathi sokuvalelwa. Ngaleso sikhathi, yayingekho i-replica, ngakho-ke ukuphindaphinda kwethu kwakuhlanganisa amaqoqo amabili ahlukene futhi azimele lapho sithole khona idatha nge-remoteWrite.
Lapha ngizobhukha ukuthi lapho sisuka ku-VictoriaMetrics Single Node kuya ku-VictoriaMetrics Cluster Version, sisahlala nezinsiza ezifanayo ezisetshenzisiwe, okungukuthi eyinhloko inkumbulo. Lena cishe indlela idatha yethu, okungukuthi ukusetshenziswa kwezinsiza, esatshalaliswe ngayo.
Isifaniso sesivele sengeziwe lapha. Konke lokhu sikuhlanganise saba yiqoqo elikhulu. Yonke idatha yethu yabiwe futhi iyaphindwa.
Iqoqo lonke linamaphuzu okungena angu-N, okusho ukuthi i-Prometheus ingakwazi ukwengeza idatha nge-HAPROXY. Lapha sinaleli phuzu lokungena. Futhi ngale ndawo yokungena ungangena useGrafana.
Esimweni sethu, i-HAPROXY ukuphela kwembobo ekhethwa ngabameleli, bayifake nezinye izinsiza ngaphakathi kwaleli qoqo. Kithina, bekungenzeki ukwenza ikheli elilodwa; bekufanele senze izindawo zokungena ezimbalwa, ngoba imishini uqobo esebenza kuyo iqoqo le-VictoriaMetrics itholakala ezindaweni ezihlukene zomhlinzeki wefu ofanayo, i.e. hhayi ngaphakathi kwefu lethu, kodwa ngaphandle. .
Sinesexwayiso. Siyayisebenzisa. Sisebenzisa i-alertmanager evela ku-Prometheus. Sisebenzisa i-Opsgenie neTelegram njengesiteshi sokulethwa kwezixwayiso. KuTelegramu batheleka besuka ku-dev, mhlawumbe okuthile okuvela ku-prod, kodwa ikakhulukazi into yezibalo, edingwa onjiniyela. Futhi i-Opsgenie ibalulekile. Lezi yizingcingo, ukuphathwa kwezigameko.
Umbuzo waphakade: "Ubani oqapha ukuqapha?" Esimweni sethu, ukuqapha ukuqapha kuyaziqapha, ngoba sisebenzisa i-vmagent ku-node ngayinye. Futhi njengoba ama-node ethu esatshalaliswa kuzo zonke izikhungo zedatha ezihlukene zomhlinzeki ofanayo, isikhungo sedatha ngasinye sinesiteshi saso, sizimele, futhi noma ngabe ubuchopho obuhlukene bufika, sisazothola izexwayiso. Yebo, kuzoba nokuningi kwabo, kodwa kungcono ukuthola izexwayiso eziningi kunokungabikho.
Siphetha uhlu lwethu ngokuqaliswa kwe-HA.
Futhi ngokuqhubekayo ngithanda ukuqaphela ulwazi lokuxhumana nomphakathi we-VictoriaMetrics. Kwabonakala kukuhle kakhulu. Izinsizwa ziyasabela. Bazama ukuphenya kuwo wonke amacala anikezwayo.
Ngiqale izinkinga ku-GitHub. Zaxazululwa ngokushesha okukhulu. Kukhona ezinye izinkinga ezimbalwa ezingavalwanga ngokuphelele, kodwa sengiyabona kukhodi esebenza ngale ndlela eqhubekayo.
Ubuhlungu obuyinhloko kimi ngesikhathi sokuphindaphinda kwakuwukuthi uma ngivala i-node, ngakho-ke imizuzwana engu-30 yokuqala i-vminsert ayikwazanga ukuqonda ukuthi kwakungekho i-backend. Lokhu manje sekunqunyiwe. Futhi ngokoqobo ngomzuzwana noma emibili, idatha ithathwa kuwo wonke ama-node asele, futhi isicelo siyayeka ukulinda leyo nodi engekho.
Ngesinye isikhathi besifuna i-VictoriaMetrics ibe i-VictoriaMetrics opharetha. Samlinda. Manje sakha uhlaka olusebenzayo lokuthi u-opharetha we-VictoriaMetrics athathe yonke imithetho yokubala kusengaphambili, njll. I-Prometheus, ngoba sisebenzisa imithetho eza no-opharetha we-Prometheus.
Kuneziphakamiso zokuthuthukisa ukuqaliswa kweqoqo. Ngiwabalule ngenhla.
Futhi ngempela ngifuna ukwehlisela isampula. Esimweni sethu, ukuncishiswa kwesampula kuyadingeka kuphela kumathrendi wokubuka. Uma sikhuluma nje, imethrikhi eyodwa inganele kimi emini. Lezi zindlela ziyadingeka unyaka, emithathu, emihlanu, iminyaka eyishumi. Futhi inani elilodwa lemethrikhi lanele impela.
- Siyabazi ubuhlungu, njengabanye ozakwethu, lapho sisebenzisa i-Prometheus.
- Sizikhethele i-VictoriaMetrics.
- Ilinganisa kahle ngokuqondile nangokuvundlile.
- Singasabalalisa izingxenye ezihlukene ezinombolweni ezihlukene zamanodi kuqoqo, sizikhawulele ngenkumbulo, sengeze inkumbulo, njll.
Sizosebenzisa i-VictoriaMetrics ekhaya ngoba siyithande kakhulu. Yilokhu okwakukhona futhi osekukhona.
Amakhodi ambalwa e-QR engxoxo ye-VictoriaMetrics, engixhumana nabo, irada yobuchwepheshe ye-LeroyMerlin.
Source: www.habr.com