Ngiphakamisa ukuthi ufunde umbhalo wombiko ka-Roman Khavronenko "ExtendedPromQL"
Kafushane ngami. Igama lami ngingumRoma. Ngisebenza e-CloudFlare futhi ngihlala eLondon. Kodwa futhi ngingumnakekeli we-VictoriaMetrics.
Futhi yimi umbhali ngoba Grafana kanye ingummeleli omncane we-ClickHouse.
Sizoqala ngengxenye yokuqala, ebizwa ngokuthi “Ubunzima Bokuhumusha” futhi kuyo ngizokhuluma ngeqiniso lokuthi noma yiluphi ulimi noma nje ulimi lokuxhumana lubaluleke kakhulu. Ngoba le yindlela oyidlulisela ngayo imicabango yakho komunye umuntu noma uhlelo, ukuthi usenza kanjani isicelo. Abantu abaku-inthanethi bayaphikisana ngokuthi yiluphi ulimi olungcono - i-java noma olunye. Ngokwami, nginqume ukuthi ngidinga ukukhetha ngokusho komsebenzi, ngoba konke lokhu kuqondile.
Ake siqale kusukela ekuqaleni. Iyini i-PromQL? I-PromQL iwulimi lombuzo we-Prometheus. Lena yindlela esenza ngayo imibuzo ku-Prometheus ukuze sithole idatha yochungechunge lwesikhathi.
Ithini idatha yochungechunge lwesikhathi? Ngokwezwi nezwi, lawa amapharamitha amathathu.
Lokhu:
- Sibhekeni?
- Uma sibheka.
- Futhi libonisa inani elingakanani?
Uma ubheka leli shadi (leli shadi livela ocingweni lami elibonisa izibalo zami zesinyathelo), lingakwazi ukuphendula le mibuzo ngokushesha.
Sibheka izinyathelo. Sibona incazelo futhi sibona isikhathi lapho sibheka khona. Okungukuthi, uma ubheka lo mdwebo, ungasho kalula ukuthi ngeSonto ngihambe izinyathelo ezingaba ngu-15. Lena idatha yochungechunge lwesikhathi.
Manje "masihlukanise" (siguqule) zibe enye imodeli yedatha ngendlela yethebula. Lapha futhi sinalokho esikubukayo. Lapha ngengeze idatha eyengeziwe eyengeziwe, esizoyibiza nge-meta-data, okungukuthi akumina engidlule kulokhu, kodwa abantu ababili, isibonelo, u-Jay no-Silent Bob. Yilokhu esikubukayo; ikhombisani futhi ikhombisa nini lelo nani.
Manje ake sizame ukugcina yonke le datha kusizindalwazi. Isibonelo, ngithathe i-syntax ye-ClickHouse. Futhi lapha sakha ithebula elilodwa elibizwa ngokuthi "Izinyathelo", okungukuthi esikubukayo. Kukhona isikhathi lapho sibheka khona; lokho ekubonisayo kanye nedatha ethile yemetha lapho sizogcina khona ukuthi ubani: u-Jay no-Silent Bob.
Futhi ukuzama ukukubona ngeso lengqondo konke lokhu, sizosebenzisa iGrafana ngoba, okokuqala nje, yinhle.
Futhi sizosebenzisa le plugin. Kunezizathu ezimbili zalokhu. Eyokuqala yingoba ngiyibhalile. Futhi ngazi kahle ukuthi kunzima kangakanani ukukhipha idatha yochungechunge lwesikhathi kusuka ku-ClickHouse ukuze uyibonise eGrafana.
Sizoyibonisa kuphaneli yegrafu. Leli iphaneli edume kakhulu e-Grafana, ebonisa ukuncika kwevelu ngesikhathi, ngakho sidinga amapharamitha amabili kuphela.
Masibhale umbuzo olula - indlela yokubonisa izibalo zesinyathelo e-Grafana, sigcina le datha ku-ClickHouse, kuthebula esilidalile. Futhi sibhala lesi sicelo esilula. Sikhetha ezinyathelweni. Sikhetha inani bese sikhetha isikhathi lalawa manani, okungukuthi amapharamitha amathathu afanayo esikhulume ngawo.
Futhi ngenxa yalokho, sizothola igrafu efana nale. Ubani owaziyo ukuthi kungani exakile?
Kunjalo, sidinga ukuhlela ngesikhathi.
Futhi ekugcineni sizothola isimiso esingcono, kodwa esixakile. Ubani owaziyo ukuthi kungani? Kulungile, kukhona ababambiqhaza ababili, futhi thina e-Grafana sinikeza uchungechunge lwezikhathi ezimbili, ngoba uma ubheka imodeli yedatha futhi, khona-ke uchungechunge lwesikhathi ngasinye luyinhlanganisela eyingqayizivele yegama nawo wonke amalebula angukhiye wenani.
Ngakho-ke, kudingeka sikhethe umuntu othile. Sikhetha uJay.
Ake sidwebe futhi. Manje igrafu ibukeka njengeqiniso. Manje lolu uhlelo olujwayelekile futhi yonke into isebenza kahle.
Futhi cishe uyazi ukuthi ungayenza kanjani into efanayo, kodwa e-Prometheus nge-PromQL. Into efana nale. Kalula kancane. Ake sikuhlephule konke. Sathatha Izinyathelo. Bese uhlunga ngoJay. Asicacisi lapha ukuthi sidinga ukuthola inani futhi asikhethi isikhathi.
Manje ake sizame ukubala isivinini sokunyakaza sika-Jay noma u-Silent Bob. Ku-ClickHouse sizodinga ukwenza i-rungingDifference, okungukuthi, sibale umehluko phakathi kwamapheya amaphoyinti futhi siwahlukanise ngesikhathi ukuze sithole isivinini esiqondile. Isicelo sizobukeka kanjena.
Futhi izobonisa cishe lawa manani, okungukuthi u-Silent Bob noma u-Jay uthatha cishe izinyathelo ezingu-1,8 ngomzuzwana.
Futhi ku-Prometheus uyazi ukuthi ungakwenza kanjani lokhu futhi. Kulula kakhulu kunangaphambili.
Futhi ukwenza kube lula futhi ukukwenza eGrafana, ngengeze lesi sisonga, esibukeka sifana kakhulu ne-PromQL. Ibizwa ngokuthi Rate Macros noma noma yini ofuna ukuyibiza. E-Grafana uvele ubhale “izinga”, kodwa endaweni ethile ekujuleni liguquke libe yilesi sicelo esikhulu. Futhi akudingeki nokuthi uyibheke, ikhona ndawana thize, kodwa ugcina isikhathi esiningi, ngoba ukubhala imibuzo emikhulu kangaka ye-SQL kuhlala kubiza. Ungenza iphutha kalula bese ungaqondi ukuthi kwenzekani isikhathi eside.
Futhi lesi isicelo esingazange singene ngisho kwisilayidi esisodwa futhi kwadingeka ngisihlukanise sibe amakholomu amabili. Lesi futhi isicelo ku-ClickHouse, okwenza isilinganiso esifanayo, kodwa kuzo zombili uchungechunge lwesikhathi: Silent Bob noJay, ukuze sibe nochungechunge lwezikhathi ezimbili kuphaneli. Futhi lokhu kakade kunzima kakhulu, ngombono wami.
Futhi ngokusho kuka-Prometheus kuzoba yisamba (isilinganiso). Nge-ClickHouse, ngenze i-macro ehlukile ebizwa ngokuthi i-RateColumns, ebukeka njengombuzo ku-Prometheus.
Siyibhekile futhi kubonakala sengathi i-PromQL ipholile kakhulu, kodwa inemikhawulo.
Lokhu:
- KHETHA okulinganiselwe.
- I-Borderline JOIN.
- AKUKHO UKUBA nokusekelwa.
Futhi uma usebenze nayo isikhathi eside, khona-ke uyazi ukuthi ngezinye izikhathi kunzima kakhulu ukwenza okuthile ku-PromQL, kodwa ku-SQL ungenza cishe yonke into, ngoba zonke lezi zinketho esisanda kukhuluma ngazo zingenziwa ku-SQL. . Kodwa ingabe kuyoba lula ukuyisebenzisa? Futhi lokhu kungenza ngicabange ukuthi ulimi olunamandla kakhulu lungase lungabi lula kakhulu ngaso sonke isikhathi.
Ngakho-ke, ngezinye izikhathi udinga ukukhetha ulimi lomsebenzi. Kufana noBatman elwa noSuperman. Kuyacaca ukuthi uSuperman unamandla, kodwa uBatman wakwazi ukumnqoba ngoba uyasebenza futhi wayazi kahle ukuthi wenzani.
Futhi ingxenye elandelayo i-Extending PromQL.
Nakulokhu futhi mayelana ne-VictoriaMetrics. Iyini i-VictoriaMetrics? Lena isizindalwazi sochungechunge lwesikhathi, siku-OpenSource, sisabalalisa izinguqulo zayo ezilodwa kanye nezihlanganisiwe. Ngokusho kwama-benchmarks ethu, kuyashesha kunanoma yini esemakethe manje futhi ukucindezelwa kuyafana, okungukuthi abantu bangempela babika ukucindezelwa okungaba ngu-0,4 bytes ngephuzu ngalinye, kuyilapho i-Prometheus ingu-1,2-1,4.
Seseka okungaphezu nje kuka-Prometheus. Sisekela i-InfluxDB, Graphite, OpenTSDB.
Ungakwazi “ukusibhalela”, okungukuthi, ungadlulisela idatha endala.
Futhi sisebenza ngokuphelele ne-Prometheus ne-Grafana, okungukuthi sisekela injini ye-PromQL. Futhi eGrafana ungamane uguqule iphoyinti lokugcina le-Prometheus libe i-VictoriaMetrics futhi wonke amadeshibhodi akho azosebenza njengoba enzile.
Kodwa ungasebenzisa futhi izici ezengeziwe ezihlinzekwa yi-VictoriaMetrics.
Sizodlula ngokushesha izici esizengezile.
Yeqa ipharamitha yesikhawu - ungashiya amapharamitha wesikhawu e-Grafana. Uma ungafuni ukuthola amagrafu angajwayelekile lapho usondeza/usondeza iphaneli, kuyanconywa ukusebenzisa okuguquguqukayo. $__interval. Lolu wushintsho lwangaphakathi lwe-Grafana futhi ikhetha ibanga ledatha ngokwalo. Futhi i-VictoriaMetrics ngokwayo ingaqonda ukuthi lobu bubanzi kufanele bube yini. Futhi awudingi ukubuyekeza zonke izicelo zakho. Kuzoba lula kakhulu.
Umsebenzi wesibili ireferensi yesikhawu. Ungasebenzisa lesi sikhawu ezinkulumweni zakho. Ungaphindaphinda, uhlukanise, udlulise, ubhekisele kukho.
Okulandelayo umndeni womsebenzi we-rollup. Umsebenzi we-Rollup uguqula noma yiluphi uchungechunge lwakho lwesikhathi lube uchungechunge lwesikhathi oluthathu oluhlukene. Lawa amaminithi, ubuningi kanye nesilinganiso. Lokhu ngikuthola kulula kakhulu ngoba kwesinye isikhathi kungakhombisa ezinye izinto ezingaphandle kanye nokungalungile.
Futhi uma wenza nje ukucasuka noma ukukala, lapho-ke uzophuthelwa ezinye izimo lapho uchungechunge lwesikhathi lungaziphathi ngendlela obulindele ngayo. Ngalo msebenzi kulula kakhulu ukukubona, ake sithi i-max isuka ku-avg kakhulu.
Okulandelayo ukuguquguquka okuzenzakalelayo. Okuzenzakalelayo - lokhu kusho ukuthi yiliphi inani okufanele silidwebe ku-Grafana uma singenalo uchungechunge lwesikhathi okwamanje. Kwenzeka nini lokhu? Ake sithi ukhipha ama-metrics amaphutha. Futhi unesicelo esipholile kangangokuthi lapho uqala, awunawo amaphutha futhi awekho amaphutha amahora amathathu alandelayo noma ngisho nosuku. Futhi unamadeshibhodi abonisa ubudlelwano ukusuka empumelelweni kuye emaphutheni. Futhi ngeke bakubonise lutho ngoba awunayo imethrikhi yephutha. Futhi ngokuzenzakalelayo ungacacisa noma yini.
Keep_last_Value – ilondoloza inani lokugcina lemethrikhi uma lingekho. Uma u-Prometheus engayitholi phakathi nemizuzu engu-5 ngemva kokuklwebheka okulandelayo, khona-ke lapha sizokhumbula inani layo lokugcina futhi amashadi akho ngeke aphinde aphuke.
I-Scrape_interval - ibonisa ukuthi i-Prometheus iqoqa kangaki idatha ku-metric yakho, futhi ngaziphi izikhathi. Lapha ungabona iphasi, isibonelo.
Ukushintsha ilebula kuyisici esidumile. Kodwa sicabanga ukuthi kuyinkimbinkimbi ngoba kuthatha izingxabano eziphelele. Futhi awudingi nje ukukhumbula izimpikiswano ezi-5, kodwa futhi ukhumbule ukulandelana kwazo.
Ngakho-ke, kungani ungazenzi zibe lula? Okusho ukuthi, yihlukanise ibe yimisebenzi emincane ene-syntax eqondakalayo.
Futhi manje ingxenye ejabulisayo. Kungani sicabanga ukuthi le PromQL enwetshiwe? Ngoba sisekela i-Common Table Expressions. Ungakwazi ukulandela ikhodi ye-QR (), bona izixhumanisi ezinezibonelo, endaweni yokudlala, lapho ungasebenzisa khona imibuzo ngqo ku-VictoriaMetrics ngaphandle kokuyifaka esipheqululini.
Futhi kuyini lokhu? Lesi sicelo esingenhla siyisicelo esidume kakhulu. Ngicabanga ukuthi kunoma iyiphi ideshibhodi ezinkampanini eziningi usebenzisa isihlungi esifanayo kuyo yonke into. Ngokuvamile kunjalo. Kodwa uma udinga ukungeza isihlungi esisha, kufanele ubuyekeze iphaneli ngayinye, noma ulande ideshibhodi, uyivule nge-JSON, thola okunye esikhundleni, nakho okuthatha isikhathi. Kungani ungagcini leli nani kokuguquguqukayo futhi ulisebenzise kabusha? Lokhu kubukeka, ngokubona kwami, kulula kakhulu futhi kucacile.
Isibonelo, uma ngidinga ukubuyekeza izihlungi ku-Grafana kuzo zonke izicelo, futhi ideshibhodi ingaba nkulu noma kungaba neziningana zazo. Futhi ngingathanda ukuyixazulula kanjani le nkinga eGrafana?
Ngixazulula le nkinga ngale ndlela: Ngenza i-commonFilter futhi ngichaze lesi sihlungi kuso, bese ngisisebenzisa kabusha emibuzweni. Kodwa uma wenza okufanayo manje, ngeke kusebenze ngoba iGrafana ayikuvumeli ukuthi usebenzise okuguquguqukayo ngaphakathi kokuguquguquka kwemibuzo. Futhi kuyamangaza.
Futhi ngakho ngenze inketho ekuvumela ukuthi wenze lokhu. Futhi uma unentshisekelo noma ufuna isici esinjalo, bese usisekela noma ungasithandi uma ungawuthandi lo mbono.
Okuningi mayelana ne-PromQL kunwetshiwe. Lapha asichazi kuphela okuguquguqukayo, kodwa umsebenzi wonke. Futhi sikubiza ngokuthi ru (ukusetshenziswa kwezinsiza). Futhi lo msebenzi wamukela izinsiza zamahhala, umkhawulo wensiza kanye nesihlungi. I-syntax ibonakala ilula. Futhi kulula kakhulu ukusebenzisa lo msebenzi nokubala iphesenti lememori yamahhala esinayo. Okusho ukuthi, ingakanani inkumbulo esinayo, uyini umkhawulo kanye nendlela yokuhlunga. Kubukeka kukuhle kakhulu uma ukubhale konke, usebenzisa kabusha izihlungi ezifanayo, ngoba kungaphenduka umbuzo omkhulu, omkhulu.
Futhi nasi isibonelo sesicelo esikhulu kangaka, esikhulu. Isuka kudeshibhodi esemthethweni ye-NodeExporter ye-Grafana. Kodwa angiqondi kahle ukuthi kwenzekani lapha. Okungukuthi, yiqiniso, ngiyaqonda uma ubhekisisa, kodwa inani labakaki linganciphisa ngokushesha isisusa sokuqonda ukuthi kwenzekani lapha. Futhi kungani ungakwenzi kube lula futhi kucace?
Isibonelo, njengalokhu, ukuhlukanisa izinto ezibalulekile noma izingxenye zibe okuguquguqukayo. Bese wenza izibalo zakho eziyisisekelo. Lokhu sekufana nokuhlela, yilokhu engingathanda ukukubona esikhathini esizayo eGrafana.
Nasi isibonelo sesibili sokuthi singakwenza kanjani lokhu kube lula nakakhulu uma sesivele sinalo msebenzi we-ru, futhi usuvele ukhona ngqo ku-VictoriaMetrics. Bese uvele udlulise inani eligcinwe kunqolobane olimemezele ku-CTE.
Sengike ngakhuluma ngokuthi kubaluleke kangakanani ukusebenzisa ulimi olufanele lokuhlela. Futhi, mhlawumbe, yonke inkampani yaseGrafana inokuthile okwehlukile okwenzekayo. Futhi mhlawumbe futhi unikeza ukufinyelela ku-Grafana kubathuthukisi bakho, futhi abathuthukisi benza into yabo. Futhi bonke bakwenza ngendlela ehlukile. Kodwa ngangifuna ukuthi ngandlela-thile ifane, okungukuthi, ukuyinciphisa ibe sezingeni elivamile.
Ake sithi awunabo nje onjiniyela besistimu, mhlawumbe unochwepheshe, ama-devops noma ama-SRE. Mhlawumbe unongoti abazi ukuthi kuyini ukuqapha, abakwaziyo ukuthi iGrafana iyini, okusho ukuthi sebeneminyaka besebenza nayo futhi bazi kahle ukuthi kwenziwa kanjani kahle. Futhi sebebhale lokhu izikhathi eziyi-100 futhi bakuchazela wonke umuntu, kodwa ngesizathu esithile akekho olalelayo.
Kuthiwani uma bengafaka lolu lwazi ngqo ku-Grafana ukuze abanye abasebenzisi baphinde basebenzise izici? Futhi uma bebedinga ukubala iphesenti lememori yamahhala, bebevele basebenzise umsebenzi. Kuthiwani uma abadali babathumeli, kanye nomkhiqizo wabo, baphinde banikeze isethi yemisebenzi yokuthi bangasebenza kanjani ngamamethrikhi abo, ngoba bazi kahle ukuthi ayini lawa mamethrikhi kanye nendlela yokubala ngendlela efanele?
Lokhu akukho ngempela. Yilokhu engikwenzile mina. Lokhu ukusekelwa komtapo wolwazi eGrafana. Ake sithi abafana abenze i-NodeExporter benze engikhulume ngakho. Futhi banikeze nesethi yemisebenzi.
Okusho ukuthi, kubukeka kanjena. Uxhuma le labhulali ku-Grafana, ungena ekuhleleni futhi kubhalwe kalula ku-JSON ukuthi kusetshenzwa kanjani ngale metric. Okusho ukuthi, isethi ethile yemisebenzi, incazelo yabo nokuthi baphenduka babe yini.
Ngicabanga ukuthi lokhu kungaba usizo, ngoba eGrafana ungabhala kanjalo nje. Futhi u-Grafana "ukutshela" ukuthi kunomsebenzi othize futhi othize ovela kumtapo wezincwadi othize - masiwusebenzise. Ngicabanga ukuthi lokho kungaba kuhle kakhulu.
Okuncane mayelana ne-VictoriaMetrics. Senza izinto eziningi ezithakazelisayo. Funda izindatshana zethu mayelana nokucindezelwa, mayelana nemincintiswano yethu nezinye izinhlelo zokusebenza zedatha yochungechunge lwesikhathi, incazelo yethu yokuthi usebenza kanjani ne-PromQL, ngoba kusenenqwaba yabaqalayo kulokhu, kanye nokuqina okuqondile kanye nokubhekana ne-Thanos.
Imibuzo:
Ngizoqala umbuzo wami ngendaba yempilo elula. Ngenkathi ngiqala ukusebenzisa i-Grafana, ngabhala umbuzo ophoqayo owawunemigqa emi-5 ubude. Umphumela wokugcina uyigrafu ekholisa kakhulu. Lolu hlelo selucishe luye ekukhiqizweni. Kodwa lapho sihlolisisa, kwavela ukuthi le grafu ikhombisa umbhedo ophelele ongahlangene neqiniso, nakuba izinombolo ziwela phakathi kwebanga ebesilindele ukulibona. Futhi umbuzo wami. Sinemitapo yolwazi, sinemisebenzi, kodwa sizibhala kanjani izivivinyo zeGrafana? Ubhale isicelo esiyinkimbinkimbi lapho isinqumo sebhizinisi sincike - uku-oda isitsha sangempela samaseva noma unga-oda. Futhi njengoba sazi, lo msebenzi odweba igrafu ufana neqiniso. Ngiyabonga.
Ngiyabonga ngombuzo. Kunezingxenye ezimbili. Okokuqala, ngithola umbono, ngokusekelwe kokuhlangenwe nakho kwami, ukuthi abasebenzisi abaningi, lapho bebheka amashadi abo, abaqondi ukuthi babonisani. Ngesizathu esithile, abantu bahle kakhulu ekuvezeni izaba zanoma yikuphi ukudidayo okwenzeka kumagrafu, ngisho noma kuyiphutha ngaphakathi komsebenzi. Futhi ingxenye yesibili - kubonakala kimi ukuthi ukusebenzisa imisebenzi enjalo kungaba indlela engcono kakhulu yokuxazulula inkinga yakho, esikhundleni sokuthi unjiniyela wakho ngamunye enze ukuhlela kwakhe umthamo nokwenza amaphutha ngokungenzeka okuthile.
Ungahlola kanjani?
Ungahlola kanjani? Cishe cha.
Njengesivivinyo eGrafana.
Ngabe iGrafana ihlangene ngani nayo? U-Grafana uhumusha lesi sicelo ngokuqondile ku-DataSource.
Ukwengeza kancane kumapharamitha.
Cha, akukho okwengezwa eGrafana. Kungase kube nemingcele ye-GET, njengokuthi, ithi, isinyathelo. Ayicaciswanga ngokusobala, kodwa ungayikhipha, noma ungase ungayishiyi, kodwa yengezwa ngokuzenzakalelayo. Ngeke ubhale izivivinyo lapha. Angicabangi ukuthi kufanele sithembele kuGrafana njengomthombo weqiniso lapha.
Siyabonga ngombiko! Siyabonga ngokuminyanisa! Ushilo ukwenza imephu eguquguqukayo kugrafu, ukuthi eGrafana awukwazi ukusebenzisa okuguquguqukayo ngaphakathi kokuguquguquka. Uyazi ukuthi ngiqonde ukuthini?
Yebo.
Lokhu kwaqale kwangiphatha kabi ngesikhathi ngifuna ukwakha isexwayiso eGrafana. Futhi lapho udinga ukwenza isexwayiso kumsingathi ngamunye ngokwehlukana. Le nto oyenzile, ingabe iyazisebenzela izexwayiso e-Grafana?
Uma i-Grafana ingafinyeleli okuguquguqukayo ngokuhlukile, khona-ke yebo, izosebenza. Kodwa iseluleko sami esokuthi ungasebenzisi neze isexwayiso e-Grafana, kungcono usebenzise i-alertmanager.
Yebo, ngiyayisebenzisa, kodwa kubonakale kulula ukumisa eGrafana, kodwa ngiyabonga ngezeluleko!
Source: www.habr.com