Ndicebisa ukuba ufunde okushicilelweyo kwengxelo ka-2019 ka-Alexander Valyalkin "Hamba ulungelelwaniso kwiVictoriaMetrics"
I-DBMS ekhawulezayo kunye neyoyikisayo yokugcina kunye nokusetyenzwa kwedatha ngohlobo lothotho lwexesha (irekhodi lifom yexesha kunye neseti yamaxabiso ahambelana neli xesha, umzekelo, efunyenwe ngokuvota ngamaxesha athile ubume beenzwa okanye ukuqokelela iimetrics).
Nali ikhonkco kwividiyo yale ngxelo -
Nceda usixelele ngesiqu sakho. NdinguAlexander Valyalkin. Apha . Ndiyayithanda iGo kunye nokwenza ngcono ukusebenza. Ndibhale ezininzi iilayibrari eziluncedo kwaye azikho luncedo kangako. Baqala nokuba fast, okanye nge quick isimaphambili.
Ngoku ndisebenza kwiVictoriaMetrics. Yintoni kwaye ndenza ntoni apho? Ndiza kuthetha ngale nto kule ntetho.
Ulwandlalo lwengxelo luhamba ngolu hlobo lulandelayo:
- Okokuqala, ndiza kukuxelela ukuba yintoni iVictoriaMetrics.
- Emva koko ndiza kukuxelela ukuba yintoni uchungechunge lwexesha.
- Emva koko ndiza kukuxelela ukuba i-database yochungechunge lwexesha isebenza njani.
- Okulandelayo, ndiza kukuxelela malunga noyilo lwedatha: yintoni equlethwe ngayo.
- Kwaye ke masiqhubele phambili kwi-optimizations enayo iVictoriaMetrics. Olu lulungiselelo lwesalathiso esijikelelweyo kunye nolungiselelo lophumezo lwebitset kwiGo.
Ngaba kukho nabani na kubaphulaphuli owaziyo ukuba yintoni iVictoriaMetrics? Wow, abantu abaninzi sele besazi. Ziindaba ezimnandi. Kwabo bangaziyo, le yidatabase yothotho lwexesha. Isekelwe kwi-ClickHouse architecture, kwezinye iinkcukacha zokuphunyezwa kweClickHouse. Umzekelo, kwizinto ezifana nezi: MergeTree, ubalo olunxuseneyo kuzo zonke iicore processors ezikhoyo kunye nokuphucula ukusebenza ngokusebenza kwiibhloko zedatha ezibekwe kwicache yeprosesa.
I-VictoriaMetrics ibonelela ngoxinzelelo lwedatha olungcono kunezinye ii-database zexesha.
Ilinganisa ngokuthe nkqo - oko kukuthi, unokongeza iiprosesa ezininzi, ngaphezulu kwe-RAM kwikhompyuter enye. I-VictoriaMetrics iya kusebenzisa ngempumelelo ezi zixhobo zikhoyo kwaye iya kuphucula imveliso yomda.
I-VictoriaMetrics iphinda ilinganise ngokuthe tye - oko kukuthi, unokongeza iinodi ezongezelelweyo kwiqela le-VictoriaMetrics, kwaye ukusebenza kwayo kuya kunyuka phantse ngokulandelelana.
Njengoko uqikelele, iVictoriaMetrics yidatabase ekhawulezayo, kuba andikwazi ukubhala abanye. Kwaye kubhaliwe kwiGo, ke ndithetha ngayo kule ndibano.
Ngubani owaziyo ukuba yintoni uthotho lwexesha? Kwakhona wazi abantu abaninzi. Uluhlu lwexesha luluhlu lwezibini (timestamp, Π·Π½Π°ΡΠ΅Π½ΠΈΠ΅), apho ezi zibini zihlelwe ngokwexesha. Ixabiso linani lendawo edadayo - float64.
Uluhlu lwexesha ngalinye luchongwa ngokukodwa ngesitshixo. Siqulathe ntoni esi sitshixo? Iqulathe iseti engenanto yexabiso eliphambili.
Nanku umzekelo woluhlu lwamaxesha. Isitshixo solu luhlu luluhlu lwezibini: __name__="cpu_usage" ligama le metric, instance="my-server" - le yikhompyuter apho le metric iqokelelwa khona, datacenter="us-east" - eli liziko ledatha apho le khompyutha ikhona.
Siye sagqiba ngegama lothotho lwexesha elibandakanya izibini ezintathu zexabiso eliphambili. Eli qhosha lihambelana noluhlu lwezibini (timestamp, value). t1, t3, t3, ..., tN - ezi zizitampu zexesha, 10, 20, 12, ..., 15 - amaxabiso ahambelanayo. Oku kukusetyenziswa kwecpu ngexesha elinikiweyo kuthotho olunikiweyo.
Lunokusetyenziswa phi uthotho lwamaxesha? Ngaba ukho umntu onayo nayiphi na ingcamango?
- Kwi-DevOps, unokulinganisa i-CPU, i-RAM, inethiwekhi, i-rps, inani leempazamo, njl.
- IoT - sinokulinganisa ubushushu, uxinzelelo, ulungelelwaniso lwe-geo kunye nenye into.
- Kwakhona ngemali - sinokubeka iliso kumaxabiso azo zonke iintlobo zesitokhwe kunye neemali.
- Ukongeza, uchungechunge lwexesha lunokusetyenziswa ekubekeni iliso kwiinkqubo zemveliso kwiifektri. Sinabasebenzisi abasebenzisa iVictoriaMetrics ukujonga iiinjini zomoya, kwiirobhothi.
- Uthotho lwexesha lukwaluncedo ekuqokeleleni ulwazi oluvela kuluvo lwezixhobo ezahlukeneyo. Umzekelo, kwi-injini; ukulinganisa uxinzelelo lwetayara; kumlinganiselo wesantya, umgama; ukulinganisa ukusetyenziswa kwepetroli, njl.
- Uthotho lwamaxesha lunokusetyenziselwa ukujonga iinqwelomoya. Inqwelomoya nganye inebhokisi emnyama eqokelela uthotho lwexesha kwiiparamitha ezahlukeneyo zempilo yenqwelomoya. Uthotho lwexesha lukwasetyenziswa kwishishini le-aerospace.
- Ukhathalelo lwempilo luxinzelelo lwegazi, i-pulse, njl.
Kusenokubakho izicelo ezininzi endizilibeleyo, kodwa ndiyathemba ukuba uyakuqonda ukuba uthotho lwexesha lusetyenziswa ngokusebenzayo kwihlabathi lanamhlanje. Kwaye umthamo wokusetyenziswa kwabo ukhula minyaka le.
Kutheni ufuna isiseko sedatha yexesha? Kutheni ungasebenzisi isiseko sedatha esiqhelekileyo sobudlelwane ukugcina uthotho lwexesha?
Ngenxa yokuba uchungechunge lwexesha luhlala luqulethe ulwazi oluninzi, okunzima ukuyigcina kunye nokusebenza kwiinkcukacha eziqhelekileyo. Ke ngoko, oovimba bedatha abakhethekileyo bothotho lwexesha bavele. Ezi ziseko zigcina amanqaku ngempumelelo (timestamp, value) ngesitshixo esinikiweyo. Banikezela nge-API yokufunda idatha egciniweyo ngesitshixo, ngesitshixo esisodwa-ixabiso lesibini, okanye ngababini bexabiso eliphambili, okanye nge-regexp. Ngokomzekelo, ufuna ukufumana umthwalo we-CPU yazo zonke iinkonzo zakho kwiziko ledatha eMelika, ngoko kufuneka usebenzise lo mbuzo-mbuzo.
Ngokuqhelekileyo uluhlu lweenkcukacha zedatha zibonelela ngeelwimi ezikhethekileyo zemibuzo kuba uthotho lwexesha lwe-SQL alufanelekanga kakhulu. Nangona kukho i-database exhasa i-SQL, ayifanelekanga kakhulu. Buza iilwimi ezinje , , , . Ndiyathemba ukuba kukho umntu okhe weva nokuba ibe nye kwezi lwimi. Abantu abaninzi mhlawumbi bevile ngePromQL. Olu lulwimi lombuzo wePrometheus.
Yile nto imbonakalo yesiseko sedatha yangoku kusetyenziswa iVictoriaMetrics njengomzekelo.
Inezahlulo ezimbini. Olu lugcino lwesalathiso esijikelelweyo kunye nogcino lwamaxabiso othotho lwexesha. Ezi zokugcina zahluliwe.
Xa irekhodi entsha ifika kwisiseko sedatha, siqala ukufikelela kwisalathiso esiguqulweyo ukufumana isichongi sexesha leseti enikiweyo. label=value kwimetric enikiweyo. Sifumana esi sichongi kwaye sigcine ixabiso kwivenkile yedatha.
Xa isicelo sifika ukufumana kwakhona idatha kwi-TSDB, siqala ukuya kwisalathiso esiguqulweyo. Masifumane yonke into timeseries_ids iirekhodi ezihambelana nale seti label=value. Kwaye emva koko sifumana yonke idatha efunekayo kwi-warehouse yedatha, efakwe kwi-indexed by timeseries_ids.
Makhe sijonge umzekelo wendlela uthotho lwedatha esebenza ngayo umbuzo ongenayo okhethiweyo.
- Okokuqala ufumana yonke into
timeseries_idsukusuka kwisalathiso esiguqulweyo esiqulathe izibini ezinikiweyolabel=value, okanye ukwanelisa intetho eqhelekileyo enikiweyo. - Emva koko ifumana zonke iingongoma zedatha kugcino lwedatha ngexesha elinikiweyo lezo zifunyenweyo
timeseries_ids. - Emva koku, i-database yenza izibalo ezithile kula manqaku edatha, ngokwesicelo somsebenzisi. Kwaye emva koko ibuyisela impendulo.
Kule ntetho ndiza kukuxelela ngenxalenye yokuqala. Olu lukhangelo timeseries_ids ngesalathiso esijikiweyo. Unokubukela malunga nenxalenye yesibini kunye nenxalenye yesithathu kamva , okanye linda de ndilungiselele ezinye iingxelo :)
Masiqhubele phambili kwisalathiso esiguqulweyo. Abaninzi banokucinga ukuba oku kulula. Ngubani owaziyo ukuba yintoni isalathiso esigqwethiweyo kwaye sisebenza njani? Owu, abasebaninzi abantu. Makhe sizame ukuqonda ukuba yintoni na.
Ngokwenene ilula. Sisichazi-magama nje esenza imaphu yesitshixo sexabiso. Yintoni isitshixo? Esi sibini label=valuephi label ΠΈ value - le yimigca. Kwaye amaxabiso yiseti timeseries_ids, equka isibini esinikiweyo label=value.
Isalathiso esijikiweyo sikuvumela ukuba ufumane yonke into ngokukhawuleza timeseries_ids, abaye banikela label=value.
Ikwakuvumela ukuba ufumane ngokukhawuleza timeseries_ids ixesha uthotho izibini eziliqela label=value, okanye izibini label=regexp. Kwenzeka njani oku? Ngokufumana ukuhlangana kweseti timeseries_ids ngesibini ngasinye label=value.
Makhe sijonge uphumezo olwahlukeneyo lwesalathiso esijikelelweyo. Masiqale ngeyona ndlela ilula yokuphumeza engenangqondo. Ukhangeleka ngolu hlobo.
Umsebenzi getMetricIDs ufumana uluhlu lweentambo. Umgca ngamnye uqulethe label=value. Lo msebenzi ubuyisela uluhlu metricIDs.
Ingaba isebenza kanjani? Apha sinenguqu yehlabathi ebizwa invertedIndex. Esi sisichazi-magama esiqhelekileyo (map), eya kwenza imephu yomtya ukusika ints. Umgca uqulathe label=value.
Ukuphunyezwa komsebenzi: fumana metricIDs eyokuqala label=value, emva koko sidlula kuyo yonke enye into label=value, siyayifumana metricIDs ukwenzela bona. Kwaye ubize umsebenzi intersectInts, eza kuxutyushwa ngezantsi. Kwaye lo msebenzi ubuyisela ukudibana kolu luhlu.
Njengoko ubona, ukuphumeza isalathiso esijikiweyo akukho nzima kakhulu. Kodwa olu luphunyezo olungenangqondo. Zeziphi izinto ezingalunganga enazo? Eyona nto ingalunganga yokuphunyezwa kwe-naive kukuba isalathiso esinjalo sigcinwe kwi-RAM. Emva kokuqalisa kwakhona isicelo silahlekelwa sesi salathisi. Akukho kugcinwa kwesisalathiso kwidiski. Isalathiso esinjalo esijikiweyo asinakwenzeka ukuba silungele isiseko sedatha.
I-drawback yesibini nayo inxulumene nememori. Isalathisi esijikiweyo kufuneka singene kwi-RAM. Ukuba idlula ubungakanani be-RAM, kuyacaca ukuba siya kuphuma-kwimpazamo yememori. Kwaye inkqubo ayizukusebenza.
Le ngxaki ingasonjululwa kusetyenziswa izisombululo esele zilungisiwe ezifana , okanye .
Ngamafutshane, sifuna i-database evumela ukuba senze imisebenzi emithathu ngokukhawuleza.
- Umsebenzi wokuqala urekhoda
ΠΊΠ»ΡΡ-Π·Π½Π°ΡΠ΅Π½ΠΈΠ΅kule database. Uyenza le nto ngokukhawuleza, aphoΠΊΠ»ΡΡ-Π·Π½Π°ΡΠ΅Π½ΠΈΠ΅ziintambo ezingenamkhethe. - Umsebenzi wesibini kukukhangela ngokukhawuleza ixabiso usebenzisa iqhosha elinikiweyo.
- Kwaye umsebenzi wesithathu lukhangelo olukhawulezayo lwawo onke amaxabiso ngesimaphambili esinikiweyo.
I-LevelDB kunye ne-RocksDB - ezi nkcukacha zaphuhliswa nguGoogle kunye ne-Facebook. Kuqale kwafika iLevelDB. Emva koko abafana abavela ku-Facebook bathatha i-LevelDB kwaye baqala ukuyiphucula, benza iRocksDB. Ngoku phantse zonke iinkcukacha zangaphakathi zisebenza kwi-RocksDB ngaphakathi kwe-Facebook, kubandakanywa nezo zidluliselwe kwi-RocksDB kunye ne-MySQL. Bamthiya igama .
Isalathiso esijikiweyo sinokuphunyezwa kusetyenziswa i-LevelDB. Yenziwa njani? Sigcina njengesitshixo label=value. Kwaye ixabiso sisichongi soluhlu lwexesha apho isibini sikhoyo label=value.
Ukuba sinothotho lwexesha elininzi ngepere enikiweyo label=value, ngoko ke kuya kubakho imiqolo emininzi kwesi siseko sedatha kunye nesitshixo esifanayo nesahlukileyo timeseries_ids. Ukufumana uluhlu lwazo zonke timeseries_ids, eqala ngale nto label=prefix, senza uluhlu lokuskena apho esi siseko sedatha silungiselelwe. Oko kukuthi, sikhetha yonke imigca eqala ngayo label=prefix kwaye ufumane okuyimfuneko timeseries_ids.
Nanku ukuphunyezwa kwesampulu yendlela enokujongeka ngayo kwiGo. Sinesalathisi esijikiweyo. Le yi-LevelDB.
Umsebenzi uyafana nophunyezo olungenangqondo. Iphinda uzalisekiso olungenangqondo phantse umgca ngomgca. Inqaku kuphela kukuba endaweni yokujika map sifikelela kwisalathisi esijikiweyo. Sifumana onke amaxabiso okokuqala label=value. Emva koko sidlula kuzo zonke izibini eziseleyo label=value kwaye ufumane iiseti ezihambelanayo zeemetricIDs zabo. Emva koko sifumana isiphambuka.
Yonke into ibonakala ilungile, kodwa kukho iingxaki kwesi sisombululo. I-VictoriaMetrics ekuqaleni iphumeze isalathiso esiguquliweyo esekwe kwi-LevelDB. Kodwa ekugqibeleni kwafuneka ndiyeke.
Ngoba? Ngenxa yokuba i-LevelDB iyacotha kunokuphunyezwa kokungenangqondo. Kuphunyezo olungenangqondo, lunikwe isitshixo esinikiweyo, sifumana ngokukhawuleza isilayi sonke metricIDs. Lo ngumsebenzi okhawulezayo kakhulu - isilayi sonke silungele ukusetyenziswa.
Kwi-LevelDB, ngalo lonke ixesha umsebenzi ubizwa GetValues kufuneka uhambe kuyo yonke imigca eqala ngayo label=value. Kwaye ufumane ixabiso lomgca ngamnye timeseries_ids. Kwezinjalo timeseries_ids qokelela iqhekeza lezi timeseries_ids. Ngokucacileyo, oku kucotha kakhulu kunokufikelela kwimephu eqhelekileyo ngesitshixo.
I-drawback yesibini kukuba i-LevelDB ibhalwe kwi-C. Ukubiza imisebenzi ye-C ukusuka kwi-Go ayikhawulezi kakhulu. Kuthatha amakhulu ee nanoseconds. Oku akukhawulezi kakhulu, kuba xa kuthelekiswa nomnxeba oqhelekileyo womsebenzi obhaliweyo ekuhambeni, othatha i-1-5 nanoseconds, umehluko ekusebenzeni ngamashumi amaxesha. KwiVictoriaMetrics esi yayisisiphene esibulalayo :)
Ngoko ke ndabhala eyam ukuphunyezwa kwesalathisi esiguqulelweyo. Uye wambiza .
I-Mergeset isekwe kulwakhiwo lwedatha ye-MergeTree. Olu lwakhiwo lwedatha lubolekwe kwiClickHouse. Ngokucacileyo, ukudibanisa kufuneka kulungiselelwe ukukhangela okukhawulezayo timeseries_ids ngokwesitshixo esinikiweyo. I-Mergeset ibhalwe ngokupheleleyo kwi-Go. Ungabona . Ukuphunyezwa kokudibanisa kukwifolda . Ungazama ukufumanisa ukuba kuqhubeka ntoni apho.
Ukudibanisa i-API ifana kakhulu ne-LevelDB kunye ne-RocksDB. Oko kukuthi, ikuvumela ukuba ugcine ngokukhawuleza iirekhodi ezintsha apho kwaye ukhethe ngokukhawuleza iirekhodi ngesimaphambili esinikiweyo.
Siza kuthetha malunga nokungalunganga kokudibanisa kamva. Ngoku makhe sithethe malunga nokuba zeziphi iingxaki eziye zavela kunye neVictoriaMetrics kwimveliso xa kusetyenziswa isalathiso esiguquliweyo.
Kwakutheni ukuze bavele?
Isizathu sokuqala lizinga eliphezulu le-churn. Ukuguqulelwa kwisiRashiya, olu lutshintsho rhoqo kuthotho lwexesha. Oku kuxa uthotho lwexesha luphela kwaye uthotho olutsha luqala, okanye uthotho lwexesha olutsha luqala. Kwaye oku kwenzeka rhoqo.
Isizathu sesibini linani elikhulu loluhlu lwexesha. Ekuqaleni, xa ukubeka iliso kwakufumana ukuthandwa, inani lexesha lexesha lalincinci. Umzekelo, kwikhompyuter nganye kufuneka ubeke iliso kwi-CPU, imemori, inethiwekhi kunye nomthwalo wediski. 4 uthotho lwexesha kwikhompyuter nganye. Masithi uneekhompyuter ezili-100 kunye nothotho lwamaxesha angama-400. Oku kuncinci kakhulu.
Ngokuhamba kwexesha, abantu bafumanisa ukuba banokulinganisa ulwazi oluthe kratya. Umzekelo, linganisa umthwalo ongewona weprosesa yonke, kodwa ngokwahlukileyo kwisiseko seprosesa nganye. Ukuba unama-cores angama-40 eprosesa, ke unamaxesha angama-40 ngaphezulu kwexesha lokulinganisa umthwalo weprosesa.
Kodwa akuphelelanga apho. Undoqo weprosesa nganye unokuba namazwe amaninzi, njengokungenzi nto, xa ingasebenzi. Kwaye usebenze kwindawo yomsebenzisi, usebenze kwindawo ye-kernel nakwamanye amazwe. Kwaye imeko nganye enjalo inokulinganiswa njengoluhlu lwexesha olwahlukileyo. Oku kwandisa inani lemiqolo ngamaxesha angama-7-8.
Kwimetric enye sifumene 40 x 8 = 320 metrics kwikhompyuter enye nje. Phinda-phinda nge-100, sifumana ama-32 endaweni ye-000.
Emva koko uKubernetes weza. Kwaye iye yaba mandundu kuba iKubernetes inokusingatha iinkonzo ezininzi ezahlukeneyo. Inkonzo nganye eKubernetes ineepod ezininzi. Kwaye konke oku kufuneka kubekwe esweni. Ukongeza, sinokusasazwa rhoqo kweenguqulelo ezintsha zeenkonzo zakho. Kwinguqulelo entsha nganye, uthotho lwexesha olutsha kufuneka lwenziwe. Ngenxa yoko, inani lexesha lexesha likhula ngokukhawuleza kwaye sijongene nengxaki yenani elikhulu lexesha, elibizwa ngokuba yi-high-cardinality. IVictoriaMetrics ihlangabezana nayo ngempumelelo xa ithelekiswa nolunye uluhlu lwedatha yedatha.
Makhe sijonge ngakumbi kwireyithi ephezulu ye-churn. Yintoni ebangela izinga eliphezulu lokuvuthwa kwemveliso? Ngenxa yokuba ezinye iintsingiselo zeelebhile kunye neethegi zihlala zitshintsha.
Ngokomzekelo, thatha i-Kubernetes, enombono deployment, o.k. xa uguqulelo olutsha lwesicelo sakho lukhutshiwe. Ngesizathu esithile, abaphuhlisi be-Kubernetes bagqibe ekubeni bongeze i-id yokuthunyelwa kwileyibhile.
Oku kwakhokelela entwenini? Ngaphezu koko, ngokusasazwa okutsha ngalunye, lonke ixesha elidala liphazamisekile, kwaye endaweni yazo, uchungechunge lwexesha elitsha luqala ngexabiso elitsha lelebhile. deployment_id. Kunokubakho amakhulu amawaka kunye nezigidi zemiqolo enjalo.
Into ebalulekileyo malunga nayo yonke le nto kukuba inani elipheleleyo loluhlu lwexesha likhula, kodwa inani lexesha elisebenzayo ngoku kunye nokufumana idatha lihlala lihlala lihleli. Le meko ibizwa ngokuba yi-high churn rate.
Eyona ngxaki iphambili yesantya esiphezulu se-churn kukuqinisekisa isantya sokukhangela rhoqo kulo lonke uthotho lwexesha kwiseti enikiweyo yeelebhile kwisithuba sexesha elithile. Ngokuqhelekileyo eli lithuba lexesha leyure yokugqibela okanye usuku lokugqibela.
Indlela yokusombulula le ngxaki? Nalu ukhetho lokuqala. Oku kukwahlula isalathisi esijikiweyo sibe ngamalungu azimeleyo ngexesha. Oko kukuthi, ixesha elithile lidlula, sigqibezela ukusebenza ngesalathiso sangoku esiguqulweyo. Kwaye wenze isalathisi esitsha. Elinye ixesha lidlula, sidala enye kunye nenye.
Kwaye xa sithatha iisampulu kwezi zalathisi ziguqulweyo, sifumana iseti yezalathisi ezijikiweyo eziwela phakathi kwesithuba esinikiweyo. Kwaye, ngokufanelekileyo, sikhetha i-id yochungechunge lwexesha ukusuka apho.
Oku konga izixhobo kuba akunyanzelekanga ukuba sijonge iindawo ezingaweliyo kwixesha elinikiweyo. Oko kukuthi, ngokuqhelekileyo, ukuba sikhetha idatha yeyure yokugqibela, ngoko kwixesha elidlulileyo siyatsiba izicelo.
Kukho enye inketho yokusombulula le ngxaki. Oku kukugcina usuku ngalunye uluhlu olwahlukileyo lwee-ID zothotho lwexesha olwenzeka ngolo suku.
I-advanteji yesi sisombululo kwisisombululo sangaphambili kukuba asiphindi ulwazi lwexesha elinganyamalaliyo ekuhambeni kwexesha. Zihlala zikhona kwaye azitshintshi.
Ukungalungi kukuba eso sisombululo sinzima kakhulu ukusiphumeza kwaye kunzima ukusilungisa. Kwaye iVictoriaMetrics yakhetha esi sisombululo. Le yindlela okwenzeka ngayo ngokwembali. Esi sisombululo sikwasebenza kakuhle xa kuthelekiswa nangaphambili. Ngenxa yokuba esi sisombululo asizange siphunyezwe ngenxa yokuba kuyimfuneko ukuphindaphinda idatha kwisahlulelo ngasinye soluhlu lwexesha olungatshintshiyo, o.k.t. olungapheli ngokuhamba kwexesha. I-VictoriaMetrics yayilungiselelwe ngokuyintloko ukusetyenziswa kwendawo yediski, kwaye ukuphunyezwa kwangaphambili kwenza ukusetyenziswa kwendawo yediski kubi. Kodwa oku kuphunyezwa kulungeleka ngakumbi ekunciphiseni ukusetyenziswa kwendawo yediski, ngoko kwakhethwa.
Kwafuneka ndilwe naye. Umzabalazo yayikukuba kolu phumezo kusafuneka ukhethe inani elikhulu kakhulu timeseries_ids yedata kunaxa isalathisi esijikiweyo sikwahlulwe ixesha.
Sayicombulula njani le ngxaki? Siyisombulule ngendlela yoqobo - ngokugcina iziphawuli zothotho lwamaxesha aliqela kungeniso lwesalathiso esijikelelweyo endaweni yesiphawuli esinye. Oko kukuthi, sinesitshixo label=value, eyenzeka ngalo lonke ixesha uthotho. Kwaye ngoku sigcina ezininzi timeseries_ids kwingeniso enye.
Nanku umzekelo. Ngaphambili besinamangeno e-N, kodwa ngoku sinengeniso enye esimaphambili sayo sifana nazo zonke ezinye. Kungeno lwangaphambili, ixabiso liqulathe lonke ixesha ii-ID zothotho.
Oku kwenza ukuba kukwazeke ukunyusa isantya sokuskena kweso salathisi siguqulweyo ukuya kumaxesha ali-10. Kwaye kwasivumela ukuba sinciphise ukusetyenziswa kwememori kwi-cache, kuba ngoku sigcina umtya label=value kube kanye kuphela kwindawo yokugcina kunye amaxesha N. Kwaye lo mgca unokuba mkhulu ukuba ugcina imigca emide kwiithegi zakho kunye neeleyibhile, uKubernetes athanda ukuyityhala apho.
Olunye ukhetho lokukhawulezisa ukukhangela kwisalathiso esiguqulweyo kukusharding. Ukwenza izalathisi ezininzi eziguqulweyo endaweni yesinye kunye nokwabelana ngedatha phakathi kwazo ngesitshixo. Le iseti key=value umphunga. Oko kukuthi, sifumana izalathisi ezininzi ezizimeleyo eziguqulweyo, esinokubuza ngazo ngokunxuseneyo kwiiprosesa ezininzi. Ukuphunyezwa kwangaphambili kuvunyelwe kuphela ukusebenza kwimowudi yeprosesa enye, oko kukuthi, ukuskena idatha kwisiseko esinye kuphela. Esi sisombululo sikuvumela ukuba uskene idatha kwiicores ezininzi ngexesha elinye, njengoko iClickHouse ithanda ukwenza. Yile nto siceba ukuyiphumeza.
Ngoku makhe sibuyele kwiigusha zethu - kumsebenzi we-intersection timeseries_ids. Makhe siqwalasele ukuba yintoni ukuphunyezwa enokubakho. Lo msebenzi ikuvumela ukuba ufumane timeseries_ids kwiseti enikiweyo label=value.
Inketho yokuqala kukuphunyezwa ngokungenangqondo. Izirhintyela ezimbini. Apha sifumana igalelo lomsebenzi intersectInts izilayi ezibini - a ΠΈ b. Kwimveliso, kufuneka ibuyele kuthi ukuhlangana kwezi zilayi.
Uzalisekiso olungenangqondo lukhangeleka ngolu hlobo. Siphindaphinda onke amaxabiso ukusuka kwisilayi a, ngaphakathi kwesi sixhobo sihamba kuwo onke amaxabiso esilayi b. Kwaye siyabathelekisa. Ukuba ziyahambelana, ngoko sifumene isiphambuka. Kwaye uyigcine kuyo result.
Ziziphi izinto ezingeloncedo? Ubunzima be-quadratic yintsilelo yayo ephambili. Umzekelo, ukuba imilinganiselo yakho isilayi a ΠΈ b isigidi esinye ngexesha, ke lo msebenzi soze ubuyisele impendulo kuwe. Kuba kuya kufuneka yenze itriliyoni enye, nto leyo ininzi nakwiikhompyuter zanamhlanje.
Ukuphunyezwa kwesibini kusekelwe kwimephu. Senza imaphu. Sibeka onke amaxabiso ukusuka kwisilayi kule mephu a. Emva koko sihamba ngeqhekeza kwi-loop eyahlukileyo b. Kwaye sijonga ukuba eli xabiso lisuka kwisilayi b kwimephu. Ukuba ikhona, yongeze kwisiphumo.
Ziziphi iingenelo? Inzuzo kukuba kukho ubunzima bomgca kuphela. Oko kukuthi, umsebenzi uya kwenza ngokukhawuleza kakhulu kwizilayi ezinkulu. Ngesilayi sobukhulu besigidi, lo msebenzi uza kuphunyezwa ngokuphindaphindwa kwezigidi ezi-2, ngokuchaseneyo netriliyoni ephindaphindwayo yomsebenzi wangaphambili.
Icala elisezantsi kukuba lo msebenzi ufuna inkumbulo eninzi ukwenza le mephu.
I-drawback yesibini yi-overhead enkulu ye-hashing. Le drawback ayibonakali kakhulu. Kwaye nakuthi kwakungabonakali kakhulu, ngoko ke ekuqaleni kwi-VictoriaMetrics ukuphunyezwa kwendlela yokuhlangana yayihamba ngemephu. Kodwa ke iprofayili ibonise ukuba ixesha eliphambili leprosesa lichithwa ukubhala kwimephu kunye nokujonga ubukho bexabiso kule mephu.
Kutheni ixesha le-CPU lichithwa kwezi ndawo? Kuba iGo yenza umsebenzi we-hashing kule migca. Oko kukuthi, ibala i-hash yesitshixo ukuze ikwazi ukufikelela kuyo kwisalathiso esinikiweyo kwiHashMap. Umsebenzi wokubala we-hash ugqitywe kumashumi e-nanoseconds. Oku kuyacotha kwiVictoriaMetrics.
Ndigqibe ekubeni ndiphumeze i-bitset elungiselelwe ngokukodwa le meko. Yile ndlela ukudibana kwezilayi ezibini ngoku kujongeka ngayo. Apha sidala i-bitset. Songeza izinto ukusuka kwisilayi sokuqala kuyo. Emva koko sijonga ubukho bezi zinto kwisiqwenga sesibini. Kwaye wongeze kwisiphumo. Oko kukuthi, phantse akukho kwahluka kumzekelo wangaphambili. Ekuphela kwento elapha kukuba sitshintshe ukufikelela kwimephu ngemisebenzi yesiko add ΠΈ has.
Ekuboneni kuqala, kubonakala ngathi oku kufuneka kusebenze kancinci, ukuba ngaphambili imephu eqhelekileyo yayisetyenziswa apho, kwaye ke eminye imisebenzi ibizwa ngokuba, kodwa iprofayili ibonisa ukuba le nto isebenza ngokuphindwe kayi-10 ngokukhawuleza kunemephu eqhelekileyo kwimeko yeVictoriaMetrics.
Ukongeza, isebenzisa imemori encinci kakhulu xa kuthelekiswa nokuphunyezwa kwemephu. Kuba sigcina amasuntswana apha endaweni yeebhayithi ezisibhozo.
Ukungalungi kolu phumezo kukuba ayicacanga kangako, hayi into encinci.
Enye i-drawback enokuthi abaninzi bangayiqapheli kukuba oku kuphunyezwa akunakusebenza kakuhle kwezinye iimeko. Oko kukuthi, ilungiselelwe imeko ethile, kule meko yokudibana kwe-ID zexesha le-VictoriaMetrics. Oku akuthethi ukuba ifanelekile kuzo zonke iimeko. Ukuba isetyenziswe ngokungalunganga, asiyi kufumana ukunyuka komsebenzi, kodwa impazamo ephuma kwimemori kunye nokunciphisa ukusebenza.
Makhe siqwalasele ukuphunyezwa kwesi sakhiwo. Ukuba ufuna ukujonga, ibekwe kwimithombo yeVictoriaMetrics, kwifolda . Yenzelwe ngokukodwa imeko yeVictoriaMetrics, apho timeseries_id lixabiso lamasuntswana angama-64, apho amasuntswana angama-32 ahlala engatshintshi kwaye ngawokugqibela angama-32 atshintshayo.
Olu lwakhiwo lwedatha alugcinwanga kwidiski, lusebenza kuphela kwinkumbulo.
Nantsi i-API yayo. Ayintsonkothanga kakhulu. I-API ilungiselelwe ngokukodwa kumzekelo othile wokusebenzisa iVictoriaMetrics. Oko kukuthi, akukho misebenzi engeyomfuneko apha. Nantsi imisebenzi esetyenziswa ngokucacileyo yiVictoriaMetrics.
Kukho imisebenzi add, eyongeza amaxabiso amatsha. Kukho umsebenzi has, ejonga amaxabiso amatsha. Kwaye kukho umsebenzi del, esusa amaxabiso. Kukho umsebenzi womncedisi len, ebuyisela ubungakanani besethi. Umsebenzi clone clones kakhulu. Kwaye umsebenzi appendto iguqula le seti ibe sisilayi timeseries_ids.
Yile nto ukuphunyezwa kwesi sakhiwo sedatha kubonakala ngathi. iseti inezinto ezimbini:
-
ItemsCountngumhlaba womncedisi ukubuyisela ngokukhawuleza inani lezinto kwiseti. Kuyakwenzeka ukwenza ngaphandle kwale ndawo yokuncedisa, kodwa bekufuneka yongezwe apha kuba iVictoriaMetrics ihlala ibuza ubude be-bitset kwii-algorithms zayo. -
Intsimi yesibini
buckets. Oku sisilayi ukusuka kwisakhiwobucket32. Isakhiwo ngasinye sigcinahiintsimi. La ngamasuntswana angama-32 aphezulu. Kwaye izilayi ezibini -b16hisΠΈbucketsΠΈΠ·bucket16izakhiwo.
Iibhithi eziphezulu ze-16 zenxalenye yesibini yesakhiwo se-64-bit zigcinwe apha. Kwaye apha ii-bitsets zigcinelwe amasuntswana ali-16 ebhayithi nganye.
Bucket64 iqulathe uluhlu uint64. Ubude bubalwa kusetyenziswa ezi zigxina. Kwinye bucket16 ubuninzi bunokugcinwa 2^16=65536 kancinci. Ukuba uyahlulahlula ngo8, ngoko yi 8 kilobytes. Ukuba uyahlulahlula ngo8 kwakhona, yi1000 uint64 intsingiselo. Yiyo i Bucket16 β esi sisakhiwo sethu se-8-kilobyte.
Makhe sijonge indlela enye yeendlela zesi sakhiwo ukongeza ixabiso elitsha iphunyezwa.
Konke kuqala nge uint64 iintsingiselo. Sibala amasuntswana angama-32 aphezulu, sibala amasuntswana angama-32 asezantsi. Masihambe kuyo yonke into buckets. Sithelekisa iibhithi eziphezulu ze-32 kwibhakethi nganye kunye nexabiso elongezelelweyo. Kwaye ukuba ziyahambelana, ngoko sibiza umsebenzi add kwisakhiwo b32 buckets. Kwaye udibanise amasuntswana angama-32 apho. Kwaye ukuba ibuyile true, ngoko oku kuthetha ukuba songeze ixabiso elinjalo apho kwaye asizange sibe nexabiso elinjalo. Ukuba iyabuya false, ngoko intsingiselo enjalo yayisele ikho. Emva koko sandisa inani lezinto kwisakhiwo.
Ukuba asikayifumani le oyifunayo bucket kunye nexabiso le-hi elifunekayo, ngoko sibiza umsebenzi addAlloc, eya kuvelisa entsha bucket, ukongeza kwisakhiwo sebhakethi.
Oku kukuphunyezwa komsebenzi b32.add. Iyafana nokuphunyezwa kwangaphambili. Sibala awona masuntswana ali-16 abalulekileyo, amasuntswana ali-16 ubuncinane.
Emva koko sihamba kuwo onke amasuntswana angama-16 aphezulu. Sifumana iimatshisi. Kwaye ukuba kukho umdlalo, sibiza indlela yokongeza, esiya kuyiqwalasela kwiphepha elilandelayo bucket16.
Kwaye nali elona nqanaba lisezantsi, ekufuneka liphuculwe kangangoko kunokwenzeka. Sibala i uint64 ixabiso id kwisilayi kancinane kwaye kwakhona bitmask. Le yimaski yexabiso elinikiweyo le-64-bit, elinokusetyenziswa ukujonga ubukho bale bit, okanye ukuseta. Sijonga ukubona ukuba le ntwana isetiwe kwaye iyisete, kwaye sibuyisele ubukho. Oku kukuphunyezwa kwethu, okusivumele ukuba sikhawulezise ukusebenza kwee-ID ze-intersecting of time series ngamaxesha angama-10 xa kuthelekiswa neemephu eziqhelekileyo.
Ukongeza kolu lungiselelo, iVictoriaMetrics inolunye ulungiselelo oluninzi. Uninzi lwezi zilungiso zongezwa ngesizathu, kodwa emva kokwenza iprofayili yekhowudi kwimveliso.
Lo ngowona mgaqo uphambili wokwenza usetyenziso- musa ukongeza ukulungelelaniswa ucinga ukuba kuya kubakho umqobo apha, kuba kungavela ukuba akusayi kubakho mqobo apho. Ukuphucula ngokuqhelekileyo kuthoba umgangatho wekhowudi. Ke ngoko, kuyafaneleka ukukhulisa kuphela emva kokwenza iprofayili kwaye kukhethwa kwimveliso, ukuze le yidatha yokwenyani. Ukuba nabani na onomdla, ungajonga ikhowudi yemvelaphi yeVictoriaMetrics kwaye ujonge ezinye izinto ezikhona.
Ndinombuzo malunga ne-bitset. Ifana kakhulu nokuphunyezwa kwe-bool ye-C++ ye-vector, i-bitset ephuculweyo. Ngaba uthathe ukuphunyezwa apho?
Hayi, hayi ukusuka apho. Xa kuphunyezwa le bitset, ndandikhokelwa lulwazi lwesakhiwo sezi ids timeseries, ezisetyenziswa VictoriaMetrics. Kwaye ulwakhiwo lwazo lunjalo kangangokuba amasuntswana angama-32 aphezulu ahlala rhoqo. Amasuntswana angama-32 asezantsi angatshintsha. Okukhona isezantsi kancinci, kokukhona inokutshintsha. Ngoko ke, oku kuphunyezwa kulungiselelwe ngokukodwa olu lwakhiwo lwedatha. Ukuphunyezwa kweC ++, ngokokwazi kwam, kulungiselelwe imeko ngokubanzi. Ukuba ulungiselela imeko ngokubanzi, oku kuthetha ukuba ayizukuba yeyona ilungileyo kwimeko ethile.
Ndikwacebisa ukuba ujonge ingxelo ka-Alexey Milovid. Malunga nenyanga edlulileyo, wathetha malunga nokwenza ngcono kwiClickHouse kwiingcali ezithile. Utsho nje ukuba kwimeko eqhelekileyo, ukuphunyezwa kwe-C ++ okanye okunye ukuphunyezwa kulungiselelwe ukusebenza kakuhle kumndilili esibhedlele. Inokwenza kakubi kunokuphunyezwa kolwazi oluthile olufana nolwethu, apho saziyo ukuba amasuntswana angama-32 aphezulu ahlala rhoqo.
Ndinombuzo wesibini. Ngowuphi umahluko osisiseko kwi-InfluxDB?
Zininzi iiyantlukwano ezisisiseko. Ngokumalunga nokusebenza kunye nokusetyenziswa kwememori, i-InfluxDB kwiimvavanyo ibonisa amaxesha e-10 ngaphezulu kokusetyenziswa kwememori kwi-high cardinality time series, xa ubuninzi babo, umzekelo, izigidi. Ngokomzekelo, i-VictoriaMetrics idla i-1 GB kwimiqolo esebenzayo yesigidi, ngelixa i-InfluxDB idla i-10 GB. Kwaye ngumahluko omkhulu lowo.
Umahluko wesibini osisiseko kukuba i-InfluxDB ineelwimi ezinemibuzo engaqhelekanga-i-Flux kunye ne-InfluxQL. Azikho lula kakhulu ukusebenza kunye nothotho lwexesha xa kuthelekiswa , exhaswa yiVictoriaMetrics. I-PromQL lulwimi lombuzo oluvela kwi-Prometheus.
Kwaye omnye umahluko kukuba i-InfluxDB inemodeli yedatha engaqhelekanga kancinane, apho umgca ngamnye unokugcina amasimi amaninzi ngeseti eyahlukileyo yeethegi. Le migca yahlulwe ngokubhekele phaya ngokweetheyibhile ezahlukeneyo. Ezi ngxaki zongezelelweyo ziwenza nzima umsebenzi olandelayo kwesi siseko sedatha. Kunzima ukuxhasa nokuqonda.
KwiVictoriaMetrics yonke into ilula kakhulu. Apho, uthotho lwexesha ngalinye liyisitshixo-ixabiso. Ixabiso yiseti yamanqaku - (timestamp, value), kwaye isitshixo yiseti label=value. Akukho mahluko phakathi kwamabala kunye nemilinganiselo. Ikuvumela ukuba ukhethe nayiphi na idatha kwaye emva koko udibanise, udibanise, uthabathe, uphindaphinde, uhlukanise, ngokungafaniyo ne-InfluxDB apho izibalo phakathi kwemigca eyahlukeneyo zingekaphunyezwa ngokokwazi kwam. Nokuba ziphunyeziwe, kunzima, kufuneka ubhale ikhowudi eninzi.
Ndinombuzo ocacisayo. Ngaba ndiqonde ngokuchanekileyo ukuba bekukho uhlobo lwengxaki othethe ngayo, ukuba esi salathiso siguqulweyo asingeni kwinkumbulo, ngoko kukho ukwahlula apho?
Kuqala, ndibonise uzalisekiso olungenangqondo lwesalathiso esijikiweyo kwimephu yeGo esemgangathweni. Olu phunyezo alufanelekanga kwiziko ledatha kuba esi salathisi siguqulweyo asigcinwanga kwidiski, kwaye isiseko sedatha kufuneka sigcine kwidiski ukuze le datha ihlale ikhona xa kuqalwa kwakhona. Kolu phumezo, xa uqala kwakhona isicelo, isalathisi sakho esiguqulweyo siya kunyamalala. Kwaye uya kuphulukana nokufikelela kuyo yonke idatha kuba awuyi kukwazi ukuyifumana.
Mholo! Enkosi ngengxelo! Igama lam ndinguPavel. Ndisuka eWildberries. Ndinemibuzo embalwa kuwe. Umbuzo omnye. Ngaba ucinga ukuba ukhethe umgaqo owahlukileyo xa usakha ulwakhiwo lwesicelo sakho kwaye wahlulahlula idatha ngokuhamba kwexesha, mhlawumbi ubuya kukwazi ukunqumla idatha xa ukhangela, ngokusekwe kuphela kwinto yokuba isahlulelo esinye sinedatha enye Ixesha, oko kukuthi, ngexesha elinye kwaye awuzukuba nexhala malunga nokuba amaqhekeza akho asasazeke ngokwahlukileyo? Inombolo yombuzo 2 - ekubeni usebenzisa i-algorithm efanayo kunye ne-bitset nayo yonke enye into, mhlawumbi uzame ukusebenzisa imiyalelo yeprosesa? Mhlawumbi ukhe wazama ukulungiselelwa okunjalo?
Ndiphendule eyesibini kwangoko. Asikafiki kwelo nqanaba okwangoku. Kodwa ukuba kuyimfuneko, siya kufika apho. Kwaye owokuqala, wawuthini umbuzo?
Uxoxe ngeemeko ezimbini. Kwaye bathi bakhetha eyesibini enokuphunyezwa okuntsokothileyo. Kwaye abazange bakhethe eyokuqala, apho idatha yahlulahlulwe ngexesha.
Ewe. Kwimeko yokuqala, umthamo opheleleyo wesalathiso uya kuba mkhulu, kuba kwisahlulelo ngasinye kuya kufuneka sigcine idatha ephindwe kabini yolu luhlu lwexesha oluqhubekayo kuzo zonke ezi zahlulelo. Kwaye ukuba ixesha lakho loluhlu lwe-churn lincinci, o.k.t uthotho olufanayo luhlala lusetyenziswa, ke kwimeko yokuqala siya kuphulukana kakhulu nesixa sendawo yedisk ethathiweyo xa kuthelekiswa nemeko yesibini.
Kwaye ke-ewe, ukwahlula ixesha lukhetho oluhle. UPrometheus uyayisebenzisa. Kodwa uPrometheus unenye ingxaki. Xa udibanisa ezi ziqwenga zedatha, kufuneka igcine kwimemori ulwazi lwemeta yazo zonke iilebhile kunye namaxesha. Ke ngoko, ukuba iziqwenga zedatha ezidibanisayo zikhulu, ke ukusetyenziswa kwememori konyuka kakhulu ngexesha lokudibanisa, ngokungafaniyo neVictoriaMetrics. Xa kudityaniswa, iVictoriaMetrics ayityi nkumbulo kwaphela; zimbini kuphela iikhilobhayithi ezisetyenzisiweyo, nokuba ingakanani na iingceba zedatha ezidityanisiweyo.
I-algorithm oyisebenzisayo isebenzisa imemori. Iphawula iithegi zoluhlu lwamaxesha eziqulathe amaxabiso. Kwaye ngale ndlela ujonga ubukho obudityanisiweyo kuluhlu lwedatha enye nakwenye. Kwaye uyaqonda ukuba i-intersect yenzeka okanye hayi. Ngokuqhelekileyo, ii-database zisebenzisa i-cursors kunye ne-iterators ezigcina umxholo wazo wangoku kwaye ziqhube idatha ehleliweyo ngenxa yobunzima obulula bale misebenzi.
Kutheni singasebenzisi iikhesa ukunqumla idatha?
Ewe.
Sigcina imiqolo ehleliweyo kwi-LevelDB okanye i-mergeset. Singahambisa ikhesa kwaye sifumane isiphambuka. Kutheni singayisebenzisi? Kuba iyacotha. Kuba izikhesa zithetha ukuba kufuneka ubize umsebenzi kumgca ngamnye. Ifowuni yomsebenzi yi-5 nanoseconds. Kwaye ukuba une 100 imigca, kuyavela ukuba sichitha isiqingatha sesekhondi sifowunela umsebenzi.
Kukho into enjalo, ewe. Kwaye umbuzo wam wokugqibela. Umbuzo unokuvakala ungaqhelekanga. Kutheni kungenakwenzeka ukufunda zonke ii-aggregates eziyimfuneko ngelixa idatha ifika kwaye igcinwe kwifom efunekayo? Kutheni ugcine imiqulu emikhulu kwezinye iinkqubo ezifana neVictoriaMetrics, iClickHouse, njl., kwaye emva koko uchithe ixesha elininzi kuzo?
Ndiza kunika umzekelo ukuze icace. Masithi sisebenza njani isixhobo sokulinganisa isantya sento yokudlala? Irekhoda umgama owuhambileyo, lonke ixesha uyongeza kwixabiso elinye, kwaye okwesibini-ixesha. Kwaye iyahlula. Kwaye ufumana isantya esiphakathi. Unokwenza into efanayo. Dibanisa zonke iinyani eziyimfuneko kubhabho.
Kulungile, ndiyawuqonda umbuzo. Umzekelo wakho unendawo yawo. Ukuba uyazi ukuba zeziphi ii-aggregates ozifunayo, ke lo lolona phunyezo lulungileyo. Kodwa ingxaki kukuba abantu bagcina ezi metrics, enye idatha kwi-ClickHouse kwaye abakazi ukuba baya kudibanisa njani kwaye bahluze kwixesha elizayo, ngoko kufuneka bagcine yonke idatha eluhlaza. Kodwa ukuba uyazi ukuba kufuneka ubale into emyinge, kutheni ungayibali endaweni yokugcina inqwaba yamaxabiso aluhlaza apho? Kodwa oku kuphela ukuba uyazi kakuhle into oyifunayo.
Ngendlela, oovimba beenkcukacha zokugcina uthotho lwexesha lwenkxaso yokubalwa kwee-aggregates. Umzekelo, uPrometheus uyaxhasa . Oko kukuthi, oku kunokwenziwa ukuba uyazi ukuba zeziphi iiyunithi oza kuzifuna. I-VictoriaMetrics ayinayo le nto okwangoku, kodwa ihlala ilandelwa yi-Prometheus, apho oku kunokwenziwa kwimithetho yokurekhoda.
Umzekelo, kumsebenzi wam wangaphambili bendidinga ukubala inani leziganeko kwifestile etyibilikayo kwiyure yokugqibela. Ingxaki kukuba kwafuneka ndenze ukuphunyezwa kwesiko kwi-Go, oko kukuthi inkonzo yokubala le nto. Le nkonzo ekugqibeleni yayingeyonto encinci, kuba kunzima ukuyibala. Ukuphunyezwa kunokuba lula ukuba ufuna ukubala ezinye ii-aggregates ngamaxesha amiselweyo. Ukuba ufuna ukubala iziganeko kwifestile etyibilikayo, ngoko ayilula njengoko ibonakala. Ndicinga ukuba oku akukaphunyezwa kwi-ClickHouse okanye kwii-database ze-timeseries, kuba kunzima ukuphumeza.
Kwaye omnye umbuzo. Sasithetha nje nge-avareji, kwaye ndakhumbula ukuba kwakukho into efana neGraphite ene-Carbon backend. Kwaye wayesazi indlela yokunciphisa idatha endala, oko kukuthi, ukushiya inqaku elinye ngomzuzu, inqaku elinye ngeyure, njl njl. Ngokomgaqo, oku kukulungele kakhulu ukuba sifuna idatha ekrwada, xa sithetha, inyanga, kunye nayo yonke enye into enokuyenza. icuthwe . Kodwa iPrometheus kunye neVictoriaMetrics ayikuxhasi oku kusebenza. Ngaba icwangcisiwe ukuyixhasa? Ukuba akunjalo, kutheni kungenjalo?
Enkosi ngombuzo. Abasebenzisi bethu babuza lo mbuzo ngamaxesha athile. Babuza ukuba siyakongeza nini inkxaso yokuthoba isampulu. Kukho iingxaki ezininzi apha. Okokuqala, wonke umsebenzisi uyaqonda downsampling into eyahlukileyo: umntu ufuna ukufumana naliphi na inqaku elingenasizathu kwisithuba esinikiweyo, umntu ufuna ubuninzi, ubuncinci, amaxabiso aphakathi. Ukuba iinkqubo ezininzi zibhala idatha kwisiseko sedatha yakho, ngoko awukwazi ukuyidibanisa yonke. Kungenzeka ukuba inkqubo nganye ifuna ukuncitshiswa okwahlukileyo. Kwaye oku kunzima ukuphumeza.
Kwaye into yesibini kukuba i-VictoriaMetrics, efana ne-ClickHouse, ilungiselelwe ukusebenza kwixabiso elikhulu ledatha ekrwada, ngoko inokufola imigca yeebhiliyoni ngaphantsi kwesibini ukuba unama-cores amaninzi kwinkqubo yakho. Amanqaku othotho lwexesha lokuskena kwiVictoriaMetrics - 50 amanqaku ngesekondi nganye. Kwaye lo msebenzi ulinganisa kwii-cores ezikhoyo. Oko kukuthi, ukuba unama-cores angama-000, umzekelo, uya kujonga amanqaku ebhiliyoni ngomzuzwana. Kwaye le propati yeVictoriaMetrics kunye neClickHouse inciphisa imfuno yokunciphisa.
Olunye uphawu kukuba iVictoriaMetrics icinezela ngempumelelo le datha. Ukunyanzeliswa kwi-avareji kwimveliso ukusuka kwi-0,4 ukuya kwi-0,8 bytes kwinqanaba ngalinye. Inqaku ngalinye sisitampu sexesha + ixabiso. Kwaye icinezelwe ibe ngaphantsi kwe-byte enye ngokomndilili.
Sergey. Ndinombuzo. Leliphi ixesha elincinane lokurekhoda umyinge?
Imillisecond enye. Kutshanje sibe nencoko nabanye abaphuhlisi bedatabase bexesha. Ubuncinci bexesha labo ngumzuzwana omnye. Kwaye kwiGraphite, umzekelo, ikwayinto yesibini. Kwi-OpenTSDB ikwangumzuzwana omnye. I-InfluxDB ine-nanosecond echanekileyo. Kwi-VictoriaMetrics yimillisecond enye, kuba kwi-Prometheus yimillisecond enye. Kwaye iVictoriaMetrics yaqala yaphuhliswa njengendawo yokugcina kude yePrometheus. Kodwa ngoku inokugcina idatha kwezinye iinkqubo.
Umntu endithethe naye uthi banokuchaneka okwesibini ukuya kweyesibini - kwanele kubo kuba kuxhomekeke kuhlobo lwedatha egcinwa kwi-database ye-time series. Ukuba le datha ye-DevOps okanye idatha evela kwiziseko zophuhliso, apho uqokelela khona ngezihlandlo ze-30 imizuzwana, ngomzuzu, ngoko okwesibini ukuchaneka kwanele, awudingi nto encinci. Kwaye ukuba uqokelela le datha kwiinkqubo zokurhweba rhoqo, ngoko udinga ukuchaneka kwe-nanosecond.
Ukuchaneka kweMillisecond kwiVictoriaMetrics ikwafanelekile kwimeko ye-DevOps, kwaye inokulungela uninzi lwamatyala endiwakhankanye ekuqaleni kwengxelo. Ekuphela kwento enokuthi ingafaneleki kuyo ziinkqubo zokurhweba rhoqo.
Enkosi! Kwaye omnye umbuzo. Yintoni ukuhambelana kwiPromQL?
Ukuhambelana ngokupheleleyo ngasemva. IVictoriaMetrics ixhasa ngokupheleleyo iPromQL. Ukongeza, yongeza umsebenzi owongezelelweyo oqhubela phambili kwiPromQL, ebizwa ngokuba . Kukho intetho kuYouTube malunga nalo msebenzi owandisiweyo. Ndathetha kwiMonitoring Meetup ngentwasahlobo eSt.
Umzila weTelegram .
Ngabasebenzisi ababhalisiweyo kuphela abanokuthatha inxaxheba kuphando. , ndiyacela.
Yintoni ekunqandayo ukuba utshintshele kwiVictoriaMetrics njengogcino lwakho lwexesha elide lwePrometheus? (Bhala kwizimvo, ndiza kuyongeza kwi-poll))
-
71,4%Andisebenzisi iPrometheus5
-
28,6%Bendingazi ngeVictoriaMetrics2
Bali-7 abasebenzisi abavotileyo. Abasebenzisi abasi-12 abakhange.
umthombo: www.habr.com