Ndikupangira kuti muwerenge zomwe zalembedwa kumapeto kwa 2019 ndi Alexander Valyalkin "Pitani kukhathamiritsa mu VictoriaMetrics"
- DBMS yachangu komanso yowopsa yosungira ndikusintha zidziwitso mumtundu wanthawi (mbiri imapanga nthawi ndi gulu lazinthu zomwe zikugwirizana ndi nthawi ino, mwachitsanzo, zopezedwa ndikuvotera kwakanthawi kwa sensa kapena kusonkhanitsa miyeso).
Nawu ulalo wa kanema wa lipoti ili -
Tiuzeni za inu nokha. Ndine Alexander Valyalkin. Pano . Ndine wokonda za Go ndi kukhathamiritsa magwiridwe antchito. Ndinalemba mabuku ambiri othandiza komanso osathandiza. Amayamba ndi kaya fast, kapena ndi quick chiyambi.
Pano ndikugwira ntchito pa VictoriaMetrics. Ndi chiyani ndipo ndikuchita chiyani pamenepo? Ndilankhula za izi mu chiwonetserochi.
Ndemanga ya lipoti ili motere:
- Choyamba, ndikuwuzani zomwe VictoriaMetrics ndi.
- Kenako ndikuuzeni nthawi zotsatizana.
- Kenako ndikuuzani momwe mndandanda wanthawi yayitali umagwirira ntchito.
- Chotsatira, ndikuwuzani za kamangidwe ka database: zomwe zimapangidwira.
- Kenako tiyeni tipitirire ku zokometsera zomwe VictoriaMetrics ili nazo. Uku ndi kukhathamiritsa kwa index inverted ndi kukhathamiritsa kwa kukhazikitsa kwa bitset mu Go.
Kodi pali aliyense mwa omvera akudziwa kuti VictoriaMetrics ndi chiyani? Wow, anthu ambiri akudziwa kale. Ndi nkhani yabwino. Kwa omwe sakudziwa, iyi ndi database yanthawi yayitali. Zimatengera kamangidwe ka ClickHouse, pazambiri za ClickHouse kukhazikitsa. Mwachitsanzo, monga: MergeTree, kuwerengera kofananira pama processor cores onse omwe alipo ndi kukhathamiritsa kwa magwiridwe antchito pogwira midadada ya data yomwe imayikidwa mu cache ya purosesa.
VictoriaMetrics imapereka compression yabwinoko ya data kuposa nkhokwe zina zanthawi.
Imakula molunjika - ndiye kuti, mutha kuwonjezera mapurosesa ambiri, RAM yochulukirapo pakompyuta imodzi. VictoriaMetrics idzagwiritsa ntchito bwino zinthu zomwe zilipo ndipo ipititsa patsogolo zokolola zambiri.
VictoriaMetrics imakhalanso ndi miyeso yopingasa - ndiye kuti, mutha kuwonjezera ma node owonjezera ku gulu la VictoriaMetrics, ndipo magwiridwe ake azikwera pafupifupi mzere.
Monga mumaganizira, VictoriaMetrics ndi nkhokwe yachangu, chifukwa sindingathe kulemba ena. Ndipo zalembedwa mu Go, kotero ine ndikulankhula za izo pa msonkhano uno.
Ndani akudziwa kuti mndandanda wa nthawi ndi chiyani? Amadziwanso anthu ambiri. Mndandanda wa nthawi ndi magulu awiriawiri (timestamp, значение), kumene awiriawiriwa amasanjidwa ndi nthawi. Mtengo ndi nambala yoyandama - float64.
Nthawi iliyonse imadziwika ndi kiyi. Kodi funguloli lili ndi chiyani? Zimapangidwa ndi gulu lopanda kanthu la mapeya amtengo wapatali.
Nachi chitsanzo cha mndandanda wa nthawi. Chinsinsi cha mndandanda uwu ndi mndandanda wa awiriawiri: __name__="cpu_usage" ndi dzina la metric, instance="my-server" - iyi ndi kompyuta yomwe metric iyi imasonkhanitsidwa, datacenter="us-east" - iyi ndi data center yomwe kompyuta iyi ili.
Tinamaliza ndi dzina lanthawi yokhala ndi magawo atatu amtengo wapatali. Chinsinsi ichi chikufanana ndi mndandanda wa awiriawiri (timestamp, value). t1, t3, t3, ..., tN - Izi ndi zizindikiro za nthawi, 10, 20, 12, ..., 15 - mfundo zogwirizana. Uku ndiye kugwiritsa ntchito cpu panthawi yoperekedwa pamndandanda womwe waperekedwa.
Kodi mndandanda wa nthawi ungagwiritsidwe ntchito kuti? Kodi alipo amene ali ndi lingaliro?
- Mu DevOps, mutha kuyeza CPU, RAM, network, rps, kuchuluka kwa zolakwika, ndi zina zambiri.
- IoT - titha kuyeza kutentha, kuthamanga, ma geo coordinates ndi zina.
- Komanso ndalama - titha kuyang'anira mitengo yamitundu yonse ndi ndalama.
- Kuphatikiza apo, mndandanda wanthawi ukhoza kugwiritsidwa ntchito powunikira njira zopangira m'mafakitale. Tili ndi ogwiritsa ntchito VictoriaMetrics kuyang'anira ma turbines amphepo, a maloboti.
- Mndandanda wanthawi umathandizanso kusonkhanitsa zambiri kuchokera ku masensa a zida zosiyanasiyana. Mwachitsanzo, kwa injini; kuyeza kuthamanga kwa tayala; poyezera liwiro, mtunda; poyezera kumwa mafuta a petulo, etc.
- Mndandanda wa nthawi ungagwiritsidwenso ntchito kuyang'anira ndege. Ndege iliyonse imakhala ndi bokosi lakuda lomwe limasonkhanitsa nthawi zosiyanasiyana za thanzi la ndege. Mndandanda wa nthawi umagwiritsidwanso ntchito m'makampani opanga ndege.
- Zaumoyo ndi kuthamanga kwa magazi, kugunda, etc.
Pakhoza kukhala mapulogalamu ambiri omwe ndinayiwala, koma ndikuyembekeza kuti mukumvetsa kuti mndandanda wa nthawi ukugwiritsidwa ntchito kwambiri masiku ano. Ndipo kuchuluka kwa ntchito zawo kukukulirakulira chaka chilichonse.
Chifukwa chiyani mukufunikira database yotsatizana nthawi? Chifukwa chiyani simungagwiritse ntchito nkhokwe yanthawi zonse kuti musunge nthawi?
Chifukwa mndandanda wa nthawi nthawi zambiri umakhala ndi zidziwitso zambiri, zomwe zimakhala zovuta kuzisunga ndikuzikonza m'malo osungira wamba. Chifukwa chake, zida zapadera zamagulu anthawi zidawonekera. Maziko awa amasunga bwino mfundo (timestamp, value) ndi kiyi yopatsidwa. Amapereka API yowerengera deta yosungidwa ndi kiyi, ndi makiyi amodzi amtengo wapatali, kapena mawiri angapo amtengo wapatali, kapena ndi regexp. Mwachitsanzo, mukufuna kupeza kuchuluka kwa CPU pazantchito zanu zonse pamalo opangira data ku America, ndiye muyenera kugwiritsa ntchito funso lachinyengoli.
Nthawi zambiri nkhokwe zotsatsira nthawi zimapereka zilankhulo zapadera zamafunso chifukwa mndandanda wanthawi wa SQL suyenerana bwino. Ngakhale pali nkhokwe zomwe zimathandizira SQL, sizoyenera kwambiri. Funsani zilankhulo monga , , , . Ndikukhulupirira kuti wina wamva chimodzi mwa zilankhulo izi. Anthu ambiri mwina adamvapo za PromQL. Ichi ndi chilankhulo cha funso la Prometheus.
Umu ndi momwe kamangidwe kamakono kamndandanda wamakono amawonekera pogwiritsa ntchito VictoriaMetrics mwachitsanzo.
Lili ndi magawo awiri. Uku ndikusungira kwa index yotembenuzidwa ndi kusungirako mindandanda yanthawi. nkhokwe izi analekanitsidwa.
Mbiri yatsopano ikafika munkhokwe, timapeza kalozera wolowera kuti tipeze chizindikiritso cha nthawi ya seti yomwe yaperekedwa. label=value kwa metric yomwe yapatsidwa. Timapeza chizindikiritsochi ndikusunga mtengo mu sitolo ya data.
Pempho likabwera kuti titenge deta kuchokera ku TSDB, timapita kaye ku index inverted. Tiyeni titenge chirichonse timeseries_ids zolemba zomwe zikugwirizana ndi seti iyi label=value. Ndiyeno timapeza zofunikira zonse kuchokera ku malo osungiramo deta, olembedwa ndi timeseries_ids.
Tiyeni tiwone chitsanzo cha momwe mndandanda wanthawi yayitali umasinthira funso lomwe likubwera.
- Choyamba amapeza chilichonse
timeseries_idskuchokera pa index inverted yomwe ili ndi awiriawiri operekedwalabel=value, kapena kukhutiritsa mawu operekedwa nthawi zonse. - Kenako amapeza mfundo zonse deta ku yosungirako deta pa nthawi imeneyi kwa anthu opezeka
timeseries_ids. - Pambuyo pake, databaseyo imapanga mawerengedwe ena pazigawo za deta, malinga ndi pempho la wogwiritsa ntchito. Ndipo pambuyo pake imabwezera yankho.
Mu ulaliki uwu ndikuuzani za gawo loyamba. Uku ndikufufuza timeseries_ids ndi inverted index. Mutha kuyang'ana za gawo lachiwiri ndi gawo lachitatu pambuyo pake , kapena dikirani mpaka ndikonze malipoti ena :)
Tiyeni tipite ku inverted index. Ambiri angaganize kuti izi ndi zophweka. Ndani akudziwa chomwe index inverted ndi momwe imagwirira ntchito? O, osati anthu ochuluka kwambiri. Tiyeni tiyese kumvetsetsa kuti ndi chiyani.
Ndizosavuta. Ndi dikishonale chabe yomwe imayika chinsinsi cha mtengo. Kodi kiyi ndi chiyani? Banja ili label=valuekumene label и value - awa ndi mizere. Ndipo ma values ndi seti timeseries_ids, zomwe zikuphatikizapo awiri operekedwa label=value.
Inverted index imakupatsani mwayi wopeza chilichonse mwachangu timeseries_ids, amene apereka label=value.
Komanso amalola kuti mwamsanga kupeza timeseries_ids mndandanda wa nthawi kwa awiriawiri angapo label=value, kapena kwa maanja label=regexp. Kodi izi zimachitika bwanji? Popeza mphambano ya seti timeseries_ids kwa gulu lililonse label=value.
Tiyeni tiwone machitidwe osiyanasiyana a index inverted. Tiyeni tiyambe ndi kukhazikitsa kosavuta kopanda nzeru. Iye akuwoneka chonchi.
ntchito getMetricIDs amapeza mndandanda wa zingwe. Mzere uliwonse uli ndi label=value. Ntchitoyi imabweretsanso mndandanda metricIDs.
Zimagwira ntchito bwanji? Pano tili ndi kusintha kwapadziko lonse komwe kumatchedwa invertedIndex. Ili ndi dikishonale yokhazikika (map), yomwe imapanga mapu a chingwe kuti adule ints. Mzere uli ndi label=value.
Kukhazikitsa ntchito: get metricIDs kwa woyamba label=value, kenako timadutsa china chilichonse label=value, tikumvetsa metricIDs kwa iwo. Ndipo kuitana ntchito intersectInts, zomwe zidzakambidwe pansipa. Ndipo ntchitoyi imabweretsanso mphambano ya mindandanda iyi.
Monga mukuonera, kukhazikitsa index inverted sikovuta kwambiri. Koma uku ndikukhazikitsa kopanda nzeru. Ndi kuipa kotani? Choyipa chachikulu pakukhazikitsa kopanda nzeru ndikuti index yosinthika yotere imasungidwa mu RAM. Titayambiranso kugwiritsa ntchito timataya index iyi. Palibe kusungidwa kwa index iyi ku disk. Mlozera wotembenuzidwa woterewu ndi wokayikitsa kuti uyenera kukhala wankhokwe.
The drawback yachiwiri imakhudzananso ndi kukumbukira. Mlozera wolowetsedwa uyenera kulowa mu RAM. Ngati ipitilira kukula kwa RAM, ndiye kuti mwachiwonekere tituluka - kuchokera pakulakwitsa kukumbukira. Ndipo pulogalamuyi sigwira ntchito.
Vutoli litha kuthetsedwa pogwiritsa ntchito njira zomwe zapangidwa kale monga , kapena .
Mwachidule, timafunikira database yomwe imatilola kuchita ntchito zitatu mwachangu.
- Ntchito yoyamba ndikujambula
ключ-значениеku database iyi. Iye amachita izi mofulumira kwambiri, kumeneключ-значениеndi zingwe zosasintha. - Ntchito yachiwiri ndikufufuza mwachangu mtengo pogwiritsa ntchito kiyi yoperekedwa.
- Ndipo ntchito yachitatu ndikufufuza mwachangu zamtengo wapatali ndi chilembo choperekedwa.
LevelDB ndi RocksDB - nkhokwe izi zidapangidwa ndi Google ndi Facebook. Choyamba anabwera LevelDB. Kenako anyamata ochokera ku Facebook adatenga LevelDB ndikuyamba kuyikonza, adapanga RocksDB. Tsopano pafupifupi nkhokwe zonse zamkati zimagwira ntchito pa RocksDB mkati mwa Facebook, kuphatikiza zomwe zasamutsidwa ku RocksDB ndi MySQL. Iwo anamutcha dzina .
Mndandanda wolowetsedwa ukhoza kukhazikitsidwa pogwiritsa ntchito LevelDB. Kodi kuchita izo? Timasunga ngati kiyi label=value. Ndipo mtengo ndi chizindikiritso cha nthawi yomwe awiriwa alipo label=value.
Ngati tili ndi mndandanda wanthawi zambiri wokhala ndi peyala yoperekedwa label=value, ndiye padzakhala mizere yambiri mu database iyi yokhala ndi kiyi yomweyo komanso yosiyana timeseries_ids. Kuti mupeze mndandanda wa zonse timeseries_ids, zomwe zimayamba ndi izi label=prefix, timapanga masikelo osiyanasiyana omwe databaseyi imakongoletsedwa. Ndiye kuti, timasankha mizere yonse yomwe imayamba label=prefix ndi kupeza zofunika timeseries_ids.
Nayi chitsanzo cha kukhazikitsa momwe zingawonekere mu Go. Tili ndi index inverted. Izi ndi LevelDB.
Ntchitoyi ndi yofanana ndi kukhazikitsa kwa naive. Ikubwereza kukhazikitsa mosadziwa pafupifupi mzere ndi mzere. Mfundo yokhayo ndi yakuti m'malo motembenukira ku map timapeza index inverted. Timapeza zikhalidwe zonse zoyambira label=value. Kenako timadutsa awiriawiri otsalawo label=value ndi kuwapezera ma metricID ofananira nawo. Ndiye ife tikupeza mphambano.
Chilichonse chikuwoneka kuti chili bwino, koma pali zovuta panjira iyi. VictoriaMetrics poyambilira adagwiritsa ntchito index inverted kutengera LevelDB. Koma pamapeto pake ndinayenera kusiya.
Chifukwa chiyani? Chifukwa LevelDB ndiyochedwa kuposa kukhazikitsa mosazindikira. Pakukhazikitsa kopanda nzeru, kupatsidwa kiyi yopatsidwa, nthawi yomweyo timapeza gawo lonse metricIDs. Iyi ndi ntchito yofulumira kwambiri - chidutswa chonsecho chakonzeka kugwiritsidwa ntchito.
Mu LevelDB, nthawi iliyonse ntchito imatchedwa GetValues muyenera kudutsa mizere yonse yomwe imayambira label=value. Ndipo pezani mtengo wa mzere uliwonse timeseries_ids. Za zoterozo timeseries_ids sonkhanitsani chidutswa cha izi timeseries_ids. Mwachiwonekere, izi ndizochedwa kwambiri kusiyana ndi kungopeza mapu wamba ndi kiyi.
Chotsalira chachiwiri ndi chakuti LevelDB yalembedwa mu C. Kuitana C ntchito kuchokera ku Go sikuthamanga kwambiri. Zimatengera mazana a nanoseconds. Izi sizofulumira kwambiri, chifukwa poyerekeza ndi kuyitana kwanthawi zonse komwe kumalembedwa, komwe kumatenga 1-5 nanoseconds, kusiyana kwa magwiridwe antchito ndi kakhumi. Kwa VictoriaMetrics ichi chinali cholakwika chachikulu :)
Chifukwa chake ndidalemba kukhazikitsidwa kwanga kwa index inverted. Ndipo adamuyitana .
Mergeset imachokera ku MergeTree data structure. Deta iyi idabwerekedwa ku ClickHouse. Mwachiwonekere, mergeset iyenera kukonzedwa kuti isakasaka mwachangu timeseries_ids malinga ndi kiyi yomwe wapatsidwa. Mergeset imalembedwa kwathunthu mu Go. Mutha kuwona . Kukhazikitsa kwa mergeset kuli mufoda . Mutha kuyesa kudziwa zomwe zikuchitika kumeneko.
Mergeset API ndi yofanana kwambiri ndi LevelDB ndi RocksDB. Ndiye kuti, zimakulolani kuti musunge mwachangu zolemba zatsopano pamenepo ndikusankha mwachangu zolemba ndi prefix yoperekedwa.
Tikambirana za kuipa kwa mergeset pambuyo pake. Tsopano tiyeni tikambirane zovuta zomwe zidabuka ndi VictoriaMetrics pakupanga pokhazikitsa index yosinthika.
Chifukwa chiyani iwo anawuka?
Chifukwa choyamba ndi kuchuluka kwa churn. Kutanthauziridwa ku Chirasha, uku ndikusintha pafupipafupi kwanthawi yayitali. Apa ndi pamene mndandanda wa nthawi umatha ndipo mndandanda watsopano umayamba, kapena nthawi zambiri zatsopano zimayamba. Ndipo izi zimachitika kawirikawiri.
Chifukwa chachiwiri ndi kuchuluka kwa nthawi. Poyambirira, pamene kuyang'anira kunali kutchuka, chiwerengero cha nthawi chinali chochepa. Mwachitsanzo, pa kompyuta iliyonse muyenera kuwunika CPU, kukumbukira, maukonde ndi litayamba katundu. 4 nthawi mndandanda pa kompyuta. Tiyerekeze kuti muli ndi makompyuta 100 ndi nthawi 400. Izi ndizochepa kwambiri.
M'kupita kwa nthawi, anthu anaganiza kuti akhoza kuyeza zambiri granular. Mwachitsanzo, yesani kuchuluka kwa purosesa yonse, koma padera pa purosesa iliyonse. Ngati muli ndi ma processor cores 40, ndiye kuti muli ndi nthawi 40 yochulukirapo kuti muyese kuchuluka kwa purosesa.
Koma si zokhazo. Chigawo chilichonse cha purosesa chikhoza kukhala ndi zigawo zingapo, monga zopanda pake, pamene sichigwira ntchito. Komanso gwiritsani ntchito malo ogwiritsa ntchito, gwiritsani ntchito kernel space ndi mayiko ena. Ndipo dera lililonse lotere lingathenso kuyesedwa ngati nthawi yosiyana. Izi zimawonjezera kuchuluka kwa mizere ndi nthawi 7-8.
Kuchokera ku metric imodzi timapeza 40 x 8 = 320 metrics pa kompyuta imodzi yokha. Kuchulukitsa ndi 100, timapeza 32 m'malo mwa 000.
Kenako Kubernetes adabwera. Ndipo zidafika poipa chifukwa Kubernetes amatha kukhala ndi mautumiki osiyanasiyana. Ntchito iliyonse ku Kubernetes imakhala ndi ma pod ambiri. Ndipo zonsezi ziyenera kuyang'aniridwa. Kuphatikiza apo, timatumiza mitundu yatsopano ya mautumiki anu mosalekeza. Pa mtundu uliwonse watsopano, mndandanda wa nthawi zatsopano uyenera kupangidwa. Chotsatira chake, chiwerengero cha nthawi chimakula kwambiri ndipo tikukumana ndi vuto la chiwerengero chachikulu cha nthawi, chomwe chimatchedwa high-cardinality. VictoriaMetrics amalimbana nazo bwino poyerekeza ndi nkhokwe zina zanthawi.
Tiyeni tione mwatsatanetsatane kuchuluka kwa churn. Nchiyani chimayambitsa kuchulukirachulukira kochulukira pamapangidwe? Chifukwa matanthauzo ena a zilembo ndi ma tag akusintha nthawi zonse.
Mwachitsanzo, tengani Kubernetes, yomwe ili ndi lingaliro deployment, mwachitsanzo pamene pulogalamu yanu yatsopano yatulutsidwa. Pazifukwa zina, opanga Kubernetes adaganiza zowonjezera ID yotumizira palembalo.
Kodi izi zidatsogolera ku chiyani? Kuphatikiza apo, pakutumiza kwatsopano kulikonse, mndandanda wakale wakale umasokonekera, ndipo m'malo mwake, mndandanda wanthawi zatsopano umayamba ndi mtengo watsopano. deployment_id. Pakhoza kukhala mazana a zikwi ngakhalenso mamiliyoni a mizere yoteroyo.
Chofunika kwambiri pa zonsezi ndikuti chiwerengero cha nthawi zonse chimakula, koma chiwerengero cha nthawi zomwe zikugwira ntchito komanso kulandira deta zimakhalabe nthawi zonse. Dzikoli limatchedwa high churn rate.
Vuto lalikulu la kuchuluka kwa churn ndikuwonetsetsa kuthamanga kosalekeza kwa mndandanda wanthawi zonse pamalemba omwe aperekedwa pakanthawi inayake. Nthawi zambiri iyi ndi nthawi ya ola lomaliza kapena tsiku lomaliza.
Kodi kuthetsa vutoli? Nayi njira yoyamba. Uku ndikugawaniza index inverted kukhala magawo odziyimira pawokha pakapita nthawi. Ndiye kuti, pakadutsa nthawi, timamaliza kugwira ntchito ndi index yosinthika yomwe ilipo. Ndipo pangani index inverted yatsopano. Nthawi ina ikadutsa, timapanga ina ndi ina.
Ndipo tikamayesa kuchokera ku ma inverted indices, timapeza gulu la ma inverted indices omwe amagwera mkati mwanthawi yomwe yaperekedwa. Ndipo, moyenerera, timasankha id ya mndandanda wa nthawi kuchokera pamenepo.
Izi zimapulumutsa zothandizira chifukwa sitiyenera kuyang'ana magawo omwe sagwera mkati mwa nthawi yomwe yaperekedwa. Ndiye kuti, nthawi zambiri, ngati tisankha deta ya ola lomaliza, ndiye kuti nthawi yapitayi timadumpha mafunso.
Palinso njira ina yothetsera vutoli. Uku ndikusungira tsiku lililonse mndandanda wosiyana wa ma ID omwe adachitika tsikulo.
Ubwino wa yankholi kuposa yankho lapitalo ndikuti sitimabwereza zambiri zomwe sizitha pakapita nthawi. Iwo amakhalapo nthawi zonse ndipo sasintha.
Choyipa chake ndichakuti njira yotereyi ndi yovuta kuigwiritsa ntchito komanso yovuta kuyithetsa. Ndipo VictoriaMetrics adasankha yankho ili. Umu ndi momwe zidachitikira m'mbiri. Njira imeneyi imagwiranso ntchito bwino poyerekeza ndi yapitayi. Chifukwa yankho ili silinakhazikitsidwe chifukwa chakuti ndikofunikira kubwereza deta mu gawo lililonse la mndandanda wa nthawi zomwe sizisintha, mwachitsanzo, zomwe sizimatha pakapita nthawi. VictoriaMetrics idakonzedweratu kuti igwiritse ntchito malo a disk, ndipo kukhazikitsidwa koyambirira kunapangitsa kuti kugwiritsa ntchito malo a disk kuipire. Koma kukhazikitsa uku ndikoyenera kuchepetsa kugwiritsa ntchito malo a disk, kotero adasankhidwa.
Ndinayenera kumenyana naye. Kulimbana kunali kuti pakukhazikitsa uku mukufunikabe kusankha nambala yokulirapo timeseries_ids pa data kuposa pamene index inverted igawika nthawi.
Tinathetsa bwanji vuto limeneli? Tidazithetsa m'njira yoyambirira - posunga zozindikiritsa zotsatizana zingapo pamndandanda uliwonse wopindika m'malo mwa chizindikiritso chimodzi. Ndiko kuti, tili ndi kiyi label=value, zomwe zimachitika nthawi zonse. Ndipo tsopano ife kupulumutsa angapo timeseries_ids mu kulowa kumodzi.
Nachi chitsanzo. M'mbuyomu tinali ndi zolembera za N, koma tsopano tili ndi cholowera chimodzi chomwe chiyambi chake ndi chofanana ndi ena onse. Pazolemba zam'mbuyomu, mtengowo uli ndi ma ID anthawi zonse.
Izi zidapangitsa kuti ziwonjezeke liwiro la kusanthula kwa index yotembenuzidwa mpaka nthawi 10. Ndipo zinatilola kuti tichepetse kukumbukira kukumbukira kwa cache, chifukwa tsopano timasunga chingwe label=value kamodzi kokha mu posungira pamodzi N nthawi. Ndipo mzerewu ukhoza kukhala wawukulu ngati musunga mizere yayitali m'ma tag anu ndi zilembo, zomwe Kubernetes amakonda kukankhira pamenepo.
Njira ina yofulumizitsa kusaka pa index inverted ndi sharding. Kupanga ma index angapo otembenuzidwa m'malo mwa amodzi ndikugawana data pakati pawo ndi kiyi. Izi ndi seti key=value nthunzi. Ndiye kuti, timapeza ma index angapo odziyimira pawokha, omwe titha kufunsa mofananira pama processor angapo. Zomwe zidachitika m'mbuyomu zimangololeza kugwira ntchito munjira imodzi ya purosesa, mwachitsanzo, kusanthula deta pachinthu chimodzi chokha. Yankho ili limakupatsani mwayi wosanthula ma cores angapo nthawi imodzi, monga ClickHouse amakonda kuchita. Izi ndi zomwe tikukonzekera kukhazikitsa.
Tsopano tiyeni tibwerere ku nkhosa zathu - ku ntchito ya mphambano timeseries_ids. Tiyeni tiwone zomwe zingagwiritsidwe ntchito. Ntchitoyi imakulolani kuti mupeze timeseries_ids kwa seti yopatsidwa label=value.
Njira yoyamba ndiyo kukhazikitsa mosaganizira. Mitundu yambiri yamaluwa. Apa tikupeza zolowetsamo intersectInts magawo awiri - a и b. Pakutulutsa, iyenera kutibwezera kwa ife mphambano ya magawo awa.
Kukhazikitsa kopanda pake kumawoneka chonchi. Timabwereza mfundo zonse kuchokera pagawo a, mkati mwa chipikachi timadutsa pamakhalidwe onse a kagawo b. Ndipo timawafananiza. Ngati zikugwirizana, ndiye kuti tapeza mphambano. Ndipo sungani mkati result.
Kodi kuipa kwake ndi kotani? Quadratic zovuta ndiye drawback yake yayikulu. Mwachitsanzo, ngati miyeso yanu ndi kagawo a и b miliyoni imodzi panthawi, ndiye kuti ntchitoyi sidzakubwezerani yankho. Chifukwa idzafunika kupanga thililiyoni imodzi kubwereza, zomwe zimakhala zambiri ngakhale makompyuta amakono.
Kukhazikitsa kwachiwiri kumatengera mapu. Timapanga mapu. Timayika zikhalidwe zonse kuchokera pagawo pamapu awa a. Kenako timadutsa mu kagawo kakang'ono kosiyana b. Ndipo timayang'ana ngati mtengowu ukuchokera pagawo b mu map. Ngati ilipo, yonjezerani ku zotsatira zake.
Kodi ubwino wake ndi wotani? Ubwino wake ndikuti pali zovuta zofananira zokha. Ndiye kuti, ntchitoyi izichita mwachangu kwambiri pamagawo akulu. Pakagawo kakang'ono miliyoni, ntchitoyi ichitika mobwerezabwereza 2 miliyoni, mosiyana ndi ma thililiyoni a ntchito yapitayi.
Choyipa ndichakuti ntchitoyi imafuna kukumbukira zambiri kuti ipange mapu.
Yachiwiri drawback ndi lalikulu pamwamba pa hashing. Izi drawback si zoonekeratu. Ndipo kwa ife sizinali zoonekeratu, kotero poyamba mu VictoriaMetrics kukhazikitsidwa kwa mphambano kunali kudzera pamapu. Koma kufotokoza kwawonetsa kuti nthawi yayikulu yopangira purosesa imagwiritsidwa ntchito polemba mapu ndikuyang'ana ngati pali phindu pamapuwa.
Chifukwa chiyani nthawi ya CPU ikuwonongeka m'malo awa? Chifukwa Go imagwira ntchito ya hashing pamizere iyi. Ndiko kuti, imawerengera hashi ya kiyi kuti muthe kuyipeza pamndandanda womwe wapatsidwa mu HashMap. Ntchito yowerengera hashi imamalizidwa mu makumi a nanoseconds. Izi ndizochedwa kwa VictoriaMetrics.
Ndinaganiza zokhazikitsa bitset wokometsedwa makamaka pankhaniyi. Izi ndi momwe mphambano ya magawo awiri ikuwonekera tsopano. Apa tikupanga bitset. Timawonjezera zinthu kuchokera pagawo loyamba. Kenako timayang'ana kupezeka kwa zinthu izi mugawo lachiwiri. Ndipo onjezerani ku zotsatira zake. Ndiko kuti, pafupifupi palibe chosiyana ndi chitsanzo chapitacho. Chokhacho apa ndikuti tasintha mwayi wopeza mapu ndi magwiridwe antchito add и has.
Poyang'ana koyamba, zikuwoneka kuti izi ziyenera kugwira ntchito pang'onopang'ono, ngati kale mapu okhazikika ankagwiritsidwa ntchito kumeneko, ndiyeno ntchito zina zimatchedwa, koma kufotokozera kumasonyeza kuti chinthu ichi chimagwira ntchito 10 mofulumira kuposa mapu ovomerezeka pa nkhani ya VictoriaMetrics.
Kuphatikiza apo, imagwiritsa ntchito kukumbukira kocheperako poyerekeza ndi kukhazikitsa mapu. Chifukwa tikusunga ma bits apa m'malo mwa ma baiti eyiti.
Choyipa cha kukhazikitsa uku ndikuti sizowoneka bwino, osati zazing'ono.
Choyipa china chomwe ambiri sangazindikire ndikuti kukhazikitsa uku sikungagwire bwino nthawi zina. Ndiye kuti, imakonzedwera mlandu wina wake, pamlanduwu wamakambirano a nthawi ya VictoriaMetrics. Izi sizikutanthauza kuti ndizoyenera milandu yonse. Ngati ikugwiritsidwa ntchito molakwika, sitipeza chiwonjezeko cha magwiridwe antchito, koma cholakwika chosakumbukira komanso kuchepa kwa magwiridwe antchito.
Tiyeni tilingalire kukhazikitsidwa kwa dongosololi. Ngati mukufuna kuyang'ana, ili mu VictoriaMetrics sources, mufoda . Imakonzedwa makamaka pamilandu ya VictoriaMetrics, komwe timeseries_id ndi mtengo wa 64-bit, pomwe ma bits 32 oyamba amakhala osasinthasintha ndipo ma 32 omaliza okha amasintha.
Dongosolo la datali silinasungidwe pa disk, limagwira ntchito pokumbukira.
Nayi API yake. Sizovuta kwambiri. API imapangidwa makamaka ku chitsanzo chapadera chogwiritsa ntchito VictoriaMetrics. Ndiko kuti, palibe ntchito zosafunikira pano. Nazi ntchito zomwe zimagwiritsidwa ntchito momveka bwino ndi VictoriaMetrics.
Pali ntchito add, zomwe zimawonjezera zikhalidwe zatsopano. Pali ntchito has, yomwe imayang'ana zatsopano. Ndipo pali ntchito del, zomwe zimachotsa makhalidwe abwino. Pali ntchito yothandizira len, zomwe zimabweretsanso kukula kwa seti. Ntchito clone clones kwambiri. Ndipo ntchito appendto amasintha seti iyi kukhala kagawo timeseries_ids.
Izi ndi zomwe kukhazikitsidwa kwa dongosolo la deta likuwonekera. set ili ndi zinthu ziwiri:
-
ItemsCountndi gawo lothandizira kubweza mwachangu kuchuluka kwa zinthu mu seti. Zitha zotheka kuchita popanda gawo lothandizirali, koma limayenera kuonjezedwa pano chifukwa VictoriaMetrics nthawi zambiri imafunsa za kutalika kwa bitset pama algorithms ake. -
Munda wachiwiri ndi
buckets. Ichi ndi kagawo kamangidwebucket32. Kapangidwe kalikonse kamasungahimunda. Awa ndi ma bits 32 apamwamba. Ndipo magawo awiri -b16hisиbucketskuchokerabucket16zomangamanga.
Ma bits 16 apamwamba a gawo lachiwiri la kapangidwe ka 64-bit amasungidwa pano. Ndipo apa ma bitset amasungidwa kumunsi kwa 16 bits iliyonse.
Bucket64 imakhala ndi gulu uint64. Kutalika kumawerengedwa pogwiritsa ntchito zosinthazi. Mmodzi bucket16 pazipita akhoza kusungidwa 2^16=65536 pang'ono. Ngati mugawa izi ndi 8, ndiye kuti ndi 8 kilobytes. Mukagawanitsanso 8, ndi 1000 uint64 tanthauzo. Ndiko kuti Bucket16 - ichi ndi dongosolo lathu la 8-kilobyte.
Tiyeni tiwone momwe imodzi mwa njira zamapangidwe awa zowonjezerera mtengo watsopano zimatsatiridwa.
Zonse zimayamba ndi uint64 matanthauzo. Timawerengera ma bits 32 apamwamba, timawerengera ma bits 32 apansi. Tiyeni tidutse chirichonse buckets. Timayerekeza ma bits 32 apamwamba mu chidebe chilichonse ndi mtengo womwe ukuwonjezedwa. Ndipo ngati zikugwirizana, ndiye kuti timatcha ntchitoyi add mu dongosolo b32 buckets. Ndipo onjezani ma bits 32 otsika pamenepo. Ndipo ngati idabwerera true, ndiye izi zikutanthauza kuti tawonjezera mtengo wotere pamenepo ndipo tinalibe mtengo wotere. Ngati ibwerera false, ndiye tanthauzo loterolo linalipo kale. Kenaka timawonjezera chiwerengero cha zinthu zomwe zimapangidwira.
Ngati sitinapeze yomwe mukufuna bucket ndi kufunika kwa hi-value, ndiye timayitana ntchitoyo addAlloc, yomwe idzatulutsa yatsopano bucket, kuwonjezera pa kapangidwe ka ndowa.
Uku ndikukhazikitsa ntchitoyo b32.add. Ndizofanana ndi zomwe zakhazikitsidwa kale. Timawerengera ma bits 16 ofunikira kwambiri, ocheperako 16.
Kenako timadutsa ma bits 16 onse apamwamba. Timapeza machesi. Ndipo ngati pali machesi, timatcha njira yowonjezera, yomwe tidzakambirana patsamba lotsatira bucket16.
Ndipo apa pali gawo lotsika kwambiri, lomwe liyenera kukonzedwa momwe mungathere. Timawerengera uint64 mtengo wa id mu kagawo kakang'ono komanso bitmask. Ichi ndi chigoba cha mtengo wopatsidwa 64-bit, womwe ungagwiritsidwe ntchito kuyang'ana kukhalapo kwa pang'ono, kapena kuyiyika. Timayang'ana kuti tiwone ngati pang'ono iyi yakhazikitsidwa ndikuyiyika, ndikubwereranso kukhalapo. Uku ndiye kukhazikitsa kwathu, zomwe zidatipangitsa kuti tifulumizitse kugwiritsa ntchito ma ID a nthawi yodutsa nthawi 10 poyerekeza ndi mamapu wamba.
Kuphatikiza pa kukhathamiritsa uku, VictoriaMetrics ilinso ndi zina zambiri zokhathamiritsa. Zambiri mwazowonjezerazi zidawonjezeredwa pazifukwa, koma pambuyo polemba ma code pakupanga.
Ili ndiye lamulo lalikulu la kukhathamiritsa - osawonjezera kukhathamiritsa poganiza kuti pakhala vuto pano, chifukwa zitha kupezeka kuti sipadzakhala botolo pamenepo. Kukhathamiritsa nthawi zambiri kumatsitsa mtundu wa code. Chifukwa chake, ndikofunikira kukhathamiritsa pokhapokha mutapanga mbiri komanso makamaka pakupanga, kuti izi zikhale zenizeni. Ngati wina ali ndi chidwi, mutha kuyang'ana pa VictoriaMetrics source code ndikuwunikanso kukhathamiritsa kwina komwe kulipo.
Ndili ndi funso lokhudza bitset. Zofanana kwambiri ndi kukhazikitsa kwa C ++ vector bool, optimized bitset. Kodi mwatenga kukhazikitsa kuchokera pamenepo?
Ayi, osati kuchokera pamenepo. Pokhazikitsa bitset iyi, ndidatsogozedwa ndi chidziwitso cha mapangidwe a ma ID awa, omwe amagwiritsidwa ntchito mu VictoriaMetrics. Ndipo mawonekedwe awo ndi oti ma bits 32 apamwamba amakhala osasinthasintha. Ma bits otsika a 32 amatha kusintha. M'munsi pang'ono, nthawi zambiri imatha kusintha. Chifukwa chake, kukhazikitsidwa uku ndikokometsedwa kwadongosolo la data. Kukhazikitsa kwa C ++, monga momwe ndikudziwira, kumakonzedweratu pazochitika zonse. Ngati mukonzekeretsa nkhani yonse, izi zikutanthauza kuti sizingakhale zabwino kwambiri pazochitika zinazake.
Ndikulangizanso kuti muwone lipoti la Alexey Milovid. Pafupifupi mwezi wapitawo, adalankhula za kukhathamiritsa mu ClickHouse pazapaderadera. Amangonena kuti nthawi zambiri, kukhazikitsidwa kwa C ++ kapena kukhazikitsidwa kwina kumapangidwira kuti azigwira ntchito bwino kuchipatala. Zitha kuchita moyipa kwambiri kuposa kugwiritsa ntchito chidziwitso chachindunji monga chathu, pomwe tikudziwa kuti ma bits 32 apamwamba amakhala osasintha.
Ndili ndi funso lachiwiri. Kodi pali kusiyana kotani pakati pa InfluxDB?
Pali zosiyana zambiri. Pankhani ya magwiridwe antchito ndi kukumbukira kukumbukira, InfluxDB pamayesero ikuwonetsa kugwiritsa ntchito kukumbukira nthawi 10 pazotsatira zanthawi yayitali, mukakhala ndi zambiri, mwachitsanzo, mamiliyoni. Mwachitsanzo, VictoriaMetrics imadya 1 GB pamizere yogwira miliyoni miliyoni, pomwe InfluxDB imadya 10 GB. Ndipo ndiko kusiyana kwakukulu.
Kusiyana kwachiwiri kwakukulu ndikuti InfluxDB ili ndi zilankhulo zachilendo - Flux ndi InfluxQL. Iwo si yabwino kwambiri ntchito ndi mndandanda wa nthawi poyerekeza , yomwe imathandizidwa ndi VictoriaMetrics. PromQL ndi chilankhulo chochokera ku Prometheus.
Ndipo kusiyana kwinanso ndikuti InfluxDB ili ndi mtundu wa data wachilendo pang'ono, pomwe mzere uliwonse umatha kusunga magawo angapo okhala ndi ma tag osiyanasiyana. Mizere iyi imagawidwanso m'matebulo osiyanasiyana. Zowonjezereka izi zimasokoneza ntchito yotsatira ndi database iyi. Ndizovuta kuthandizira ndikumvetsetsa.
Mu VictoriaMetrics zonse ndizosavuta. Kumeneko, mndandanda wa nthawi iliyonse uli ndi mtengo wapatali. Mtengo ndi gulu la mfundo - (timestamp, value), ndipo fungulo ndi seti label=value. Palibe kulekana pakati pa minda ndi miyeso. Zimakulolani kuti musankhe deta iliyonse ndikuphatikiza, kuwonjezera, kuchotsa, kuchulukitsa, kugawa, mosiyana ndi InfluxDB kumene kuwerengera pakati pa mizere yosiyana sikunakwaniritsidwebe monga momwe ndikudziwira. Ngakhale atakhazikitsidwa, ndizovuta, muyenera kulemba ma code ambiri.
Ndili ndi funso lomveka bwino. Kodi ndimamvetsetsa bwino kuti panali vuto linalake lomwe mudalikamba, kuti index yotembenuzidwayi sikwanira pamtima, ndiye kuti pali kugawa pamenepo?
Choyamba, ndidawonetsa kukhazikitsidwa kopanda nzeru kwa index inverted pamapu okhazikika a Go. Kukhazikitsa uku sikuli koyenera pazosungidwa chifukwa cholozerachi sichinasungidwe ku diski, ndipo nkhokwe iyenera kusungidwa ku diski kuti detayi ikhalepo poyambiranso. Pakukhazikitsa uku, mukayambitsanso pulogalamuyo, index yanu yolowera idzazimiririka. Ndipo mudzataya mwayi wopeza deta yonse chifukwa simungathe kuzipeza.
Moni! Zikomo chifukwa cha lipoti! Dzina langa ndine Pavel. Ndimachokera ku Wildberries. Ndili ndi mafunso angapo kwa inu. Funso limodzi. Kodi mukuganiza kuti ngati mutasankha mfundo ina pomanga kamangidwe ka ntchito yanu ndikugawanitsa deta pakapita nthawi, ndiye kuti mwina mukanatha kuphatikizira deta pofufuza, potengera kuti gawo limodzi lili ndi deta yamtundu umodzi. nthawi, ndiye kuti, mu nthawi imodzi ndipo simuyenera kuda nkhawa kuti zidutswa zanu zabalalika mosiyana? Funso nambala 2 - popeza mukugwiritsa ntchito algorithm yofananira ndi bitset ndi china chilichonse, ndiye mwina mwayesa kugwiritsa ntchito malangizo a purosesa? Mwinamwake mwayesapo kukhathamiritsa koteroko?
Ndiyankha yachiwiri nthawi yomweyo. Sitinafikebe mpaka pano. Koma ngati kuli kofunikira, tidzafika kumeneko. Ndipo loyamba, funso linali chiyani?
Munakambirana zochitika ziwiri. Ndipo iwo adanena kuti adasankha yachiwiri ndi kukhazikitsa zovuta kwambiri. Ndipo sanakonde woyamba, pomwe deta imagawidwa ndi nthawi.
Inde. Poyamba, chiwerengero chonse cha index chidzakhala chokulirapo, chifukwa pagawo lililonse tiyenera kusunga deta yobwereza ya nthawi zomwe zikupitilira magawo onsewa. Ndipo ngati nthawi yanu yochulukirachulukira ndi yaying'ono, i.e. mndandanda womwewo umagwiritsidwa ntchito nthawi zonse, ndiye kuti poyamba tingataye zochulukirapo pakuchuluka kwa disk space yomwe imakhalapo poyerekeza ndi yachiwiri.
Ndipo kotero - inde, kugawa nthawi ndi njira yabwino. Prometheus amagwiritsa ntchito. Koma Prometheus ali ndi vuto lina. Pophatikiza zidutswa za datazi, ziyenera kusungidwa mu memory meta zambiri zamalebulo onse ndi mndandanda wanthawi. Chifukwa chake, ngati zidutswa za data zomwe zimaphatikiza ndi zazikulu, ndiye kuti kukumbukira kumawonjezeka kwambiri pakuphatikiza, mosiyana ndi VictoriaMetrics. Mukaphatikiza, VictoriaMetrics sagwiritsa ntchito kukumbukira konse; ma kilobytes angapo okha ndi omwe amadyedwa, mosasamala kanthu za kukula kwa zidutswa za data.
Algorithm yomwe mukugwiritsa ntchito imagwiritsa ntchito kukumbukira. Imalemba ma tag a mndandanda wanthawi omwe ali ndi zopatsa. Ndipo mwanjira iyi mumayang'ana kupezeka kwawiri mugulu limodzi la data komanso lina. Ndipo mumamvetsetsa ngati mphambano inachitika kapena ayi. Nthawi zambiri, ma database amagwiritsa ntchito ma cursors ndi obwereza omwe amasunga zomwe zili pano ndikudutsa mu data yosanjidwa chifukwa chazovuta za izi.
Chifukwa chiyani sitigwiritsa ntchito zolozera kuti tidutse deta?
Inde.
Timasunga mizere yosankhidwa mu LevelDB kapena mergeset. Titha kusuntha cholozera ndikupeza mphambano. Chifukwa chiyani sitigwiritsa ntchito? Chifukwa ndi pang'onopang'ono. Chifukwa zolozera zikutanthauza kuti muyenera kuyitanira ntchito pamzere uliwonse. Kuyimba foni ndi 5 nanoseconds. Ndipo ngati muli ndi mizere 100, ndiye kuti timathera theka la sekondi ndikuyimba ntchitoyi.
Pali chinthu choterocho, inde. Ndipo funso langa lomaliza. Funso likhoza kumveka lachilendo. N’chifukwa chiyani n’zosatheka kuŵerenga zophatikiza zonse zofunika panthawi imene deta ifika ndi kuisunga mumpangidwe wofunikira? Chifukwa chiyani kusunga ma voliyumu akulu mumakina ena monga VictoriaMetrics, ClickHouse, ndi zina zambiri, ndiyeno kumathera nthawi yochuluka pa iwo?
Ndipereka chitsanzo kuti ndimveke bwino. Tiye tinene kuti kachidole kakang'ono ka speedometer kamagwira ntchito bwanji? Imalemba mtunda womwe mwayenda, nthawi zonse ndikuwonjezera pamtengo umodzi, ndipo yachiwiri - nthawi. Ndipo amagawanitsa. Ndipo amapeza liwiro lapakati. Inu mukhoza kuchita za chinthu chomwecho. Onjezani mfundo zonse zofunika pa ntchentche.
Chabwino, ndamva funso. Chitsanzo chanu chili ndi malo ake. Ngati mukudziwa zomwe mukufunikira, ndiye kuti iyi ndiye njira yabwino kwambiri. Koma vuto ndiloti anthu amasunga ma metrics awa, deta ina mu ClickHouse ndipo sadziwa momwe angasankhire ndikusefa m'tsogolomu, kotero ayenera kusunga deta yonse yaiwisi. Koma ngati mukudziwa kuti muyenera kuwerengera chinthu pafupifupi, ndiye bwanji osawerengera m'malo mosunga mulu wamtengo wapatali pamenepo? Koma izi zimangokhala ngati mukudziwa zomwe mukufuna.
Mwa njira, ma database osungira nthawi amathandizira kuwerengera kwamagulu. Mwachitsanzo, Prometheus amathandizira . Ndiye kuti, izi zitha kuchitika ngati mukudziwa mayunitsi omwe mungafune. VictoriaMetrics ilibe izi, koma nthawi zambiri imatsogozedwa ndi Prometheus, momwe izi zitha kuchitikira m'malamulo obwereza.
Mwachitsanzo, mu ntchito yanga yapitayi ndimayenera kuwerengera kuchuluka kwa zochitika pawindo lotsetsereka pa ola lapitalo. Vuto ndiloti ndinayenera kupanga chizolowezi mu Go, mwachitsanzo, ntchito yowerengera chinthu ichi. Utumiki uwu pamapeto pake unali wosachepera, chifukwa ndizovuta kuwerengera. Kukhazikitsa kungakhale kosavuta ngati mukufuna kuwerengera ma aggregates panthawi yokhazikika. Ngati mukufuna kuwerengera zochitika pawindo lotsetsereka, ndiye kuti sizophweka monga momwe zikuwonekera. Ndikuganiza kuti izi sizinakwaniritsidwebe mu ClickHouse kapena m'ma database a timeseries, chifukwa ndizovuta kukhazikitsa.
Ndipo funso linanso. Tinkangolankhula za kuchulukana, ndipo ndinakumbukira kuti kale panali chinthu chonga Graphite chokhala ndi Carbon backend. Ndipo ankadziwa kuonda akale deta, ndiye kusiya mfundo imodzi pa mphindi, mfundo imodzi pa ola, etc. Mfundo, izi ndi yabwino kwambiri ngati tifunika yaiwisi deta, ndi kulankhula, kwa mwezi, ndi china chirichonse angathe. kuchepetsedwa. Koma Prometheus ndi VictoriaMetrics sizigwirizana ndi izi. Kodi chinalinganizidwa kuchichirikiza? Ngati sichoncho, chifukwa chiyani?
Zikomo chifukwa cha funso. Ogwiritsa ntchito amafunsa funsoli nthawi ndi nthawi. Amafunsa kuti tidzawonjezera liti chithandizo chotsitsa. Pali mavuto angapo pano. Choyamba, wogwiritsa ntchito aliyense amamvetsetsa downsampling chinachake chosiyana: wina akufuna kuti apeze mfundo iliyonse pa nthawi yomwe wapatsidwa, wina akufuna kuti zikhale zopambana, zochepa, zapakati. Ngati machitidwe ambiri amalembera deta ku database yanu, ndiye kuti simungathe kuziphatikiza pamodzi. Zitha kukhala kuti dongosolo lililonse limafunikira kupatulira kosiyana. Ndipo izi ndizovuta kukhazikitsa.
Ndipo chinthu chachiwiri ndichakuti VictoriaMetrics, monga ClickHouse, imakongoletsedwa kuti igwire ntchito ndi kuchuluka kwakukulu kwa data yaiwisi, kotero imatha kukokera mizere mabiliyoni osakwana sekondi imodzi ngati muli ndi ma cores ambiri m'dongosolo lanu. Kusanthula nthawi ku VictoriaMetrics - mapointi 50 pa sekondi iliyonse pachimake. Ndipo magwiridwe antchito awa amakulira ku ma cores omwe alipo. Ndiye kuti, ngati muli ndi ma cores 000, mwachitsanzo, mudzasanthula mfundo biliyoni pamphindikati. Ndipo katundu wa VictoriaMetrics ndi ClickHouse amachepetsa kufunika kotsitsa.
Chinthu china ndi chakuti VictoriaMetrics imakanikiza bwino izi. Kuponderezana kwapakati pakupanga kumachokera ku 0,4 mpaka 0,8 bytes pa mfundo iliyonse. Mfundo iliyonse ndi chizindikiro cha nthawi + mtengo. Ndipo imapanikizidwa kukhala yocheperapo imodzi mwa avareji.
SERGEY. Ndili ndi funso. Kodi nthawi yochepa yojambulira quantum ndi iti?
Mphindi imodzi. Posachedwapa tidacheza ndi opanga ma database ena anthawi. Nthawi yawo yochepa ndi sekondi imodzi. Ndipo mu Graphite, mwachitsanzo, ndi sekondi imodzi. Mu OpenTSDB ilinso sekondi imodzi. InfluxDB ili ndi kulondola kwa nanosecond. Ku VictoriaMetrics ndi millisecond imodzi, chifukwa ku Prometheus ndi millisecond imodzi. Ndipo VictoriaMetrics poyambirira idapangidwa ngati malo osungira akutali a Prometheus. Koma tsopano akhoza kupulumutsa deta ku machitidwe ena.
Munthu amene ndinalankhula naye akunena kuti ali ndi kulondola kwachiwiri mpaka kwachiwiri - ndizokwanira kwa iwo chifukwa zimatengera mtundu wa deta yomwe ikusungidwa mumndandanda wa nthawi. Ngati izi ndi deta ya DevOps kapena deta yochokera kuzinthu zowonongeka, komwe mumasonkhanitsa pazigawo za masekondi a 30, pamphindi, ndiye kuti kulondola kwachiwiri ndikokwanira, simukusowa chilichonse chochepa. Ndipo ngati musonkhanitsa deta iyi kuchokera ku machitidwe apamwamba a malonda, ndiye kuti mukufunikira kulondola kwa nanosecond.
Kulondola kwa Millisecond mu VictoriaMetrics ndikoyeneranso mlandu wa DevOps, ndipo kumatha kukhala koyenera milandu yambiri yomwe ndidatchula koyambirira kwa lipotilo. Chinthu chokhacho chomwe sichingakhale choyenera ndi machitidwe apamwamba ochita malonda.
Zikomo! Ndipo funso lina. Kodi kuyanjana mu PromQL ndi chiyani?
Kugwirizana kwathunthu mmbuyo. VictoriaMetrics imathandizira PromQL kwathunthu. Kuphatikiza apo, imawonjezera magwiridwe antchito apamwamba mu PromQL, yomwe imatchedwa . Pali zokamba pa YouTube zokhudzana ndi magwiridwe antchito awa. Ndinalankhula pa Monitoring Meetup m’ngululu ku St.
Kanema wa uthengawo .
Ogwiritsa ntchito olembetsedwa okha ndi omwe angatenge nawo gawo pa kafukufukuyu. chonde.
Nchiyani chikukulepheretsani kusintha kupita ku VictoriaMetrics ngati chosungira chanu chanthawi yayitali cha Prometheus? (Lembani mu ndemanga, ndikuwonjezera pavoti))
-
71,4%Sindigwiritsa ntchito Prometheus5
-
28,6%Sindimadziwa za VictoriaMetrics2
Ogwiritsa 7 adavota. Ogwiritsa ntchito 12 adakana.
Source: www.habr.com