{"id":80733,"date":"2020-05-08T13:42:31","date_gmt":"2020-05-08T11:42:31","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/go-optimizations-in-victoriametrics-aleksandr-valyalkin"},"modified":"2020-05-08T13:42:31","modified_gmt":"2020-05-08T11:42:31","slug":"go-optimizations-in-victoriametrics-aleksandr-valyalkin","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/go-optimizations-in-victoriametrics-aleksandr-valyalkin","title":{"rendered":"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><strong>Pon\u00fakame v\u00e1m mo\u017enos\u0165 obozn\u00e1mi\u0165 sa s prepisom spr\u00e1vy z konca roka 2019 od Alexandra Valjalkina \u201eOptimaliz\u00e1cie Go v VictoriaMetrics\u201c<\/strong><\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/victoriametrics.com\/\">VictoriaMetrics<\/a><\/noindex> \u2014 r\u00fdchly a \u0161k\u00e1lovate\u013en\u00fd DBMS na ukladanie a spracovanie \u00fadajov vo forme \u010dasov\u00fdch r\u00e1dov (z\u00e1znam vytv\u00e1ra \u010das a s\u00fabor hodnoty s\u00favisiaceho s t\u00fdmto \u010dasom, napr\u00edklad z\u00edskan\u00e9 pravideln\u00fdm zis\u0165ovan\u00edm stavu senzorov alebo zbieran\u00edm metr\u00edk).<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/dd13ec10594aeb138be29ce439cc476a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Tu je odkaz na video tejto spr\u00e1vy \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/youtu.be\/MZ5P21j_HLE\">https:\/\/youtu.be\/MZ5P21j_HLE<\/a><\/noindex><\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/docs.google.com\/presentation\/d\/1k7OjHvxTHA7669MFwsNTCx8hII-a8lNvpmQetLxmrEU\/edit?usp=sharing\">Prezent\u00e1cie<\/a><\/noindex><\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/0073390d6dbcc6ab3e5305907ac6a229.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Povieme si nie\u010do o mne. Ja som Alexander Valjalkin. Tu je <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/valyala\">m\u00f4j \u00fa\u010det na GitHub<\/a><\/noindex>. Zauj\u00edmam sa o Go a optimaliz\u00e1ciu v\u00fdkonu. Nap\u00edsal som ve\u013ea r\u00f4znych u\u017eito\u010dn\u00fdch a menej u\u017eito\u010dn\u00fdch kni\u017en\u00edc. Za\u010d\u00ednaj\u00fa bu\u010f s <code>fast<\/code>, alebo s <code>quick<\/code> prefixom. <\/p>\n<p><\/p>\n<p>V s\u00fa\u010dasnosti pracujem na VictoriaMetrics. \u010co to je a \u010do tam rob\u00edm? O tom v\u00e1m poviem v tejto prezent\u00e1cii. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/c9194bc3fee1f615d838979bec12f7d0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pl\u00e1n prezent\u00e1cie je tak\u00fd:<\/p>\n<p><\/p>\n<ul>\n<li>Na za\u010diatku v\u00e1m poviem, \u010do je VictoriaMetrics. <\/li>\n<li>Potom v\u00e1m poviem, \u010do s\u00fa \u010dasov\u00e9 rady. <\/li>\n<li>Potom v\u00e1m vysvetl\u00edm, ako funguje datab\u00e1za \u010dasov\u00fdch r\u00e1d.<\/li>\n<li>Potom v\u00e1m poviem o architekt\u00fare datab\u00e1zy: z \u010doho je zlo\u017een\u00e1.<\/li>\n<li>A potom prejdeme k optimaliz\u00e1ci\u00e1m, ktor\u00e9 existuj\u00fa vo VictoriaMetrics. To je optimaliz\u00e1cia inverzn\u00e9ho indexu a optimaliz\u00e1cia realiz\u00e1cie bitset na Go.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/3e0244b6b6fe952f660e4a094778921a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Vie niekto v publiku, \u010do je VictoriaMetrics? Ni\u010d si, u\u017e ve\u013ea \u013eud\u00ed to vie. To je dobr\u00e1 spr\u00e1va. Pre t\u00fdch, ktor\u00ed nevedia \u2013 je to datab\u00e1za pre \u010dasov\u00e9 rady. Je postaven\u00e1 na architekt\u00fare ClickHouse, na niektor\u00fdch detailoch implement\u00e1cie ClickHouse. Napr\u00edklad na tak\u00fdch, ako: MergeTree, paraleln\u00e9 v\u00fdpo\u010dty na v\u0161etk\u00fdch dostupn\u00fdch jadr\u00e1ch procesora a optimaliz\u00e1cia v\u00fdkonu prostredn\u00edctvom pr\u00e1ce s blokmi \u00fadajov, ktor\u00e9 sa umiest\u0148uj\u00fa do cache procesora. <\/p>\n<p><\/p>\n<p>VictoriaMetrics poskytuje najlep\u0161ie kompresie \u00fadajov v porovnan\u00ed s in\u00fdmi datab\u00e1zami pre \u010dasov\u00e9 rady. <\/p>\n<p><\/p>\n<p>\u0160k\u00e1lovanie je vertik\u00e1lne \u2014 t. j. m\u00f4\u017eete prida\u0165 viac procesorov, viac opera\u010dnej pam\u00e4te na jednom po\u010d\u00edta\u010di. VictoriaMetrics \u00faspe\u0161ne vyu\u017eije tieto dostupn\u00e9 zdroje a zv\u00fd\u0161i line\u00e1rny v\u00fdkon.<\/p>\n<p><\/p>\n<p>VictoriaMetrics sa tie\u017e horizont\u00e1lne \u0161k\u00e1luje \u2014 t. j. m\u00f4\u017eete prida\u0165 \u010fal\u0161ie uzly do klastrov VictoriaMetrics, a jej v\u00fdkon sa takmer line\u00e1rne zvy\u0161uje.<\/p>\n<p><\/p>\n<p>Ako ste sa zhodli, VictoriaMetrics je r\u00fdchla datab\u00e1za, preto\u017ee o in\u00fdch nem\u00f4\u017eem p\u00edsa\u0165. A je nap\u00edsan\u00e1 v Go, tak\u017ee o nej hovor\u00edm na tomto meetupe.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/0121c9baead722eb1fe22fb6f900d4ad.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Kto vie, \u010do je \u010dasov\u00e1 rada? Ve\u013ea \u013eud\u00ed to vie. \u010casov\u00e1 rada je s\u00e9ria p\u00e1r <code>(\u010dasov\u00e1 zna\u010dka, hodnota)<\/code>, kde s\u00fa tieto p\u00e1ry zoraden\u00e9 pod\u013ea \u010dasu. Hodnota predstavuje \u010d\u00edslo s pl\u00e1vaj\u00facou desatinnou \u010diarkou \u2013 float64.<\/p>\n<p><\/p>\n<p>Ka\u017ed\u00e1 \u010dasov\u00e1 rada je jedine\u010dne identifikovan\u00e1 k\u013e\u00fa\u010dom. Z \u010doho sa sklad\u00e1 tento k\u013e\u00fa\u010d? Sklad\u00e1 sa z nepr\u00e1zdnej mno\u017einy p\u00e1rov k\u013e\u00fa\u010d-hodnota. <\/p>\n<p><\/p>\n<p>Tu je pr\u00edklad \u010dasovej rady. K\u013e\u00fa\u010dom tejto rady je zoznam p\u00e1rov: <code>__name__=\"cpu_usage\"<\/code> \u2013 to je n\u00e1zov metriky, <code>instance=\"my-server\"<\/code> \u2014 to je po\u010d\u00edta\u010d, na ktorom bola t\u00e1to metrika zhroma\u017eden\u00e1, <code>datacenter=\"us-east\"<\/code> \u2014 to je d\u00e1tov\u00e9 centrum, kde sa nach\u00e1dza tento po\u010d\u00edta\u010d.<\/p>\n<p><\/p>\n<p>M\u00e1me meno \u010dasovej rady, ktor\u00e9 pozost\u00e1va z troch p\u00e1rov k\u013e\u00fa\u010d-hodnota. Tomuto k\u013e\u00fa\u010du zodpoved\u00e1 zoznam p\u00e1rov <code>(timestamp, value)<\/code>. <code>t1, t3, t3, ..., tN<\/code> \u2014 to s\u00fa \u010dasov\u00e9 pe\u010diatky, <code>10, 20, 12, ..., 15<\/code> \u2014 zodpovedaj\u00face hodnoty. Toto je cpu-usage v danom okamihu pre dan\u00fa radu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/76cab9db0263318d355ac607ceb7e0f6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Kde m\u00f4\u017eu by\u0165 pou\u017eit\u00e9 \u010dasov\u00e9 rady? M\u00e1 niekto nejak\u00e9 n\u00e1pady? <\/p>\n<p><\/p>\n<ul>\n<li>V DevOps m\u00f4\u017eeme mera\u0165 za\u0165a\u017eenie CPU, RAM, siete, rps, po\u010det ch\u00fdb a pod. <\/li>\n<li>IoT \u2013 m\u00f4\u017eeme mera\u0165 teplotu, tlak, geografick\u00e9 s\u00faradnice a \u010fal\u0161ie.<\/li>\n<li>Rovnako vo financi\u00e1ch \u2013 m\u00f4\u017eeme monitorova\u0165 ceny r\u00f4znych akci\u00ed a mien. <\/li>\n<li>Okrem toho m\u00f4\u017eu by\u0165 \u010dasov\u00e9 rady pou\u017eit\u00e9 na monitorovanie v\u00fdrobn\u00fdch procesov vo fabrik\u00e1ch. M\u00e1me pou\u017e\u00edvate\u013eov, ktor\u00ed pou\u017e\u00edvaj\u00fa VictoriaMetrics na sledovanie vetern\u00fdch turb\u00edn, pre roboty.<\/li>\n<li>\u010casov\u00e9 rady s\u00fa tie\u017e u\u017eito\u010dn\u00e9 na zhroma\u017e\u010fovanie inform\u00e1ci\u00ed zo senzorov r\u00f4znych zariaden\u00ed. Napr\u00edklad pre motor; na meranie tlaku v pneumatik\u00e1ch; na meranie r\u00fdchlosti, vzdialenosti; na meranie spotreby benz\u00ednu a pod.<\/li>\n<li>\u010casov\u00e9 rady sa m\u00f4\u017eu tie\u017e pou\u017e\u00edva\u0165 na monitorovanie lietadiel. V ka\u017edom lietadle je \u010dierna skrinka, ktor\u00e1 zhroma\u017e\u010fuje \u010dasov\u00e9 rady pre r\u00f4zne parametre zdravia lietadla. \u010casov\u00e9 rady sa tie\u017e pou\u017e\u00edvaj\u00fa v leteckom priemysle. <\/li>\n<li>Zdravotn\u00e1 starostlivos\u0165 \u2013 to s\u00fa krvn\u00fd tlak, pulz at\u010f.<\/li>\n<\/ul>\n<p><\/p>\n<p>Mo\u017eno existuj\u00fa e\u0161te in\u00e9 aplik\u00e1cie, na ktor\u00e9 som zabudol, ale d\u00fafam, \u017ee ste pochopili, \u017ee \u010dasov\u00e9 rady s\u00fa akt\u00edvne vyu\u017e\u00edvan\u00e9 v modernej dobe. A objem ich pou\u017e\u00edvania ka\u017ed\u00fdm rokom rastie.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/f1da29d372163751268b6a8f86836d04.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na \u010do sl\u00fa\u017ei datab\u00e1za pre \u010dasov\u00e9 rady? Pre\u010do nie je mo\u017en\u00e9 pou\u017ei\u0165 be\u017en\u00fa rela\u010dn\u00fa datab\u00e1zu na uchov\u00e1vanie \u010dasov\u00fdch r\u00e1dov?<\/p>\n<p><\/p>\n<p>Preto\u017ee v \u010dasov\u00fdch radoch zvy\u010dajne existuje ve\u013ek\u00e9 mno\u017estvo inform\u00e1ci\u00ed, ktor\u00e9 je \u0165a\u017ek\u00e9 uchov\u00e1va\u0165 a spracov\u00e1va\u0165 v be\u017en\u00fdch datab\u00e1zach. Preto vznikli \u0161pecializovan\u00e9 datab\u00e1zy pre \u010dasov\u00e9 rady. Tieto datab\u00e1zy efekt\u00edvne ukladaj\u00fa body <code>(timestamp, value)<\/code> s dan\u00fdm k\u013e\u00fa\u010dom. Poskytuj\u00fa API na \u010d\u00edtanie ulo\u017een\u00fdch \u00fadajov pod\u013ea k\u013e\u00fa\u010da, jednotliv\u00fdch p\u00e1rov k\u013e\u00fa\u010d-hodnota, alebo pod\u013ea nieko\u013ek\u00fdch tak\u00fdch p\u00e1rov, alebo pod\u013ea regexp. Napr\u00edklad, ak chcete n\u00e1js\u0165 za\u0165a\u017eenie procesora v\u0161etk\u00fdch va\u0161ich slu\u017eieb v d\u00e1tovom centre v Amerike, potrebujete pou\u017ei\u0165 tak\u00fdto pseudo-dotaz.<\/p>\n<p><\/p>\n<p>\u0160pecializovan\u00e9 datab\u00e1zy pre \u010dasov\u00e9 rady zvy\u010dajne pon\u00fakaj\u00fa \u0161pecializovan\u00e9 dotazovacie jazyky, preto\u017ee SQL pre \u010dasov\u00e9 rady nie je ve\u013emi vhodn\u00fd. Aj ke\u010f existuj\u00fa datab\u00e1zy, ktor\u00e9 podporuj\u00fa SQL, nie je to ide\u00e1lne. Lep\u0161ie sa hodia tak\u00e9 dotazovacie jazyky, ako s\u00fa <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/@valyala\/promql-tutorial-for-beginners-9ab455142085\">PromQL<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/docs.influxdata.com\/influxdb\/v1.8\/query_language\/spec\/\">InfluxQL<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/www.influxdata.com\/products\/flux\/\">Flux<\/a><\/noindex>, <noindex><a rel=\"nofollow\" href=\"https:\/\/code.kx.com\/q\/\">Q<\/a><\/noindex>. D\u00fafam, \u017ee niekto po\u010dul aspo\u0148 o jednom z t\u00fdchto jazykov. O PromQL ur\u010dite po\u010dulo mnoho \u013eud\u00ed. To je dotazovac\u00ed jazyk pre Prometheus.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/59e5b6d82a2f7861077bc5fe34519ee4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Takto vyzer\u00e1 architekt\u00fara modernej datab\u00e1zy pre \u010dasov\u00e9 rady na pr\u00edklade VictoriaMetrics.<\/p>\n<p><\/p>\n<p>Sklad\u00e1 sa z dvoch \u010dast\u00ed. Je to \u00falo\u017eisko pre inverzovan\u00fd index a \u00falo\u017eisko pre hodnoty \u010dasov\u00fdch r\u00e1dov. Tieto \u00falo\u017eisk\u00e1 s\u00fa oddelen\u00e9. <\/p>\n<p><\/p>\n<p>Ke\u010f pr\u00edde nov\u00fd z\u00e1znam do datab\u00e1zy, najprv sa obraciame na inverzovan\u00fd index, aby sme na\u0161li identifik\u00e1tor \u010dasov\u00e9ho radu pod\u013ea dan\u00e9ho s\u00faboru <code>label=value<\/code> pre t\u00fato metriku. N\u00e1jdeme tento identifik\u00e1tor a ulo\u017e\u00edme hodnotu do \u00falo\u017eiska \u00fadajov.<\/p>\n<p><\/p>\n<p>Ke\u010f pr\u00edde nejak\u00fd dopyt na v\u00fdber \u00fadajov z TSDB, v prvom rade sa ponor\u00edme do inverzovan\u00e9ho indexu. Z\u00edskame v\u0161etky <code>timeseries_ids<\/code> z\u00e1znamy, ktor\u00e9 zodpovedaj\u00fa dan\u00e9mu s\u00faboru <code>label=value<\/code>. A potom z\u00edskame v\u0161etky potrebn\u00e9 \u00fadaje z \u00falo\u017eiska \u00fadajov, indexovan\u00e9ho pod\u013ea <code>timeseries_ids<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/eb9b36fa3c6de2188b97b56ad3839b2c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Zv\u00e1\u017eme pr\u00edklad, ako datab\u00e1za pre \u010dasov\u00e9 rady spracov\u00e1va prich\u00e1dzaj\u00faci select-dotaz.<\/p>\n<p><\/p>\n<ul>\n<li>V prvom rade z\u00edska v\u0161etky <code>timeseries_ids<\/code> z inverzovan\u00e9ho indexu, ktor\u00e9 obsahuj\u00fa dan\u00e9 p\u00e1ry <code>label=value<\/code>, alebo sp\u013a\u0148aj\u00fa dan\u00e9 regul\u00e1rne v\u00fdrazy.<\/li>\n<li>Potom z\u00edska v\u0161etky data points z \u00falo\u017eiska \u00fadajov na danom \u010dasovom intervale pre n\u00e1jden\u00e9 <code>timeseries_ids<\/code>.<\/li>\n<li>Po tomto datab\u00e1za vykon\u00e1va nejak\u00e9 v\u00fdpo\u010dty nad t\u00fdmito d\u00e1tov\u00fdmi bodmi pod\u013ea po\u017eiadavky pou\u017e\u00edvate\u013ea. A potom vracia odpove\u010f.<\/li>\n<\/ul>\n<p><\/p>\n<p>V tejto prezent\u00e1cii v\u00e1m poviem o prvej \u010dasti. Je to vyh\u013ead\u00e1vanie <code>timeseries_ids<\/code> pod\u013ea invertovan\u00e9ho indexu. O druhej a tretej \u010dasti si m\u00f4\u017eete nesk\u00f4r pozrie\u0165 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\">zdroje VictoriaMetrics<\/a><\/noindex>, alebo po\u010dka\u0165, k\u00fdm priprav\u00edm \u010fal\u0161ie predn\u00e1\u0161ky \ud83d\ude42<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/3a7dab707dd1defb9bc674a342052a03.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prejdeme k invertovan\u00e9mu indexu. Mnoh\u00fdm sa to m\u00f4\u017ee zda\u0165 jednoduch\u00e9. Kto vie, \u010do je invertovan\u00fd index a ako funguje? O, u\u017e nie je tak ve\u013ea \u013eud\u00ed. Pok\u00fasme sa pochopi\u0165, \u010do to je. <\/p>\n<p><\/p>\n<p>Na prv\u00fd poh\u013ead je to jednoduch\u00e9. Je to len slovn\u00edk, ktor\u00fd zobrazuje k\u013e\u00fa\u010d na hodnotu. \u010co je to k\u013e\u00fa\u010d? Tento p\u00e1r <code>label=value<\/code>, kde <code>label<\/code> a <code>value<\/code> je re\u0165azec. A hodnoty s\u00fa s\u00fabor <code>timeseries_ids<\/code>, ktor\u00fd obsahuje dan\u00fd p\u00e1r <code>label=value<\/code>.<\/p>\n<p><\/p>\n<p>Invertovan\u00fd index umo\u017e\u0148uje r\u00fdchlo n\u00e1js\u0165 v\u0161etky <code>timeseries_ids<\/code>, ktor\u00e9 obsahuj\u00fa dan\u00e9 <code>label=value<\/code>.<\/p>\n<p><\/p>\n<p>A takisto umo\u017e\u0148uje r\u00fdchlo n\u00e1js\u0165 <code>timeseries_ids<\/code> \u010dasov\u00e9 rady pre viacero p\u00e1rov <code>label=value<\/code>, alebo pre p\u00e1ry <code>label=regexp<\/code>. Ako to funguje? Pomocou zis\u0165ovania prieniku mno\u017einy <code>timeseries_ids<\/code> pre ka\u017ed\u00fd p\u00e1r <code>label=value<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/837a15a078e8c9422c721f3f072c0c09.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pozrime sa na r\u00f4zne implement\u00e1cie invertovan\u00e9ho indexu. Za\u010dneme s najjednoduch\u0161ou naivnou implement\u00e1ciou. Vyzer\u00e1 to takto. <\/p>\n<p><\/p>\n<p>Funkcia <code>getMetricIDs<\/code> z\u00edskava zoznam re\u0165azcov. Ka\u017ed\u00fd re\u0165azec obsahuje <code>label=value<\/code>. T\u00e1to funkcia vracia zoznam <code>metricIDs<\/code>.<\/p>\n<p><\/p>\n<p>Ako to funguje? Tu m\u00e1me glob\u00e1lnu premenn\u00fa, ktor\u00e1 sa vol\u00e1 <code>invertedIndex<\/code>. Je to be\u017en\u00fd slovn\u00edk (<code>map<\/code>), ktor\u00fd zobrazuje re\u0165azec na slice int-ov. Re\u0165azec obsahuje <code>label=value<\/code>.<\/p>\n<p><\/p>\n<p>Implement\u00e1cia funkcie: vyber\u00e1me <code>metricIDs<\/code> pre prv\u00fd <code>label=value<\/code>, potom prech\u00e1dzame v\u0161etk\u00fdmi ostatn\u00fdmi <code>label=value<\/code>, vyber\u00e1me <code>metricIDs<\/code> pre nich. A vol\u00e1me funkciu <code>intersectInts<\/code>, o ktorej sa bude hovori\u0165 nesk\u00f4r. A t\u00e1to funkcia vr\u00e1ti prienik t\u00fdchto zoznamov.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/709beca79884a9693098781974c26f4d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako vid\u00edte, implement\u00e1cia invertovan\u00e9ho indexu nie je ve\u013emi zlo\u017eit\u00e1. Ale je to naivn\u00e1 implement\u00e1cia. Ak\u00e9 m\u00e1 nev\u00fdhody? Hlavn\u00e1 nev\u00fdhoda naivnej implement\u00e1cie je v tom, \u017ee tak\u00fd invertovan\u00fd index m\u00e1me ulo\u017een\u00fd v opera\u010dnej pam\u00e4ti. Po re\u0161tarte aplik\u00e1cie tento index strat\u00edme. Nie je \u017eiadne ulo\u017eenie tohto indexu na disk. Pre datab\u00e1zu tak\u00fd invertovan\u00fd index asi \u0165a\u017eko vyhovuje.<\/p>\n<p><\/p>\n<p>Druh\u00fd nedostatok s\u00favis\u00ed taktie\u017e s pam\u00e4\u0165ou. Inverzn\u00fd index mus\u00ed by\u0165 ulo\u017een\u00fd v RAM. Ak presiahne ve\u013ekos\u0165 opera\u010dnej pam\u00e4te, je jasn\u00e9, \u017ee dostaneme - chyba nedostatku pam\u00e4te. A program nebude fungova\u0165.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/495394168ac03aa8cbcf56ccad1cb2b8.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tento probl\u00e9m je mo\u017en\u00e9 vyrie\u0161i\u0165 pomocou hotov\u00fdch rie\u0161en\u00ed, ako s\u00fa <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/google\/leveldb\">LevelDB<\/a><\/noindex>, alebo <noindex><a rel=\"nofollow\" href=\"https:\/\/rocksdb.org\/\">RocksDB<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p>Ak to zhrnieme, potrebujeme datab\u00e1zu, ktor\u00e1 umo\u017e\u0148uje r\u00fdchlo vykona\u0165 tri oper\u00e1cie. <\/p>\n<p><\/p>\n<ul>\n<li>Prv\u00e1 oper\u00e1cia \u2013 to je z\u00e1pis <code>k\u013e\u00fa\u010d-hodnota<\/code> do tejto datab\u00e1zy. Rob\u00ed to ve\u013emi r\u00fdchlo, kde <code>k\u013e\u00fa\u010d-hodnota<\/code> \u2013 to s\u00fa \u013eubovo\u013en\u00e9 re\u0165azce. <\/li>\n<li>Druh\u00e1 oper\u00e1cia \u2013 to je r\u00fdchle vyh\u013ead\u00e1vanie hodnoty pod\u013ea zadan\u00e9ho k\u013e\u00fa\u010da.<\/li>\n<li>A tretia oper\u00e1cia \u2013 to je r\u00fdchle vyh\u013ead\u00e1vanie v\u0161etk\u00fdch hodn\u00f4t pod\u013ea zadan\u00e9ho prefixu. <\/li>\n<\/ul>\n<p><\/p>\n<p>LevelDB a RocksDB \u2013 tieto datab\u00e1zy vyvinuli Google a Facebook. Najprv sa objavila LevelDB. Potom chalani z Facebooku vzali LevelDB a za\u010dali ju vylep\u0161ova\u0165, vytvorili RocksDB. Teraz na RocksDB vo Facebooku pracuj\u00fa takmer v\u0161etky intern\u00e9 datab\u00e1zy, vr\u00e1tane toho, \u017ee pre\u0161li na RocksDB aj MySQL. Nazvali to <noindex><a rel=\"nofollow\" href=\"http:\/\/myrocks.io\/\">MyRocks<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p>Inverzn\u00fd index je mo\u017en\u00e9 realizova\u0165 pomocou LevelDB. Ako to urob\u00edme? Ulo\u017e\u00edme ako k\u013e\u00fa\u010d <code>label=value<\/code>. A ako hodnotu \u2013 identifik\u00e1tor \u010dasovej rady, kde je pr\u00edtomn\u00fd p\u00e1r <code>label=value<\/code>.<\/p>\n<p><\/p>\n<p>Ak m\u00e1me ve\u013ea \u010dasov\u00fdch r\u00e1d s dan\u00fdm p\u00e1rom <code>label=value<\/code>, bude v tejto datab\u00e1ze ve\u013ea riadkov so zhodn\u00fdm k\u013e\u00fa\u010dom a r\u00f4znymi <code>timeseries_ids<\/code>. Aby sme z\u00edskali zoznam v\u0161etk\u00fdch <code>timeseries_ids<\/code>, ktor\u00e9 za\u010d\u00ednaj\u00fa na dan\u00fd <code>label=prefix<\/code>, urob\u00edme range scan, na ktor\u00fd je t\u00e1to datab\u00e1za optimalizovan\u00e1. T. j. vyberieme v\u0161etky riadky, ktor\u00e9 za\u010d\u00ednaj\u00fa s <code>label=prefix<\/code> a z\u00edskame potrebn\u00e9 <code>timeseries_ids<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/a40877ecb758ac3bc1979bcd568d7e83.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Toto je pribli\u017en\u00e1 realiz\u00e1cia, ako by vyzerala v Go. M\u00e1me inverzn\u00fd index. Toto je LevelDB.<\/p>\n<p><\/p>\n<p>Funkcia je rovnak\u00e1 ako pri naivnej realiz\u00e1cii. Takmer riadok za riadkom opakuje naivn\u00fa realiz\u00e1ciu. Jedin\u00e1 vec je, \u017ee namiesto pr\u00edstupu k <code>map<\/code> pristupujeme k inverzn\u00e9mu indexu. Z\u00edskavame v\u0161etky hodnoty pre prv\u00fd <code>label=value<\/code>. Potom prech\u00e1dzame v\u0161etk\u00fdmi zost\u00e1vaj\u00facimi p\u00e1rmi <code>label=value<\/code> a z\u00edskavame zodpovedaj\u00face s\u00fapravy metricIDs pre ne. N\u00e1sledne nach\u00e1dzame prienik. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/de11b15286816ada04b22727ff78e43d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Zdalo by sa, \u017ee je v\u0161etko v poriadku, ale v tomto rie\u0161en\u00ed s\u00fa nedostatky. VictoriaMetrics najprv realizovala inverzn\u00fd index na z\u00e1klade LevelDB. Ale nakoniec sa musela tohto pr\u00edstupu vzda\u0165.<\/p>\n<p><\/p>\n<p>Pre\u010do? Preto\u017ee LevelDB je pomal\u0161\u00ed ako naivn\u00e1 realiz\u00e1cia. Pri naivnej realiz\u00e1cii pod\u013ea zadan\u00e9ho k\u013e\u00fa\u010da z\u00edskavame hne\u010f cel\u00fd slice. <code>metricIDs<\/code>. Tento ve\u013emi r\u00fdchly proces \u2013 cel\u00fd slice je pripraven\u00fd na pou\u017eitie.<\/p>\n<p><\/p>\n<p>V LevelDB sa pri ka\u017edom zavolan\u00ed funkcie <code>GetValues<\/code> mus\u00edte prejs\u0165 v\u0161etky riadky, ktor\u00e9 za\u010d\u00ednaj\u00fa na <code>label=value<\/code>. A pre ka\u017ed\u00fd riadok z\u00edska\u0165 hodnotu <code>timeseries_ids<\/code>. Z tak\u00fdchto <code>timeseries_ids<\/code> zhroma\u017edi\u0165 slice t\u00fdchto <code>timeseries_ids<\/code>. Je o\u010dividn\u00e9, \u017ee je to ove\u013ea pomal\u0161ie ako jednoduch\u00fd pr\u00edstup k be\u017en\u00e9mu mapu pod\u013ea k\u013e\u00fa\u010da.<\/p>\n<p><\/p>\n<p>Druh\u00fd nedostatok je, \u017ee LevelDB je nap\u00edsan\u00fd v C. Volanie C-funkci\u00ed z Go nie je ve\u013emi r\u00fdchle. Trv\u00e1 to stovky nanosek\u00fand. Nie je to ve\u013emi r\u00fdchle, preto\u017ee v porovnan\u00ed s be\u017en\u00fdm volan\u00edm funkcie nap\u00edsanej v Go, ktor\u00e1 trv\u00e1 1-5 nanosek\u00fand, je rozdiel v v\u00fdkonnosti v desiatkach n\u00e1sobkov. Pre VictoriaMetrics to bol fat\u00e1lny nedostatok \ud83d\ude42<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/47cace825807b7b062064c2e790dc9d3.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Preto som nap\u00edsal vlastn\u00fa implement\u00e1ciu inverzn\u00e9ho indexu. A nazval ju <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/tree\/master\/lib\/mergeset\">mergeset<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p>Mergeset je zalo\u017een\u00fd na d\u00e1tovej \u0161trukt\u00fare MergeTree. T\u00e1to d\u00e1tov\u00e1 \u0161trukt\u00fara je prevzat\u00e1 z ClickHouse. Je zrejm\u00e9, \u017ee mergeset mus\u00ed by\u0165 optimalizovan\u00fd pre r\u00fdchle vyh\u013ead\u00e1vanie <code>timeseries_ids<\/code> pod\u013ea zadan\u00e9ho k\u013e\u00fa\u010da. Mergeset je nap\u00edsan\u00fd cel\u00fd v Go. M\u00f4\u017eete si pozrie\u0165 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\">zdrojov\u00e9 k\u00f3dy VictoriaMetrics na GitHub<\/a><\/noindex>. Implement\u00e1cia mergeset sa nach\u00e1dza v prie\u010dinku <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/tree\/master\/lib\/mergeset\">\/lib\/mergeset<\/a><\/noindex>. M\u00f4\u017eete sa pok\u00fasi\u0165 zisti\u0165, \u010do sa tam deje.<\/p>\n<p><\/p>\n<p>API mergeset je ve\u013emi podobn\u00e9 LevelDB a RocksDB. To znamen\u00e1, \u017ee umo\u017e\u0148uje r\u00fdchlo ulo\u017ei\u0165 nov\u00e9 z\u00e1znamy a r\u00fdchlo vybra\u0165 z\u00e1znamy pod\u013ea zadan\u00e9ho prefixu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/845704ed36f70beb468907d22701af59.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>O nedostatkoch mergeset si povieme nesk\u00f4r. Teraz sa porozpr\u00e1vame o tom, ak\u00e9 probl\u00e9my vznikli s VictoriaMetrics v produkcii pri implement\u00e1cii inverzn\u00e9ho indexu.<\/p>\n<p><\/p>\n<p>Pre\u010do vznikli?<\/p>\n<p><\/p>\n<p>Prv\u00e1 pr\u00ed\u010dina \u2013 je vysok\u00e1 miera churn. V preklade do sloven\u010diny \u2013 to je \u010dast\u00e1 zmena \u010dasov\u00fdch radov. To sa stane, ke\u010f \u010dasov\u00fd rad kon\u010d\u00ed a za\u010d\u00edna nov\u00fd rad, alebo za\u010d\u00edna ve\u013ea nov\u00fdch \u010dasov\u00fdch radov. A to sa deje \u010dasto.<\/p>\n<p><\/p>\n<p>Druh\u00e1 pr\u00ed\u010dina \u2013 je ve\u013ek\u00e9 mno\u017estvo \u010dasov\u00fdch radov. Na za\u010diatku, ke\u010f monitorovanie naberalo na popularite, po\u010det \u010dasov\u00fdch radov bol mal\u00fd. Napr\u00edklad, pre ka\u017ed\u00fd po\u010d\u00edta\u010d je potrebn\u00e9 monitorova\u0165 za\u0165a\u017eenie procesora, pam\u00e4te, siete a disku. 4 \u010dasov\u00e9 rady na ka\u017ed\u00fd po\u010d\u00edta\u010d. M\u00e1te napr\u00edklad 100 po\u010d\u00edta\u010dov a 400 \u010dasov\u00fdch radov. To je ve\u013emi m\u00e1lo. <\/p>\n<p><\/p>\n<p>V priebehu \u010dasu si \u013eudia vymysleli, \u017ee je mo\u017en\u00e9 mera\u0165 podrobnej\u0161ie inform\u00e1cie. Napr\u00edklad mera\u0165 za\u0165a\u017eenie nie cel\u00e9ho procesora, ale jednotliv\u00fdch procesorov\u00fdch jadier. Ak m\u00e1te 40 procesorov\u00fdch jadier, dostanete tak 40-kr\u00e1t viac \u010dasov\u00fdch radov na meranie za\u0165a\u017eenia procesora. <\/p>\n<p><\/p>\n<p>Ale to e\u0161te nie je v\u0161etko. Ka\u017ed\u00e9 procesorov\u00e9 jadro m\u00f4\u017ee ma\u0165 nieko\u013eko stavov, ako je idle, ke\u010f nepracuje. Takisto m\u00f4\u017ee pracova\u0165 v u\u017e\u00edvate\u013eskom prostred\u00ed, v jadre a v in\u00fdch stavoch. A ka\u017ed\u00fd tak\u00fdto stav sa d\u00e1 tie\u017e mera\u0165 ako samostatn\u00fd \u010dasov\u00fd rad. To dodato\u010dne zvy\u0161uje po\u010det radov 7-8-kr\u00e1t.<\/p>\n<p><\/p>\n<p>Z jednej metriky sme dostali 40 x 8 = 320 metr\u00edk len pre jeden po\u010d\u00edta\u010d. N\u00e1sob\u00edme to 100 a dostaneme 32 000 namiesto 400. <\/p>\n<p><\/p>\n<p>Potom sa objavil Kubernetes. A situ\u00e1cia sa e\u0161te zhor\u0161ila, preto\u017ee v Kubernetes m\u00f4\u017ee hosti\u0165 ve\u013ea r\u00f4znych slu\u017eieb. Ka\u017ed\u00e1 slu\u017eba v Kubernetes sa sklad\u00e1 z mnoh\u00fdch podov. A to v\u0161etko je potrebn\u00e9 monitorova\u0165. Okrem toho m\u00e1me neust\u00e1ly deployment nov\u00fdch verzi\u00ed va\u0161ich slu\u017eieb. Pre ka\u017ed\u00fa nov\u00fa verziu je potrebn\u00e9 vytvori\u0165 nov\u00e9 \u010dasov\u00e9 rady. Nakoniec po\u010det \u010dasov\u00fdch radov exponenci\u00e1lne rastie a \u010del\u00edme probl\u00e9mu vysok\u00e9ho kardin\u00e1lneho mno\u017estva \u010dasov\u00fdch radov. VictoriaMetrics s n\u00edm \u00faspe\u0161ne bojuje v porovnan\u00ed s in\u00fdmi datab\u00e1zami pre \u010dasov\u00e9 rady. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/c11b7ce6d3294ae420243f72636af4b9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Po\u010fme sa podrobnej\u0161ie pozrie\u0165 na vysok\u00fa miera obratu. \u010co sp\u00f4sobuje vysok\u00fa mieru obratu v produkcii? Preto\u017ee niektor\u00e9 hodnoty zna\u010diek a tagov sa neust\u00e1le menia.<\/p>\n<p><\/p>\n<p>Napr\u00edklad vezmeme Kubernetes, v ktorom existuje pojem <code>deployment<\/code>, t. j. ke\u010f sa vyd\u00e1va nov\u00e1 verzia va\u0161ej aplik\u00e1cie. V\u00fdvoj\u00e1ri Kubernetes z nejak\u00e9ho d\u00f4vodu rozhodli prida\u0165 ID deploymentu do labelu.<\/p>\n<p><\/p>\n<p>\u010co to sp\u00f4sobilo? To, \u017ee pri ka\u017edom novom deployment\u2019e s\u00fa v\u0161etky star\u00e9 \u010dasov\u00e9 rady preru\u0161en\u00e9 a namiesto nich za\u010d\u00ednaj\u00fa nov\u00e9 \u010dasov\u00e9 rady s novou hodnotou labelu. <code>deployment_id<\/code>. Tak\u00fdch radov m\u00f4\u017ee by\u0165 stovky tis\u00edc a dokonca mili\u00f3ny.<\/p>\n<p><\/p>\n<p>D\u00f4le\u017eitou vlastnos\u0165ou tohto v\u0161etk\u00e9ho je to, \u017ee celkov\u00fd po\u010det \u010dasov\u00fdch radov rastie, ale po\u010det \u010dasov\u00fdch radov, ktor\u00e9 s\u00fa v danom okamihu akt\u00edvne a na ktor\u00e9 prich\u00e1dzaj\u00fa d\u00e1ta, zost\u00e1va kon\u0161tantn\u00fd. Tento stav sa naz\u00fdva \u2013 vysok\u00e1 miera obratu.<\/p>\n<p><\/p>\n<p>Hlavn\u00fd probl\u00e9m vysok\u00e9ho churn rate spo\u010d\u00edva v zabezpe\u010den\u00ed nepretr\u017eitej r\u00fdchlosti vyh\u013ead\u00e1vania v\u0161etk\u00fdch \u010dasov\u00fdch radov pod\u013ea zadan\u00e9ho s\u00faboru \u0161t\u00edtkov za ur\u010dit\u00fd \u010dasov\u00fd interval. Zvy\u010dajne ide o \u010dasov\u00fd interval za posledn\u00fa hodinu alebo za posledn\u00fd de\u0148. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/ac18482b87cc37624d163b30c68cf7be.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako tento probl\u00e9m vyrie\u0161i\u0165? Tu je prv\u00e1 mo\u017enos\u0165. Rozdeli\u0165 invertovan\u00fd index na nez\u00e1visl\u00e9 \u010dasti pod\u013ea \u010dasu. T.j. prejde ur\u010dit\u00fd \u010dasov\u00fd interval, prestaneme pracova\u0165 s aktu\u00e1lnym invertovan\u00fdm indexom a vytvor\u00edme nov\u00fd invertovan\u00fd index. Prejde \u010fal\u0161\u00ed \u010dasov\u00fd interval, vytvor\u00edme \u010fal\u0161\u00ed a e\u0161te jeden. <\/p>\n<p><\/p>\n<p>A pri dotaze z t\u00fdchto invertovan\u00fdch indexov n\u00e1jdeme s\u00fabor invertovan\u00fdch indexov, ktor\u00e9 sa nach\u00e1dzaj\u00fa v zadanom intervale. A odtia\u013e vyberieme id \u010dasov\u00fdch radov. <\/p>\n<p><\/p>\n<p>To umo\u017e\u0148uje u\u0161etri\u0165 zdroje, preto\u017ee nemus\u00edme prehliada\u0165 \u010dasti, ktor\u00e9 nespadn\u00fa do zadan\u00e9ho intervalu. T.j. zvy\u010dajne, ak vyber\u00e1me \u00fadaje za posledn\u00fa hodinu, pre predch\u00e1dzaj\u00face \u010dasov\u00e9 intervaly prehliadame po\u017eiadavky. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/1afa3a88caa7423941ae3494b9cec706.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje \u010fal\u0161ia mo\u017enos\u0165 rie\u0161enia tohto probl\u00e9mu. Ulo\u017ei\u0165 pre ka\u017ed\u00fd de\u0148 samostatn\u00fd zoznam id \u010dasov\u00fdch radov, ktor\u00e9 sa za tento de\u0148 vyskytli.<\/p>\n<p><\/p>\n<p>V\u00fdhodou tohto rie\u0161enia v porovnan\u00ed s predch\u00e1dzaj\u00facim je, \u017ee neduplikuje inform\u00e1cie o \u010dasov\u00fdch radoch, ktor\u00e9 nezmizn\u00fa s \u010dasom. Neust\u00e1le existuj\u00fa a nemenia sa. <\/p>\n<p><\/p>\n<p>Nev\u00fdhodou je, \u017ee tak\u00e9to rie\u0161enie je zlo\u017eitej\u0161ie na implement\u00e1ciu a \u0165a\u017e\u0161ie na lad\u011bn\u00ed. A VictoriaMetrics si vybrala toto rie\u0161enie. Historicky sa to tak vyvinulo. Toto rie\u0161enie sa tie\u017e ukazuje dobre, v porovnan\u00ed s predch\u00e1dzaj\u00facim. Preto\u017ee toto rie\u0161enie nebolo implementovan\u00e9 kv\u00f4li tomu, \u017ee mus\u00edme duplikova\u0165 \u00fadaje v ka\u017edom partition pre \u010dasov\u00e9 rady, ktor\u00e9 sa nemenia, t.j. ktor\u00e9 nezmizn\u00fa s \u010dasom. VictoriaMetrics bola najprv optimalizovan\u00e1 pre spotrebu diskov\u00e9ho priestoru, a predch\u00e1dzaj\u00faca implement\u00e1cia zhor\u0161ovala spotrebu diskov\u00e9ho priestoru. A toto rie\u0161enie je lep\u0161ie prisp\u00f4soben\u00e9 na minimaliz\u00e1ciu spotreby diskov\u00e9ho priestoru, preto bolo vybran\u00e9. <\/p>\n<p><\/p>\n<p>Museli sme s \u0148ou bojova\u0165. Boj spo\u010d\u00edval v tom, \u017ee v tejto implement\u00e1cii je st\u00e1le potrebn\u00e9 vybera\u0165 ove\u013ea v\u00e4\u010d\u0161ie mno\u017estvo <code>timeseries_ids<\/code> pre \u00fadaje, ako ke\u010f je invertovan\u00fd index rozdelen\u00fd pod\u013ea \u010dasu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/5b10e61aaa803428ed3bfe31815f09d3.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako sme vyrie\u0161ili tento probl\u00e9m? Rie\u0161ili sme ho origin\u00e1lnym sp\u00f4sobom - ulo\u017een\u00edm nieko\u013ek\u00fdch identifik\u00e1torov \u010dasov\u00fdch radov v ka\u017edom z\u00e1pise invertovan\u00e9ho indexu namiesto jedn\u00e9ho identifik\u00e1tora. T. j. m\u00e1me k\u013e\u00fa\u010d <code>label=value<\/code>, ktor\u00fd sa vyskytuje v ka\u017edom \u010dasovom rade. A teraz uchov\u00e1vame nieko\u013eko <code>timeseries_ids<\/code> v jednom z\u00e1pise.<\/p>\n<p><\/p>\n<p>Toto je pr\u00edklad. Predt\u00fdm sme mali N z\u00e1znamov, a teraz m\u00e1me jeden z\u00e1znam, ktor\u00e9ho prefix je rovnak\u00fd ako u v\u0161etk\u00fdch ostatn\u00fdch. Predch\u00e1dzaj\u00faci z\u00e1znam obsahoval v\u0161etky id \u010dasov\u00fdch radov. <\/p>\n<p><\/p>\n<p>To umo\u017enilo zv\u00fd\u0161i\u0165 r\u00fdchlos\u0165 skenovania tak\u00e9ho invertovan\u00e9ho indexu a\u017e desiatkr\u00e1t. A umo\u017enilo zn\u00ed\u017ei\u0165 spotrebu pam\u00e4te pre cache, preto\u017ee teraz uchov\u00e1vame re\u0165azec <code>label=value<\/code> iba raz v cache spolu s N-kr\u00e1t. A tento re\u0165azec m\u00f4\u017ee by\u0165 ve\u013ek\u00fd, ak m\u00e1te v tagoch a labels dlh\u00e9 re\u0165azce, ktor\u00e9 tam r\u00e1d vklad\u00e1 Kubernetes.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/ae4f38a440203bad2d03cf2abd1faafe.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u010eal\u0161\u00ed sp\u00f4sob, ako ur\u00fdchli\u0165 vyh\u013ead\u00e1vanie v invertovanom indexe, je sharding. Vytvorenie nieko\u013ek\u00fdch invertovan\u00fdch indexov namiesto jedn\u00e9ho a shardovanie d\u00e1t medzi nimi pod\u013ea k\u013e\u00fa\u010da. To je s\u00fabor <code>key=value<\/code> p\u00e1rov. T. j. z\u00edskame nieko\u013eko nez\u00e1visl\u00fdch invertovan\u00fdch indexov, ktor\u00e9 m\u00f4\u017eeme paralelne vyh\u013ead\u00e1va\u0165 na viacer\u00fdch procesoroch. Predch\u00e1dzaj\u00face implement\u00e1cie umo\u017enili pracova\u0165 iba v jednoprocesorovom re\u017eime, t. j. skenova\u0165 d\u00e1ta iba na jednom jadre. Toto rie\u0161enie umo\u017e\u0148uje skenova\u0165 d\u00e1ta okam\u017eite na nieko\u013ek\u00fdch jadr\u00e1ch, ako to r\u00e1d rob\u00ed ClickHouse. To pl\u00e1nujeme realizova\u0165.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/5ced6446efbf8ded8d7fb91694a999b0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A teraz sa vr\u00e1\u0165me k na\u0161im baranom - k funkcii prieniku <code>timeseries_ids<\/code>. Pozrime sa, ak\u00e9 m\u00f4\u017eu by\u0165 implement\u00e1cie. T\u00e1to funkcia umo\u017e\u0148uje n\u00e1js\u0165 <code>timeseries_ids<\/code> pre dan\u00fa sadu <code>label=value<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/073a155018a91120c6996c324772f9af.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prv\u00e1 mo\u017enos\u0165 je naivn\u00e1 implement\u00e1cia. Dva vnoren\u00e9 cykly. Takto z\u00edskavame ako vstup pre funkciu <code>intersectInts<\/code> dve slices - <code>a<\/code> a <code>b<\/code>. Na v\u00fdstupe by mal vr\u00e1ti\u0165 prienik t\u00fdchto slices.<\/p>\n<p><\/p>\n<p>Naivn\u00e1 implement\u00e1cia vyzer\u00e1 takto. Prejdeme v\u0161etk\u00fdmi hodnotami zo slice <code>a<\/code>, vn\u00fatri tohto cyklu prejdeme v\u0161etk\u00fdmi hodnotami slice <code>b<\/code>. A porovn\u00e1me ich. Ak sa zhoduj\u00fa, znamen\u00e1 to, \u017ee sme na\u0161li prienik. A ulo\u017e\u00edme ho do <code>result<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/b89dcbb4f6d73377f9cf4f50169d6dd7.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ak\u00e9 s\u00fa nev\u00fdhody? Kv\u00e1dratick\u00e1 zlo\u017eitos\u0165 - to je jej hlavn\u00e1 nev\u00fdhoda. Napr\u00edklad, ak m\u00e1te ve\u013ekosti slice <code>a<\/code> a <code>b<\/code> ak sa vr\u00e1tite k jedn\u00e9mu mili\u00f3nu, t\u00e1to funkcia v\u00e1m nikdy nevr\u00e1ti odpove\u010f. Preto\u017ee bude musie\u0165 vykona\u0165 jeden bili\u00f3n iter\u00e1ci\u00ed, \u010do je ve\u013emi ve\u013ea aj pre modern\u00e9 po\u010d\u00edta\u010de.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/bad26b0092a2fd7fc813bcb79a9138ce.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Druh\u00e1 implement\u00e1cia je zalo\u017een\u00e1 na mape. Vytvor\u00edme mapu. Ulo\u017e\u00edme do tejto mapy v\u0161etky hodnoty zo slice. <code>a<\/code>. Potom prejdeme samostatn\u00fdm cyklom cez slice. <code>b<\/code>. A kontrolujeme, \u010di t\u00e1to hodnota zo slice <code>b<\/code> existuje v mape. Ak \u00e1no, prid\u00e1me ju do v\u00fdsledku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/1e610acf641a9cfe4c80aac1fe26044c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ak\u00e9 s\u00fa v\u00fdhody? V\u00fdhoda je v tom, \u017ee tu je len line\u00e1rna zlo\u017eitosti. T. j. funkcia sa vykon\u00e1 omnoho r\u00fdchlej\u0161ie pre ve\u013ek\u00e9 ve\u013ekosti slices. Pre mili\u00f3nov\u00fa ve\u013ekos\u0165 slice sa t\u00e1to funkcia vykon\u00e1 za 2 mili\u00f3ny iter\u00e1ci\u00ed, na rozdiel od bili\u00f3na iter\u00e1ci\u00ed, ako v predch\u00e1dzaj\u00facej funkcii.<\/p>\n<p><\/p>\n<p>A nev\u00fdhodou je, \u017ee t\u00e1to funkcia vy\u017eaduje viac pam\u00e4te na vytvorenie tejto mapy.<\/p>\n<p><\/p>\n<p>Druh\u00e1 nev\u00fdhoda \u2013 to je ve\u013ek\u00fd overhead pri hashovan\u00ed. T\u00e1to nev\u00fdhoda nie je ve\u013emi zrejm\u00e1. A pre n\u00e1s to tie\u017e nebolo ve\u013emi zrejm\u00e9, preto\u017ee na za\u010diatku v VictoriaMetrics bola implement\u00e1cia intersection cez mapu. Ale potom profilovanie uk\u00e1zalo, \u017ee v\u00e4\u010d\u0161ina procesorov\u00e9ho \u010dasu sa tr\u00e1vi na zapisovan\u00ed do mapy a na kontrole pr\u00edtomnosti hodnoty v tejto mape.<\/p>\n<p><\/p>\n<p>Pre\u010do sa na t\u00fdchto miestach tr\u00e1vi procesorov\u00fd \u010das? Preto\u017ee v t\u00fdchto riadkoch Go vykon\u00e1va oper\u00e1ciu hashovania. T. j. po\u010d\u00edta hash z k\u013e\u00fa\u010da, aby sa mohol potom odkazova\u0165 na dan\u00fd index v HashMap. Oper\u00e1cia v\u00fdpo\u010dtu hashu sa vykon\u00e1va za desatiny nanosek\u00fand. To je pomal\u00e9 pre VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/969f10d875d88998e958894ca28b1181.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rozhodol som sa implementova\u0165 bitset, optimalizovan\u00fd \u0161peci\u00e1lne pre tento pr\u00edpad. Takto teraz vyzer\u00e1 prekr\u00fdvanie dvoch slices. Tu vytvor\u00edme bitset. Prid\u00e1me do neho prvky z prv\u00e9ho slice. Potom kontrolujeme pr\u00edtomnos\u0165 t\u00fdchto prvkov v druhom slice. A prid\u00e1vame ich do v\u00fdsledku. T. j. takmer sa to nel\u00ed\u0161i od predch\u00e1dzaj\u00faceho pr\u00edkladu. Jedin\u00e9, \u010do sme tu nahradili, je pr\u00edstup k mape vlastn\u00fdmi funkciami <code>add<\/code> a <code>has<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/b4ec0753d30d9684868a031f605632c0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na prv\u00fd poh\u013ead sa zd\u00e1, \u017ee to mus\u00ed by\u0165 pomal\u0161ie, ak predt\u00fdm bola pou\u017eit\u00e1 \u0161tandardn\u00e1 mapa a teraz sa volaj\u00fa nejak\u00e9 funkcie, ale profilovanie ukazuje, \u017ee t\u00e1to vec pracuje 10-kr\u00e1t r\u00fdchlej\u0161ie ako \u0161tandardn\u00e1 mapa pre pr\u00edpad s VictoriaMetrics.<\/p>\n<p><\/p>\n<p>Okrem toho vyu\u017e\u00edva ove\u013ea menej pam\u00e4te v porovnan\u00ed s implement\u00e1ciou na mape. Preto\u017ee tu uchov\u00e1vame bity namiesto osembajtov\u00fdch hodn\u00f4t.<\/p>\n<p><\/p>\n<p>Nev\u00fdhodou takejto realiz\u00e1cie je, \u017ee nie je tak\u00e1 zrejm\u00e1, nie trivi\u00e1lna. <\/p>\n<p><\/p>\n<p>\u010eal\u0161ia nev\u00fdhoda, ktor\u00fa si mnoh\u00ed nemusia v\u0161imn\u00fa\u0165 \u2013 je to, \u017ee t\u00e1to realiz\u00e1cia m\u00f4\u017ee zle fungova\u0165 v niektor\u00fdch pr\u00edpadoch. T. j. je optimalizovan\u00e1 pre konkr\u00e9tny pr\u00edpad, pre tento pr\u00edpad prekr\u00fdvania ids \u010dasov\u00fdch radov VictoriaMetrics. To neznamen\u00e1, \u017ee bude vyhovova\u0165 v\u0161etk\u00fdm pr\u00edpadom. Ak sa pou\u017eije nespr\u00e1vne, m\u00f4\u017eeme z\u00edska\u0165 nie n\u00e1rast v\u00fdkonu, ale out of memory error a spomalenie v\u00fdkonu. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/94bd174afe43e67b3cf279dc7a1be17c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Zv\u00e1\u017eme realiz\u00e1ciu tejto \u0161trukt\u00fary. Ak chcete nazrie\u0165, nach\u00e1dza sa v zdrojov\u00fdch k\u00f3doch VictoriaMetrics, v prie\u010dinku <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/tree\/master\/lib\/uint64set\">lib\/uint64set<\/a><\/noindex>. Je optimalizovan\u00e1 presne pre pr\u00edpad VictoriaMetrics, kde <code>timeseries_id<\/code> je 64-bitov\u00e1 hodnota, kde prv\u00fdch 32 bitov je v\u00e4\u010d\u0161inou kon\u0161tantn\u00fdch a menia sa len posledn\u00e9 32 bity.<\/p>\n<p><\/p>\n<p>T\u00e1to d\u00e1tov\u00e1 \u0161trukt\u00fara sa neuklad\u00e1 na disk, pracuje len v pam\u00e4ti. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/e24ca65f860e2b04037e073f7acae0c1.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tu je jej API. Nie je ve\u013emi zlo\u017eit\u00e1. API je prisp\u00f4soben\u00e9 presne pre konkr\u00e9tny pr\u00edklad pou\u017eitia VictoriaMetrics. T. j. tu nie s\u00fa zbyto\u010dn\u00e9 funkcie. Tu s\u00fa funkcie, ktor\u00e9 jasne pou\u017e\u00edva VictoriaMetrics.<\/p>\n<p><\/p>\n<p>S\u00fa tu funkcie <code>add<\/code>, ktor\u00e1 prid\u00e1va nov\u00e9 hodnoty. Je tu funkcia <code>has<\/code>, ktor\u00e1 kontroluje nov\u00e9 hodnoty. A je tu funkcia <code>del<\/code>, ktor\u00e1 odstra\u0148uje hodnoty. Je tu pomocn\u00e1 funkcia <code>len<\/code>, ktor\u00e1 vracia ve\u013ekos\u0165 mno\u017einy. Funkcia <code>klon<\/code> klonuje mno\u017einu. A funkcia <code>appendto<\/code> prev\u00e1dza t\u00fato mno\u017einu na slice <code>timeseries_ids<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/874b04042d13f342d39f8f393e7fb78c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tu je, ako vyzer\u00e1 realiz\u00e1cia tejto d\u00e1tovej \u0161trukt\u00fary. V set je dva prvky:<\/p>\n<p><\/p>\n<ul>\n<li>\n<p><code>ItemsCount<\/code> \u2013 je to pomocn\u00e9 pole, aby r\u00fdchlo vr\u00e1til po\u010det prvkov v set. Mohli by sme sa bez tohto pomocn\u00e9ho po\u013ea ob\u00eds\u0165, ale museli sme ho prida\u0165, preto\u017ee VictoriaMetrics \u010dasto zis\u0165uje v svojich algoritmoch d\u013a\u017eku bitset.<\/p>\n<p>\n<\/li>\n<li>\n<p>Druh\u00e9 pole \u2013 je to <code>buckets<\/code>. To je slice zo \u0161trukt\u00fary <code>bucket32<\/code>. V ka\u017edej \u0161trukt\u00fare sa uchov\u00e1va <code>hi<\/code> pole. To s\u00fa horn\u00e9 32 bity. A dva slice \u2013 <code>b16his<\/code> a <code>buckets<\/code> z <code>bucket16<\/code> \u0161trukt\u00far. <\/p>\n<p>\n<\/li>\n<\/ul>\n<p><\/p>\n<p>Toto uchov\u00e1va horn\u00fdch 16 bitov druhej \u010dasti 64-bitovej \u0161trukt\u00fary. A tu s\u00fa bitsety pre mlad\u0161\u00edch 16 bitov ka\u017ed\u00e9ho bajtu. <\/p>\n<p><\/p>\n<p><code>Bucket64<\/code> sa sklad\u00e1 z po\u013ea <code>uint64<\/code>. D\u013a\u017eka sa vypo\u010d\u00edta pomocou t\u00fdchto kon\u0161t\u00e1nt. V jednom <code>bucket16<\/code> maxim\u00e1lne m\u00f4\u017ee by\u0165 ulo\u017een\u00fdch <code>2^16=65536<\/code> bitov. Ak to vydel\u00edme 8, je to 8 kilobajtov. Ak to vydel\u00edme znova 8, je to 1000 <code>uint64<\/code> hodnotu. T. j. <code>Bucket16<\/code> \u2013 to je pre n\u00e1s 8-kilobajtov\u00e1 \u0161trukt\u00fara. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/1cca4e6bfafa3408a3c95e0118e26d80.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pozrime sa, ako je implementovan\u00e1 jedna z met\u00f3d \u0161trukt\u00fary na pridanie novej hodnoty. <\/p>\n<p><\/p>\n<p>V\u0161etko za\u010d\u00edna s <code>uint64<\/code> hodnotou. Vypo\u010d\u00edtame horn\u00fdch 32 bitov, vypo\u010d\u00edtame doln\u00fdch 32 bitov. Prejdeme v\u0161etk\u00fdmi <code>buckets<\/code>. Porovn\u00e1me horn\u00fdch 32 bitov v ka\u017edom bucket s prid\u00e1vanou hodnotou. A ak sa zhoduj\u00fa, zavol\u00e1me funkciu <code>add<\/code> v \u0161trukt\u00fare b32 <code>buckets<\/code>. A prid\u00e1me tam doln\u00fdch 32 bitov. A ak to vr\u00e1tilo <code>true<\/code>, znamen\u00e1 to, \u017ee sme pridali tak\u00fa hodnotu a tak\u00fa hodnotu sme e\u0161te nemali. Ak vr\u00e1ti <code>nepravda<\/code>, tak\u00e1 hodnota u\u017e existovala. Potom zv\u00fd\u0161ime po\u010det prvkov v \u0161trukt\u00fare. <\/p>\n<p><\/p>\n<p>Ak sme nena\u0161li potrebn\u00fd <code>bucket<\/code> s potrebn\u00fdm hi-hodnotou, potom zavol\u00e1me funkciu <code>addAlloc<\/code>, ktor\u00e1 pridelen\u00ed nov\u00fd <code>bucket<\/code>, prid\u00e1vaj\u00fac ho do bucket-\u0161trukt\u00fary.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/c5d1e5de767f47c7c40e01e12b6d0617.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Toto je implement\u00e1cia funkcie <code>b32.add<\/code>. Je podobn\u00e1 predch\u00e1dzaj\u00facej implement\u00e1cii. Vypo\u010d\u00edtame horn\u00fdch 16 bitov, doln\u00fdch 16 bitov.<\/p>\n<p><\/p>\n<p>Potom prejdeme v\u0161etk\u00fdmi horn\u00fdmi 16 bitmi. N\u00e1jdeme zhody. A pri zhod\u00e1ch zavol\u00e1me met\u00f3du add, ktor\u00fa presk\u00famame na nasleduj\u00facej str\u00e1nke pre <code>bucket16<\/code>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/e98c4ef316f3a894fd7019c3c85696f7.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A tu je najni\u017e\u0161ia \u00farove\u0148, ktor\u00e1 mus\u00ed by\u0165 maxim\u00e1lne optimalizovan\u00e1. Vypo\u010d\u00edtavame pre <code>uint64<\/code> id hodnotu v slice biti, ako aj <code>bitmask<\/code>. Toto je maska pre dan\u00fa 64-bitov\u00fa hodnotu, pod\u013ea ktorej m\u00f4\u017eeme skontrolova\u0165 pr\u00edtomnos\u0165 tohto bitu, alebo ho nastavi\u0165. Kontrolujeme pr\u00edtomnos\u0165 tohto bitu, ak je nastaven\u00fd, nastavujeme ho a vraciame pr\u00edtomnos\u0165. Tak\u00e1to realiz\u00e1cia n\u00e1m umo\u017enila zr\u00fdchli\u0165 oper\u00e1ciu prieniku ids \u010dasov\u00fdch radov 10-n\u00e1sobne v porovnan\u00ed s be\u017en\u00fdmi mapami.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/4dfa5a6142829a3551be22d6c9c3ab92.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Vo VictoriaMetrics okrem tejto optimaliz\u00e1cie existuje mno\u017estvo \u010fal\u0161\u00edch optimaliz\u00e1ci\u00ed. V\u00e4\u010d\u0161ina z t\u00fdchto optimaliz\u00e1ci\u00ed bola pridan\u00e1 nie n\u00e1hodne, ale po profilovan\u00ed k\u00f3du v produkcii.<\/p>\n<p><\/p>\n<p>Toto je hlavn\u00e9 pravidlo optimaliz\u00e1cie \u2013 neprid\u00e1va\u0165 optimaliz\u00e1ciu, predpokladaj\u00fac, \u017ee tu bude \u00fazke miesto, preto\u017ee sa m\u00f4\u017ee uk\u00e1za\u0165, \u017ee tam \u00fazke miesto nie je. Optimaliz\u00e1cia zvy\u010dajne zhor\u0161uje kvalitu k\u00f3du. Preto je dobr\u00e9 optimalizova\u0165 len po profilovan\u00ed a ide\u00e1lne v produkcii, aby to boli re\u00e1lne d\u00e1ta. Kto m\u00e1 z\u00e1ujem, m\u00f4\u017ee sa pozrie\u0165 na zdroje VictoriaMetrics a presk\u00fama\u0165 \u010fal\u0161ie optimaliz\u00e1cie, ktor\u00e9 tam s\u00fa.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Go optimaliz\u00e1cie vo VictoriaMetrics. Alexander Valjalkin\" src=\"\/wp-content\/uploads\/2020\/05\/7585c4dc782627bac5b649e6a55e2ffb.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><em>M\u00e1m ot\u00e1zku o bitset. Ve\u013emi to pripom\u00edna implement\u00e1ciu C++ vector bool, optimalizovan\u00fd bitset. Vzali ste implement\u00e1ciu odtia\u013e?<\/em><\/p>\n<p><\/p>\n<p>Nie, to nie je odtia\u013e. Pri realiz\u00e1cii tohto bitsetu som sa orientoval na znalosti \u0161trukt\u00fary t\u00fdchto ids timeseries, ktor\u00e9 sa pou\u017e\u00edvaj\u00fa vo VictoriaMetrics. Ich \u0161trukt\u00fara je tak\u00e1, \u017ee horn\u00fdch 32 bitov je v\u00e4\u010d\u0161inou kon\u0161tantn\u00fdch. Doln\u00fdch 32 bitov sa m\u00f4\u017ee meni\u0165. \u010c\u00edm ni\u017e\u0161\u00ed bit, t\u00fdm \u010dastej\u0161ie sa m\u00f4\u017ee meni\u0165. Preto je t\u00e1to implement\u00e1cia optimalizovan\u00e1 pr\u00e1ve pre t\u00fato \u0161trukt\u00faru d\u00e1t. Implement\u00e1cia v C++, pokia\u013e viem, je optimalizovan\u00e1 pre v\u0161eobecn\u00fd pr\u00edpad. Ak sa rob\u00ed optimaliz\u00e1cia pre v\u0161eobecn\u00fd pr\u00edpad, znamen\u00e1 to, \u017ee nebude najoptim\u00e1lnej\u0161ia pre konkr\u00e9tny pr\u00edpad.<\/p>\n<p><\/p>\n<p>Odpor\u00fa\u010dam v\u00e1m pozrie\u0165 si tie\u017e predn\u00e1\u0161ku Alexeja Milovida. Pred mesiacom hovoril o optimaliz\u00e1ci\u00e1ch v ClickHouse pre konkr\u00e9tne \u0161pecializ\u00e1cie. Hovor\u00ed, \u017ee v be\u017enom pr\u00edpade je implement\u00e1cia v C++ alebo in\u00e1 implement\u00e1cia zameran\u00e1 na dobr\u00fa pr\u00e1cu v priemere. M\u00f4\u017ee fungova\u0165 hor\u0161ie ako \u0161pecializovan\u00e1 implement\u00e1cia pre konkr\u00e9tne znalosti, ako m\u00e1me my, ke\u010f vieme, \u017ee horn\u00fdch 32 bitov je v\u00e4\u010d\u0161inou kon\u0161tantn\u00fdch.<\/p>\n<p><\/p>\n<p><em>M\u00e1m druh\u00fa ot\u00e1zku. Ak\u00fd je z\u00e1sadn\u00fd rozdiel od InfluxDB?<\/em><\/p>\n<p><\/p>\n<p>Z\u00e1sadn\u00fdch rozdielov je mnoho. Ak hovor\u00edme o v\u00fdkonne a spotrebe pam\u00e4te, InfluxDB v testoch ukazuje desa\u0165n\u00e1sobne vy\u0161\u0161iu spotrebu pam\u00e4te pre high cardinality \u010dasov\u00e9 rady, ke\u010f ich m\u00e1te ve\u013ea, napr\u00edklad mili\u00f3ny. Napr\u00edklad, VictoriaMetrics spotrebuje 1 GB na mili\u00f3n akt\u00edvnych radov, zatia\u013e \u010do InfluxDB pritom spotrebuje 10 Gb. A to je ve\u013ek\u00fd rozdiel. <\/p>\n<p><\/p>\n<p>Druh\u00fd z\u00e1sadn\u00fd rozdiel spo\u010d\u00edva v tom, \u017ee InfluxDB m\u00e1 zvl\u00e1\u0161tne jazyky dopytov \u2013 Flux a InfluxQL. Nie s\u00fa pr\u00edli\u0161 pohodln\u00e9 na pr\u00e1cu s \u010dasov\u00fdmi radmi v porovnan\u00ed s <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/@valyala\/promql-tutorial-for-beginners-9ab455142085\">PromQL<\/a><\/noindex>, ktor\u00fd je podporovan\u00fd vo VictoriaMetrics. PromQL je jazyk dopytov z Prometheus.<\/p>\n<p><\/p>\n<p>A e\u0161te jeden rozdiel \u2013 InfluxDB m\u00e1 trochu zvl\u00e1\u0161tny model d\u00e1t, kde v ka\u017edom riadku m\u00f4\u017ee by\u0165 ulo\u017een\u00fdch nieko\u013eko fields s r\u00f4znymi sadami tagov. Tieto riadky s\u00fa e\u0161te rozdelen\u00e9 na r\u00f4zne tabu\u013eky. Tieto \u010fal\u0161ie zlo\u017eitosti komplikuj\u00fa n\u00e1sledn\u00fa pr\u00e1cu s touto datab\u00e1zou. Je \u0165a\u017ek\u00e9 ju udr\u017eiava\u0165 a ch\u00e1pa\u0165.<\/p>\n<p><\/p>\n<p>Vo VictoriaMetrics je v\u0161etko ove\u013ea jednoduch\u0161ie. Ka\u017ed\u00fd \u010dasov\u00fd rad predstavuje k\u013e\u00fa\u010d-hodnota. Hodnota je s\u00fabor bodov \u2013 <code>(timestamp, value)<\/code>, a k\u013e\u00fa\u010d je s\u00fabor <code>label=value<\/code>. Neexistuje \u017eiadne rozdelenie na fields a measurements. To v\u00e1m umo\u017e\u0148uje vybra\u0165 ak\u00e9ko\u013evek \u00fadaje a n\u00e1sledne ich kombinova\u0165, s\u010d\u00edtava\u0165, od\u010d\u00edta\u0165, n\u00e1sobi\u0165, deli\u0165, na rozdiel od InfluxDB, kde v\u00fdpo\u010dty medzi r\u00f4znymi radmi zatia\u013e neboli implementovan\u00e9, pokia\u013e viem. Ak boli implementovan\u00e9, s\u00fa zlo\u017eit\u00e9 a treba nap\u00edsa\u0165 ve\u013ea k\u00f3du. <\/p>\n<p><\/p>\n<p><em>M\u00e1m dopl\u0148uj\u00facu ot\u00e1zku. Spr\u00e1vne som pochopil, \u017ee bola nejak\u00e1 probl\u00e9m, o ktorom ste hovorili, \u017ee tento inverzn\u00fd index sa nezmest\u00ed do pam\u00e4te, preto je tam partitioning?<\/em><\/p>\n<p><\/p>\n<p>Na za\u010diatku som uk\u00e1zal naivn\u00fa implement\u00e1ciu invertovan\u00e9ho indexu na \u0161tandardnej map\u011b v Go. Tak\u00e1to implement\u00e1cia nie je vhodn\u00e1 pre datab\u00e1zy, preto\u017ee tento invertovan\u00fd index sa neuklad\u00e1 na disk a datab\u00e1za by mala uklada\u0165 na disk, aby ste mali tieto \u00fadaje dostupn\u00e9 po re\u0161tarte. V tejto implement\u00e1cii v\u00e1m invertovan\u00fd index po re\u0161tarte aplik\u00e1cie zmizne. A strat\u00edte pr\u00edstup ku v\u0161etk\u00fdm \u00fadajom, preto\u017ee ich nebudete m\u00f4c\u0165 n\u00e1js\u0165. <\/p>\n<p><\/p>\n<p><em>Dobr\u00fd de\u0148! \u010eakujem za predn\u00e1\u0161ku! Vol\u00e1m sa Pavel. Som zo spolo\u010dnosti Wildberries. M\u00e1m na v\u00e1s nieko\u013eko ot\u00e1zok. Prv\u00e1 ot\u00e1zka. Mysl\u00edte si, \u017ee keby ste si vybrali in\u00fd princ\u00edp pri budovan\u00ed architekt\u00fary va\u0161ej aplik\u00e1cie a partitionovali \u00fadaje pod\u013ea \u010dasu, tak by ste mohli robi\u0165 priese\u010dn\u00e9 vyh\u013ead\u00e1vanie \u00fadajov, zalo\u017een\u00e9 len na tom, \u017ee v jednej partition s\u00fa \u00fadaje za jeden \u010dasov\u00fd \u00fasek, t.j. za jeden \u010dasov\u00fd interval a nemuseli by ste sa ob\u00e1va\u0165, \u017ee m\u00e1te k\u00fasky roztr\u00fasen\u00e9 r\u00f4zne? Druh\u00e1 ot\u00e1zka \u2014 ke\u010f\u017ee implementujete tak\u00fdto algoritmus s bitsetom a v\u0161etk\u00fdm ostatn\u00fdm, mo\u017eno ste sa sna\u017eili pou\u017ei\u0165 in\u0161trukcie procesora? Mo\u017eno ste sk\u00fa\u0161ali tak\u00e9 optimaliz\u00e1cie?<\/em><\/p>\n<p><\/p>\n<p>Na druh\u00fa rovno odpoviem. E\u0161te sme sa k tomu nedostali. Ale ak to bude potrebn\u00e9, dostaneme sa. A prv\u00e1 ot\u00e1zka, ak\u00e1 bola?<\/p>\n<p><\/p>\n<p><em>Diskutovali ste o dvoch scen\u00e1roch. A povedali ste, \u017ee ste si vybrali druh\u00fd s komplikovanej\u0161ou implement\u00e1ciou. A neuprednostnili ste prv\u00fd, kde s\u00fa \u00fadaje partitionovan\u00e9 pod\u013ea \u010dasu.<\/em> <\/p>\n<p><\/p>\n<p>\u00c1no. V prvom pr\u00edpade by bol celkov\u00fd objem indexu v\u00e4\u010d\u0161\u00ed, preto\u017ee v ka\u017edej partition by sme museli uchov\u00e1va\u0165 duplicity \u00fadajov pre tie \u010dasov\u00e9 rady, ktor\u00e9 prebiehaj\u00fa cez v\u0161etky tieto partition. A ak m\u00e1te n\u00edzku churn rate \u010dasov\u00fdch radov, t. j. neust\u00e1le sa pou\u017e\u00edvaj\u00fa rovnak\u00e9 rady, v prvom pr\u00edpade by sme ove\u013ea viac stratili na obsaden\u00ed diskov\u00e9ho priestoru v porovnan\u00ed s druh\u00fdm pr\u00edpadom.<\/p>\n<p><\/p>\n<p>Tak\u017ee \u2013 \u00e1no, \u010dasov\u00e9 partitionovanie je dobr\u00e1 mo\u017enos\u0165. Pou\u017e\u00edva to aj Prometheus. Ale Prometheus m\u00e1 in\u00fa nev\u00fdhodu. Pri zl\u00fa\u010den\u00ed t\u00fdchto kusov \u00fadajov potrebuj\u00fa uchov\u00e1va\u0165 v pam\u00e4ti meta-inform\u00e1cie o v\u0161etk\u00fdch labeloch a \u010dasov\u00fdch radoch. Preto, ak s\u00fa kusy \u00fadajov ve\u013ek\u00e9, ktor\u00e9 zl\u00fa\u010di, pam\u00e4\u0165ov\u00e1 spotreba pri zl\u00fa\u010den\u00ed ve\u013emi rastie, na rozdiel od VictoriaMetrics. Pri zl\u00fa\u010den\u00ed VictoriaMetrics v\u00f4bec nepou\u017e\u00edva pam\u00e4\u0165, tam sa spotrebov\u00e1va len nieko\u013eko kilobajtov, nez\u00e1visle od ve\u013ekost\u00ed zl\u00fa\u010den\u00fdch kusov \u00fadajov.<\/p>\n<p><\/p>\n<p><em>Algoritmus, ktor\u00fd m\u00e1te, vyu\u017e\u00edva pam\u00e4\u0165. V nej sa ozna\u010duj\u00fa timeseries-\u0161t\u00edtky, na ktor\u00fdch s\u00fa hodnoty. A takto kontrolujete pr\u00edtomnos\u0165 v jednom d\u00e1tovom poli a v druhom. A ch\u00e1pete \u2013 do\u0161lo k prieniku alebo nie. Zvy\u010dajne v datab\u00e1zach implementuj\u00fa kurzory, iter\u00e1tory, ktor\u00e9 uchov\u00e1vaj\u00fa svoj aktu\u00e1lny stav a prech\u00e1dzaj\u00fa trieden\u00fdmi \u00fadajmi, \u010d\u00edm m\u00e1te jednoduch\u00fa zlo\u017eitost' tak\u00fdchto oper\u00e1ci\u00ed.<\/em> <\/p>\n<p><\/p>\n<p>Pre\u010do nepou\u017e\u00edvame kurzory na prienik \u00fadajov?<\/p>\n<p><\/p>\n<p><em>\u00c1no.<\/em> <\/p>\n<p><\/p>\n<p>M\u00e1me v LevelDB alebo v mergeset pr\u00e1ve usporiadan\u00e9 riadky. M\u00f4\u017eeme \u00eds\u0165 kurzorom a n\u00e1js\u0165 prienik. A pre\u010do ich nepou\u017e\u00edvame? Preto\u017ee \u2013 je to pomal\u00e9. Preto\u017ee kurzory predpokladaj\u00fa, \u017ee na ka\u017ed\u00fd riadok treba zavola\u0165 funkciu. Zavolanie funkcie trv\u00e1 5 nanosek\u00fand. A ak m\u00e1te 100 000 000 riadkov, tak str\u00e1came pol sekundy len na volanie funkcie.<\/p>\n<p><\/p>\n<p><em>Tak\u00e9to nie\u010do existuje, \u00e1no. A moja posledn\u00e1 ot\u00e1zka. Ot\u00e1zka, mo\u017eno, znie trochu \u010dudne. Pre\u010do v momente prich\u00e1dzania \u00fadajov nie je mo\u017en\u00e9 po\u010d\u00edta\u0165 v\u0161etky potrebn\u00e9 agreg\u00e1ty a uchov\u00e1va\u0165 ich v po\u017eadovanej forme? Pre\u010do uchov\u00e1va\u0165 obrovsk\u00e9 objemy v nejak\u00fdch syst\u00e9moch ako VictoriaMetrics, ClickHouse a podobne, aby sme potom m\u00ed\u0148ali ve\u013ea \u010dasu na ne?<\/em><\/p>\n<p><\/p>\n<p><em>Pr\u00edklad, ktor\u00fd objasn\u00ed veci. Povedzme, ako funguje mal\u00fd hra\u010dk\u00e1rsky r\u00fdchlomer? Zaznamen\u00e1va vzdialenos\u0165, ktor\u00fa ste pre\u0161li, neust\u00e1le ju s\u010d\u00edta s jednou hodnotou, druhou - \u010dasom. A del\u00ed. A dost\u00e1va priemern\u00fa r\u00fdchlos\u0165. Robi\u0165 sa to d\u00e1 podobne. S\u010d\u00edta\u0165 na cest\u00e1ch v\u0161etky potrebn\u00e9 fakty.<\/em><\/p>\n<p><\/p>\n<p>Dobre, rozumiem ot\u00e1zke. V\u00e1\u0161 pr\u00edklad m\u00e1 svoje opodstatnenie. Ak viete, ak\u00e9 agreg\u00e1ty potrebujete, je to najlep\u0161ia realiz\u00e1cia. Probl\u00e9m je v tom, \u017ee \u013eudia ukladaj\u00fa tieto metriky, nejak\u00e9 d\u00e1ta do ClickHouse a nevedia, ako ich v bud\u00facnosti bud\u00fa agregova\u0165, filtrova\u0165, preto musia uchov\u00e1va\u0165 v\u0161etky surov\u00e9 d\u00e1ta. Ale ak viete, \u017ee potrebujete vypo\u010d\u00edta\u0165 nie\u010do priemern\u00e9, pre\u010do to nevypo\u010d\u00edta\u0165, namiesto toho, aby ste tam ukladali hromadu surov\u00fdch hodn\u00f4t? Ale len v pr\u00edpade, \u017ee presne viete, \u010do potrebujete.<\/p>\n<p><\/p>\n<p>Mimochodom, datab\u00e1zy na uchov\u00e1vanie \u010dasov\u00fdch radov podporuj\u00fa po\u010d\u00edtanie agreg\u00e1tov. Napr\u00edklad, Prometheus podporuje <noindex><a rel=\"nofollow\" href=\"https:\/\/prometheus.io\/docs\/prometheus\/latest\/configuration\/recording_rules\/\">registra\u010dn\u00e9 pravidl\u00e1<\/a><\/noindex>. T. j. to sa d\u00e1 urobi\u0165, ak viete, ak\u00e9 agreg\u00e1ty budete potrebova\u0165. V VictoriaMetrics to zatia\u013e nie je, ale pred \u0148ou sa zvy\u010dajne in\u0161taluje Prometheus, v ktorom to m\u00f4\u017eete urobi\u0165 v registra\u010dn\u00fdch pravidl\u00e1ch.<\/p>\n<p><\/p>\n<p>Napr\u00edklad, na predch\u00e1dzaj\u00facej pr\u00e1ci sme potrebovali po\u010d\u00edta\u0165 po\u010det udalost\u00ed v pos\u00favacom okne za posledn\u00fa hodinu. Probl\u00e9m bol v tom, \u017ee sme museli vytvori\u0165 vlastn\u00fa realiz\u00e1ciu v Go, t. j. slu\u017ebu na po\u010d\u00edtanie tejto veci. T\u00e1to slu\u017eba bola nakoniec zlo\u017eit\u00e1, preto\u017ee je \u0165a\u017ek\u00e9 to po\u010d\u00edta\u0165. Realiz\u00e1cia m\u00f4\u017ee by\u0165 jednoduch\u00e1, ak potrebujete po\u010d\u00edta\u0165 nejak\u00e9 agreg\u00e1ty na pevn\u00fdch \u010dasov\u00fdch intervaloch. Ak chcete po\u010d\u00edta\u0165 udalosti v pos\u00favacom okne, nie je to tak\u00e9 jednoduch\u00e9, ako sa zd\u00e1. Mysl\u00edm, \u017ee to st\u00e1le nie je implementovan\u00e9 v ClickHouse alebo v baz\u00e1ch \u010dasov\u00fdch radov, preto\u017ee je to zlo\u017eit\u00e1 realiz\u00e1cia.<\/p>\n<p><\/p>\n<p><em>A e\u0161te jedna ot\u00e1zka. Teraz sme hovorili o priemern\u00ed a spomenul som si, \u017ee kedysi existoval tak\u00fd syst\u00e9m ako Graphite s backendom Carbon. A ten dok\u00e1zal star\u00e9 d\u00e1ta zriedi\u0165, t. j. necha\u0165 jeden bod za min\u00fatu, jeden bod za hodinu at\u010f. V podstate je to dos\u0165 praktick\u00e9, ak potrebujeme surov\u00e9 d\u00e1ta, povedzme za mesiac, a v\u0161etko ostatn\u00e9 m\u00f4\u017eeme riedi\u0165. Ale Prometheus a VictoriaMetrics t\u00fato funk\u010dnos\u0165 nepodporuj\u00fa. Je pl\u00e1novan\u00e9 jej zavedenie? Ak nie, pre\u010do?<\/em><\/p>\n<p><\/p>\n<p>\u010eakujeme za ot\u00e1zku. Na\u0161i pou\u017e\u00edvatelia sa \u010dasto p\u00fdtaj\u00fa, kedy prid\u00e1me podporu downsamplingu. Existuje nieko\u013eko probl\u00e9mov. Po prv\u00e9, ka\u017ed\u00fd pou\u017e\u00edvate\u013e m\u00e1 na mysli <code>downsampling<\/code> nie\u010do in\u00e9: niekto chce z\u00edska\u0165 ak\u00fdko\u013evek bod v danom intervale, niekto chce maxim\u00e1lne, minim\u00e1lne alebo priemern\u00e9 hodnoty. Ak do va\u0161ej datab\u00e1zy ukladaj\u00fa \u00fadaje r\u00f4zne syst\u00e9my, nemo\u017eno ich spracova\u0165 rovnak\u00fdm sp\u00f4sobom. M\u00f4\u017ee sa sta\u0165, \u017ee pre ka\u017ed\u00fd syst\u00e9m bude potrebn\u00e9 pou\u017ei\u0165 odli\u0161n\u00fd downsampling. A to je zlo\u017eit\u00e1 realiz\u00e1cia.<\/p>\n<p><\/p>\n<p>A druh\u00e1 vec \u2013 VictoriaMetrics, rovnako ako ClickHouse, je optimalizovan\u00e1 na pr\u00e1cu s ve\u013ek\u00fdmi objemami surov\u00fdch d\u00e1t, tak\u017ee dok\u00e1\u017ee spracova\u0165 miliardu riadkov za menej ako sekundu, ak m\u00e1te vo svojom syst\u00e9me ve\u013ea jadier. Skenovanie \u010dasov\u00fdch bodov vo VictoriaMetrics \u2013 50 000 000 bodov za sekundu na jedno jadro. A t\u00e1to v\u00fdkonnos\u0165 sa \u0161k\u00e1luje na dostupn\u00e9 jadr\u00e1. T. j. ak m\u00e1te 20 jadier, napr\u00edklad, skenovanie miliardy bodov za sekundu je mo\u017en\u00e9. A t\u00e1to vlastnos\u0165 VictoriaMetrics a ClickHouse zni\u017euje potrebu downsamplingu.<\/p>\n<p><\/p>\n<p>\u010eal\u0161ou vlastnos\u0165ou je, \u017ee VictoriaMetrics efekt\u00edvne komprimuje tieto d\u00e1ta. Priemerne dosiahne kompresiu od 0,4 do 0,8 bajtov na bod. Ka\u017ed\u00fd bod pozost\u00e1va z \u010dasovej pe\u010diatky + hodnoty. A je komprimovan\u00fd na menej ako jeden bajt priemerne. <\/p>\n<p><\/p>\n<p><em>Sergej. M\u00e1m ot\u00e1zku. Ak\u00fd je minim\u00e1lny \u010dasov\u00fd kvant z\u00e1pisu?<\/em><\/p>\n<p><\/p>\n<p>Jedna milisekunda. Ned\u00e1vno sme mali rozhovor s in\u00fdmi v\u00fdvoj\u00e1rmi datab\u00e1z pre \u010dasov\u00e9 rady. Ich minim\u00e1lny \u010dasov\u00fd kvant je jedna sekunda. V Graphite, napr\u00edklad, tie\u017e jedna sekunda. V OpenTSDB tie\u017e jedna sekunda. V InfluxDB je presnos\u0165 na nanosekundu. Vo VictoriaMetrics je to jedna milisekunda, preto\u017ee v Prometheus je jedna milisekunda. A VictoriaMetrics bola vyvinut\u00e1 ako extern\u00e9 \u00falo\u017eisko pre Prometheus p\u00f4vodne. Ale teraz m\u00f4\u017ee uklada\u0165 d\u00e1ta aj z in\u00fdch syst\u00e9mov. <\/p>\n<p><\/p>\n<p>\u010clovek, s ktor\u00fdm som hovoril, povedal, \u017ee ich presnos\u0165 je sekundov\u00e1 \u2014 to im sta\u010d\u00ed, preto\u017ee to z\u00e1vis\u00ed od typu \u00fadajov, ktor\u00e9 sa ukladaj\u00fa do datab\u00e1zy \u010dasov\u00fdch r\u00e1d. Ak ide o DevOps \u00fadaje alebo \u00fadaje z infra\u0161trukt\u00fary, kde ich zbierate s intervalom 30 sek\u00fand, za min\u00fatu, tak tam je sekundov\u00e1 presnos\u0165 dostato\u010dn\u00e1, menej u\u017e netreba. Ak v\u0161ak zbierate tieto \u00fadaje z high frequency trading systems, tam je potrebn\u00e1 presnos\u0165 na nanosekundu.<\/p>\n<p><\/p>\n<p>Milisekundov\u00e1 presnos\u0165 vo VictoriaMetrics je vhodn\u00e1 nielen pre DevOps pr\u00edpady, ale m\u00f4\u017ee by\u0165 vhodn\u00e1 aj pre v\u00e4\u010d\u0161inu pr\u00edpadov, ktor\u00e9 som spomenul na za\u010diatku prezent\u00e1cie. Jedin\u00e1 vec, pre ktor\u00fa nemus\u00ed by\u0165 vhodn\u00e1, s\u00fa syst\u00e9my na obchodovanie s vysokou frekvenciou.<\/p>\n<p><\/p>\n<p><em>\u010eakujem! A e\u0161te jedna ot\u00e1zka. Ak\u00e1 je kompatibilita s PromQL?<\/em><\/p>\n<p><\/p>\n<p>\u00dapln\u00e1 sp\u00e4tn\u00e1 kompatibilita. VictoriaMetrics plne podporuje PromQL. Okrem toho prid\u00e1va aj \u010fal\u0161iu roz\u0161\u00edren\u00fa funk\u010dnos\u0165 do PromQL, ktor\u00e1 sa naz\u00fdva <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/VictoriaMetrics\/VictoriaMetrics\/wiki\/MetricsQL\">MetricsQL<\/a><\/noindex>. \u010co sa t\u00fdka tejto roz\u0161\u00edrenej funk\u010dnosti, existuje prezent\u00e1cia na YouTube. O tom som rozpr\u00e1val na Monitoring Meetup na jar v Petrohrade.<\/p>\n<p><\/p>\n<p>Telegram kan\u00e1l <noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/VictoriaMetrics_ru1\">VictoriaMetrics<\/a><\/noindex>.<\/p>\n<p class=\"for_users_only_msg\">Len registrovan\u00ed pou\u017e\u00edvatelia sa m\u00f4\u017eu z\u00fa\u010dastni\u0165 prieskumu. <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/auth\/login\/\">Prihl\u00e1ste sa<\/a><\/noindex>, pros\u00edm.<\/p>\n<h2 class=\"default-block__polling-title\">\u010co v\u00e1m br\u00e1ni prejs\u0165 na VictoriaMetrics ako dlhodob\u00e9 \u00falo\u017eisko pre Prometheus? (Nap\u00ed\u0161te do koment\u00e1rov, prid\u00e1m to do prieskumu))<\/h2>\n<ul class=\"poll-result\">\n<li class=\"poll-result__item\">\n<p>                <strong class=\"poll-result__data-percent  poll-result__data-percent_winner\">71,4%<\/strong>Nepou\u017e\u00edvam Prometheus5<\/p>\n<\/li>\n<li class=\"poll-result__item\">\n<p>                <strong class=\"poll-result__data-percent\">28,6%<\/strong>Nevedel som o VictoriaMetrics2<\/p>\n<\/li>\n<\/ul>\n<p>    Hlasovalo 7 pou\u017e\u00edvate\u013eov. Zdr\u017ealo sa 12 pou\u017e\u00edvate\u013eov.<br \/>\n<br \/>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/500844\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043a\u043e\u043d\u0446\u0430 2019 \u0433\u043e\u0434\u0430 \u0410\u043b\u0435\u043a\u0441\u0430\u043d\u0434\u0440\u0430 \u0412\u0430\u043b\u044f\u043b\u043a\u0438\u043d\u0430 &quot;Go optimizations in VictoriaMetrics&quot; VictoriaMetrics \u2014 \u0431\u044b\u0441\u0442\u0440\u0430\u044f \u0438 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u0430\u044f \u0421\u0423\u0411\u0414 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0434\u0430\u043d\u043d\u044b\u0445 \u0432 \u0444\u043e\u0440\u043c\u0435 \u0432\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0433\u043e \u0440\u044f\u0434\u0430 (\u0437\u0430\u043f\u0438\u0441\u044c \u043e\u0431\u0440\u0430\u0437\u0443\u0435\u0442 \u0432\u0440\u0435\u043c\u044f \u0438 \u043d\u0430\u0431\u043e\u0440 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u044d\u0442\u043e\u043c\u0443 \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0439, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0445 \u0447\u0435\u0440\u0435\u0437 \u043f\u0435\u0440\u0438\u043e\u0434\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u043e\u043f\u0440\u043e\u0441 \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u044f \u0434\u0430\u0442\u0447\u0438\u043a\u043e\u0432 \u0438\u043b\u0438 \u0441\u0431\u043e\u0440 \u043c\u0435\u0442\u0440\u0438\u043a). \u0412\u043e\u0442 \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u0432\u0438\u0434\u0435\u043e \u044d\u0442\u043e\u0433\u043e \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u2014 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":80734,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-80733","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043a\u043e\u043d\u0446\u0430 2019 \u0433\u043e\u0434\u0430 \u0410\u043b\u0435\u043a\u0441\u0430\u043d\u0434\u0440\u0430 \u0412\u0430\u043b\u044f\u043b\u043a\u0438\u043d\u0430 &quot;Go optimizations in VictoriaMetrics&quot;\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/go-optimizations-in-victoriametrics-aleksandr-valyalkin\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47Go optimizations in VictoriaMetrics. \u0410\u043b\u0435\u043a\u0441\u0430\u043d\u0434\u0440 \u0412\u0430\u043b\u044f\u043b\u043a\u0438\u043d | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043a\u043e\u043d\u0446\u0430 2019 \u0433\u043e\u0434\u0430 \u0410\u043b\u0435\u043a\u0441\u0430\u043d\u0434\u0440\u0430 \u0412\u0430\u043b\u044f\u043b\u043a\u0438\u043d\u0430 &quot;Go optimizations in VictoriaMetrics&quot;\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/go-optimizations-in-victoriametrics-aleksandr-valyalkin\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-05-08T11:42:31+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-05-08T11:42:31+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Optimaliz\u00e1cie Go vo VictoriaMetrics. Alexander Valjalkin | ProHoster","description":"Odpor\u00fa\u010dam v\u00e1m obozn\u00e1mi\u0165 sa s preh\u013eadom spr\u00e1vy z konca roka 2019 od Alexandra Valjalkina \"Optimaliz\u00e1cie Go vo VictoriaMetrics\"","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/go-optimizations-in-victoriametrics-aleksandr-valyalkin","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47Go optimizations in VictoriaMetrics. \u0410\u043b\u0435\u043a\u0441\u0430\u043d\u0434\u0440 \u0412\u0430\u043b\u044f\u043b\u043a\u0438\u043d | ProHoster","og:description":"\u041f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u044e \u043e\u0437\u043d\u0430\u043a\u043e\u043c\u0438\u0442\u044c\u0441\u044f \u0441 \u0440\u0430\u0441\u0448\u0438\u0444\u0440\u043e\u0432\u043a\u043e\u0439 \u0434\u043e\u043a\u043b\u0430\u0434\u0430 \u043a\u043e\u043d\u0446\u0430 2019 \u0433\u043e\u0434\u0430 \u0410\u043b\u0435\u043a\u0441\u0430\u043d\u0434\u0440\u0430 \u0412\u0430\u043b\u044f\u043b\u043a\u0438\u043d\u0430 &quot;Go optimizations in VictoriaMetrics&quot;","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/go-optimizations-in-victoriametrics-aleksandr-valyalkin","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-05-08T11:42:31+00:00","article:modified_time":"2020-05-08T11:42:31+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"80733","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 16:12:22","updated":"2022-09-29 05:16:51","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/80733","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=80733"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/80733\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/80734"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=80733"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=80733"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=80733"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}