{"id":94153,"date":"2020-09-13T19:42:30","date_gmt":"2020-09-13T17:42:30","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev"},"modified":"2020-09-13T19:42:30","modified_gmt":"2020-09-13T17:42:30","slug":"victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","title":{"rendered":"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/d84a6783fbf35e9d96afd263b5cd4578.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics \u2014 r\u00fdchla a \u0161k\u00e1lovate\u013en\u00e1 DBMS na ukladanie a spracovanie d\u00e1t vo forme \u010dasovej rady (z\u00e1znam vytv\u00e1ra \u010das a s\u00fabor hodn\u00f4t s\u00favisiacich s t\u00fdmto \u010dasom, napr\u00edklad z\u00edskan\u00fdch prostredn\u00edctvom periodick\u00e9ho prieskumu stavu senzorov alebo zberu metr\u00edk).<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\n<center><div class=\"youtube-placeholder\" data-id=\"74swsWqf0Uc\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/74swsWqf0Uc\/hqdefault.jpg\" alt=\"Prehrajte video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/df849870a0432a863e34eb11387c9fab.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Vol\u00e1m sa Kolobaev Pavel. DevOps, SRE, LeroyMerlin, v\u0161etko ako k\u00f3d \u2013 to v\u0161etko sa t\u00fdka n\u00e1s: m\u0148a a ostatn\u00fdch zamestnancov LeroyMerlin.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/b7b63999cef8d80148bf666c2bce335a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/bit.ly\/3jf1fIK\">https:\/\/bit.ly\/3jf1fIK<\/a><\/noindex><\/p>\n<p><\/p>\n<p>Existuje cloud postaven\u00fd na OpenStacku. Tam je mal\u00fd odkaz na technick\u00fd radar. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/e99d4aeab2a62bc46cfa9aaadb9590af.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Syst\u00e9m je postaven\u00fd na hardv\u00e9ri Kubernetes, ako aj na v\u0161etk\u00fdch sprievodn\u00fdch slu\u017eb\u00e1ch OpenStacku a logovania. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/c19c24b9006162eaf79f8f9bdc5aa14a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na\u0161a sch\u00e9ma bola tak\u00e1to po\u010das v\u00fdvoja. Ke\u010f sme to v\u0161etko vyv\u00edjali, mali sme oper\u00e1tora Prometheus, ktor\u00fd uchov\u00e1val d\u00e1ta priamo v klasteri K8s. Automaticky nach\u00e1dzal potrebn\u00e9 \u00fadaje na skenovanie a ukladal ich pod nohy, zjednodu\u0161ene povedan\u00e9. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/ebaf6b3d32483c405a1ae69d0ea7e3ea.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u0161etky d\u00e1ta mus\u00edme prenies\u0165 mimo klaster Kubernetes, preto\u017ee ak sa nie\u010do stane, mus\u00edme rozumie\u0165, \u010do a kde. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/36f921aa715ed00b9ac26ae3861a73b8.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prv\u00e9 rie\u0161enie \u2013 pou\u017e\u00edvame feder\u00e1ciu, ke\u010f m\u00e1me extern\u00fd Prometheus, a ke\u010f pristupujeme ku klastrom Kubernetes pomocou mechanizmu feder\u00e1cie.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/11033de585d88cfaee1edeec908da55a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale tu vznikaj\u00fa mal\u00e9 probl\u00e9my. V na\u0161om pr\u00edpade probl\u00e9my za\u010dali, ke\u010f sme mali 250 000 metr\u00edk, a ke\u010f ich bolo 400 000, uvedomili sme si, \u017ee takto nem\u00f4\u017eeme pracova\u0165. Zv\u00fd\u0161ili sme scrape_timeout na 25 sek\u00fand.<\/p>\n<p><\/p>\n<p>Pre\u010do sme to museli urobi\u0165? Prometheus za\u010d\u00edna odpo\u010d\u00edtava\u0165 \u010das \u010dasov\u00e9ho limitu od za\u010diatku momentu prieskumu. A nez\u00e1le\u017e\u00ed na tom, \u017ee d\u00e1ta e\u0161te te\u010d\u00fa. Ak sa v ur\u010denom \u010dasovom obdob\u00ed nedaj\u00fa skompilova\u0165 d\u00e1ta a rel\u00e1cia nebude uzavret\u00e1 cez http, pova\u017euje sa rel\u00e1cia za ne\u00faspe\u0161n\u00fa a d\u00e1ta sa nedostan\u00fa do samotn\u00e9ho Prometheus.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/c5d8f034cbf80638775ecc68b6ad8f21.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u0161etci poznaj\u00fa grafy, ktor\u00e9 z\u00edskavame, ke\u010f \u010das\u0165 d\u00e1t ch\u00fdba. Grafy s\u00fa z\u00fabkat\u00e9 a to n\u00e1m nevyhovuje. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/8caa1939ac95d26163e98ed2bd12fd6a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u010eal\u0161ou mo\u017enos\u0165ou je shardovanie na z\u00e1kladne dvoch r\u00f4znych Prometheus cez ten ist\u00fd mechanizmus feder\u00e1cie.<\/p>\n<p><\/p>\n<p>Napr\u00edklad jednoducho vzia\u0165 a shardova\u0165 ich pod\u013ea mena. To sa d\u00e1 tie\u017e pou\u017ei\u0165, ale rozhodli sme sa posun\u00fa\u0165 \u010falej. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/c5715365ac494689194ad4d305a7be08.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Budeme musie\u0165 teraz tieto shard-y nejako spracova\u0165. M\u00f4\u017eeme vzia\u0165 promxy, ktor\u00fd sa pozrie do oblasti shardu, a multiplexuje d\u00e1ta. Funguje s dvoma shardmi ako s jedn\u00fdm vstupn\u00fdm bodom. To sa d\u00e1 implementova\u0165 cez promxy, ale je to zatia\u013e pr\u00edli\u0161 zlo\u017eit\u00e9. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/672984d1ca13743acb4b8102b74d0eb2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prv\u00e1 mo\u017enos\u0165 \u2013 chceme sa vzda\u0165 mechanzmu federation, preto\u017ee je ve\u013emi pomal\u00fd. <\/p>\n<p><\/p>\n<p>V\u00fdvoj\u00e1ri Prometheus jasne hovoria: \u201eChlapi, pou\u017e\u00edvajte in\u00e9 TimescaleDB, preto\u017ee nebudeme dlho podporova\u0165 uchov\u00e1vanie metr\u00edk\u201c. To nie je ich \u00faloha. <img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/dfc59949ec16fe731d339d7bf477edff.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na papier si zapisujeme, \u017ee naozaj potrebujeme export von, aby sme nemuseli v\u0161etko uchov\u00e1va\u0165 na jednom mieste. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/a0e566b89c15df9c17c0750ce8512044.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Druh\u00fd nedostatok \u2013 je to spotreba pam\u00e4te. \u00c1no, ch\u00e1pem, \u017ee mnoh\u00ed povedia, \u017ee v roku 2020 p\u00e1r gigabajtov pam\u00e4te stoj\u00ed len p\u00e1r centov, ale napriek tomu. <\/p>\n<p><\/p>\n<p>Teraz m\u00e1me dev a prod prostredie. V dev to je okolo 9 gigabajtov na 350 000 metr\u00edk. V prod to je nie\u010do vy\u0161e 14 gigabajtov na 780 000 metr\u00edk. Pri tom je retention time len 30 min\u00fat. To je zl\u00e9. A teraz vysvetl\u00edm pre\u010do.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/35ad6d2792fd4dbf68b15851f24eb047.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rob\u00edme v\u00fdpo\u010det, t. j. pri mili\u00f3n a pol metr\u00edk, a u\u017e sme bl\u00edzko tomuto \u010d\u00edslu, na etape n\u00e1vrhu dost\u00e1vame 35-37 gigabajtov pam\u00e4te. Ale u\u017e pri 4 mili\u00f3noch metr\u00edk je potrebn\u00e9 okolo 90 gigabajtov pam\u00e4te. T. j. to bolo vypo\u010d\u00edtan\u00e9 pod\u013ea vzorca, ktor\u00fd poskytli v\u00fdvoj\u00e1ri Prometheus. Pozreli sme sa na korel\u00e1ciu a pochopili sme, \u017ee nechceme plati\u0165 p\u00e1r mili\u00f3nov za server len na monitorovanie. <\/p>\n<p><\/p>\n<p>Na\u0161a potreba strojov sa nebude len zvy\u0161ova\u0165, ale aj monitorujeme samotn\u00e9 virtu\u00e1lne stroje. Preto \u010d\u00edm viac virtu\u00e1lnych strojov, t\u00fdm viac r\u00f4znych metr\u00edk at\u010f. Bude to \u0161peci\u00e1lny rast n\u00e1\u0161ho klastra z h\u013eadiska metr\u00edk.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/d0aa8f00c7bd511b128bfa38cd280d03.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>S diskov\u00fdm priestorom to nie je a\u017e tak zl\u00e9, ale radi by sme to zlep\u0161ili. Za 15 dn\u00ed sme z\u00edskali celkom 120 gigabajtov, z ktor\u00fdch 100 je komprimovan\u00fdch d\u00e1t, 20 s\u00fa nekomprimovan\u00e9 d\u00e1ta, ale v\u017edy chceme menej. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/04dcc202029ce2433ae5cfb379c51181.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pod\u013ea toho zapisujeme \u010fal\u0161\u00ed bod \u2013 je to ve\u013ek\u00e1 spotreba zdrojov, ktor\u00e9 chceme \u0161etri\u0165, preto\u017ee nechceme, aby n\u00e1\u0161 klaster monitorovania zjedol viac zdrojov ne\u017e n\u00e1\u0161 klaster, ktor\u00fd spravuje OpenStack.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/1708dcd413d890f18149414616f6264e.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje e\u0161te jedna nev\u00fdhoda Prometheus, ktor\u00fa sme si uvedomili, a to je ak\u00e9ko\u013evek obmedzenie pam\u00e4te. S Prometheus je to ove\u013ea hor\u0161ie, preto\u017ee nem\u00e1 \u017eiadne tak\u00e9to mo\u017enosti. Pou\u017e\u00edva\u0165 obmedzenie v doku nie je tie\u017e mo\u017enos\u0165. Ak by n\u00e1hodou spadol RAF a mal 20-30 gigabajtov, obnovenie potrv\u00e1 ve\u013emi dlho. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/95a19ff217d1601e9ee8dbe1f1b11ba2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>To je \u010fal\u0161\u00ed d\u00f4vod, pre\u010do n\u00e1m Prometheus nevyhovuje, t. j. nie je mo\u017en\u00e9 obmedzi\u0165 spotrebu pam\u00e4te. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/77280b358370327e4285c90795833a23.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00f4\u017eeme prejs\u0165 na tak\u00fato sch\u00e9mu. T\u00e1to sch\u00e9ma je potrebn\u00e1 na organiz\u00e1ciu HA klastera. Chceme, aby na\u0161e metriky boli v\u017edy a v\u0161ade dostupn\u00e9, aj ke\u010f spadne server, ktor\u00fd tieto metriky uchov\u00e1va. A preto mus\u00edme postavi\u0165 tak\u00fato sch\u00e9mu. <\/p>\n<p><\/p>\n<p>T\u00e1to sch\u00e9ma hovor\u00ed, \u017ee budeme ma\u0165 duplicitu shardov a t\u00fdm p\u00e1dom aj duplicitu n\u00e1kladov na pou\u017eit\u00e9 zdroje. Je mo\u017en\u00e9 ju takmer horizont\u00e1lne \u0161k\u00e1lova\u0165, no aj tak bude spotreba zdrojov obrovsk\u00e1. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/461a4c163693af4700d69a0667aea667.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Odp\u00edsali sme si nedostatky v porad\u00ed, v akom sme ich identifikovali:<\/p>\n<p><\/p>\n<ul>\n<li>Je potrebn\u00e9 vyvies\u0165 metriky von.<\/li>\n<li>Ve\u013ek\u00e1 spotreba zdrojov.<\/li>\n<li>Nie je mo\u017en\u00e9 obmedzi\u0165 spotrebu pam\u00e4te.<\/li>\n<li>Zlo\u017eit\u00e1 a n\u00e1ro\u010dn\u00e1 implement\u00e1cia HA.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/d748b9c51767e37b9248e77871172d77.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rozhodli sme sa, \u017ee ideme pre\u010d od Prometheus ako od \u00falo\u017eiska. <\/p>\n<p><\/p>\n<p>Ur\u010dili sme si \u010fal\u0161ie po\u017eiadavky, ktor\u00e9 potrebujeme. Tieto s\u00fa:<\/p>\n<p><\/p>\n<ul>\n<li>Podpora promql, preto\u017ee u\u017e je nap\u00edsan\u00e9 ve\u013ea pod Prometheus: dotazy, upozornenia. <\/li>\n<li>A m\u00e1me tu e\u0161te Grafanu, ktor\u00e1 je rovnak\u00fdm sp\u00f4sobom nap\u00edsan\u00e1 pod Prometheus ako backend. Nechceme prepisova\u0165 dashboardy. <\/li>\n<li>Chceme postavi\u0165 norm\u00e1lnu architekt\u00faru HA. <\/li>\n<li>Chceme zn\u00ed\u017ei\u0165 spotrebu ak\u00fdchko\u013evek zdrojov. <\/li>\n<li>Existuje e\u0161te mal\u00fd detail. Nem\u00f4\u017eeme pou\u017e\u00edva\u0165 r\u00f4zne cloudov\u00e9 syst\u00e9my zberu metrik. Nevieme, \u010do sa n\u00e1m tam ulet\u00ed do t\u00fdch metr\u00edk. A ke\u010f\u017ee tam m\u00f4\u017ee od\u00eds\u0165 \u010doko\u013evek, mus\u00edme sa obmedzi\u0165 na lok\u00e1lne umiestnenie. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/53bf7dd09cec6fbd42569702b800cf08.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na v\u00fdber bol mal\u00fd rozsah. Zhroma\u017edili sme v\u0161etko, \u010do sme za\u017eili. Pozreli sme sa na str\u00e1nku Prometheus v sekcii integracie, pre\u010d\u00edtali sme mno\u017estvo \u010dl\u00e1nkov a pos\u00fadili, \u010do je vlastne k dispoz\u00edcii. A pre n\u00e1s sme si vybrali VictoriaMetrics ako n\u00e1hradu za Prometheus.<\/p>\n<p><\/p>\n<p>Pre\u010do? Preto\u017ee:<\/p>\n<p><\/p>\n<ul>\n<li>Ovl\u00e1da promql.<\/li>\n<li>M\u00e1 modul\u00e1rnu architekt\u00faru. <\/li>\n<li>Nevy\u017eaduje zmeny v Grafane.<\/li>\n<li>A to najd\u00f4le\u017eitej\u0161ie \u2013 mo\u017eno budeme poskytova\u0165 \u00falo\u017eisko metrik v r\u00e1mci na\u0161ej spolo\u010dnosti ako slu\u017ebu, preto sa vopred pozer\u00e1me na obmedzenia r\u00f4zneho druhu, aby pou\u017e\u00edvatelia mohli v obmedzenej podobe vyu\u017ei\u0165 v\u0161etky zdroje klastera, preto\u017ee existuje \u0161anca, \u017ee bude multitenancy.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/137a425e463ce524e873c962c0bc27f7.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rob\u00edme prv\u00e9 porovnanie. Berieme ten ist\u00fd Prometheus vn\u00fatri klastera, k nemu sa prip\u00e1ja extern\u00fd Prometheus. Prid\u00e1vame cez remoteWrite VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/5357d81a4fc38a2178cd589f36fce063.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Hne\u010f na za\u010diatok spomeniem, \u017ee sme tu zaregistrovali mierny n\u00e1rast CPU za strany VictoriaMetrics. Na wiki VictoriaMetrics s\u00fa uveden\u00e9 parametre, ktor\u00e9 funguj\u00fa najlep\u0161ie. Overili sme ich. V\u00fdrazne zn\u00ed\u017eili spotrebu CPU.<\/p>\n<p><\/p>\n<p>V na\u0161om pr\u00edpade sa spotreba pam\u00e4te Prometheus, ktor\u00fd je v klastri Kubernetes, nezv\u00fd\u0161ila v\u00fdrazne. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/7c70bfb0a50bcbd1e0976b70ae5b7b3a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Porovn\u00e1vame dva zdroje d\u00e1t t\u00fdch ist\u00fdch \u00fadajov. V Prometheus vid\u00edme v\u0161etky ch\u00fdbaj\u00face \u00fadaje. Vo VictoriaMetrics je v\u0161etko v poriadku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/b2d7f9b451f87366e186b25450cb52fa.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u00fdsledky testov s diskov\u00fdm priestorom. V Prometheus sme dostali celkovo 120 gigabajtov. V VictoriaMetrics dost\u00e1vame u\u017e 4 gigabajty za de\u0148. Tam je trochu in\u00fd mechanizmus, ako sme zvyknut\u00ed vidie\u0165 v Prometheus. T. j. \u00fadaje s\u00fa dos\u0165 dobre komprimovan\u00e9 za de\u0148, za polhodinu. S\u00fa u\u017e dobre stla\u010den\u00e9 za de\u0148, za polhodinu, aj ke\u010f sa e\u0161te bud\u00fa zl\u00fa\u010di\u0165 \u00fadaje. Nakoniec sme u\u0161etrili na diskovom priestore.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/ce1f6edf70595aac40aeaae70c286bed.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>E\u0161te \u0161etr\u00edme na spotrebe pam\u00e4\u0165ov\u00fdch zdrojov. Na\u0161a testovacia in\u0161tancia Prometheus bola nasaden\u00e1 na virtu\u00e1lnom stroji \u2013 8 jadier, 24 gigabajtov. Prometheus spotrebov\u00e1va prakticky v\u0161etko. Spadol kv\u00f4li OOM Killer. Pri tom do\u0148 pr\u00fadilo len 900 000 akt\u00edvnych metr\u00edk. To je pribli\u017ene 25 000-27 000 metr\u00edk za sekundu. <\/p>\n<p><\/p>\n<p>VictoriaMetrics bola spusten\u00e1 na dvojjadrovom virtu\u00e1lnom stroji s 8 gigabajtmi RAM. Podarilo sa n\u00e1m dosiahnu\u0165 dobr\u00e9 fungovanie VictoriaMetrics \u00fapravou niektor\u00fdch parametroch na 8-gigabajtovom zariaden\u00ed. Nakoniec sme sa zmestili do 7 gigabajtov. Z\u00edskali sme pritom vy\u0161\u0161iu r\u00fdchlos\u0165 prenosu obsahu, teda metr\u00edk, ne\u017e v pr\u00edpade Prometheus. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/e4c175aee859bc95a4ee08e8b3fdcc22.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Spotreba CPU sa v porovnan\u00ed s Prometheus v\u00fdrazne zlep\u0161ila. Tu Prometheus spotrebov\u00e1va 2,5 jadra, k\u00fdm VictoriaMetrics len 0,25 jadra. Na za\u010diatku \u2013 0,5 jadra. Po\u010das zl\u00fa\u010denia sa to m\u00f4\u017ee zv\u00fd\u0161i\u0165 na jedno jadro, ale to sa st\u00e1va ve\u013emi zriedka.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/dcdbc7d70642130949b0c65b43a58f7f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V na\u0161om pr\u00edpade sme sa rozhodli pre VictoriaMetrics z jasn\u00fdch d\u00f4vodov, chceli sme u\u0161etri\u0165 a podarilo sa n\u00e1m to. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/457576073885b9153041589bb2f53ee4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ihne\u010f vylu\u010dujeme dva body \u2013 export metr\u00edk a vysok\u00fa spotrebu zdrojov. Ost\u00e1va n\u00e1m rie\u0161i\u0165 dva body, ktor\u00e9 sme si e\u0161te ponechali. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/9ee296c7838abd243a893c8e34b60711.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Hne\u010f na za\u010diatok uvediem, \u017ee ViktoriaMetrics uva\u017eujeme ako \u00falo\u017eisko metr\u00edk. Ale ke\u010f\u017ee chceme s najv\u00e4\u010d\u0161ou pravdepodobnos\u0165ou pon\u00faknu\u0165 VictoriaMetrics ako \u00falo\u017eisko pre cel\u00e9 Leroy, potrebujeme obmedzi\u0165 t\u00fdch, ktor\u00ed bud\u00fa tento klaster pou\u017e\u00edva\u0165, aby n\u00e1m ho nezni\u010dili. <\/p>\n<p><\/p>\n<p>Existuje skvel\u00e1 mo\u017enos\u0165, ktor\u00e1 umo\u017e\u0148uje obmedzi\u0165 \u010das, objem d\u00e1t a \u010das vykonania. <\/p>\n<p><\/p>\n<p>A takisto je tu vynikaj\u00faca mo\u017enos\u0165 obmedzi\u0165 spotrebu pam\u00e4te, \u010d\u00edm m\u00f4\u017eeme n\u00e1js\u0165 ten prav\u00fd balans, ktor\u00fd n\u00e1m umo\u017en\u00ed dosiahnu\u0165 norm\u00e1lnu r\u00fdchlos\u0165 pr\u00e1ce a primeran\u00fa spotrebu zdrojov. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/039d8b7f24e50a2e62211eef30ef0812.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>T\u00fdmto sme od\u0161krtli \u010fal\u0161\u00ed bod, t. j. nem\u00f4\u017eeme obmedzi\u0165 spotrebu pam\u00e4te.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/7444f68188a835392f8c6ad82c3c26d1.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V prv\u00fdch iter\u00e1ci\u00e1ch sme testovali VictoriaMetrics Single Node. Potom prech\u00e1dzame na VictoriaMetrics Cluster Version.<\/p>\n<p><\/p>\n<p>Tu n\u00e1m otv\u00e1raj\u00fa mo\u017enosti distrib\u00facie r\u00f4znych slu\u017eieb vo VictoriaMetrics v z\u00e1vislosti od toho, na \u010dom bud\u00fa be\u017ea\u0165 a ak\u00e9 zdroje spotreb\u00favaj\u00fa. Je to ve\u013emi flexibiln\u00e9 a pohodln\u00e9 rie\u0161enie. Pou\u017eili sme to na vlastnej ko\u017ei. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/4bcadbd80d7efcdbcbda6c79929ab279.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Hlavn\u00fdmi komponentmi VictoriaMetrics Cluster Version s\u00fa vmstorage. T\u00fdch m\u00f4\u017ee by\u0165 N po\u010det. V na\u0161om pr\u00edpade s\u00fa zatia\u013e dva. <\/p>\n<p><\/p>\n<p>A je tu vminsert. To je proxy server, ktor\u00fd n\u00e1m umo\u017e\u0148uje: realizova\u0165 shardovanie medzi v\u0161etk\u00fdmi storages, o ktor\u00fdch sme mu povedali, a taktie\u017e umo\u017e\u0148uje replik\u00e1ciu, t. j. budete ma\u0165 aj shardovanie, aj replik\u00e1ciu. <\/p>\n<p><\/p>\n<p>Vminsert podporuje protokoly OpenTSDB, Graphite, InfluxDB a remoteWrite od Prometheus.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/6b4ac44513d7bfa7e0b234aa0f2fa3f5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje tie\u017e vmselect. Jeho hlavnou \u00falohou je \u00eds\u0165 do vmstorage, z\u00edska\u0165 od nich d\u00e1ta, deduplova\u0165 tieto d\u00e1ta a odovzda\u0165 ich klientovi. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/768db0f7dbd17223b7057d33ceccd12d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Je tu skvel\u00e1 vec ako vmagent. Ve\u013emi n\u00e1m to p\u00e1\u010di. Umo\u017e\u0148uje to konfigurova\u0165 presne tak, ako Prometheus, a pritom robi\u0165 v\u0161etko presne ako Prometheus. T. j. zbiera metriky z r\u00f4znych ent\u00edt, slu\u017eieb a odosiela ich do vminsert. Potom u\u017e v\u0161etko z\u00e1vis\u00ed od v\u00e1s.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/429caa4d3296810c688e7c972ab6d35c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u010eal\u0161ia vynikaj\u00faca slu\u017eba je vmalert, ktor\u00e1 umo\u017e\u0148uje ako backend pou\u017e\u00edva\u0165 VictoriaMetrics, z\u00edskava\u0165 od vminsert a odosiela\u0165 spracovan\u00e9 d\u00e1ta do vmselect. Spracov\u00e1va samotn\u00e9 alerty a rules. V pr\u00edpade alertov dost\u00e1vame alert cez alertmanager.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/67d6264ffd5e061e168261586b4e6369.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje komponent wmauth. Mohol by sa pou\u017e\u00edva\u0165, a mo\u017eno aj nie (e\u0161te sme sa s t\u00fdm nerozhodli) ako syst\u00e9m autoriz\u00e1cie pri multitenancy verzi\u00e1ch klastrov. Podporuje remoteWrite pre Prometheus a m\u00f4\u017ee autorizova\u0165 na z\u00e1klade URL, presnej\u0161ie druhej jej \u010dasti, kam m\u00f4\u017eete alebo nem\u00f4\u017eete p\u00edsa\u0165. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/2a6a97eedc36ce71d660e44e0b736581.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje tie\u017e vmbackup, vmrestore. To je v podstate obnova a z\u00e1lohovanie v\u0161etk\u00fdch d\u00e1t. Ovl\u00e1da S3, GCS, file.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/ec11f7a981bb80daaf6891001d80a1c9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prv\u00e1 iter\u00e1cia n\u00e1\u0161ho klastru bola uskuto\u010dnen\u00e1 po\u010das karant\u00e9ny. V tom \u010dase neexistovala replik\u00e1cia, preto na\u0161a iter\u00e1cia pozost\u00e1vala z dvoch r\u00f4znych a nez\u00e1visl\u00fdch klastrov, do ktor\u00fdch sme cez remoteWrite prij\u00edmali \u00fadaje. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/2a76181fc6c0ffb6b09ad5bcdb33909c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tu chcem poveda\u0165, \u017ee ke\u010f sme prech\u00e1dzali z VictoriaMetrics Single Node na VictoriaMetrics Cluster Version, zostali sme v rovnak\u00fdch spotrebovan\u00fdch zdrojoch, t.j. hlavn\u00e9 je pam\u00e4\u0165. Dalo sa to takto distribuova\u0165 na\u0161e \u00fadaje, t.j. spotreba zdrojov.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/2d22446669e71b490226a789294707fc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tu u\u017e bola pridan\u00e1 replik\u00e1cia. Spojili sme v\u0161etko do jedn\u00e9ho relat\u00edvne ve\u013ek\u00e9ho klastru. V\u0161etky \u00fadaje m\u00e1me ako shardovan\u00e9, tak aj replikovan\u00e9. <\/p>\n<p><\/p>\n<p>Cel\u00fd klaster m\u00e1 N bodov pr\u00edstupu, t.j. Prometheus m\u00f4\u017ee prid\u00e1va\u0165 \u00fadaje cez HAPROXY. Tu m\u00e1me tento pr\u00edstupov\u00fd bod. A cez tento pr\u00edstupov\u00fd bod sa d\u00e1 vstupova\u0165 do Grafany. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/4a797ed22c57cab7c10b2635f53941df.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V na\u0161om pr\u00edpade je HAPROXY jedin\u00fd port, ktor\u00fd proxyzuje select, insert a ostatn\u00e9 slu\u017eby do tohto klastru. V na\u0161om pr\u00edpade nebolo mo\u017en\u00e9 vytvori\u0165 jednu adresu, museli sme vytvori\u0165 nieko\u013eko pr\u00edstupov\u00fdch bodov, preto\u017ee samotn\u00e9 virtu\u00e1lne stroje, na ktor\u00fdch be\u017e\u00ed VictoriaMetrics-klaster, s\u00fa v r\u00f4znych z\u00f3nach jedn\u00e9ho poskytovate\u013ea cloudu, t.j. nie vo vn\u00fatri n\u00e1\u0161ho cloudu, ale vonku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/76bd7e59b58795ceada54977a933fb6d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00e1me alerting. Pou\u017e\u00edvame ho. Pou\u017e\u00edvame alertmanager od Prometheus. Ako kan\u00e1l doru\u010dovania alertov vyu\u017e\u00edvame Opsgenie a Telegram. V Telegrame sa objavuj\u00fa upozornenia od dev, mo\u017eno nie\u010do od prod, ale sk\u00f4r nie\u010do \u0161tatistick\u00e9, potrebn\u00e9 in\u017einierom. A Opsgenie \u2013 je kritick\u00e9. Toto s\u00fa telefon\u00e1ty, spr\u00e1va incidentov. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/b06abbf9eac39450a290e1a776df415f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ve\u010dn\u00fd ot\u00e1zka: \u201eKto monitoruje monitoring?\u201c. V na\u0161om pr\u00edpade monitoring monitoruje s\u00e1m monitoring, preto\u017ee pou\u017e\u00edvame vmagent na ka\u017edom uzle. A preto\u017ee na\u0161e uzly s\u00fa rozmiestnen\u00e9 po r\u00f4znych d\u00e1tov\u00fdch centr\u00e1ch jedn\u00e9ho poskytovate\u013ea, v ka\u017edom d\u00e1tovom centre m\u00e1me svoj kan\u00e1l, s\u00fa nez\u00e1visl\u00e9 a aj keby pri\u0161lo k rozdeleniu mysle, st\u00e1le dostaneme alerty. \u00c1no, bude ich viac, ale je lep\u0161ie dosta\u0165 viac alertov ako \u017eiadne. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/20e94b612f00619015da7e6ebc1e6afb.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Koniec na\u0161ej zoznamu s implement\u00e1ciou HA.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/f7cac236b8f99d8f2f12288e565c7de0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A chcel by som tie\u017e zd\u00f4razni\u0165 sk\u00fasenos\u0165 s komunik\u00e1ciou s komunitou VictoriaMetrics. T\u00e1to sk\u00fasenos\u0165 bola ve\u013emi pozit\u00edvna. Chlapci s\u00fa ochotn\u00ed. Sna\u017eia sa pochopi\u0165 ka\u017ed\u00fd pr\u00edpad, ktor\u00fd sa navrhuje. <\/p>\n<p><\/p>\n<p>Zalo\u017eil som issues na GitHub. R\u00fdchlo boli vyrie\u0161en\u00e9. Existuje e\u0161te p\u00e1r issues, ktor\u00e9 nie s\u00fa \u00faplne uzavret\u00e9, ale u\u017e pod\u013ea k\u00f3du vid\u00edm, \u017ee pr\u00e1ca t\u00fdmto smerom prebieha. <\/p>\n<p><\/p>\n<p>Hlavn\u00e1 boles\u0165 po\u010das iter\u00e1ci\u00ed pre m\u0148a bola v tom, \u017ee ak vypnem uzol, prv\u00fdch 30 sek\u00fand nemal vminsert pochopenie, \u017ee backend neexistuje. To je teraz u\u017e vyrie\u0161en\u00e9. A doslova za sekundu alebo dve sa \u00fadaje u\u017e zbieraj\u00fa zo v\u0161etk\u00fdch zost\u00e1vaj\u00facich uzlov a dopyt prest\u00e1va \u010daka\u0165 na ten ch\u00fdbaj\u00faci uzol. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/722ad280977c2f377c5afe36eb309c96.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Chceli sme v nejakom momente od VictoriaMetrics, aby to bol VictoriaMetrics operator. \u010cakali sme na\u0148. Teraz sme v akt\u00edvnej f\u00e1ze budovania ob\u00e1lky nad VictoriaMetrics operator, aby sme z\u00edskali v\u0161etky pravidl\u00e1 predpo\u010d\u00edtania a pod. Prometheus, preto\u017ee dostato\u010dne akt\u00edvne pou\u017e\u00edvame pravidl\u00e1, ktor\u00e9 prich\u00e1dzaj\u00fa s oper\u00e1torom Prometheus.<\/p>\n<p><\/p>\n<p>Existuj\u00fa n\u00e1vrhy na zlep\u0161enie klastrovej implement\u00e1cie. Vystavil som ich vy\u0161\u0161ie. <\/p>\n<p><\/p>\n<p>A tie\u017e ve\u013emi t\u00fa\u017eime po downsamplingu. V na\u0161om pr\u00edpade je downsampling potrebn\u00fd v\u00fdhradne na preh\u013eadanie trendov. Hrubo povedan\u00e9, jedna metrika po\u010das d\u0148a mi sta\u010d\u00ed. Tieto trendy potrebujeme na rok, tri, p\u00e4\u0165, desa\u0165 rokov. A jedna hodnota metriky \u00faplne sta\u010d\u00ed.<br \/>\n<img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/01073d8f9b16e1094daef3e36dd97e44.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<ul>\n<li>Poc\u00edtili sme boles\u0165, ako aj niektor\u00ed na\u0161i kolegovia, pri pou\u017e\u00edvan\u00ed Prometheus.<\/li>\n<li>Pre n\u00e1s sme si vybrali VictoriaMetrics. <\/li>\n<li>Dobre sa \u0161k\u00e1luje vertik\u00e1lne aj horizont\u00e1lne. <\/li>\n<li>M\u00f4\u017eeme rozlo\u017ei\u0165 r\u00f4zne komponenty na r\u00f4zny po\u010det uzlov v klastri, obmedzi\u0165 ich pod\u013ea objemu pam\u00e4te, prida\u0165 pam\u00e4\u0165 at\u010f. <\/li>\n<\/ul>\n<p><\/p>\n<p>Budeme pou\u017e\u00edva\u0165 VictoriaMetrics, preto\u017ee sa n\u00e1m ve\u013emi p\u00e1\u010dila. Toto bolo a \u010do sa stalo. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitoring s\u00fakromn\u00fdch cloudov. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/e04b66e5e2b765143797d3c983d4029c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/VictoriaMetrics_ru1\">https:\/\/t.me\/VictoriaMetrics_ru1<\/a><\/noindex><\/p>\n<p><\/p>\n<p>P\u00e1r qr k\u00f3dov na chat VictoriaMetrics, moje kontakty, technick\u00fd radar LeroyMerlin.<\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/518680\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>VictoriaMetrics \u2014 \u0431\u044b\u0441\u0442\u0440\u0430\u044f \u0438 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u0430\u044f \u0421\u0423\u0411\u0414 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0434\u0430\u043d\u043d\u044b\u0445 \u0432 \u0444\u043e\u0440\u043c\u0435 \u0432\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0433\u043e \u0440\u044f\u0434\u0430 (\u0437\u0430\u043f\u0438\u0441\u044c \u043e\u0431\u0440\u0430\u0437\u0443\u0435\u0442 \u0432\u0440\u0435\u043c\u044f \u0438 \u043d\u0430\u0431\u043e\u0440 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u044d\u0442\u043e\u043c\u0443 \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0439, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0445 \u0447\u0435\u0440\u0435\u0437 \u043f\u0435\u0440\u0438\u043e\u0434\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u043e\u043f\u0440\u043e\u0441 \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u044f \u0434\u0430\u0442\u0447\u0438\u043a\u043e\u0432 \u0438\u043b\u0438 \u0441\u0431\u043e\u0440 \u043c\u0435\u0442\u0440\u0438\u043a). \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041a\u043e\u043b\u043e\u0431\u0430\u0435\u0432 \u041f\u0430\u0432\u0435\u043b. DevOps, SRE, LeroyMerlin, \u0432\u0441\u0435 \u043a\u0430\u043a \u043a\u043e\u0434 \u2013 \u044d\u0442\u043e \u0432\u0441\u0435 \u043f\u0440\u043e \u043d\u0430\u0441: \u043f\u0440\u043e \u043c\u0435\u043d\u044f \u0438 \u043f\u0440\u043e \u0434\u0440\u0443\u0433\u0438\u0445 \u0441\u043e\u0442\u0440\u0443\u0434\u043d\u0438\u043a\u043e\u0432 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":94154,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-94153","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47VictoriaMetrics \u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043f\u0440\u0438\u0432\u0430\u0442\u043d\u044b\u0445 \u043e\u0431\u043b\u0430\u043a\u043e\u0432. \u041f\u0430\u0432\u0435\u043b \u041a\u043e\u043b\u043e\u0431\u0430\u0435\u0432 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-09-13T17:42:30+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-09-13T17:42:30+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47VictoriaMetrics a monitorovanie s\u00fakromn\u00fdch cloudov. Pavel Kolobaev | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47VictoriaMetrics \u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043f\u0440\u0438\u0432\u0430\u0442\u043d\u044b\u0445 \u043e\u0431\u043b\u0430\u043a\u043e\u0432. \u041f\u0430\u0432\u0435\u043b \u041a\u043e\u043b\u043e\u0431\u0430\u0435\u0432 | ProHoster","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-09-13T17:42:30+00:00","article:modified_time":"2020-09-13T17:42:30+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"94153","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 11:32:22","updated":"2022-09-28 11:09:52","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/94153","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=94153"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/94153\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/94154"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=94153"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=94153"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=94153"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}