{"id":94153,"date":"2020-09-13T19:42:30","date_gmt":"2020-09-13T17:42:30","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev"},"modified":"2020-09-13T19:42:30","modified_gmt":"2020-09-13T17:42:30","slug":"victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","title":{"rendered":"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/d84a6783fbf35e9d96afd263b5cd4578.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>VictoriaMetrics \u2014 rychl\u00e1 a \u0161k\u00e1lovateln\u00e1 datab\u00e1ze pro uchov\u00e1v\u00e1n\u00ed a zpracov\u00e1n\u00ed dat ve form\u011b \u010dasov\u00fdch \u0159ad (z\u00e1znam vytv\u00e1\u0159\u00ed \u010das a sadu odpov\u00eddaj\u00edc\u00edch hodnot, nap\u0159\u00edklad z\u00edskan\u00fdch prost\u0159ednictv\u00edm periodick\u00e9ho dotazov\u00e1n\u00ed stavu senzor\u016f nebo shroma\u017e\u010fov\u00e1n\u00ed metrik).<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\n<center><div class=\"youtube-placeholder\" data-id=\"74swsWqf0Uc\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/74swsWqf0Uc\/hqdefault.jpg\" alt=\"P\u0159ehr\u00e1t video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/df849870a0432a863e34eb11387c9fab.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Jmenuji se Kolobaev Pavel. DevOps, SRE, LeroyMerlin, v\u0161echno jako k\u00f3d \u2013 to v\u0161echno se n\u00e1s t\u00fdk\u00e1: m\u011b a ostatn\u00edch zam\u011bstnanc\u016f LeroyMerlin.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/b7b63999cef8d80148bf666c2bce335a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/bit.ly\/3jf1fIK\">https:\/\/bit.ly\/3jf1fIK<\/a><\/noindex><\/p>\n<p><\/p>\n<p>Existuje cloud zalo\u017een\u00fd na OpenStacku. Tam je mal\u00fd odkaz na technick\u00fd radar. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/e99d4aeab2a62bc46cfa9aaadb9590af.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Bylo postaveno na z\u00e1klad\u011b hardware Kubernetes, a tak\u00e9 na v\u0161ech souvisej\u00edc\u00edch slu\u017eb\u00e1ch s OpenStackem a logov\u00e1n\u00edm. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/c19c24b9006162eaf79f8f9bdc5aa14a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Schema, kterou jsme m\u011bli p\u0159i v\u00fdvoji, byla takov\u00e1. Kdy\u017e jsme to v\u0161echno vyv\u00edjeli, m\u011bli jsme oper\u00e1tora Prometheus, kter\u00fd uchov\u00e1val data uvnit\u0159 samotn\u00e9ho clustera K8s. Automaticky nalezl to, co bylo pot\u0159eba scrapovat, a ulo\u017eil si to pod nohy, tak \u0159\u00edkaj\u00edc. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/ebaf6b3d32483c405a1ae69d0ea7e3ea.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u0161echna data budeme muset vyv\u00e9st mimo cluster Kubernetes, proto\u017ee pokud se n\u011bco stane, mus\u00edme ch\u00e1pat, co a kde. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/36f921aa715ed00b9ac26ae3861a73b8.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prvn\u00ed \u0159e\u0161en\u00ed \u2013 pou\u017e\u00edv\u00e1me federation, kdy\u017e m\u00e1me extern\u00ed Prometheus a proch\u00e1zej\u00ed do clusteru Kubernetes prost\u0159ednictv\u00edm mechanismu federation.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/11033de585d88cfaee1edeec908da55a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ale vznikaj\u00ed zde drobn\u00e9 probl\u00e9my. V na\u0161em p\u0159\u00edpad\u011b probl\u00e9my za\u010daly, kdy\u017e jsme m\u011bli 250 000 metrik, a kdy\u017e jich bylo 400 000, uv\u011bdomili jsme si, \u017ee takto pracovat nem\u016f\u017eeme. Zv\u011bt\u0161ili jsme scrape_timeout na 25 sekund.<\/p>\n<p><\/p>\n<p>Pro\u010d jsme to museli ud\u011blat? Prometheus za\u010d\u00edn\u00e1 po\u010d\u00edtat \u010das timeoutu od za\u010d\u00e1tku odb\u011bru. A nez\u00e1le\u017e\u00ed na tom, \u017ee data st\u00e1le te\u010dou. Pokud se b\u011bhem dan\u00e9ho \u010dasov\u00e9ho \u00faseku data neslou\u010dila a relace nebude uzav\u0159ena p\u0159es http, pak se pova\u017euje, \u017ee relace je ne\u00fasp\u011b\u0161n\u00e1 a data se nedostanou do samotn\u00e9ho Prometheus.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/c5d8f034cbf80638775ecc68b6ad8f21.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u0161em zn\u00e1m\u00e9 grafy, kter\u00e9 dost\u00e1v\u00e1me, kdy\u017e \u010d\u00e1st dat chyb\u00ed. Grafy jsou trhan\u00e9 a to n\u00e1s neuspokojuje. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/8caa1939ac95d26163e98ed2bd12fd6a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dal\u0161\u00ed mo\u017enost \u2013 to je shardov\u00e1n\u00ed na z\u00e1klad\u011b dvou r\u016fzn\u00fdch Prometheus skrze stejn\u00fd mechanismus federation.<\/p>\n<p><\/p>\n<p>Nap\u0159\u00edklad je jednodu\u0161e vz\u00edt a shardovat je podle jm\u00e9na. To lze tak\u00e9 pou\u017e\u00edt, ale rozhodli jsme se posunout d\u00e1l. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/c5715365ac494689194ad4d305a7be08.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nyn\u00ed budeme muset tyto shards n\u011bjak zpracovat. M\u016f\u017eeme vz\u00edt promxy, kter\u00fd zajde do oblasti shardu a zmultiplikuje data. Pracuje s dv\u011bma shardy jako s jedin\u00fdm vstupn\u00edm bodem. To lze realizovat prost\u0159ednictv\u00edm promxy, ale zat\u00edm je to p\u0159\u00edli\u0161 slo\u017eit\u00e9. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/672984d1ca13743acb4b8102b74d0eb2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prvn\u00ed mo\u017enost \u2013 chceme se vzd\u00e1t mechanismu federation, proto\u017ee je velmi pomal\u00fd. <\/p>\n<p><\/p>\n<p>V\u00fdvoj\u00e1\u0159i Promethea jasn\u011b \u0159\u00edkaj\u00ed: \u201eLidi, pou\u017e\u00edvejte jin\u00e9 TimescaleDB, proto\u017ee nebudeme dlouhodob\u011b podporovat uchov\u00e1v\u00e1n\u00ed metrik.\u201c To nen\u00ed jejich \u00fakol. <img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/dfc59949ec16fe731d339d7bf477edff.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Na pap\u00edr si zapisujeme, \u017ee pot\u0159ebujeme export ven, abychom nemuseli uchov\u00e1vat v\u0161echno na jednom m\u00edst\u011b. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/a0e566b89c15df9c17c0750ce8512044.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Druh\u00fdm nedostatkem je spot\u0159eba pam\u011bti. Ano, ch\u00e1pu, \u017ee mnoz\u00ed \u0159eknou, \u017ee v roce 2020 p\u00e1r gigabajt\u016f pam\u011bti nestoj\u00ed moc, ale i tak. <\/p>\n<p><\/p>\n<p>Aktu\u00e1ln\u011b m\u00e1me dev a prod prost\u0159ed\u00ed. V dev to je asi 9 gigabajt\u016f na 350 000 metrik. V prod je to n\u011bco p\u0159es 14 gigabajt\u016f na 780 000 metrik. P\u0159itom retention time m\u00e1me pouh\u00fdch 30 minut. To je \u0161patn\u011b. A te\u010f vysv\u011btl\u00edm pro\u010d.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/35ad6d2792fd4dbf68b15851f24eb047.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prov\u00e1d\u00edme v\u00fdpo\u010dty, tj. p\u0159i jednom a p\u016fl milionu metrik, co\u017e u\u017e je velmi bl\u00edzko, z\u00edsk\u00e1me na \u00farovni projektov\u00e1n\u00ed 35-37 gigabajt\u016f pam\u011bti. Ale u\u017e p\u0159i 4 milionech metrik pot\u0159ebujeme asi 90 gigabajt\u016f pam\u011bti. Tato \u010d\u00edsla byla vypo\u010d\u00edt\u00e1na podle vzorce, kter\u00fd poskytuj\u00ed v\u00fdvoj\u00e1\u0159i Promethea. Zjistili jsme korelaci a do\u0161li jsme k tomu, \u017ee nechceme platit p\u00e1r milion\u016f za server jen na monitorov\u00e1n\u00ed. <\/p>\n<p><\/p>\n<p>Nejen\u017ee se zv\u00fd\u0161\u00ed po\u010det stroj\u016f, ale monitorujeme tak\u00e9 samotn\u00e9 virtu\u00e1ln\u00ed stroje. Tak\u017ee \u010d\u00edm v\u00edce virtu\u00e1ln\u00edch stroj\u016f, t\u00edm v\u00edce r\u016fzn\u00fdch metrik atd. Na\u0161e klastry se budou specificky rozr\u016fstat v r\u00e1mci metrik.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/d0aa8f00c7bd511b128bfa38cd280d03.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Co se diskov\u00e9ho prostoru t\u00fd\u010de, nen\u00ed to tak \u0161patn\u00e9, ale cht\u011bli bychom to zlep\u0161it. Za 15 dn\u00ed jsme z\u00edskali celkem 120 gigabajt\u016f, z nich\u017e 100 je komprimovan\u00fdch dat a 20 nekomprimovan\u00fdch, ale v\u017edy chceme m\u00e9n\u011b. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/04dcc202029ce2433ae5cfb379c51181.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Odpov\u00eddaj\u00edc\u00edm zp\u016fsobem zapisujeme je\u0161t\u011b jeden bod \u2013 je to vysok\u00e1 spot\u0159eba zdroj\u016f, kter\u00e9 bychom cht\u011bli \u0161et\u0159it, proto\u017ee nechceme, aby n\u00e1\u0161 monitorovac\u00ed klastr zab\u00edral v\u00edce zdroj\u016f ne\u017e n\u00e1\u0161 klastr, kter\u00fd spravuje OpenStack.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/1708dcd413d890f18149414616f6264e.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje je\u0161t\u011b jeden nedostatek Promethea, kter\u00fd jsme si uv\u011bdomili, a to je alespo\u0148 n\u011bjak\u00e9 omezen\u00ed pam\u011bti. S Prometheem je to mnohem hor\u0161\u00ed, proto\u017ee nem\u00e1 \u017e\u00e1dn\u00e9 takov\u00e9 mo\u017enosti. Pou\u017e\u00edvat omezen\u00ed v Dockeru tak\u00e9 nen\u00ed \u0159e\u0161en\u00ed. Pokud n\u00e1hodou spadne RAF a m\u00e1te tam 20-30 gigabajt\u016f, bude trvat docela dlouho, ne\u017e se znovu zvedne. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/95a19ff217d1601e9ee8dbe1f1b11ba2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>To je dal\u0161\u00ed d\u016fvod, pro\u010d n\u00e1m Prometheus nevyhovuje, t. j. nemo\u017enost omezit spot\u0159ebu pam\u011bti. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/77280b358370327e4285c90795833a23.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u016f\u017eeme p\u0159ej\u00edt na tento model. Tento model pot\u0159ebujeme pro organizaci HA clusteru. Chceme, aby na\u0161e metriky byly v\u017edy a v\u0161ude dostupn\u00e9, i p\u0159i v\u00fdpadku serveru, kter\u00fd tyto metriky uchov\u00e1v\u00e1. A t\u00edm p\u00e1dem budeme muset postavit tuto strukturu. <\/p>\n<p><\/p>\n<p>Tento model \u0159\u00edk\u00e1, \u017ee budeme m\u00edt duplikaci shard\u016f a t\u00edm p\u00e1dem i duplikaci n\u00e1klad\u016f na spot\u0159ebov\u00e1van\u00e9 zdroje. Lze ji t\u00e9m\u011b\u0159 horizont\u00e1ln\u011b \u0161k\u00e1lovat, ale i p\u0159esto budou n\u00e1klady na zdroje ohromn\u00e9. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/461a4c163693af4700d69a0667aea667.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nev\u00fdhody v po\u0159ad\u00ed, jak jsme si je zapsali:<\/p>\n<p><\/p>\n<ul>\n<li>Je pot\u0159eba exportovat metriky ven.<\/li>\n<li>Velk\u00e1 spot\u0159eba zdroj\u016f.<\/li>\n<li>Nelze omezit spot\u0159ebu pam\u011bti.<\/li>\n<li>Slo\u017eit\u00e1 a n\u00e1ro\u010dn\u00e1 implementace HA.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/d748b9c51767e37b9248e77871172d77.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rozhodli jsme se, \u017ee se od Promethea distancujeme jako od \u00falo\u017ei\u0161t\u011b. <\/p>\n<p><\/p>\n<p>Ur\u010dili jsme si tak\u00e9 dal\u0161\u00ed po\u017eadavky, kter\u00e9 pot\u0159ebujeme. To zahrnuje:<\/p>\n<p><\/p>\n<ul>\n<li>Podporu promql, proto\u017ee ji\u017e bylo naps\u00e1no mnoho v\u011bc\u00ed pro Prometheus: dotazy, upozorn\u011bn\u00ed. <\/li>\n<li>A m\u00e1me zde tak\u00e9 Grafanu, kter\u00e1 je rovn\u011b\u017e naps\u00e1na pod Prometheus jako backend. Nechceme p\u0159episovat panely. <\/li>\n<li>Chceme postavit adekv\u00e1tn\u00ed HA architekturu. <\/li>\n<li>Chceme sn\u00ed\u017eit spot\u0159ebu v\u0161ech zdroj\u016f. <\/li>\n<li>Je tu je\u0161t\u011b mal\u00fd detail. Nem\u016f\u017eeme pou\u017e\u00edt r\u016fzn\u00e9 cloudov\u00e9 syst\u00e9my pro sb\u011br metrik. Nev\u00edme, co se n\u00e1m dostane do t\u011bchto metrik zat\u00edm. A proto\u017ee se tam m\u016f\u017ee dostat cokoliv, mus\u00edme se omezit na lok\u00e1ln\u00ed um\u00edst\u011bn\u00ed. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/53bf7dd09cec6fbd42569702b800cf08.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u00fdb\u011br nebyl velk\u00fd. Shrom\u00e1\u017edili jsme v\u0161e, s \u010d\u00edm m\u00e1me zku\u0161enosti. Pod\u00edvali jsme se na str\u00e1nku Prometheus v sekci integrace, p\u0159e\u010detli jsme spoustu \u010dl\u00e1nk\u016f a zjistili, co je k dispozici. A pro sebe jsme vybrali VictoriaMetrics jako n\u00e1hradu za Prometheus.<\/p>\n<p><\/p>\n<p>Pro\u010d? Proto\u017ee:<\/p>\n<p><\/p>\n<ul>\n<li>Podporuje promql.<\/li>\n<li>M\u00e1 modul\u00e1rn\u00ed architekturu. <\/li>\n<li>Nevy\u017eaduje zm\u011bny v Grafan\u011b.<\/li>\n<li>A co je nejd\u016fle\u017eit\u011bj\u0161\u00ed \u2013 mo\u017en\u00e1 budeme poskytovat \u00falo\u017ei\u0161t\u011b metrik uvnit\u0159 na\u0161\u00ed spole\u010dnosti jako slu\u017ebu, a proto se v p\u0159edstihu d\u00edv\u00e1me na omezen\u00ed r\u016fzn\u00fdch typ\u016f, aby u\u017eivatel\u00e9 mohli v n\u011bjak\u00e9 omezen\u00e9 podob\u011b pou\u017e\u00edvat v\u0161echny zdroje clusteru, proto\u017ee existuje \u0161ance, \u017ee bude multitenantn\u00ed.<\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/137a425e463ce524e873c962c0bc27f7.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>D\u011bl\u00e1me prvn\u00ed srovn\u00e1n\u00ed. Bereme stejn\u00fd Prometheus uvnit\u0159 clusteru, ke kter\u00e9mu p\u0159istupuje extern\u00ed Prometheus. P\u0159id\u00e1v\u00e1me p\u0159es remoteWrite VictoriaMetrics.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/5357d81a4fc38a2178cd589f36fce063.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Hned na za\u010d\u00e1tku uvedu, \u017ee jsme zde zaznamenali m\u00edrn\u00e9 zv\u00fd\u0161en\u00ed spot\u0159eby CPU ze strany VictoriaMetrics. Na wiki VictoriaMetrics je uvedeno, kter\u00e9 parametry jsou nejvhodn\u011bj\u0161\u00ed. Otestovali jsme je. Velmi dob\u0159e sn\u00ed\u017eily spot\u0159ebu pr\u00e1v\u011b CPU.<\/p>\n<p><\/p>\n<p>V na\u0161em p\u0159\u00edpad\u011b se spot\u0159eba pam\u011bti Prometheus, kter\u00fd b\u011b\u017e\u00ed v clusteru Kubernetes, zv\u00fd\u0161ila jen m\u00edrn\u011b. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/7c70bfb0a50bcbd1e0976b70ae5b7b3a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Srovn\u00e1v\u00e1me dva datov\u00e9 zdroje stejn\u00fdch dat. V Prometheus vid\u00edme v\u0161echny ty sam\u00e9 chyb\u011bj\u00edc\u00ed \u00fadaje. Ve VictoriaMetrics je v\u0161e v po\u0159\u00e1dku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/b2d7f9b451f87366e186b25450cb52fa.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u00fdsledky test\u016f s diskov\u00fdm prostorem. V Prometheus jsme z\u00edskali celkem 120 gigabajt\u016f. Ve VictoriaMetrics z\u00edsk\u00e1v\u00e1me ji\u017e 4 gigabajty denn\u011b. Maj\u00ed zde pon\u011bkud odli\u0161n\u00fd mechanismus, ne\u017e na kter\u00fd jsme zvykl\u00ed z Prometheus. Tedy data se zde dost dob\u0159e komprimuj\u00ed za den, za p\u016fl hodiny. Jsou ji\u017e dob\u0159e komprimov\u00e1na za den, za p\u016fl hodiny, i kdy\u017e se pak je\u0161t\u011b data m\u00edchaj\u00ed. Nakonec jsme u\u0161et\u0159ili na diskov\u00e9m prostoru.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/ce1f6edf70595aac40aeaae70c286bed.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dal\u0161\u00ed \u00faspora je na spot\u0159eb\u011b pam\u011b\u0165ov\u00fdch zdroj\u016f. Na\u0161e Prometheus byla v dob\u011b test\u016f nasazena na virtu\u00e1ln\u00edm stroji \u2013 8 jader, 24 gigabajt\u016f. Prometheus vyu\u017e\u00edv\u00e1 t\u00e9m\u011b\u0159 v\u0161echny dostupn\u00e9 prost\u0159edky. Zkolabovala kv\u016fli OOM Killeru. P\u0159itom do n\u00ed proudilo pouze 900 000 aktivn\u00edch metrik. To je p\u0159ibli\u017en\u011b 25 000-27 000 metrik za sekundu. <\/p>\n<p><\/p>\n<p>VictoriaMetrics byla spu\u0161t\u011bna na dvouj\u00e1drov\u00e9 virtu\u00e1ln\u00ed ma\u0161in\u011b s 8 gigabajty RAM. Dok\u00e1zali jsme donutit VictoriaMetrics, aby fungovala dob\u0159e, kdy\u017e jsme nastavili n\u011bkter\u00e9 parametry na t\u00e9 8-gigabajtov\u00e9 ma\u0161in\u011b. Nakonec jsme se ve\u0161li do 7 gigabajt\u016f. P\u0159itom jsme dos\u00e1hli rychlosti dod\u00e1v\u00e1n\u00ed obsahu, tj. metrik, dokonce vy\u0161\u0161\u00ed ne\u017e u Prometheus. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/e4c175aee859bc95a4ee08e8b3fdcc22.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Co se t\u00fd\u010de CPU, situace se v\u00fdrazn\u011b zlep\u0161ila ve srovn\u00e1n\u00ed s Prometheus. Zde Prometheus spot\u0159ebov\u00e1v\u00e1 2,5 j\u00e1dra, zat\u00edmco VictoriaMetrics pouze 0,25 j\u00e1dra. Na za\u010d\u00e1tku \u2013 0,5 j\u00e1dra. P\u0159i slu\u010dov\u00e1n\u00ed to obvykle dosahuje jednoho j\u00e1dra, ale to se st\u00e1v\u00e1 extr\u00e9mn\u011b vz\u00e1cn\u011b.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/dcdbc7d70642130949b0c65b43a58f7f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V na\u0161em p\u0159\u00edpad\u011b jsme se rozhodli pro VictoriaMetrics z dob\u0159e zn\u00e1m\u00fdch d\u016fvod\u016f; cht\u011bli jsme \u0161et\u0159it a skute\u010dn\u011b jsme u\u0161et\u0159ili. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/457576073885b9153041589bb2f53ee4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Okam\u017eit\u011b vylu\u010dujeme dva body \u2013 export metrik a vysokou spot\u0159ebu zdroj\u016f. Z\u016fst\u00e1v\u00e1 n\u00e1m rozhodnout o dvou bodech, kter\u00e9 jsme si je\u0161t\u011b ponechali. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/9ee296c7838abd243a893c8e34b60711.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tady hned dod\u00e1m, \u017ee VictoriaMetrics vid\u00edme jako \u00falo\u017ei\u0161t\u011b metrik. Ale proto\u017ee pravd\u011bpodobn\u011b budeme poskytovat VictoriaMetrics jako \u00falo\u017ei\u0161t\u011b pro cel\u00fd Leroy, pot\u0159ebujeme omezit ty, kte\u0159\u00ed budou tento cluster pou\u017e\u00edvat, aby n\u00e1m ho nezdevastovali. <\/p>\n<p><\/p>\n<p>Existuje skv\u011bl\u00e1 funkce, kter\u00e1 umo\u017e\u0148uje omezit \u010das, objem dat a dobu vykon\u00e1v\u00e1n\u00ed. <\/p>\n<p><\/p>\n<p>A tak\u00e9 je zde skv\u011bl\u00e1 mo\u017enost, kter\u00e1 umo\u017e\u0148uje omezit spot\u0159ebu pam\u011bti, \u010d\u00edm\u017e m\u016f\u017eeme nal\u00e9zt ten spr\u00e1vn\u00fd balanc, kter\u00fd n\u00e1m umo\u017en\u00ed dos\u00e1hnout norm\u00e1ln\u00ed rychlosti pr\u00e1ce a p\u0159im\u011b\u0159en\u00e9 spot\u0159eby zdroj\u016f. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/039d8b7f24e50a2e62211eef30ef0812.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dal\u0161\u00ed nev\u00fdhodou je to, \u017ee nem\u016f\u017eeme omezit spot\u0159ebu pam\u011bti.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/7444f68188a835392f8c6ad82c3c26d1.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V prvn\u00edch iterac\u00edch jsme testovali VictoriaMetrics Single Node. Nyn\u00ed p\u0159ech\u00e1z\u00edme na verzi VictoriaMetrics Cluster.<\/p>\n<p><\/p>\n<p>Zde m\u00e1me voln\u00e9 ruce co se t\u00fd\u010de rozd\u011blen\u00ed r\u016fzn\u00fdch slu\u017eeb ve VictoriaMetrics v z\u00e1vislosti na tom, na \u010dem pob\u011b\u017e\u00ed a jak\u00e9 zdroje budou spot\u0159ebov\u00e1vat. Je to velmi flexibiln\u00ed a pohodln\u00e9 \u0159e\u0161en\u00ed, kter\u00e9 jsme sami pou\u017e\u00edvali. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/4bcadbd80d7efcdbcbda6c79929ab279.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Hlavn\u00edmi komponenty verze VictoriaMetrics Cluster jsou vmstorage. M\u016f\u017ee jich b\u00fdt N mno\u017estv\u00ed, v na\u0161em p\u0159\u00edpad\u011b zat\u00edm dva. <\/p>\n<p><\/p>\n<p>A je tu tak\u00e9 vminsert. To je proxy server, kter\u00fd n\u00e1m umo\u017e\u0148uje: prov\u00e1d\u011bt shardov\u00e1n\u00ed mezi v\u0161emi \u00falo\u017ei\u0161ti, o kter\u00fdch jsme mu \u0159ekli, a tak\u00e9 umo\u017e\u0148uje m\u00edt replikaci, tj. bude m\u00edt jak shardov\u00e1n\u00ed, tak replikaci. <\/p>\n<p><\/p>\n<p>Vminsert podporuje protokoly OpenTSDB, Graphite, InfluxDB a remoteWrite od Prometheuse.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/6b4ac44513d7bfa7e0b234aa0f2fa3f5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Je tu tak\u00e9 vmselect. Jeho hlavn\u00edm \u00fakolem je vy\u017e\u00e1dat si data z vmstorage, deduplikovat je a p\u0159edat je klientovi. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/768db0f7dbd17223b7057d33ceccd12d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje skv\u011bl\u00e1 v\u011bc jako vmagent. Opravdu se n\u00e1m l\u00edb\u00ed. Umo\u017e\u0148uje konfigurovat stejn\u011b jako Prometheus a z\u00e1rove\u0148 d\u011bl\u00e1 v\u0161echno p\u0159esn\u011b tak jako Prometheus. Tj. sb\u00edr\u00e1 metriky z r\u016fzn\u00fdch entit a slu\u017eeb a odes\u00edl\u00e1 je do vminsert. D\u00e1le u\u017e z\u00e1le\u017e\u00ed na v\u00e1s.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/429caa4d3296810c688e7c972ab6d35c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dal\u0161\u00ed skv\u011blou slu\u017ebou je vmalert, kter\u00fd m\u00e1 jako backend VictoriaMetrics a z\u00edsk\u00e1v\u00e1 z vminsert a odes\u00edl\u00e1 do vmselect zpracovan\u00e1 data. Zpracov\u00e1v\u00e1 samotn\u00e9 alerty a tak\u00e9 pravidla. V p\u0159\u00edpad\u011b alert\u016f z\u00edsk\u00e1v\u00e1me alert p\u0159es alertmanager.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/67d6264ffd5e061e168261586b4e6369.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Existuje komponent wmauth. Mo\u017en\u00e1 ho budeme pou\u017e\u00edvat, a mo\u017en\u00e1 ne (je\u0161t\u011b jsme se s t\u00edm nerozhodli), jako syst\u00e9m autorizace p\u0159i multitenancy verz\u00edch klastr\u016f. Podporuje remoteWrite pro Prometheus a m\u016f\u017ee autorizovat na z\u00e1klad\u011b url, p\u0159esn\u011bji jeho druh\u00e9 \u010d\u00e1sti, kam v\u00e1m je umo\u017en\u011bno nebo zak\u00e1z\u00e1no ps\u00e1t. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/2a6a97eedc36ce71d660e44e0b736581.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Je tu tak\u00e9 vmbackup a vmrestore. To je v podstat\u011b obnova a z\u00e1lohov\u00e1n\u00ed v\u0161ech dat. Podporuje S3, GCS a soubor.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/ec11f7a981bb80daaf6891001d80a1c9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prvn\u00ed iterace na\u0161eho clusteru byla provedena b\u011bhem karant\u00e9ny. Tehdy nebyla \u017e\u00e1dn\u00e1 replikace, tak\u017ee na\u0161e iterace sest\u00e1vala ze dvou r\u016fzn\u00fdch a nez\u00e1visl\u00fdch cluster\u016f, do kter\u00fdch jsme pomoc\u00ed remoteWrite p\u0159ij\u00edmali data. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/2a76181fc6c0ffb6b09ad5bcdb33909c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Zde bych cht\u011bl uv\u00e9st, \u017ee kdy\u017e jsme p\u0159ech\u00e1zeli z VictoriaMetrics Single Node na VictoriaMetrics Cluster Version, z\u016fstali jsme na stejn\u00e9 spot\u0159eb\u011b zdroj\u016f, tedy hlavn\u011b pam\u011bti. Takto se p\u0159ibli\u017en\u011b rozd\u011blila na\u0161e data, tedy spot\u0159eba zdroj\u016f.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/2d22446669e71b490226a789294707fc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Zde byla ji\u017e p\u0159id\u00e1na replikace. V\u0161echny tyto prvky jsme spojili do jednoho relativn\u011b velk\u00e9ho clusteru. V\u0161echna data jsou jak shardov\u00e1na, tak replikov\u00e1na. <\/p>\n<p><\/p>\n<p>Cel\u00fd cluster m\u00e1 N bod\u016f p\u0159\u00edstupu, tedy Prometheus m\u016f\u017ee p\u0159id\u00e1vat data prost\u0159ednictv\u00edm HAPROXY. Tady m\u00e1me tento bod p\u0159\u00edstupu. A p\u0159es tento bod p\u0159\u00edstupu lze vstupovat pomoc\u00ed Grafany. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/4a797ed22c57cab7c10b2635f53941df.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V na\u0161em p\u0159\u00edpad\u011b je HAPROXY jedin\u00fd port, kter\u00fd zprost\u0159edkov\u00e1v\u00e1 select, insert a dal\u0161\u00ed slu\u017eby uvnit\u0159 tohoto clusteru. Nebylo mo\u017en\u00e9 vytvo\u0159it jednu adresu, museli jsme ud\u011blat n\u011bkolik bod\u016f p\u0159\u00edstupu, proto\u017ee samotn\u00e9 virtu\u00e1ln\u00ed stroje, na kter\u00fdch b\u011b\u017e\u00ed cluster VictoriaMetrics, se nach\u00e1zej\u00ed v r\u016fzn\u00fdch z\u00f3n\u00e1ch jednoho poskytovatele cloudov\u00fdch slu\u017eeb, tedy ne uvnit\u0159 na\u0161eho cloudu, ale venku. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/76bd7e59b58795ceada54977a933fb6d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00e1me alerting. Pou\u017e\u00edv\u00e1me ho. Pou\u017e\u00edv\u00e1me alertmanager od Promethea. Jako kan\u00e1l pro doru\u010dov\u00e1n\u00ed alert\u016f pou\u017e\u00edv\u00e1me Opsgenie a Telegram. Na Telegramu se shroma\u017e\u010fuj\u00ed notifikace od dev, mo\u017en\u00e1 n\u011bco od prod, ale v\u011bt\u0161inou n\u011bco takov\u00e9ho statistick\u00e9ho, pot\u0159ebn\u00e9ho in\u017een\u00fdr\u016fm. Opsgenie p\u0159edstavuje kritick\u00e9 ozn\u00e1men\u00ed. To jsou telefon\u00e1ty, spr\u00e1va incident\u016f. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/b06abbf9eac39450a290e1a776df415f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V\u011b\u010dn\u00fd dotaz: \u201eKdo monitoruje monitoring?\u201c. V na\u0161em p\u0159\u00edpad\u011b monitoring monitoruje s\u00e1m monitoring, proto\u017ee pou\u017e\u00edv\u00e1me vmagent na ka\u017ed\u00e9m uzlu. A proto\u017ee jsou na\u0161e uzly rozlo\u017eeny po r\u016fzn\u00fdch datov\u00fdch centrech jednoho poskytovatele, m\u00e1me pro ka\u017ed\u00e9 datov\u00e9 centrum vlastn\u00ed kan\u00e1l, jsou nez\u00e1visl\u00e9 a i kdy\u017e dojde k rozd\u011blen\u00ed mozku, p\u0159esto z\u00edsk\u00e1me alerty. Ano, bude jich v\u00edc, ale l\u00e9pe dostat v\u00edce alert\u016f ne\u017e \u017e\u00e1dn\u00e9. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/20e94b612f00619015da7e6ebc1e6afb.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Kon\u010d\u00edme n\u00e1\u0161 seznam realizac\u00ed HA.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/f7cac236b8f99d8f2f12288e565c7de0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A d\u00e1le bych cht\u011bl zm\u00ednit zku\u0161enost s komunitou VictoriaMetrics. Ta byla velmi pozitivn\u00ed. Kluci jsou ochotn\u00ed. Sna\u017e\u00ed se pochopit ka\u017ed\u00fd p\u0159\u00edpad, kter\u00fd je p\u0159edlo\u017een. <\/p>\n<p><\/p>\n<p>Vytv\u00e1\u0159el jsem probl\u00e9my na GitHubu. Ty byly velmi rychle vy\u0159e\u0161eny. Jsou je\u0161t\u011b n\u011bjak\u00e9 probl\u00e9my, kter\u00e9 nejsou \u00fapln\u011b uzav\u0159eny, ale u\u017e podle k\u00f3du vid\u00edm, \u017ee se na tom pracuje. <\/p>\n<p><\/p>\n<p>Hlavn\u00ed probl\u00e9m b\u011bhem iterac\u00ed pro m\u011b byl, \u017ee kdy\u017e jsem vypnul uzel, prvn\u00edch 30 sekund mi vminsert nemohl pochopit, \u017ee backend nen\u00ed k dispozici. Toto u\u017e je nyn\u00ed vy\u0159e\u0161eno. A doslova za sekundu nebo dv\u011b jsou data z\u00edsk\u00e1v\u00e1na ze v\u0161ech zb\u00fdvaj\u00edc\u00edch uzl\u016f a dotaz p\u0159est\u00e1v\u00e1 \u010dekat na chyb\u011bj\u00edc\u00ed uzel. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/722ad280977c2f377c5afe36eb309c96.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>V n\u011bjak\u00e9m okam\u017eiku jsme m\u011bli z VictoriaMetrics p\u0159\u00e1n\u00ed, aby to byl VictoriaMetrics operator. \u010cekali jsme na n\u011bj. Nyn\u00ed aktivn\u011b budujeme obal nad VictoriaMetrics operator, abychom vzali v\u0161echna pravidla pro p\u0159edpo\u010d\u00edt\u00e1v\u00e1n\u00ed atd. Prometheus, proto\u017ee dost aktivn\u011b pou\u017e\u00edv\u00e1me pravidla, kter\u00e1 p\u0159ich\u00e1zej\u00ed spolu s operatorhem Prometheus.<\/p>\n<p><\/p>\n<p>Existuj\u00ed n\u00e1vrhy na zlep\u0161en\u00ed klastrov\u00e9 implementace. Vysv\u011btlil jsem je v\u00fd\u0161e. <\/p>\n<p><\/p>\n<p>A tak\u00e9 bych velmi r\u00e1d vid\u011bl downsampling. V na\u0161em p\u0159\u00edpad\u011b je downsampling pot\u0159ebn\u00fd v\u00fdhradn\u011b pro sledov\u00e1n\u00ed trend\u016f. Jin\u00fdmi slovy, jedna metrika b\u011bhem dne mi sta\u010d\u00ed. Tyto trendy pot\u0159ebujeme na rok, na t\u0159i, na p\u011bt, na deset let. A jedno hodnotu metriky je naprosto dostate\u010dn\u00e9.<br \/>\n<img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/01073d8f9b16e1094daef3e36dd97e44.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<ul>\n<li>Poznali jsme bolest, stejn\u011b jako n\u011bkte\u0159\u00ed na\u0161i kolegov\u00e9, p\u0159i pou\u017e\u00edv\u00e1n\u00ed Prometheus.<\/li>\n<li>Vybrali jsme si VictoriaMetrics. <\/li>\n<li>Dostate\u010dn\u011b dob\u0159e se \u0161k\u00e1luje jak vertik\u00e1ln\u011b, tak horizont\u00e1ln\u011b. <\/li>\n<li>M\u016f\u017eeme rozd\u011blit r\u016fzn\u00e9 komponenty na r\u016fzn\u00e9 mno\u017estv\u00ed uzl\u016f v clusterech, omezit je podle velikosti pam\u011bti, p\u0159id\u00e1vat pam\u011b\u0165 atd. <\/li>\n<\/ul>\n<p><\/p>\n<p>Budeme u n\u00e1s pou\u017e\u00edvat VictoriaMetrics, proto\u017ee se n\u00e1m velmi l\u00edb\u00ed. Tohle byla p\u016fvodn\u00ed situace a jak to nyn\u00ed vypad\u00e1. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev\" src=\"\/wp-content\/uploads\/2020\/09\/e04b66e5e2b765143797d3c983d4029c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p><noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/VictoriaMetrics_ru1\">https:\/\/t.me\/VictoriaMetrics_ru1<\/a><\/noindex><\/p>\n<p><\/p>\n<p>Dva qr k\u00f3dy na chat VictoriaMetrics, m\u00e9 kontakty, tech radar LeroyMerlin.<\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/518680\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>VictoriaMetrics \u2014 \u0431\u044b\u0441\u0442\u0440\u0430\u044f \u0438 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u0438\u0440\u0443\u0435\u043c\u0430\u044f \u0421\u0423\u0411\u0414 \u0434\u043b\u044f \u0445\u0440\u0430\u043d\u0435\u043d\u0438\u044f \u0438 \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0434\u0430\u043d\u043d\u044b\u0445 \u0432 \u0444\u043e\u0440\u043c\u0435 \u0432\u0440\u0435\u043c\u0435\u043d\u043d\u043e\u0433\u043e \u0440\u044f\u0434\u0430 (\u0437\u0430\u043f\u0438\u0441\u044c \u043e\u0431\u0440\u0430\u0437\u0443\u0435\u0442 \u0432\u0440\u0435\u043c\u044f \u0438 \u043d\u0430\u0431\u043e\u0440 \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u044e\u0449\u0438\u0445 \u044d\u0442\u043e\u043c\u0443 \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u0437\u043d\u0430\u0447\u0435\u043d\u0438\u0439, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0445 \u0447\u0435\u0440\u0435\u0437 \u043f\u0435\u0440\u0438\u043e\u0434\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u043e\u043f\u0440\u043e\u0441 \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u044f \u0434\u0430\u0442\u0447\u0438\u043a\u043e\u0432 \u0438\u043b\u0438 \u0441\u0431\u043e\u0440 \u043c\u0435\u0442\u0440\u0438\u043a). \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u041a\u043e\u043b\u043e\u0431\u0430\u0435\u0432 \u041f\u0430\u0432\u0435\u043b. DevOps, SRE, LeroyMerlin, \u0432\u0441\u0435 \u043a\u0430\u043a \u043a\u043e\u0434 \u2013 \u044d\u0442\u043e \u0432\u0441\u0435 \u043f\u0440\u043e \u043d\u0430\u0441: \u043f\u0440\u043e \u043c\u0435\u043d\u044f \u0438 \u043f\u0440\u043e \u0434\u0440\u0443\u0433\u0438\u0445 \u0441\u043e\u0442\u0440\u0443\u0434\u043d\u0438\u043a\u043e\u0432 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":94154,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-94153","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47VictoriaMetrics \u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043f\u0440\u0438\u0432\u0430\u0442\u043d\u044b\u0445 \u043e\u0431\u043b\u0430\u043a\u043e\u0432. \u041f\u0430\u0432\u0435\u043b \u041a\u043e\u043b\u043e\u0431\u0430\u0435\u0432 | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-09-13T17:42:30+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-09-13T17:42:30+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47VictoriaMetrics a monitorov\u00e1n\u00ed soukrom\u00fdch cloud\u016f. Pavel Kolobaev | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47VictoriaMetrics \u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043f\u0440\u0438\u0432\u0430\u0442\u043d\u044b\u0445 \u043e\u0431\u043b\u0430\u043a\u043e\u0432. \u041f\u0430\u0432\u0435\u043b \u041a\u043e\u043b\u043e\u0431\u0430\u0435\u0432 | ProHoster","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/victoriametrics-i-monitoring-privatnyh-oblakov-pavel-kolobaev","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-09-13T17:42:30+00:00","article:modified_time":"2020-09-13T17:42:30+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"94153","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 11:32:22","updated":"2022-09-28 11:09:52","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/94153","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=94153"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/94153\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/94154"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=94153"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=94153"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=94153"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}