{"id":32322,"date":"2019-10-31T21:46:21","date_gmt":"2019-10-31T18:46:21","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\/"},"modified":"2019-10-31T21:46:21","modified_gmt":"2019-10-31T18:46:21","slug":"kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","title":{"rendered":"Ako sme vybudovali monitorovanie na Prometheus, Clickhouse a ELK","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Vol\u00e1m sa Anton Baderin. Pracujem v Centre vysok\u00fdch technol\u00f3gi\u00ed a zaober\u00e1m sa syst\u00e9mov\u00fdm administrat\u00edvnym. Pred mesiacom sa skon\u010dila na\u0161a firemn\u00e1 konferencia, kde sme zdie\u013eali nadobudnut\u00e9 sk\u00fasenosti s IT komunitou n\u00e1\u0161ho mesta. Hovoril som o monitorovan\u00ed webov\u00fdch aplik\u00e1ci\u00ed. Materi\u00e1l bol ur\u010den\u00fd pre \u00farove\u0148 junior alebo middle, ktor\u00ed tento proces nevytv\u00e1rali od nuly.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Ako sme vybudovali monitorovanie na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/c181509e6a10dd7562f361f328446d61.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Kame\u0148om \u00farazu, ktor\u00fd le\u017e\u00ed v z\u00e1klade akejko\u013evek monitorovacej s\u00fastavy, s\u00fa podnikov\u00e9 \u00falohy. Monitorovanie len pre zaznamen\u00e1vanie nie je nikomu zauj\u00edmav\u00e9. A \u010do chce podnik? Aby v\u0161etko fungovalo r\u00fdchlo a bez ch\u00fdb. Podnik chce proaktivitu, aby sme sami odhalili probl\u00e9my v prev\u00e1dzke slu\u017eby a maxim\u00e1lne r\u00fdchlo ich rie\u0161ili. To s\u00fa vlastne \u00falohy, ktor\u00e9 som rie\u0161il cel\u00fd minul\u00fd rok na projekte jedn\u00e9ho z na\u0161ich z\u00e1kazn\u00edkov.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2 id=\"o-proekte\">O projekte<\/h2>\n<p><\/p>\n<p>Projekt je jedn\u00fdm z najv\u00e4\u010d\u0161\u00edch programov lojality v krajine. Pom\u00e1hame maloobchodn\u00fdm sie\u0165am zvy\u0161ova\u0165 frekvenciu predaja pomocou r\u00f4znych marketingov\u00fdch n\u00e1strojov, ako s\u00fa bonusov\u00e9 karty. V projekte je celkovo 14 aplik\u00e1ci\u00ed, ktor\u00e9 funguj\u00fa na desiatich serveroch.<\/p>\n<p><\/p>\n<p>Pri veden\u00ed pohovorov som viackr\u00e1t postrehol, \u017ee administr\u00e1tori ne v\u017edy spr\u00e1vne pristupuj\u00fa k monitorovaniu webov\u00fdch aplik\u00e1ci\u00ed: st\u00e1le ve\u013ea z nich sa zameriava na metriky opera\u010dn\u00e9ho syst\u00e9mu a ob\u010das monitoruj\u00fa slu\u017eby.<\/p>\n<p><\/p>\n<p>V mojom pr\u00edpade predt\u00fdm le\u017eala v z\u00e1klade monitorovacej s\u00fastavy z\u00e1kazn\u00edka Icinga. Tento syst\u00e9m nijako nerie\u0161il uveden\u00e9 \u00falohy. \u010casto n\u00e1s klient informoval o probl\u00e9moch a niekedy n\u00e1m proste ch\u00fdbali \u00fadaje, aby sme sa dostali k pr\u00ed\u010dine.<\/p>\n<p><\/p>\n<p>Okrem toho bolo jasn\u00e9, \u017ee \u010fal\u0161\u00ed rozvoj syst\u00e9mu je bezperspekt\u00edvny. Mysl\u00edm, \u017ee t\u00ed, ktor\u00ed s\u00fa obozn\u00e1men\u00ed s Icingou, ma pochopia. Tak\u017ee sme sa rozhodli \u00faplne predeli\u0165 syst\u00e9m monitorovania webov\u00fdch aplik\u00e1ci\u00ed na projekte.<\/p>\n<p><\/p>\n<h2 id=\"prometheus\">Prometheus<\/h2>\n<p><\/p>\n<p>Vybrali sme Prometheus na z\u00e1klade troch hlavn\u00fdch ukazovate\u013eov:<\/p>\n<p><\/p>\n<ol>\n<li>Obrovsk\u00e9 mno\u017estvo dostupn\u00fdch metr\u00edk. V na\u0161om pr\u00edpade ich je 60 tis\u00edc. Samozrejme, stoj\u00ed za to poznamena\u0165, \u017ee v\u00e4\u010d\u0161ina z nich sa nevyu\u017e\u00edva (asi 95%). Na druhej strane, s\u00fa v\u0161etky relat\u00edvne lacn\u00e9. Pre n\u00e1s je to druh\u00fd extr\u00e9m v porovnan\u00ed s predt\u00fdm pou\u017e\u00edvan\u00fdm Icinga. Prid\u00e1vanie metrick\u00fdch \u00fadajov v nej bolo ve\u013emi problematick\u00e9: existuj\u00face boli drah\u00e9 (sta\u010d\u00ed sa pozrie\u0165 na zdrojov\u00fd k\u00f3d ak\u00e9hoko\u013evek pluginu). Ka\u017ed\u00fd plugin bol skriptom nap\u00edsan\u00fdm v Bash alebo Python, ktor\u00fdch spustenie je n\u00e1ro\u010dn\u00e9 z h\u013eadiska spotrebovan\u00fdch zdrojov.<\/li>\n<li>Tento syst\u00e9m spotrebov\u00e1va relat\u00edvne mal\u00e9 mno\u017estvo zdrojov. Na v\u0161etky na\u0161e metriky sta\u010d\u00ed 600 MB opera\u010dnej pam\u00e4te, 15% jedn\u00e9ho jadra a p\u00e1r desiatok IOPS. Samozrejme, treba sp\u00fa\u0161\u0165a\u0165 export\u00e9ry metr\u00edk, ale v\u0161etky s\u00fa nap\u00edsan\u00e9 v Go a tie\u017e sa nevyzna\u010duj\u00fa ve\u013ekou spotrebou. Nemysl\u00edm si, \u017ee je to v s\u00fa\u010dasn\u00fdch podmienkach probl\u00e9m. <\/li>\n<li>Umo\u017e\u0148uje prechod do Kubernetes. Vzh\u013eadom na pl\u00e1ny z\u00e1kazn\u00edka \u2013 vo\u013eba je jasn\u00e1. <\/li>\n<\/ol>\n<p><\/p>\n<h2 id=\"elk\">ELK<\/h2>\n<p><\/p>\n<p>Predt\u00fdm sme logy nesb\u00edrali a nespracov\u00e1vali. Nev\u00fdhody s\u00fa v\u0161etk\u00fdm jasn\u00e9. Vybrali sme ELK, preto\u017ee sme u\u017e mali sk\u00fasenosti s t\u00fdmto syst\u00e9mom. Skladujeme tam iba logy aplik\u00e1ci\u00ed. Hlavn\u00fdmi krit\u00e9riami vo\u013eby boli fulltextov\u00e9 vyh\u013ead\u00e1vanie a jeho r\u00fdchlos\u0165.<\/p>\n<p><\/p>\n<h2 id=\"slickhouse\">Clickhouse<\/h2>\n<p><\/p>\n<p>Na za\u010diatku sme si vybrali InfluxDB. Uvedomovali sme si potrebu zbiera\u0165 logy Nginx, \u0161tatistiky z pg_stat_statements a uchov\u00e1va\u0165 historick\u00e9 d\u00e1ta Prometheus. Influx sa n\u00e1m nep\u00e1\u010dil, preto\u017ee ob\u010das za\u010dal spotrebov\u00e1va\u0165 ve\u013ek\u00e9 mno\u017estvo pam\u00e4te a zr\u00fatil sa. Navy\u0161e, chceli sme skupinova\u0165 dotazy pod\u013ea remote_addr, ale skupinov\u00e1 funkcia v tejto DB je len pod\u013ea tagov. Tagy s\u00fa drah\u00e9 (pam\u00e4\u0165), ich po\u010det je podmiene\u010dne obmedzen\u00fd.<\/p>\n<p><\/p>\n<p>Za\u010dali sme h\u013eada\u0165 odznova. Potrebovali sme analytick\u00fa datab\u00e1zu s minim\u00e1lnou spotrebou zdrojov, ide\u00e1lne so zni\u017eovan\u00edm d\u00e1t na disku.<\/p>\n<p><\/p>\n<p>Clickhouse sp\u013a\u0148a v\u0161etky tieto krit\u00e9ri\u00e1 a nikdy sme svoj v\u00fdber ne\u013eutovali. Nep\u00ed\u0161eme do neho ve\u013ek\u00e9 objemy d\u00e1t (po\u010det vlo\u017een\u00ed je okolo p\u00e4\u0165tis\u00edc za min\u00fatu).<\/p>\n<p><\/p>\n<h2 id=\"newrelic\">NewRelic<\/h2>\n<p><\/p>\n<p>NewRelic historicky bol s nami, preto\u017ee to bol v\u00fdber z\u00e1kazn\u00edka. U n\u00e1s sa pou\u017e\u00edva ako APM.<\/p>\n<p><\/p>\n<h2 id=\"zabbix\">Zabbix<\/h2>\n<p><\/p>\n<p>Zabbix pou\u017e\u00edvame v\u00fdlu\u010dne na monitorovanie Black Box r\u00f4znych API.<\/p>\n<p><\/p>\n<h2 id=\"opredelenie-podhoda-k-monitoringu\">Definovanie pr\u00edstupu k monitorovaniu<\/h2>\n<p><\/p>\n<p>Chceli sme dekomponova\u0165 \u00falohu a t\u00fdm syst\u00e9movo usporiada\u0165 pr\u00edstup k monitorovaniu.<\/p>\n<p><\/p>\n<p>Pre tento \u00fa\u010del som na\u0161u syst\u00e9m rozdelil na nasleduj\u00face \u00farovne:<\/p>\n<p><\/p>\n<ul>\n<li>hardv\u00e9r a VMS;<\/li>\n<li>opera\u010dn\u00fd syst\u00e9m;<\/li>\n<li>syst\u00e9mov\u00e9 slu\u017eby, softv\u00e9rov\u00fd stack;<\/li>\n<li>aplik\u00e1cia;<\/li>\n<li>biznis logika.<\/li>\n<\/ul>\n<p><\/p>\n<p>\u010co je na tomto pr\u00edstupe v\u00fdhodn\u00e9:<\/p>\n<p><\/p>\n<ul>\n<li>vieme, kto je zodpovedn\u00fd za pr\u00e1cu ka\u017edej \u00farovne a na z\u00e1klade toho m\u00f4\u017eeme posiela\u0165 upozornenia;<\/li>\n<li>m\u00f4\u017eeme vyu\u017ei\u0165 \u0161trukt\u00faru pri potl\u00e1\u010dan\u00ed upozornen\u00ed \u2014 bolo by zvl\u00e1\u0161tne posiela\u0165 upozornenie o nedostupnosti datab\u00e1zy, ke\u010f je celkov\u00e1 virtu\u00e1lna ma\u0161ina nedostupn\u00e1.<\/li>\n<\/ul>\n<p><\/p>\n<p>Ke\u010f\u017ee na\u0161ou \u00falohou je odha\u013eova\u0165 poruchy v syst\u00e9me, mus\u00edme na ka\u017edej \u00farovni vy\u010dleni\u0165 ur\u010dit\u00fa sadu metr\u00edk, na ktor\u00e9 by sme mali d\u00e1va\u0165 pozor pri p\u00edsan\u00ed pravidiel pre upozornenia. Teraz prejdeme \u00farov\u0148ami \u201eVMS\u201c, \u201eOpera\u010dn\u00fd syst\u00e9m\u201c a \u201eSyst\u00e9mov\u00e9 slu\u017eby, softv\u00e9rov\u00fd stack.\u201c<\/p>\n<p><\/p>\n<h2 id=\"virtualnye-mashiny\">Virtu\u00e1lne stroje<\/h2>\n<p><\/p>\n<p><a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sk\/\"   title=\"Hosting\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"2329\">Hosting<\/a> vy\u010dle\u0148uje n\u00e1m procesor, disk, pam\u00e4\u0165 a sie\u0165. A s prv\u00fdmi dvoma sme mali probl\u00e9my. Tak\u017ee, metriky:<\/p>\n<p><\/p>\n<p>CPU stolen time \u2014 ke\u010f si k\u00fapite virtu\u00e1lny server na Amazone (napr\u00edklad t2.micro), mali by ste ch\u00e1pa\u0165, \u017ee v\u00e1m nie je pridelen\u00e9 cel\u00e9 jadro procesora, ale len kv\u00f3ta jeho \u010dasu. A ke\u010f ju vy\u010derp\u00e1te, procesor za\u010dne by\u0165 zaberan\u00fd.<\/p>\n<p><\/p>\n<p>T\u00e1to metrika umo\u017e\u0148uje sledova\u0165 tak\u00e9to momenty a prij\u00edma\u0165 rozhodnutia. Napr\u00edklad, \u010di je potrebn\u00e9 zobra\u0165 si v\u00fdkonn\u011bj\u0161\u00ed pl\u00e1n alebo rozdeli\u0165 spracovanie pozad\u00ed a po\u017eiadaviek na API na r\u00f4zne <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sk\/server\/dts-los-angeles\/\"   title=\"servera\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"3493\">servera<\/a>.<\/p>\n<p><\/p>\n<p>IOPS + CPU iowait time \u2014 prekvapivo, mnoh\u00e9 cloudov\u00e9 hostingy sa hriechom nedod\u00e1vaj\u00fa IOPS. Navy\u0161e, graf s n\u00edzkymi IOPS pre nich nie je argument. Preto je d\u00f4le\u017eit\u00e9 zbiera\u0165 aj CPU iowait. S touto dvojicou grafov \u2014 s n\u00edzkymi IOPS a vysok\u00fdm \u010dakan\u00edm na vstup a v\u00fdstup \u2014 u\u017e je mo\u017en\u00e9 sa rozpr\u00e1va\u0165 s hostingom a rie\u0161i\u0165 probl\u00e9m.<\/p>\n<p><\/p>\n<h2 id=\"operacionnaya-sistema\">Opera\u010dn\u00fd syst\u00e9m<\/h2>\n<p><\/p>\n<p>Metriky opera\u010dn\u00e9ho syst\u00e9mu:<\/p>\n<p><\/p>\n<ul>\n<li>percento dostupnej pam\u00e4te;<\/li>\n<li>akt\u00edvnos\u0165 pou\u017e\u00edvania swap: vmstat swapin, swapout;<\/li>\n<li>po\u010det dostupn\u00fdch inode a vo\u013en\u00e9ho miesta v s\u00faborovom syst\u00e9me v %<\/li>\n<li>priemern\u00e9 za\u0165a\u017eenie;<\/li>\n<li>po\u010det spojen\u00ed v stave tw;<\/li>\n<li>zaplnilos\u0165 tabu\u013eky conntrack;<\/li>\n<li>kvalitu pr\u00e1ce siete je mo\u017en\u00e9 sledova\u0165 pomocou utility ss, bal\u00edkom iproute2 \u2014 z\u00edska\u0165 z jej v\u00fdstupu hodnotu RTT spojen\u00ed a skupinova\u0165 pod\u013ea dest-portu.<\/li>\n<\/ul>\n<p><\/p>\n<p>Takisto na \u00farovni opera\u010dn\u00e9ho syst\u00e9mu n\u00e1m vznik\u00e1 tak\u00e1 entita, ako s\u00fa procesy. Je d\u00f4le\u017eit\u00e9 vy\u010dleni\u0165 v syst\u00e9me s\u00fabor procesov, ktor\u00e9 hraj\u00fa d\u00f4le\u017eit\u00fa \u00falohu v jeho pr\u00e1ci. Ak m\u00e1te nieko\u013eko pgpool, je nevyhnutn\u00e9 zbiera\u0165 inform\u00e1cie o ka\u017edom z nich.<\/p>\n<p><\/p>\n<p>S\u00fabor metr\u00edk je nasleduj\u00faci:<\/p>\n<p><\/p>\n<ul>\n<li>CPU;<\/li>\n<li>pam\u00e4\u0165 \u2014 predov\u0161etk\u00fdm rezidentn\u00e1;<\/li>\n<li>IO \u2014 ide\u00e1lne v IOPS;<\/li>\n<li>FileFd \u2014 otvoren\u00e9 a limit;<\/li>\n<li>v\u00fdznamn\u00e9 zlyhania str\u00e1nky \u2014 tak budete m\u00f4c\u0165 pochopi\u0165, ktor\u00fd proces sa swapuje.<\/li>\n<\/ul>\n<p><\/p>\n<p>Cel\u00fd monitoring je u n\u00e1s rozvinut\u00fd v Dockeri, na zber d\u00e1tov\u00fdch metr\u00edk pou\u017e\u00edvame Cadvisor. Na ostatn\u00fdch strojoch aplikujeme process-exporter.<\/p>\n<p><\/p>\n<h2 id=\"sistemnye-servisy-stek-po\">Syst\u00e9mov\u00e9 slu\u017eby, softv\u00e9rov\u00fd stack<\/h2>\n<p><\/p>\n<p>Ka\u017ed\u00e1 aplik\u00e1cia m\u00e1 svoju \u0161pecifiku a je \u0165a\u017ek\u00e9 vy\u010dleni\u0165 nejak\u00fd s\u00fabor metr\u00edk.<\/p>\n<p><\/p>\n<p>Univerz\u00e1ln\u00fdm s\u00faborom s\u00fa:<\/p>\n<p><\/p>\n<ul>\n<li>miera po\u017eiadaviek;<\/li>\n<li>po\u010det ch\u00fdb;<\/li>\n<li>latencia;<\/li>\n<li>saturation.<\/li>\n<\/ul>\n<p><\/p>\n<p>Najv\u00fdraznej\u0161ie pr\u00edklady monitorovania tejto \u00farovne u n\u00e1s \u2014 Nginx a PostgreSQL.<\/p>\n<p><\/p>\n<p>Najviac za\u0165a\u017eenou slu\u017ebou v na\u0161om syst\u00e9me je datab\u00e1za. V minulosti sme mali dostato\u010dne \u010dasto probl\u00e9my s t\u00fdm, aby sme zistili, \u010domu sa datab\u00e1za venuje.<\/p>\n<p><\/p>\n<p>Videli sme vysok\u00e9 za\u0165a\u017eenie diskov, ale slow logy ni\u010d podstatn\u00e9 neukazovali. Tento probl\u00e9m sme vyrie\u0161ili pomocou pg_stat_statements, poh\u013eadu, v ktorom sa zbiera \u0161tatistika po\u017eiadaviek.<\/p>\n<p><\/p>\n<p>To je v\u0161etko, \u010do potrebuje administr\u00e1tor.<\/p>\n<p><\/p>\n<p>Staviame grafy aktivity po\u017eiadaviek na \u010d\u00edtanie a z\u00e1pis:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Ako sme vybudovali monitorovanie na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/f838b7759221b284c06941291fa186df.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Ako sme vybudovali monitorovanie na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/ea8684ea81f21c8d762a606aeb8342a3.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>V\u0161etko je jednoduch\u00e9 a pochopite\u013en\u00e9, ka\u017edej po\u017eiadavke \u2014 vlastn\u00e1 farba.<\/p>\n<p><\/p>\n<p>Rovnako v\u00fdrazn\u00fdm pr\u00edkladom s\u00fa Nginx logy. Nie je prekvapen\u00edm, \u017ee m\u00e1lo kto ich spracov\u00e1va alebo spom\u00edna v zozname povinn\u00fdch. \u0160tandardn\u00fd form\u00e1t nie je ve\u013emi informat\u00edvny a je potrebn\u00e9 ho roz\u0161\u00edri\u0165.<\/p>\n<p><\/p>\n<p>Osobne som pridal request_time, upstream_response_time, body_bytes_sent, request_length, request_id. Staviame grafy \u010dasu odozvy a po\u010dtu ch\u00fdb:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Ako sme vybudovali monitorovanie na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/e517f15303f8fa780d40b7f7e30eb9ef.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Ako sme vybudovali monitorovanie na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/9a07ffda75300ec939edff24532fc294.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Staviame grafy \u010dasu odozvy a po\u010dtu ch\u00fdb. Pam\u00e4t\u00e1te? Hovoril som o \u00faloh\u00e1ch podnikania? Aby to bolo r\u00fdchlo a bez ch\u00fdb? Tieto ot\u00e1zky sme u\u017e dvoma grafmi vyrie\u0161ili. A pod\u013ea nich u\u017e m\u00f4\u017eeme vola\u0165 pohotov\u00fdm administr\u00e1torom.<\/p>\n<p><\/p>\n<p>Ale zostal e\u0161te jeden probl\u00e9m \u2014 zabezpe\u010di\u0165 r\u00fdchle odstr\u00e1nenie pr\u00ed\u010din incidentu.<\/p>\n<p><\/p>\n<h2 id=\"ustranenie-incidentov\">Odstr\u00e1nenie incidentov<\/h2>\n<p><\/p>\n<p>Cel\u00fd proces od identifik\u00e1cie po rie\u0161enie probl\u00e9mu je mo\u017en\u00e9 rozdeli\u0165 na nieko\u013eko krokov:<\/p>\n<p><\/p>\n<ul>\n<li>identifik\u00e1cia probl\u00e9mu;<\/li>\n<li>notifik\u00e1cia pohotov\u00e9ho administr\u00e1tora;<\/li>\n<li>reakcia na incident;<\/li>\n<li>odstr\u00e1nenie pr\u00ed\u010din.<\/li>\n<\/ul>\n<p><\/p>\n<p>Je d\u00f4le\u017eit\u00e9, aby sme to robili maxim\u00e1lne r\u00fdchlo. A ak na f\u00e1zach identifik\u00e1cie probl\u00e9mu a odoslania notifik\u00e1cie nem\u00f4\u017eeme z\u00edska\u0165 ve\u013ea \u010dasu \u2014 dve min\u00faty na ne v ka\u017edom pr\u00edpade uzm\u00fa, tak nasleduj\u00face \u2014 to je jednoducho neoran\u00e9 pole na zlep\u0161enia.<\/p>\n<p><\/p>\n<p>Predstavte si, \u017ee telefon zavel\u00ed slu\u017ebe. \u010co bude robi\u0165? H\u013eada\u0165 odpovede na ot\u00e1zky \u2014 \u010do sa pokazilo, kde sa to stalo, ako reagova\u0165? Tu s\u00fa sp\u00f4soby, ako na tieto ot\u00e1zky odpoved\u00e1me:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Ako sme vybudovali monitorovanie na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/7f33172922e646a24bb83733350b24f5.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Jednoducho zahrnieme v\u0161etky tieto inform\u00e1cie do textu upozornenia a poskytneme v \u0148om odkaz na str\u00e1nku v wiki, kde je pop\u00edsan\u00e9, ako reagova\u0165 na tento probl\u00e9m, ako ho vyrie\u0161i\u0165 a eskalova\u0165.<\/p>\n<p><\/p>\n<p>E\u0161te som ni\u010d nepovedal o \u00farovni aplik\u00e1ci\u00ed a obchodnej logiky. Bohu\u017eia\u013e, v na\u0161ich aplik\u00e1ci\u00e1ch zatia\u013e neexistuje zber metr\u00edk. Jedin\u00fdm zdrojom ak\u00fdchko\u013evek inform\u00e1ci\u00ed z t\u00fdchto \u00farovn\u00ed s\u00fa logy.<\/p>\n<p><\/p>\n<p>P\u00e1r bodov.<\/p>\n<p><\/p>\n<p>Po prv\u00e9, p\u00ed\u0161te \u0161trukt\u00farovan\u00e9 logy. Netreba zah\u0155\u0148a\u0165 kontext do textu spr\u00e1vy. S\u0165a\u017euje to ich skupinovanie a anal\u00fdzu. Logstash potrebuje ve\u013ea \u010dasu na to, aby to v\u0161etko normalizoval.<\/p>\n<p><\/p>\n<p>Po druh\u00e9, spr\u00e1vne pou\u017e\u00edvajte \u00farovne z\u00e1va\u017enosti. Ka\u017ed\u00fd jazyk m\u00e1 svoj \u0161tandard. Osobne rozde\u013eujem na \u0161tyri \u00farovne:<\/p>\n<p><\/p>\n<ol>\n<li>ch\u00fdba chyba;<\/li>\n<li>chyba na strane klienta;<\/li>\n<li>chyba na na\u0161ej strane, nestr\u00e1came peniaze, neriskujeme;<\/li>\n<li>chyba na na\u0161ej strane, str\u00e1came peniaze.<\/li>\n<\/ol>\n<p><\/p>\n<p>Na z\u00e1ver. Sna\u017ete sa budova\u0165 monitorovanie presne od obchodnej logiky. Sna\u017ete sa monitorova\u0165 samotn\u00fa aplik\u00e1ciu a pracova\u0165 u\u017e s tak\u00fdmi metrikami, ako je po\u010det predajov, po\u010det nov\u00fdch registr\u00e1ci\u00ed pou\u017e\u00edvate\u013eov, po\u010det akt\u00edvnych pou\u017e\u00edvate\u013eov v danom okamihu a tak \u010falej.<\/p>\n<p><\/p>\n<p>Ak je cel\u00e9 va\u0161e podnikanie len jedno tla\u010didlo v prehliada\u010di, je potrebn\u00e9 sledova\u0165, \u010di je stla\u010den\u00e9, \u010di funguje spr\u00e1vne. V\u0161etko ostatn\u00e9 nie je d\u00f4le\u017eit\u00e9.<\/p>\n<p><\/p>\n<p>Ak to nem\u00e1te, m\u00f4\u017eete sa pok\u00fasi\u0165 to dohna\u0165 v logoch aplik\u00e1cie, logoch Nginx a tak \u010falej, ako sme to urobili my. Mus\u00edte by\u0165 \u010do najbli\u017e\u0161ie k aplik\u00e1cii.<\/p>\n<p><\/p>\n<p>Metriky opera\u010dn\u00e9ho syst\u00e9mu s\u00fa samozrejme d\u00f4le\u017eit\u00e9, ale obchod ich nezauj\u00edma, my nedost\u00e1vame zaplaten\u00e9 za ne.<\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/449352\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c. \u041c\u0435\u0441\u044f\u0446 \u043d\u0430\u0437\u0430\u0434 \u0437\u0430\u0432\u0435\u0440\u0448\u0438\u043b\u0430\u0441\u044c \u043d\u0430\u0448\u0430 \u043a\u043e\u0440\u043f\u043e\u0440\u0430\u0442\u0438\u0432\u043d\u0430\u044f \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u044f, \u0433\u0434\u0435 \u043c\u044b \u0434\u0435\u043b\u0438\u043b\u0438\u0441\u044c \u043d\u0430\u043a\u043e\u043f\u043b\u0435\u043d\u043d\u044b\u043c \u043e\u043f\u044b\u0442\u043e\u043c \u0441 IT-\u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u043e\u043c \u043d\u0430\u0448\u0435\u0433\u043e \u0433\u043e\u0440\u043e\u0434\u0430. \u042f \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u043b \u043f\u0440\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0432\u0435\u0431-\u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439. \u041c\u0430\u0442\u0435\u0440\u0438\u0430\u043b \u043f\u0440\u0435\u0434\u043d\u0430\u0437\u043d\u0430\u0447\u0430\u043b\u0441\u044f \u0434\u043b\u044f \u0443\u0440\u043e\u0432\u043d\u044f junior \u0438\u043b\u0438 middle, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043d\u0435 \u0432\u044b\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u0438 \u044d\u0442\u043e\u0442 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0441 \u043d\u0443\u043b\u044f. \u041a\u0440\u0430\u0435\u0443\u0433\u043e\u043b\u044c\u043d\u044b\u0439 \u043a\u0430\u043c\u0435\u043d\u044c, \u043b\u0435\u0436\u0430\u0449\u0438\u0439 \u0432 \u043e\u0441\u043d\u043e\u0432\u0435 \u043b\u044e\u0431\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24139,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32322","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043d\u0430 Prometheus, Clickhouse \u0438 ELK | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T18:46:21+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T18:46:21+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Ako sme budovali monitorovanie na Prometheus, Clickhouse a ELK | ProHoster","description":"Vol\u00e1m sa Anton Baderin. Pracujem v Centre Vysok\u00fdch Technol\u00f3gi\u00ed a zaober\u00e1m sa syst\u00e9mov\u00fdm administrat\u00edvou.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043d\u0430 Prometheus, Clickhouse \u0438 ELK | ProHoster","og:description":"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:46:21+00:00","article:modified_time":"2019-10-31T18:46:21+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"32322","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-02-22 15:34:41","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:00:23","updated":"2026-02-22 15:34:41","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/32322","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=32322"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/32322\/revisions"}],"predecessor-version":[{"id":162019,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/32322\/revisions\/162019"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/24139"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=32322"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=32322"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=32322"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}