{"id":32322,"date":"2019-10-31T21:46:21","date_gmt":"2019-10-31T18:46:21","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\/"},"modified":"2019-10-31T21:46:21","modified_gmt":"2019-10-31T18:46:21","slug":"kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","title":{"rendered":"Jak jsme vybudovali monitorov\u00e1n\u00ed na Prometheus, Clickhouse a ELK","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Jmenuji se Anton Baderin. Pracuji v Centru vysok\u00fdch technologi\u00ed a zab\u00fdv\u00e1m se syst\u00e9mov\u00fdm administrativou. P\u0159ed m\u011bs\u00edcem skon\u010dila na\u0161e firemn\u00ed konference, kde jsme sd\u00edleli nabyt\u00e9 zku\u0161enosti s IT komunitou na\u0161eho m\u011bsta. Mluvil jsem o monitorov\u00e1n\u00ed webov\u00fdch aplikac\u00ed. Materi\u00e1l byl ur\u010den pro \u00farove\u0148 junior nebo middle, kte\u0159\u00ed je\u0161t\u011b nebudovali tento proces od z\u00e1kladu.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Jak jsme vybudovali monitorov\u00e1n\u00ed na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/c181509e6a10dd7562f361f328446d61.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Z\u00e1kladn\u00ed k\u00e1men, na kter\u00e9m stoj\u00ed jak\u00fdkoli syst\u00e9m monitorov\u00e1n\u00ed, je \u0159e\u0161en\u00ed obchodn\u00edch probl\u00e9m\u016f. Monitorov\u00e1n\u00ed pro samotn\u00e9 monitorov\u00e1n\u00ed nikoho nezaj\u00edm\u00e1. Co chce byznys? Aby v\u0161e fungovalo rychle a bez chyb. Byznys chce proaktivitu, aby jsme sami odhalovali probl\u00e9my v pr\u00e1ci slu\u017eby a maxim\u00e1ln\u011b rychle je odstra\u0148ovali. To jsou v podstat\u011b \u00fakoly, kter\u00e9 jsem \u0159e\u0161il cel\u00fd minul\u00fd rok na projektu jednoho z na\u0161ich klient\u016f.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2 id=\"o-proekte\">O projektu<\/h2>\n<p><\/p>\n<p>Projekt je jedn\u00edm z nejv\u011bt\u0161\u00edch program\u016f loajality v zemi. Pom\u00e1h\u00e1me maloobchodn\u00edm s\u00edt\u00edm zvy\u0161ovat frekvenci prodeje za pomoci r\u016fzn\u00fdch marketingov\u00fdch n\u00e1stroj\u016f, jako jsou bonusov\u00e9 karty. Celkem projekt zahrnuje 14 aplikac\u00ed, kter\u00e9 b\u011b\u017e\u00ed na deseti serverech.<\/p>\n<p><\/p>\n<p>P\u0159i veden\u00ed pohovor\u016f jsem opakovan\u011b zaznamenal, \u017ee administr\u00e1to\u0159i \u010dasto nep\u0159istupuj\u00ed spr\u00e1vn\u011b k monitorov\u00e1n\u00ed webov\u00fdch aplikac\u00ed: st\u00e1le mnoz\u00ed zastavuj\u00ed na metrik\u00e1ch opera\u010dn\u00edho syst\u00e9mu a jen ob\u010das monitoruj\u00ed slu\u017eby.<\/p>\n<p><\/p>\n<p>V m\u00e9m p\u0159\u00edpad\u011b d\u0159\u00edve na syst\u00e9mu monitorov\u00e1n\u00ed klienta le\u017eela Icinga. Ta nijak ne\u0159e\u0161ila v\u00fd\u0161e zm\u00edn\u011bn\u00e9 \u00fakoly. Klient n\u00e1m \u010dasto s\u00e1m hl\u00e1sil probl\u00e9my a nejednou n\u00e1m prost\u011b chyb\u011bla data, abychom se dostali k p\u0159\u00ed\u010din\u011b.<\/p>\n<p><\/p>\n<p>Krom\u011b toho bylo jasn\u00e9, \u017ee jej\u00ed dal\u0161\u00ed rozvoj je bezperspektivn\u00ed. Mysl\u00edm, \u017ee ti, kdo znaj\u00ed Icingu, m\u011b pochop\u00ed. Tak\u017ee jsme se rozhodli kompletn\u011b p\u0159epracovat syst\u00e9m monitorov\u00e1n\u00ed webov\u00fdch aplikac\u00ed v projektu.<\/p>\n<p><\/p>\n<h2 id=\"prometheus\">Prometheus<\/h2>\n<p><\/p>\n<p>Vybrali jsme Prometheus na z\u00e1klad\u011b t\u0159\u00ed hlavn\u00edch ukazatel\u016f:<\/p>\n<p><\/p>\n<ol>\n<li>Obrovsk\u00e9 mno\u017estv\u00ed dostupn\u00fdch metrik. V na\u0161em p\u0159\u00edpad\u011b je jich 60 tis\u00edc. Samoz\u0159ejm\u011b, stoj\u00ed za zm\u00ednku, \u017ee p\u0159ev\u00e1\u017en\u00e1 v\u011bt\u0161ina z nich nen\u00ed vyu\u017e\u00edv\u00e1na (asi 95 %). Na druhou stranu, v\u0161echny jsou relativn\u011b levn\u00e9. Pro n\u00e1s je to druh\u00fd extr\u00e9m ve srovn\u00e1n\u00ed s d\u0159\u00edve pou\u017e\u00edvan\u00fdm Icinga. P\u0159id\u00e1v\u00e1n\u00ed metrik v n\u00ed bylo zvl\u00e1\u0161\u0165 bolestiv\u00e9: existuj\u00edc\u00ed vy\u017eadovaly vysok\u00e9 n\u00e1klady (sta\u010d\u00ed se pod\u00edvat na zdrojov\u00e9 k\u00f3dy jak\u00e9hokoli pluginu). Ka\u017ed\u00fd plugin p\u0159edstavoval skript v Bash nebo Pythonu, jeho\u017e spu\u0161t\u011bn\u00ed nen\u00ed levn\u00e9 z hlediska spot\u0159ebovan\u00fdch zdroj\u016f.<\/li>\n<li>Tento syst\u00e9m spot\u0159ebov\u00e1v\u00e1 relativn\u011b mal\u00e9 mno\u017estv\u00ed zdroj\u016f. Na v\u0161echny na\u0161e metriky sta\u010d\u00ed 600 MB opera\u010dn\u00ed pam\u011bti, 15 % jednoho j\u00e1dra a p\u00e1r des\u00edtek IOPS. Samoz\u0159ejm\u011b, mus\u00edme spou\u0161t\u011bt export\u00e9ry metrik, ale v\u0161echny jsou naps\u00e1ny v Go a tak\u00e9 se nevyzna\u010duj\u00ed velkou n\u00e1ro\u010dnost\u00ed. Nemysl\u00edm, \u017ee by to v dne\u0161n\u00edch re\u00e1li\u00edch p\u0159edstavovalo probl\u00e9m. <\/li>\n<li>Umo\u017e\u0148uje p\u0159echod na Kubernetes. Vzhledem k pl\u00e1n\u016fm z\u00e1kazn\u00edka je v\u00fdb\u011br z\u0159ejm\u00fd. <\/li>\n<\/ol>\n<p><\/p>\n<h2 id=\"elk\">ELK<\/h2>\n<p><\/p>\n<p>D\u0159\u00edve jsme logy nesb\u00edrali a nezpracov\u00e1vali. Nedostatky jsou v\u0161em jasn\u00e9. Vybrali jsme ELK, proto\u017ee jsme ji\u017e m\u011bli zku\u0161enosti s t\u00edmto syst\u00e9mem. Uchov\u00e1v\u00e1me tam pouze logy aplikac\u00ed. Hlavn\u00edmi kriterii v\u00fdb\u011bru byly fulltextov\u00e9 vyhled\u00e1v\u00e1n\u00ed a jeho rychlost.<\/p>\n<p><\/p>\n<h2 id=\"slickhouse\">Clickhouse<\/h2>\n<p><\/p>\n<p>P\u016fvodn\u011b jsme se rozhodli pro InfluxDB. Uv\u011bdomovali jsme si pot\u0159ebu sb\u00edrat logy Nginx, statistiky z pg_stat_statements a uchov\u00e1vat historick\u00e1 data Promethea. Influx n\u00e1s neoslovil, proto\u017ee ob\u010das za\u010dal spot\u0159ebov\u00e1vat velk\u00e9 mno\u017estv\u00ed pam\u011bti a padal. Nav\u00edc jsme cht\u011bli seskupovat dotazy podle remote_addr, ale seskupov\u00e1n\u00ed v t\u00e9to DB je mo\u017en\u00e9 pouze podle tag\u016f. Tagy jsou drah\u00e9 (pam\u011b\u0165), jejich po\u010det je podm\u00edn\u011bn\u011b omezen.<\/p>\n<p><\/p>\n<p>Znovu jsme za\u010dali hledat. Pot\u0159ebovali jsme analytickou datab\u00e1zi s minim\u00e1ln\u00ed spot\u0159ebou zdroj\u016f, ide\u00e1ln\u011b se stla\u010den\u00edm dat na disku.<\/p>\n<p><\/p>\n<p>Clickhouse spl\u0148uje v\u0161echna tato krit\u00e9ria, a nikdy jsme nelitovali v\u00fdb\u011bru. Nepisujeme do n\u011bj \u017e\u00e1dn\u00e9 z\u00e1sadn\u00ed objemy dat (po\u010det vlo\u017een\u00ed je pouze kolem p\u011bti tis\u00edc za minutu).<\/p>\n<p><\/p>\n<h2 id=\"newrelic\">NewRelic<\/h2>\n<p><\/p>\n<p>NewRelic byl historicky s n\u00e1mi, proto\u017ee to byla volba z\u00e1kazn\u00edka. Pou\u017e\u00edv\u00e1me ho jako APM.<\/p>\n<p><\/p>\n<h2 id=\"zabbix\">Zabbix<\/h2>\n<p><\/p>\n<p>Zabbix vyu\u017e\u00edv\u00e1me v\u00fdhradn\u011b pro monitorov\u00e1n\u00ed Black Box r\u016fzn\u00fdch API.<\/p>\n<p><\/p>\n<h2 id=\"opredelenie-podhoda-k-monitoringu\">Definov\u00e1n\u00ed p\u0159\u00edstupu k monitorov\u00e1n\u00ed<\/h2>\n<p><\/p>\n<p>Cht\u011bli jsme \u00fakol dekomponovat a t\u00edm systematizovat p\u0159\u00edstup k monitorov\u00e1n\u00ed.<\/p>\n<p><\/p>\n<p>Pro tento \u00fa\u010del jsem na\u0161i syst\u00e9m roz\u010dlenil na n\u00e1sleduj\u00edc\u00ed \u00farovn\u011b:<\/p>\n<p><\/p>\n<ul>\n<li>\u201ehardware\u201c a VMS;<\/li>\n<li>opera\u010dn\u00ed syst\u00e9m;<\/li>\n<li>syst\u00e9mov\u00e9 slu\u017eby, softwarov\u00fd stack;<\/li>\n<li>aplikace;<\/li>\n<li>obchodn\u00ed logika.<\/li>\n<\/ul>\n<p><\/p>\n<p>Co\u017e je v\u00fdhodn\u00e9:<\/p>\n<p><\/p>\n<ul>\n<li>v\u00edme, kdo je odpov\u011bdn\u00fd za funkci ka\u017ed\u00e9 z \u00farovn\u00ed a na z\u00e1klad\u011b toho m\u016f\u017eeme pos\u00edlat upozorn\u011bn\u00ed;<\/li>\n<li>m\u016f\u017eeme pou\u017e\u00edt strukturu p\u0159i potla\u010dov\u00e1n\u00ed upozorn\u011bn\u00ed \u2014 bylo by podivn\u00e9 pos\u00edlat upozorn\u011bn\u00ed o nedostupnosti datab\u00e1ze, kdy\u017e je cel\u00e1 virtu\u00e1ln\u00ed stroj nedostupn\u00fd.<\/li>\n<\/ul>\n<p><\/p>\n<p>Jeliko\u017e je na\u0161\u00edm \u00fakolem odhalovat poruchy v syst\u00e9mu, mus\u00edme na ka\u017ed\u00e9 \u00farovni ur\u010dit ur\u010dit\u00fd soubor metrik, na kter\u00e9 bychom se m\u011bli zam\u011b\u0159it p\u0159i tvorb\u011b pravidel pro upozorn\u011bn\u00ed. D\u00e1le se pod\u00edv\u00e1me na \u00farovn\u011b \u201eVMS\u201c, \u201eOpera\u010dn\u00ed syst\u00e9m\u201c a \u201eSyst\u00e9mov\u00e9 slu\u017eby, softwarov\u00fd stack\u201c.<\/p>\n<p><\/p>\n<h2 id=\"virtualnye-mashiny\">Virtu\u00e1ln\u00ed stroje<\/h2>\n<p><\/p>\n<p><a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/cs\/\"   title=\"Hosting\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"2329\">Hosting<\/a> vymezuje n\u00e1m procesor, disk, pam\u011b\u0165 a s\u00ed\u0165. A s prvn\u00edmi dv\u011bma jsme m\u011bli probl\u00e9my. Tak\u017ee, metriky:<\/p>\n<p><\/p>\n<p>CPU stolen time \u2014 kdy\u017e si kupujete virtu\u00e1ln\u00ed stroj na Amazonu (nap\u0159. t2.micro), m\u011bli byste pochopit, \u017ee v\u00e1m nen\u00ed p\u0159id\u011bleno cel\u00e9 j\u00e1dro procesoru, ale pouze kv\u00f3ta jeho \u010dasu. A kdy\u017e tuto kv\u00f3tu vy\u010derp\u00e1te, procesor v\u00e1m za\u010dne odeb\u00edrat.<\/p>\n<p><\/p>\n<p>Tato metrika umo\u017e\u0148uje sledovat takov\u00e9 chv\u00edle a p\u0159ij\u00edmat rozhodnut\u00ed. Nap\u0159\u00edklad, jestli se vyplat\u00ed p\u0159ej\u00edt na dra\u017e\u0161\u00ed tarif, nebo rozlo\u017eit zpracov\u00e1n\u00ed pozadov\u00fdch \u00faloh a API dotaz\u016f na r\u016fzn\u00e9 <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/cs\/server\/dts-los-angeles\/\"   title=\"serveru\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"3493\">serveru<\/a>.<\/p>\n<p><\/p>\n<p>IOPS + CPU iowait time \u2014 z n\u011bjak\u00e9ho d\u016fvodu mnoho cloudov\u00fdch hosting\u016f trp\u00ed t\u00edm, \u017ee nedostate\u010dn\u011b poskytuj\u00ed IOPS. Nav\u00edc, graf s n\u00edzk\u00fdmi IOPS pro n\u011b nen\u00ed argumentem. Proto je dobr\u00e9 sb\u00edrat tak\u00e9 CPU iowait. S touto dvojic\u00ed graf\u016f \u2014 s n\u00edzk\u00fdmi IOPS a vysok\u00fdm \u010dek\u00e1n\u00edm na vstup\/v\u00fdstup \u2014 je mo\u017en\u00e9 v\u00e9st diskusi s hostingem a \u0159e\u0161it probl\u00e9m.<\/p>\n<p><\/p>\n<h2 id=\"operacionnaya-sistema\">Opera\u010dn\u00ed syst\u00e9m<\/h2>\n<p><\/p>\n<p>Metriky opera\u010dn\u00edho syst\u00e9mu:<\/p>\n<p><\/p>\n<ul>\n<li>celkov\u00e9 dostupn\u00e9 pam\u011bti v %;<\/li>\n<li>aktivity vyu\u017e\u00edv\u00e1n\u00ed swapu: vmstat swapin, swapout;<\/li>\n<li>po\u010det dostupn\u00fdch inode a voln\u00e9ho m\u00edsta na souborov\u00e9m syst\u00e9mu v %;<\/li>\n<li>pr\u016fm\u011brn\u00e9 zat\u00ed\u017een\u00ed;<\/li>\n<li>po\u010det p\u0159ipojen\u00ed ve stavu tw;<\/li>\n<li>zat\u00ed\u017een\u00ed tabulky conntrack;<\/li>\n<li>kvalitu pr\u00e1ce s\u00edt\u011b je mo\u017en\u00e9 monitorovat pomoc\u00ed utilit ss, bal\u00edku iproute2 \u2014 z\u00edskat z jeho v\u00fdstupu ukazatele RTT p\u0159ipojen\u00ed a seskupovat je podle c\u00edlov\u00e9ho portu.<\/li>\n<\/ul>\n<p><\/p>\n<p>Tak\u00e9 na \u00farovni opera\u010dn\u00edho syst\u00e9mu m\u00e1me entitu jako procesy. Je d\u016fle\u017eit\u00e9 ur\u010dit v syst\u00e9mu sadu proces\u016f, kter\u00e9 hraj\u00ed d\u016fle\u017eitou roli v jeho fungov\u00e1n\u00ed. Pokud m\u00e1te nap\u0159\u00edklad n\u011bkolik pgpool, je nutn\u00e9 sb\u00edrat informace o ka\u017ed\u00e9m z nich.<\/p>\n<p><\/p>\n<p>Sada metrik je n\u00e1sleduj\u00edc\u00ed:<\/p>\n<p><\/p>\n<ul>\n<li>CPU;<\/li>\n<li>pam\u011b\u0165 \u2014 p\u0159edev\u0161\u00edm rezidentn\u00ed;<\/li>\n<li>IO \u2014 ide\u00e1ln\u011b v IOPS;<\/li>\n<li>FileFd \u2014 otev\u0159en\u00e9 a limit;<\/li>\n<li>v\u00fdznamn\u00e9 selh\u00e1n\u00ed str\u00e1nky \u2014 tak zjist\u00edte, kter\u00fd proces swapuje.<\/li>\n<\/ul>\n<p><\/p>\n<p>Cel\u00e9 monitorov\u00e1n\u00ed m\u00e1me nasazen\u00e9 v Dockeru, pro sb\u011br metrik pou\u017e\u00edv\u00e1me Cadvisor. Na ostatn\u00edch stroj\u00edch pou\u017e\u00edv\u00e1me process-exporter.<\/p>\n<p><\/p>\n<h2 id=\"sistemnye-servisy-stek-po\">Syst\u00e9mov\u00e9 slu\u017eby, stack SW<\/h2>\n<p><\/p>\n<p>Ka\u017ed\u00e1 aplikace m\u00e1 svou specifi\u010dnost a je t\u011b\u017ek\u00e9 vybrat n\u011bjakou sadu metrik.<\/p>\n<p><\/p>\n<p>Univerz\u00e1ln\u00ed sadou jsou:<\/p>\n<p><\/p>\n<ul>\n<li>m\u00edra dotaz\u016f;<\/li>\n<li>po\u010det chyb;<\/li>\n<li>latence;<\/li>\n<li>saturace.<\/li>\n<\/ul>\n<p><\/p>\n<p>Nejv\u00fdznamn\u011bj\u0161\u00ed p\u0159\u00edklady monitorov\u00e1n\u00ed t\u00e9to \u00farovn\u011b u n\u00e1s jsou \u2014 Nginx a PostgreSQL.<\/p>\n<p><\/p>\n<p>Nejv\u00edce zat\u00ed\u017een\u00e1 slu\u017eba v na\u0161em syst\u00e9mu \u2014 datab\u00e1ze. D\u0159\u00edve jsme m\u011bli pom\u011brn\u011b \u010dasto probl\u00e9my s t\u00edm, abychom zjistili, co datab\u00e1ze d\u011bl\u00e1.<\/p>\n<p><\/p>\n<p>Vid\u011bli jsme vysokou z\u00e1t\u011b\u017e na disc\u00edch, ale slowlogy nic po\u0159\u00e1dn\u00e9ho neukazovaly. Tento probl\u00e9m jsme vy\u0159e\u0161ili pomoc\u00ed pg_stat_statements, pohledu, ve kter\u00e9m se shroma\u017e\u010fuj\u00ed statistiky o dotazech.<\/p>\n<p><\/p>\n<p>To je v\u0161e, co pot\u0159ebuje administr\u00e1tor.<\/p>\n<p><\/p>\n<p>Stav\u00edme grafy aktivity dotaz\u016f na \u010dten\u00ed a z\u00e1pis:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Jak jsme vybudovali monitorov\u00e1n\u00ed na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/f838b7759221b284c06941291fa186df.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Jak jsme vybudovali monitorov\u00e1n\u00ed na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/ea8684ea81f21c8d762a606aeb8342a3.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>V\u0161e je jednoduch\u00e9 a p\u0159ehledn\u00e9, ka\u017ed\u00e9mu dotazu \u2014 svou barvu.<\/p>\n<p><\/p>\n<p>Nejm\u00e9n\u011b jasn\u00fd p\u0159\u00edklad \u2014 Nginx logy. Nen\u00ed p\u0159ekvapen\u00edm, \u017ee je m\u00e1lo kdo parsuje nebo zmi\u0148uje v seznamu povinn\u00fdch. Standardn\u00ed form\u00e1t nen\u00ed p\u0159\u00edli\u0161 informativn\u00ed a je pot\u0159eba ho roz\u0161\u00ed\u0159it.<\/p>\n<p><\/p>\n<p>Osobn\u011b jsem p\u0159idal request_time, upstream_response_time, body_bytes_sent, request_length, request_id. Stav\u00edme grafy doby odezvy a po\u010dtu chyb:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Jak jsme vybudovali monitorov\u00e1n\u00ed na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/e517f15303f8fa780d40b7f7e30eb9ef.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Jak jsme vybudovali monitorov\u00e1n\u00ed na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/9a07ffda75300ec939edff24532fc294.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Stav\u00edme grafy doby odezvy a po\u010dtu chyb. Pamatujete? Mluvil jsem o \u00fakolech pro podnik\u00e1n\u00ed? Aby bylo v\u0161echno rychle a bez chyb? U\u017e dv\u011bma grafy jsme tyto ot\u00e1zky uzav\u0159eli. A podle nich u\u017e m\u016f\u017eeme volat pohotovostn\u00edm administr\u00e1tor\u016fm.<\/p>\n<p><\/p>\n<p>Ale z\u016fst\u00e1v\u00e1 je\u0161t\u011b jeden probl\u00e9m \u2014 zajistit rychl\u00e9 odstran\u011bn\u00ed p\u0159\u00ed\u010din incidentu.<\/p>\n<p><\/p>\n<h2 id=\"ustranenie-incidentov\">Odstran\u011bn\u00ed incident\u016f<\/h2>\n<p><\/p>\n<p>Cel\u00fd proces od zji\u0161t\u011bn\u00ed a\u017e po \u0159e\u0161en\u00ed probl\u00e9mu lze rozd\u011blit na n\u011bkolik krok\u016f:<\/p>\n<p><\/p>\n<ul>\n<li>zji\u0161t\u011bn\u00ed probl\u00e9mu;<\/li>\n<li>ozn\u00e1men\u00ed pohotovostn\u00edmu administr\u00e1torovi;<\/li>\n<li>reakce na incident;<\/li>\n<li>odstran\u011bn\u00ed p\u0159\u00ed\u010din.<\/li>\n<\/ul>\n<p><\/p>\n<p>Je d\u016fle\u017eit\u00e9, \u017ee to mus\u00edme d\u011blat co nejrychleji. A pokud v etap\u00e1ch zji\u0161t\u011bn\u00ed probl\u00e9mu a odes\u00edl\u00e1n\u00ed ozn\u00e1men\u00ed nem\u00e1me moc \u010dasu na zisk \u2014 dv\u011b minuty na n\u011b v ka\u017ed\u00e9m p\u0159\u00edpad\u011b ub\u011bhnou, pak n\u00e1sleduj\u00edc\u00ed \u2014 jsou prost\u011b neoran\u00e9 pole pro zlep\u0161en\u00ed.<\/p>\n<p><\/p>\n<p>P\u0159edstavme si, \u017ee u oper\u00e1tora zazvonil telefon. Co ud\u011bl\u00e1? Hled\u00e1 odpov\u011bdi na ot\u00e1zky \u2014 co se pokazilo, kde se to pokazilo, jak reagovat? Takhle odpov\u00edd\u00e1me na tyto ot\u00e1zky:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Jak jsme vybudovali monitorov\u00e1n\u00ed na Prometheus, Clickhouse a ELK\" src=\"\/wp-content\/uploads\/2019\/04\/7f33172922e646a24bb83733350b24f5.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Celou tuto informaci jednodu\u0161e zahrneme do textu ozn\u00e1men\u00ed, p\u0159id\u00e1me odkaz na str\u00e1nku ve wiki, kde je pops\u00e1no, jak na tento probl\u00e9m reagovat, jak ho \u0159e\u0161it a eskalovat.<\/p>\n<p><\/p>\n<p>Dosud jsem nic ne\u0159ekl o \u00farovni aplikace a obchodn\u00ed logice. Bohu\u017eel, v na\u0161ich aplikac\u00edch zat\u00edm nen\u00ed implementov\u00e1no shroma\u017e\u010fov\u00e1n\u00ed metrik. Jedin\u00fdm zdrojem alespo\u0148 n\u011bjak\u00fdch informac\u00ed z t\u011bchto \u00farovn\u00ed jsou logy.<\/p>\n<p><\/p>\n<p>P\u00e1r bod\u016f.<\/p>\n<p><\/p>\n<p>Za prv\u00e9, pi\u0161te strukturovan\u00e9 logy. Nen\u00ed t\u0159eba zahrnovat kontext do textu zpr\u00e1vy. Zt\u011b\u017euje to jejich seskupov\u00e1n\u00ed a anal\u00fdzu. Logstash pot\u0159ebuje mnoho \u010dasu na to, aby to v\u0161echno normalizoval.<\/p>\n<p><\/p>\n<p>Za druh\u00e9, spr\u00e1vn\u011b pou\u017e\u00edvejte \u00farovn\u011b z\u00e1va\u017enosti. Ka\u017ed\u00fd jazyk m\u00e1 sv\u016fj vlastn\u00ed standard. Osobn\u011b rozli\u0161uji \u010dty\u0159i \u00farovn\u011b:<\/p>\n<p><\/p>\n<ol>\n<li>\u017e\u00e1dn\u00e1 chyba;<\/li>\n<li>chyba na stran\u011b klienta;<\/li>\n<li>chyba na na\u0161\u00ed stran\u011b, neztr\u00e1c\u00edme pen\u00edze, neneseme rizika;<\/li>\n<li>chyba na na\u0161\u00ed stran\u011b, ztr\u00e1c\u00edme pen\u00edze.<\/li>\n<\/ol>\n<p><\/p>\n<p>Shr\u0148me to. M\u011bli bychom se sna\u017eit vybudovat monitoring pr\u00e1v\u011b od obchodn\u00ed logiky. Sna\u017eit se monitorovat samotnou aplikaci a operovat takov\u00fdmi metrikami, jako je po\u010det prodej\u016f, po\u010det nov\u00fdch registrac\u00ed u\u017eivatel\u016f, po\u010det aktivn\u00edch u\u017eivatel\u016f v dan\u00e9m okam\u017eiku a tak d\u00e1le.<\/p>\n<p><\/p>\n<p>Pokud je cel\u00fd v\u00e1\u0161 byznys jedn\u00edm tla\u010d\u00edtkem v prohl\u00ed\u017ee\u010di, je t\u0159eba sledovat, zda je stisknuto, zda funguje spr\u00e1vn\u011b. V\u0161echno ostatn\u00ed nen\u00ed d\u016fle\u017eit\u00e9.<\/p>\n<p><\/p>\n<p>Pokud to nem\u00e1te, m\u016f\u017eete se to sna\u017eit dohnat v log\u00e1ch aplikace, Nginx log\u00e1ch atd., jak jsme to ud\u011blali my. Mus\u00edte b\u00fdt co nejbl\u00ed\u017ee aplikaci.<\/p>\n<p><\/p>\n<p>Metriky opera\u010dn\u00edho syst\u00e9mu jsou samoz\u0159ejm\u011b d\u016fle\u017eit\u00e9, ale byznysu nejsou zaj\u00edmav\u00e9, neplat\u00ed n\u00e1m za n\u011b.<\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/449352\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c. \u041c\u0435\u0441\u044f\u0446 \u043d\u0430\u0437\u0430\u0434 \u0437\u0430\u0432\u0435\u0440\u0448\u0438\u043b\u0430\u0441\u044c \u043d\u0430\u0448\u0430 \u043a\u043e\u0440\u043f\u043e\u0440\u0430\u0442\u0438\u0432\u043d\u0430\u044f \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u044f, \u0433\u0434\u0435 \u043c\u044b \u0434\u0435\u043b\u0438\u043b\u0438\u0441\u044c \u043d\u0430\u043a\u043e\u043f\u043b\u0435\u043d\u043d\u044b\u043c \u043e\u043f\u044b\u0442\u043e\u043c \u0441 IT-\u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u043e\u043c \u043d\u0430\u0448\u0435\u0433\u043e \u0433\u043e\u0440\u043e\u0434\u0430. \u042f \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u043b \u043f\u0440\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0432\u0435\u0431-\u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439. \u041c\u0430\u0442\u0435\u0440\u0438\u0430\u043b \u043f\u0440\u0435\u0434\u043d\u0430\u0437\u043d\u0430\u0447\u0430\u043b\u0441\u044f \u0434\u043b\u044f \u0443\u0440\u043e\u0432\u043d\u044f junior \u0438\u043b\u0438 middle, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043d\u0435 \u0432\u044b\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u0438 \u044d\u0442\u043e\u0442 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0441 \u043d\u0443\u043b\u044f. \u041a\u0440\u0430\u0435\u0443\u0433\u043e\u043b\u044c\u043d\u044b\u0439 \u043a\u0430\u043c\u0435\u043d\u044c, \u043b\u0435\u0436\u0430\u0449\u0438\u0439 \u0432 \u043e\u0441\u043d\u043e\u0432\u0435 \u043b\u044e\u0431\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24139,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32322","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043d\u0430 Prometheus, Clickhouse \u0438 ELK | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T18:46:21+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T18:46:21+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Jak jsme vybudovali monitoring na Prometheus, Clickhouse a ELK | ProHoster","description":"Jmenuji se Anton Baderin. Pracuji v Centru Vysok\u00fdch Technologi\u00ed a zab\u00fdv\u00e1m se syst\u00e9mov\u00fdm administrativou.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043d\u0430 Prometheus, Clickhouse \u0438 ELK | ProHoster","og:description":"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:46:21+00:00","article:modified_time":"2019-10-31T18:46:21+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"32322","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-02-22 15:34:41","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:00:23","updated":"2026-02-22 15:34:41","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/32322","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=32322"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/32322\/revisions"}],"predecessor-version":[{"id":162019,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/32322\/revisions\/162019"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/24139"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=32322"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=32322"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=32322"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}