{"id":32322,"date":"2019-10-31T21:46:21","date_gmt":"2019-10-31T18:46:21","guid":{"rendered":"https:\/\/prohoster.info\/blog\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\/"},"modified":"2019-10-31T21:46:21","modified_gmt":"2019-10-31T18:46:21","slug":"kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","title":{"rendered":"Cum am construit monitorizarea pe Prometheus, Clickhouse \u0219i ELK","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>M\u0103 numesc Anton Baderin. Lucrez la Centrul de Tehnologii Avansate \u0219i m\u0103 ocup de administrarea sistemelor. Cu o lun\u0103 \u00een urm\u0103, s-a desf\u0103\u0219urat conferin\u021ba noastr\u0103 corporate, unde am \u00eemp\u0103rt\u0103\u0219it experien\u021ba acumulat\u0103 cu comunitatea IT din ora\u0219ul nostru. Am vorbit despre monitorizarea aplica\u021biilor web. Materialul era destinat nivelului junior sau middle, care nu au construit acest proces de la zero.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cum am construit monitorizarea pe Prometheus, Clickhouse \u0219i ELK\" src=\"\/wp-content\/uploads\/2019\/04\/c181509e6a10dd7562f361f328446d61.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Piatra de temelie pe care se bazeaz\u0103 orice sistem de monitorizare este rezolvarea problemelor de business. Monitorizarea pentru simpla existen\u021b\u0103 a acesteia nu intereseaz\u0103 pe nimeni. Ce \u00ee\u0219i dore\u0219te business-ul? Ca totul s\u0103 func\u021bioneze rapid \u0219i f\u0103r\u0103 erori. Business-ul vrea proactivitate, s\u0103 identific\u0103m noi problemele \u00een func\u021bionarea serviciului \u0219i s\u0103 le remediem c\u00e2t mai repede. Acesta este, de fapt, scopul pe care l-am urm\u0103rit tot anul trecut \u00een proiectul unuia dintre clien\u021bii no\u0219tri.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h2 id=\"o-proekte\">Despre proiect<\/h2>\n<p><\/p>\n<p>Proiectul este unul dintre cele mai mari programe de loialitate din \u021bar\u0103. Ajut\u0103m re\u021belele de retail s\u0103 creasc\u0103 frecven\u021ba v\u00e2nz\u0103rilor prin diferite instrumente de marketing, cum ar fi cardurile de bonus. \u00cen total, proiectul cuprinde 14 aplica\u021bii, care func\u021bioneaz\u0103 pe zece servere.<\/p>\n<p><\/p>\n<p>\u00cen procesul de sus\u021binere a interviurilor, am observat de multe ori c\u0103 administratorii nu abordeaz\u0103 \u00eentotdeauna corect monitorizarea aplica\u021biilor web: \u00eenc\u0103 multe dintre ei se limiteaz\u0103 la metricile sistemului de operare, monitoriz\u00e2nd sporadic serviciile.<\/p>\n<p><\/p>\n<p>\u00cen cazul meu, \u00een spatele sistemului de monitorizare al clientului se afla Icinga. Aceasta nu rezolva problemele men\u021bionate mai sus. Adesea, clientul ne informa despre probleme \u0219i nu de pu\u021bine ori ne lipseau datele pentru a ajunge la cauz\u0103.<\/p>\n<p><\/p>\n<p>\u00cen plus, era o \u00een\u021belegere clar\u0103 a lipsei de perspective pentru dezvoltarea ei ulterioar\u0103. Cred c\u0103 cei care sunt familiariza\u021bi cu Icinga m\u0103 vor \u00een\u021belege. A\u0219adar, am decis s\u0103 reproiect\u0103m complet sistemul de monitorizare a aplica\u021biilor web \u00een proiect.<\/p>\n<p><\/p>\n<h2 id=\"prometheus\">Prometheus<\/h2>\n<p><\/p>\n<p>Am ales Prometheus baz\u00e2ndu-ne pe trei indicatori principali:<\/p>\n<p><\/p>\n<ol>\n<li>Un num\u0103r imens de metrici disponibile. \u00cen cazul nostru, sunt 60 de mii. Desigur, trebuie men\u021bionat c\u0103 majoritatea dintre ele nu le utiliz\u0103m (aproape 95%). Pe de alt\u0103 parte, toate sunt relativ ieftine. Pentru noi, aceasta reprezint\u0103 o alt\u0103 extrem\u0103, comparativ cu Icinga, pe care o foloseam anterior. \u00cen cazul acesteia, ad\u0103ugarea de metrici era o durere special\u0103: cele existente erau costisitoare (e suficient s\u0103 ne uit\u0103m la sursele oric\u0103rui plugin). Orice plugin era un script \u00een Bash sau Python, a c\u0103ror execu\u021bie nu era ieftin\u0103 din punct de vedere al resurselor consumate.<\/li>\n<li>Acest sistem consum\u0103 relativ pu\u021bine resurse. Toate metricile noastre necesit\u0103 600 MB de memorie RAM, 15% dintr-un nucleu \u0219i c\u00e2teva zeci de IOPS. Desigur, trebuie s\u0103 rul\u0103m exportatori de metrici, dar to\u021bi sunt scri\u0219i \u00een Go \u0219i nu consum\u0103 mult. Nu cred c\u0103, \u00een condi\u021biile actuale, aceasta este o problem\u0103. <\/li>\n<li>Ofer\u0103 posibilitatea de a trece la Kubernetes. Av\u00e2nd \u00een vedere planurile clientului, alegerea este evident\u0103. <\/li>\n<\/ol>\n<p><\/p>\n<h2 id=\"elk\">ELK<\/h2>\n<p><\/p>\n<p>Anterior nu am colectat \u0219i procesat log-uri. Deficien\u021bele sunt clare pentru to\u021bi. Am ales ELK deoarece aveam deja experien\u021b\u0103 cu acest sistem. Acum stoc\u0103m doar log-urile aplica\u021biilor. Criteriile principale de selec\u021bie au fost c\u0103utarea text complet \u0219i viteza acesteia.<\/p>\n<p><\/p>\n<h2 id=\"slickhouse\">Clickhouse<\/h2>\n<p><\/p>\n<p>La \u00eenceput, alegerea a fost InfluxDB. Eram con\u0219tien\u021bi de necesitatea de a colecta log-urile Nginx, statisticile din pg_stat_statements, stocarea datelor istorice \u00een Prometheus. Influx nu ne-a pl\u0103cut, deoarece, din c\u00e2nd \u00een c\u00e2nd, \u00eencepea s\u0103 consume mult\u0103 memorie \u0219i se pr\u0103bu\u0219ea. \u00cen plus, ne-ar fi pl\u0103cut s\u0103 grup\u0103m cererile dup\u0103 remote_addr, iar gruparea \u00een acest SGBD era posibil\u0103 doar dup\u0103 etichete. Etichetele sunt scumpe (memorie), iar num\u0103rul lor este condi\u021bionat limitat.<\/p>\n<p><\/p>\n<p>Am re\u00eenceput c\u0103ut\u0103rile. Aveam nevoie de o baz\u0103 analitic\u0103 cu un consum minim de resurse, de preferat cu compresie a datelor pe disc.<\/p>\n<p><\/p>\n<p>Clickhouse satisface toate aceste criterii \u0219i nu ne-am pl\u00e2ns niciodat\u0103 de alegere. Nu scriem volume semnificative de date (aproximativ cinci mii de inser\u021bii pe minut).<\/p>\n<p><\/p>\n<h2 id=\"newrelic\">NewRelic<\/h2>\n<p><\/p>\n<p>NewRelic a fost istoric cu noi, deoarece a fost alegerea clientului. La noi este folosit ca APM.<\/p>\n<p><\/p>\n<h2 id=\"zabbix\">Zabbix<\/h2>\n<p><\/p>\n<p>Folosim Zabbix exclusiv pentru monitorizarea Black Box a diverselor API-uri.<\/p>\n<p><\/p>\n<h2 id=\"opredelenie-podhoda-k-monitoringu\">Definirea abord\u0103rii de monitorizare<\/h2>\n<p><\/p>\n<p>Ne-am dorit s\u0103 descompunem sarcina \u0219i, astfel, s\u0103 sistematiz\u0103m abordarea monitoriz\u0103rii.<\/p>\n<p><\/p>\n<p>Pentru aceasta, am \u00eemp\u0103r\u021bit sistemul nostru \u00een urm\u0103toarele niveluri:<\/p>\n<p><\/p>\n<ul>\n<li>\u00abhardware\u00bb \u0219i VMS;<\/li>\n<li>sistemul de operare;<\/li>\n<li>serviciile de sistem, stiva de software;<\/li>\n<li>aplica\u021bia;<\/li>\n<li>logica de afaceri.<\/li>\n<\/ul>\n<p><\/p>\n<p>Avantajele acestei abord\u0103ri:<\/p>\n<p><\/p>\n<ul>\n<li>\u0219tim cine este responsabil pentru func\u021bionarea fiec\u0103rui nivel \u0219i, \u00een func\u021bie de aceasta, putem trimite alerte;<\/li>\n<li>putem folosi structura \u00een cazul \u00een care este necesar\u0103 suprimarea alertelor - ar fi ciudat s\u0103 trimitem o alert\u0103 despre indisponibilitatea bazei de date atunci c\u00e2nd \u00eentreaga ma\u0219in\u0103 virtual\u0103 este indisponibil\u0103.<\/li>\n<\/ul>\n<p><\/p>\n<p>Av\u00e2nd \u00een vedere c\u0103 sarcina noastr\u0103 este de a identifica abaterile \u00een func\u021bionarea sistemului, trebuie s\u0103 definim un set de metrici pentru fiecare nivel, la care s\u0103 ne concentr\u0103m atunci c\u00e2nd scriem reguli de alertare. Mai departe, vom trece prin nivelurile \u201eVMS\u201d, \u201eSistemul de operare\u201d \u0219i \u201eServiciile de sistem, stiva de software\u201d.<\/p>\n<p><\/p>\n<h2 id=\"virtualnye-mashiny\">Ma\u0219ini virtuale<\/h2>\n<p><\/p>\n<p><a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/ro\/\"   title=\"G\u0103zduire\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"2329\">G\u0103zduire<\/a> ne aloc\u0103 procesor, disc, memorie \u0219i re\u021bea. \u0218i cu primele dou\u0103 am avut probleme. A\u0219adar, metricile sunt:<\/p>\n<p><\/p>\n<p>Timpul de CPU furat - atunci c\u00e2nd cump\u0103ra\u021bi o ma\u0219in\u0103 virtual\u0103 pe Amazon (de exemplu, t2.micro), trebuie s\u0103 \u00een\u021belege\u021bi c\u0103 vi se aloc\u0103 nu un nucleu \u00eentreg de procesor, ci doar o cot\u0103 din timpul s\u0103u. \u0218i c\u00e2nd o epuiza\u021bi, procesorul v\u0103 va fi furat.<\/p>\n<p><\/p>\n<p>Aceast\u0103 metric\u0103 permite monitorizarea acestor momente \u0219i luarea de decizii. De exemplu, trebuie s\u0103 trece\u021bi la un tarif mai performant sau s\u0103 separa\u021bi procesarea sarcinilor de fundal \u0219i cererile API \u00een diferite <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/ro\/server\/dts-los-angeles\/\"   title=\"server\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"3493\">server<\/a>.<\/p>\n<p><\/p>\n<p>IOPS + timpul de a\u0219teptare CPU iowait - dintr-un motiv oarecare, multe servicii de g\u0103zduire \u00een cloud nu ofer\u0103 suficiente IOPS. Mai mult, un grafic cu IOPS sc\u0103zute nu este un argument pentru ei. De aceea, este important s\u0103 colecta\u021bi \u0219i CPU iowait. Cu aceast\u0103 pereche de grafice - cu IOPS sc\u0103zute \u0219i a\u0219teptare mare de intrare-ie\u0219ire - deja pute\u021bi discuta cu furnizorul de g\u0103zduire \u0219i s\u0103 rezolva\u021bi problema.<\/p>\n<p><\/p>\n<h2 id=\"operacionnaya-sistema\">Sistem de operare<\/h2>\n<p><\/p>\n<p>Metricile sistemului de operare:<\/p>\n<p><\/p>\n<ul>\n<li>cantitatea de memorie disponibil\u0103 \u00een %;<\/li>\n<li>activitatea utiliz\u0103rii swap: vmstat swapin, swapout;<\/li>\n<li>num\u0103rul de inode-uri disponibile \u0219i spa\u021biul liber pe sistemul de fi\u0219iere \u00een %<\/li>\n<li>\u00eenc\u0103rcarea medie;<\/li>\n<li>num\u0103rul de conexiuni \u00een stare tw;<\/li>\n<li>gradul de utilizare a tabelului conntrack;<\/li>\n<li>calitatea func\u021bion\u0103rii re\u021belei poate fi monitorizat\u0103 cu ajutorul utilitarului ss, pachetul iproute2 - ob\u021bin\u00e2nd din ie\u0219irea acestuia indicatorul RTT al conexiunilor \u0219i grup\u00e2nd \u00een func\u021bie de destina\u021bie-port.<\/li>\n<\/ul>\n<p><\/p>\n<p>De asemenea, la nivelul sistemului de operare, avem o entitate numit\u0103 procese. Este important s\u0103 definim un set de procese \u00een sistem care joac\u0103 un rol esen\u021bial \u00een func\u021bionarea sa. Dac\u0103, de exemplu, ave\u021bi mai multe pgpool, atunci este necesar s\u0103 colecta\u021bi informa\u021bii despre fiecare dintre ele.<\/p>\n<p><\/p>\n<p>Setul de metrici este urm\u0103torul:<\/p>\n<p><\/p>\n<ul>\n<li>CPU;<\/li>\n<li>memoria \u2014 \u00een primul r\u00e2nd, resident\u0103;<\/li>\n<li>IO \u2014 preferabil \u00een IOPS;<\/li>\n<li>FileFd \u2014 deschise \u0219i limit\u0103;<\/li>\n<li>defec\u021biuni semnificative ale paginii \u2014 astfel ve\u021bi putea \u00een\u021belege ce proces se swap.<\/li>\n<\/ul>\n<p><\/p>\n<p>\u00centreaga monitorizare este desf\u0103\u0219urat\u0103 \u00een Docker, pentru colectarea datelor despre metrici folosim \u0421advisor. Pe celelalte ma\u0219ini aplic\u0103m process-exporter.<\/p>\n<p><\/p>\n<h2 id=\"sistemnye-servisy-stek-po\">Servicii sistemice, stiv\u0103 software<\/h2>\n<p><\/p>\n<p>Fiecare aplica\u021bie are specificul s\u0103u \u0219i este greu s\u0103 se eviden\u021bieze un set anume de metrici.<\/p>\n<p><\/p>\n<p>Setul universal este:<\/p>\n<p><\/p>\n<ul>\n<li>rata cererilor;<\/li>\n<li>num\u0103rul de erori;<\/li>\n<li>laten\u021ba;<\/li>\n<li>satura\u021bie.<\/li>\n<\/ul>\n<p><\/p>\n<p>Cele mai evidente exemple de monitorizare la acest nivel sunt Nginx \u0219i PostgreSQL.<\/p>\n<p><\/p>\n<p>Cel mai solicitat serviciu din sistemul nostru este baza de date. \u00cen trecut, aveam destul de des probleme cu descoperirea activit\u0103\u021bii bazei de date.<\/p>\n<p><\/p>\n<p>Am observat o \u00eenc\u0103rcare mare pe discuri, dar logurile de lentitudine nu ar\u0103tau nimic semnificativ. Aceast\u0103 problem\u0103 am rezolvat-o cu ajutorul pg_stat_statements, o vedere care colecteaz\u0103 statistici despre cereri.<\/p>\n<p><\/p>\n<p>Asta e tot ce trebuie adminului.<\/p>\n<p><\/p>\n<p>Construim grafice de activitate a cererilor pentru citire \u0219i scriere:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Cum am construit monitorizarea pe Prometheus, Clickhouse \u0219i ELK\" src=\"\/wp-content\/uploads\/2019\/04\/f838b7759221b284c06941291fa186df.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Cum am construit monitorizarea pe Prometheus, Clickhouse \u0219i ELK\" src=\"\/wp-content\/uploads\/2019\/04\/ea8684ea81f21c8d762a606aeb8342a3.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Totul este simplu \u0219i clar, fiecare cerere are o culoare proprie.<\/p>\n<p><\/p>\n<p>Un alt exemplu relevant sunt logurile Nginx. Nu este surprinz\u0103tor c\u0103 pu\u021bini le parseaz\u0103 sau le men\u021bioneaz\u0103 \u00een lista obligatoare. Formatul standard nu este foarte informativ \u0219i trebuie extins.<\/p>\n<p><\/p>\n<p>Personal, am ad\u0103ugat request_time, upstream_response_time, body_bytes_sent, request_length, request_id. Construim grafice de timp de r\u0103spuns \u0219i num\u0103r de erori:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Cum am construit monitorizarea pe Prometheus, Clickhouse \u0219i ELK\" src=\"\/wp-content\/uploads\/2019\/04\/e517f15303f8fa780d40b7f7e30eb9ef.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<img decoding=\"async\" alt=\"Cum am construit monitorizarea pe Prometheus, Clickhouse \u0219i ELK\" src=\"\/wp-content\/uploads\/2019\/04\/9a07ffda75300ec939edff24532fc294.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Construim grafice de timp de r\u0103spuns \u0219i num\u0103r de erori. \u00ce\u021bi aminte\u0219ti? Am vorbit despre sarcinile business-ului? S\u0103 fie rapide \u0219i f\u0103r\u0103 erori? Aceste \u00eentreb\u0103ri le-am \u00eenchis deja cu dou\u0103 grafice. \u0218i dup\u0103 ele, deja putem suna administratorii de serviciu.<\/p>\n<p><\/p>\n<p>Dar a mai r\u0103mas o problem\u0103 \u2014 asigurarea unei remedieri rapide a cauzelor incidentelor.<\/p>\n<p><\/p>\n<h2 id=\"ustranenie-incidentov\">Remedierea incidentelor<\/h2>\n<p><\/p>\n<p>\u00centregul proces, de la identificarea la solu\u021bionarea problemei, poate fi \u00eemp\u0103r\u021bit \u00een mai mul\u021bi pa\u0219i:<\/p>\n<p><\/p>\n<ul>\n<li>identificarea problemei;<\/li>\n<li>notificarea administratorului de serviciu;<\/li>\n<li>reac\u021bia la incident;<\/li>\n<li>remedierea cauzelor.<\/li>\n<\/ul>\n<p><\/p>\n<p>Este important s\u0103 facem acest lucru c\u00e2t mai repede. \u0218i dac\u0103 la etapele de identificare a problemei \u0219i trimiterea notific\u0103rii nu putem c\u00e2\u0219tiga mult timp \u2014 dou\u0103 minute vor fi necesare \u00een orice caz, etapa urm\u0103toare este pur \u0219i simplu un teritoriu neexplorat pentru \u00eembun\u0103t\u0103\u021biri.<\/p>\n<p><\/p>\n<p>S\u0103 presupunem c\u0103 telefonul de serviciu sun\u0103. Ce va face? Va c\u0103uta r\u0103spunsuri la \u00eentreb\u0103rile \u2014 ce s-a defectat, unde s-a defectat, cum s\u0103 reac\u021bioneze? Iat\u0103 cum r\u0103spundem la aceste \u00eentreb\u0103ri:<\/p>\n<p>\n<img decoding=\"async\" alt=\"Cum am construit monitorizarea pe Prometheus, Clickhouse \u0219i ELK\" src=\"\/wp-content\/uploads\/2019\/04\/7f33172922e646a24bb83733350b24f5.png\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Pur \u0219i simplu includem toate aceste informa\u021bii \u00een textul notific\u0103rii, oferind un link c\u0103tre pagina wiki care descrie cum s\u0103 reac\u021bionezi la aceast\u0103 problem\u0103, cum s\u0103 o rezolvi \u0219i s\u0103 o escaladezi.<\/p>\n<p><\/p>\n<p>\u00cenc\u0103 nu am spus nimic despre nivelul aplica\u021biei \u0219i logica de afaceri. Din p\u0103cate, aplica\u021biile noastre nu au implementat \u00eenc\u0103 colectarea metrice. Singura surs\u0103 de informa\u021bii de la aceste niveluri sunt jurnalele.<\/p>\n<p><\/p>\n<p>C\u00e2teva lucruri.<\/p>\n<p><\/p>\n<p>\u00cen primul r\u00e2nd, scrie\u021bi chei de jurnal structurate. Nu include\u021bi contextul \u00een textul mesajului. Acest lucru \u00eengreuneaz\u0103 gruparea \u0219i analiza. Logstash necesit\u0103 mult timp pentru a normaliza totul.<\/p>\n<p><\/p>\n<p>\u00cen al doilea r\u00e2nd, folosi\u021bi corect nivelurile de severitate. Fiecare limbaj are propriul standard. Personal, eviden\u021biez patru niveluri:<\/p>\n<p><\/p>\n<ol>\n<li>nu exist\u0103 erori;<\/li>\n<li>eroare de partea clientului;<\/li>\n<li>eroare de partea noastr\u0103, nu pierdem bani, nu suport\u0103m riscuri;<\/li>\n<li>eroare de partea noastr\u0103, pierdem bani.<\/li>\n<\/ol>\n<p><\/p>\n<p>\u00cen concluzie. Trebuie s\u0103 ne str\u0103duim s\u0103 construim monitorizarea bazat\u0103 pe logica de afaceri. Trebuie s\u0103 ne concentr\u0103m pe monitorizarea aplica\u021biei \u00een sine \u0219i s\u0103 oper\u0103m cu metrice precum num\u0103rul de v\u00e2nz\u0103ri, num\u0103rul de \u00eenregistr\u0103ri noi de utilizatori, num\u0103rul de utilizatori activi \u00een acel moment etc.<\/p>\n<p><\/p>\n<p>Dac\u0103 \u00eentreaga dvs. afacere este un singur buton \u00een browser, trebuie s\u0103 monitoriza\u021bi dac\u0103 acesta este ap\u0103sat, dac\u0103 func\u021bioneaz\u0103 corect. Tot restul nu conteaz\u0103.<\/p>\n<p><\/p>\n<p>Dac\u0103 nu ave\u021bi acest lucru, pute\u021bi \u00eencerca s\u0103 recupera\u021bi informa\u021biile din jurnalele aplica\u021biei, jurnalele Nginx etc., a\u0219a cum am f\u0103cut noi. Trebuie s\u0103 fi\u021bi c\u00e2t mai aproape de aplica\u021bie.<\/p>\n<p><\/p>\n<p>Metriile sistemului de operare sunt cu siguran\u021b\u0103 importante, dar nu sunt de interes pentru afaceri, nu suntem pl\u0103ti\u021bi pentru acestea.<\/p>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/449352\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c. \u041c\u0435\u0441\u044f\u0446 \u043d\u0430\u0437\u0430\u0434 \u0437\u0430\u0432\u0435\u0440\u0448\u0438\u043b\u0430\u0441\u044c \u043d\u0430\u0448\u0430 \u043a\u043e\u0440\u043f\u043e\u0440\u0430\u0442\u0438\u0432\u043d\u0430\u044f \u043a\u043e\u043d\u0444\u0435\u0440\u0435\u043d\u0446\u0438\u044f, \u0433\u0434\u0435 \u043c\u044b \u0434\u0435\u043b\u0438\u043b\u0438\u0441\u044c \u043d\u0430\u043a\u043e\u043f\u043b\u0435\u043d\u043d\u044b\u043c \u043e\u043f\u044b\u0442\u043e\u043c \u0441 IT-\u0441\u043e\u043e\u0431\u0449\u0435\u0441\u0442\u0432\u043e\u043c \u043d\u0430\u0448\u0435\u0433\u043e \u0433\u043e\u0440\u043e\u0434\u0430. \u042f \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u043b \u043f\u0440\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u0432\u0435\u0431-\u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0439. \u041c\u0430\u0442\u0435\u0440\u0438\u0430\u043b \u043f\u0440\u0435\u0434\u043d\u0430\u0437\u043d\u0430\u0447\u0430\u043b\u0441\u044f \u0434\u043b\u044f \u0443\u0440\u043e\u0432\u043d\u044f junior \u0438\u043b\u0438 middle, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043d\u0435 \u0432\u044b\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u0438 \u044d\u0442\u043e\u0442 \u043f\u0440\u043e\u0446\u0435\u0441\u0441 \u0441 \u043d\u0443\u043b\u044f. \u041a\u0440\u0430\u0435\u0443\u0433\u043e\u043b\u044c\u043d\u044b\u0439 \u043a\u0430\u043c\u0435\u043d\u044c, \u043b\u0435\u0436\u0430\u0449\u0438\u0439 \u0432 \u043e\u0441\u043d\u043e\u0432\u0435 \u043b\u044e\u0431\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":24139,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-32322","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043d\u0430 Prometheus, Clickhouse \u0438 ELK | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T18:46:21+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T18:46:21+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Cum am construit monitorizarea folosind Prometheus, Clickhouse \u0219i ELK | ProHoster","description":"M\u0103 numesc Anton Baderin. Lucrez \u00een Centrul de Tehnologii Avansate \u0219i m\u0103 ocup cu administrarea sistemelor.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u0430\u043a \u043c\u044b \u0441\u0442\u0440\u043e\u0438\u043b\u0438 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043d\u0430 Prometheus, Clickhouse \u0438 ELK | ProHoster","og:description":"\u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 \u0410\u043d\u0442\u043e\u043d \u0411\u0430\u0434\u0435\u0440\u0438\u043d. \u042f \u0440\u0430\u0431\u043e\u0442\u0430\u044e \u0432 \u0426\u0435\u043d\u0442\u0440\u0435 \u0412\u044b\u0441\u043e\u043a\u0438\u0445 \u0422\u0435\u0445\u043d\u043e\u043b\u043e\u0433\u0438\u0439 \u0438 \u0437\u0430\u043d\u0438\u043c\u0430\u044e\u0441\u044c \u0441\u0438\u0441\u0442\u0435\u043c\u043d\u044b\u043c \u0430\u0434\u043c\u0438\u043d\u0438\u0441\u0442\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kak-my-stroili-monitoring-na-prometheus-clickhouse-i-elk","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:46:21+00:00","article:modified_time":"2019-10-31T18:46:21+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"32322","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-02-22 15:34:41","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:00:23","updated":"2026-02-22 15:34:41","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/32322","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=32322"}],"version-history":[{"count":2,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/32322\/revisions"}],"predecessor-version":[{"id":162019,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/32322\/revisions\/162019"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/24139"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=32322"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=32322"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=32322"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}