{"id":52115,"date":"2019-11-01T00:00:00","date_gmt":"2019-10-31T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/monitoring-kak-servis-modulnaya-sistema-dlya-mikroservisnoj-arhitektury"},"modified":"2020-02-18T13:59:47","modified_gmt":"2020-02-18T10:59:47","slug":"monitoring-kak-servis-modulnaya-sistema-dlya-mikroservisnoj-arhitektury","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/monitoring-kak-servis-modulnaya-sistema-dlya-mikroservisnoj-arhitektury","title":{"rendered":"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Ast\u0103zi, pe proiectul nostru, pe l\u00e2ng\u0103 codul monolitic, func\u021bioneaz\u0103 zeci de microservicii. Fiecare dintre acestea necesit\u0103 monitorizare. E dificil pentru inginerii DevOps s\u0103 fac\u0103 acest lucru la astfel de volume. Am dezvoltat un sistem de monitorizare care func\u021bioneaz\u0103 ca un serviciu pentru dezvoltatori. Ace\u0219tia \u00ee\u0219i pot scrie singuri metricele \u00een sistemul de monitorizare, le pot utiliza, construi tablouri de bord pe baza acestora \u0219i ad\u0103uga alerte care se vor activa la atingerea valorilor limit\u0103. Din partea inginerilor DevOps, doar infrastructura \u0219i documenta\u021bia sunt necesare. <\/p>\n<p>Acest post este transcrierea prezent\u0103rii mele de la <noindex><a rel=\"nofollow\" href=\"http:\/\/bit.ly\/tomicro\">sec\u021biunea<\/a><\/noindex> de la RIT++. Mul\u021bi ne-au cerut s\u0103 facem versiuni textuale ale prezent\u0103rilor de acolo. Dac\u0103 ai fost la conferin\u021b\u0103 sau ai vizionat videoclipul, nu vei g\u0103si nimic nou. Iar celorlal\u021bi \u2014 bun venit sub cat. Voi povesti cum am ajuns la acest sistem, cum func\u021bioneaz\u0103 \u0219i cum pl\u0103nuim s\u0103-l actualiz\u0103m. <\/p>\n<p><img decoding=\"async\" alt=\"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii\" src=\"\/wp-content\/uploads\/2019\/11\/000b8fdc8fb16aa76ce545ff2aa4e767.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<h3>Trecut: scheme \u0219i planuri <\/h3>\n<p>\nCum am ajuns la sistemul de monitorizare existent? Pentru a r\u0103spunde acestei \u00eentreb\u0103ri, trebuie s\u0103 ne \u00eentoarcem \u00een 2015. Iat\u0103 cum ar\u0103ta atunci: <\/p>\n<p><img decoding=\"async\" alt=\"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii\" src=\"\/wp-content\/uploads\/2019\/11\/d955f9bcdb5f5a5a54720c77113ca85d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAveam aproximativ 24 de noduri care se ocupau de monitorizare. Aici exist\u0103 o mul\u021bime de cronuri, scripturi, demoni care monitorizeaz\u0103 ceva, trimit mesaje, \u00eendeplinesc func\u021bii. Ne-am g\u00e2ndit c\u0103, pe m\u0103sur\u0103 ce timpul trece, un astfel de sistem va deveni din ce \u00een ce mai pu\u021bin viabil. Nu are sens s\u0103-l dezvolt\u0103m mai departe: este prea greoi. <br \/>\nAm decis s\u0103 alegem elementele de monitorizare pe care le vom p\u0103stra \u0219i dezvolta \u0219i cele de care ne vom desp\u0103r\u021bi. Am ajuns la un num\u0103r de 19. Au r\u0103mas doar Graphite, agregatori \u0219i Grafana ca tablouri de bord. Dar cum va ar\u0103ta noul sistem? Iat\u0103 cum: <\/p>\n<p><img decoding=\"async\" alt=\"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii\" src=\"\/wp-content\/uploads\/2019\/11\/57d220113d0b76c05874b5d774bff8af.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAvem un depozit de metrice: acestea sunt Graphite, care vor fi bazate pe SSD-uri rapide, \u0219i anumite agregatoare pentru metrice. Apoi \u2014 Grafana pentru generarea tablourilor de bord \u0219i Moira ca sistem de alerte. De asemenea, am dorit s\u0103 dezvolt\u0103m un sistem pentru detectarea anomaliilor. <\/p>\n<h3>Standard: Monitorizare 2.0<\/h3>\n<p>\nA\u0219a ar\u0103tau planurile \u00een 2015. Dar trebuia s\u0103 preg\u0103tim nu doar infrastructura \u0219i serviciul \u00een sine, ci \u0219i documenta\u021bia pentru acesta. Am dezvoltat un standard corporativ pe care l-am numit Monitorizare 2.0. Care erau cerin\u021bele pentru sistem? <\/p>\n<ul>\n<li>disponibilitate continu\u0103; <\/li>\n<li>interval de stocare a metricelor = 10 secunde; <\/li>\n<li>stocarea structurat\u0103 a metricilor \u0219i tablourilor de bord; <\/li>\n<li>SLA &gt; 99,99% <\/li>\n<li>colectarea metricilor evenimentelor prin UDP (!). <\/li>\n<\/ul>\n<p>\nAm avut nevoie de UDP, deoarece avem un volum mare de trafic \u0219i evenimente care genereaz\u0103 metrici. Dac\u0103 le-am scrie pe toate simultan \u00een Graphite, stocarea s-ar pr\u0103bu\u0219i. De asemenea, am ales prefixe de prim nivel pentru toate metricile. <\/p>\n<p><img decoding=\"async\" alt=\"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii\" src=\"\/wp-content\/uploads\/2019\/11\/d24c4474b19b532a48cef4d4376287ad.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nFiecare dintre prefixe are o proprietate anume. Exist\u0103 metrici pentru servere, re\u021bele, containere, resurse, aplica\u021bii \u0219i a\u0219a mai departe. S-a realizat o filtrare clar\u0103, strict\u0103 \u0219i tipizat\u0103, unde accept\u0103m metrici de prim nivel, iar celelalte sunt pur \u0219i simplu eliminate. Asta este cum am planificat acest sistem \u00een 2015. Dar ce avem \u00een prezent? <\/p>\n<h3>Prezent: schema de interac\u021biune a componentelor de monitorizare<\/h3>\n<p>\n\u00cen primul r\u00e2nd, monitoriz\u0103m aplica\u021biile: codul nostru PHP, aplica\u021biile \u0219i microserviciile \u2014 \u00een cuvinte simple, tot ce scriu dezvoltatorii no\u0219tri. Toate aplica\u021biile trimit metrici prin UDP c\u0103tre aggregatorul Brubeck (statsd, rescris \u00een C). S-a dovedit a fi cel mai rapid \u00een urma testelor sintetice. \u0218i el trimite deja metricile agregate c\u0103tre Graphite prin TCP. <\/p>\n<p>Acesta are un tip de metrici, numit temporizatoare. Este o chestiune foarte convenabil\u0103. De exemplu, pentru fiecare conexiune a utilizatorului cu serviciul, trimite\u021bi \u00een Brubeck o metric\u0103 cu timpul de r\u0103spuns. Au sosit un milion de r\u0103spunsuri, iar aggregatorul a generat doar 10 metrici. Ave\u021bi num\u0103rul de utilizatori veniti, maximul, minima \u0219i timpul mediu de r\u0103spuns, mediana \u0219i 4 percentiles. Apoi datele sunt transmise \u00een Graphite \u0219i le vedem pe toate live. <\/p>\n<p>De asemenea, avem agregare pentru metrici privind hardware-ul, software-ul, metricile sistemului \u0219i vechea noastr\u0103 sistem de monitorizare Munin (care a func\u021bionat pentru noi p\u00e2n\u0103 \u00een 2015). Toate acestea le colect\u0103m prin demonul C din CollectD (\u00een care sunt incluse o mul\u021bime de pluginuri diferite, acesta poate interoga toate resursele sistemului de hosting pe care este instalat, trebuie doar s\u0103 specifica\u021bi \u00een configurare unde s\u0103 scrie datele) \u0219i scriem datele \u00een Graphite prin intermediul acestuia. De asemenea, suport\u0103 pluginuri Python \u0219i scripturi shell, astfel \u00eenc\u00e2t pute\u021bi scrie solu\u021bii personalizate: CollectD va colecta aceste date de la un host local sau de la distan\u021b\u0103 (presupunem c\u0103 exist\u0103 Curl) \u0219i le va trimite \u00een Graphite. <\/p>\n<p>Apoi, toate metricile pe care le-am colectat sunt trimise c\u0103tre Carbon-c-relay. Aceasta este o solu\u021bie Carbon Relay de la Graphite, recompilat\u0103 \u00een C. Este un router care centralizeaz\u0103 toate metricile pe care le trimitem de la agregatoarele noastre \u0219i le ruteaz\u0103 c\u0103tre noduri. De asemenea, \u00een timpul rut\u0103rii, verific\u0103 validitatea metricilor. Acestea, pe de o parte, trebuie s\u0103 corespund\u0103 schemei cu prefixe pe care am ar\u0103tat-o anterior \u0219i, pe de alt\u0103 parte, s\u0103 fie valide pentru Graphite. Altfel, sunt abandonate. <\/p>\n<p>Apoi, Carbon-c-relay trimite metricile c\u0103tre clusterul Graphite. Folosim Carbon-cache ca principal\u0103 solu\u021bie de stocare a metricilor, recompilat \u00een Go. Go-carbon dep\u0103\u0219e\u0219te semnificativ Carbon-cache \u00een ceea ce prive\u0219te performan\u021ba, datorit\u0103 suportului s\u0103u pentru multithreading. Acesta prime\u0219te datele \u0219i le scrie pe discuri folosind pachetul whisper (standard, scris \u00een Python). Pentru a citi datele din solu\u021biile noastre de stocare, utiliz\u0103m API-ul Graphite. Acesta func\u021bioneaz\u0103 mult mai repede dec\u00e2t standardul Graphite WEB. Ce se \u00eent\u00e2mpl\u0103 mai departe cu datele? <\/p>\n<p>Ele merg c\u0103tre Grafana. Ca surs\u0103 principal\u0103 de date, folosim clusterele noastre de Graphite, plus avem Grafana ca interfa\u021b\u0103 web pentru a afi\u0219a metricile \u0219i a construi dashboard-uri. Fiecare serviciu \u00ee\u0219i creeaz\u0103 propriul dashboard. Apoi, ei construiesc grafice pe care sunt reprezentate metricile pe care le transmit din aplica\u021biile lor. Pe l\u00e2ng\u0103 Grafana, avem \u0219i SLAM. Acesta este un demon \u00een Python care calculeaz\u0103 SLA pe baza datelor din Graphite. Dup\u0103 cum am men\u021bionat, avem mai multe zeci de microservicii, fiecare av\u00e2nd cerin\u021be specifice. Cu ajutorul SLAM, verific\u0103m documenta\u021bia \u0219i o compar\u0103m cu ceea ce avem \u00een Graphite, evalu\u00e2nd c\u00e2t de bine corespund cerin\u021bele disponibilit\u0103\u021bii serviciilor noastre. <\/p>\n<p>S\u0103 continu\u0103m: alerting. Acesta este organizat printr-un sistem puternic - Moira. Este independent\u0103, deoarece sub capot\u0103 are propriul s\u0103u Graphite. A fost dezvoltat\u0103 de echipa de la SKB \u201eKontur\u201d, scris\u0103 \u00een Python \u0219i Go, complet open source. Moira prime\u0219te acela\u0219i flux de date care se duce c\u0103tre Graphite. Dac\u0103 dintr-un anumit motiv stocarea dumneavoastr\u0103 se pr\u0103bu\u0219e\u0219te, alertingul dumneavoastr\u0103 va continua s\u0103 func\u021bioneze.<\/p>\n<p>Am implementat Moira \u00een Kubernetes, \u0219i ca baz\u0103 de date principal\u0103 folose\u0219te un cluster de servere Redis. Rezultatul este un sistem rezistent la erori. Compar\u0103 fluxul de metrici cu lista de declan\u0219atoare: dac\u0103 nu exist\u0103 men\u021biuni, atunci elimin\u0103 metrica. Astfel, este capabil\u0103 s\u0103 proceseze gigabytes de metrici pe minut. <\/p>\n<p>De asemenea, am integrat un LDAP corporativ, prin care fiecare utilizator al sistemului corporativ poate crea notific\u0103ri pentru declan\u0219atoarele existente (sau recente). Fiindc\u0103 Moira con\u021bine Graphite, sus\u021bine toate func\u021biile acestuia. A\u0219adar, mai \u00eent\u00e2i iei o linie \u0219i o copiezi \u00een Grafana. Te ui\u021bi cum sunt afi\u0219ate datele pe grafice. Apoi iei aceea\u0219i linie \u0219i o copiezi \u00een Moira. O \u00eencarci cu limite \u0219i ob\u021bii alerte. Pentru a face tot acest lucru, nu ai nevoie de cuno\u0219tin\u021be speciale. Moira poate alerta prin SMS, email, \u00een Jira, Slack\u2026 De asemenea, suport\u0103 execu\u021bia de scripturi personalizate. C\u00e2nd are loc un declan\u0219ator \u0219i este abonat\u0103 la un script personalizat sau un binar, \u00eel executeaz\u0103 \u0219i trimite JSON-ul la stdin acestui binar. Prin urmare, programul t\u0103u trebuie s\u0103-l parseze. Ce vei face cu acest JSON depinde de tine. Vrei s\u0103-l trimi\u021bi \u00een Telegram, vrei s\u0103 deschizi sarcini \u00een Jira, f\u0103 orice dore\u0219ti. <\/p>\n<p>Pentru alerte folosim \u0219i o dezvoltare proprie \u2014 Imagotag. Am adaptat un panou utilizat \u00een mod normal pentru etichetele de pre\u021b electronice \u00een magazine, pentru sarcinile noastre. Am afi\u0219at declan\u0219atoarele din Moira pe el. Acolo este indicat \u00een ce stare se afl\u0103 \u0219i c\u00e2nd au avut loc. O parte din echipa de dezvoltare a renun\u021bat la notific\u0103rile \u00een Slack \u0219i email \u00een favoarea acestui panou. <\/p>\n<p><img decoding=\"async\" alt=\"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii\" src=\"\/wp-content\/uploads\/2019\/11\/bcf454c96adaa0cff81f5e56db5b715c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDeoarece suntem o companie progresist\u0103, am monitorizat \u00een acest sistem \u0219i Kubernetes. L-am integrat \u00een sistem folosind Heapster, pe care l-am instalat \u00een cluster; acesta colecteaz\u0103 date \u0219i le trimite \u00een Graphite. Drept rezultat, schema arat\u0103 astfel: <\/p>\n<p><img decoding=\"async\" alt=\"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii\" src=\"\/wp-content\/uploads\/2019\/11\/cca0f2f9f590fc84609d48e52ebd0d21.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<\/p>\n<h3>Componentele monitoriz\u0103rii<\/h3>\n<p>Iat\u0103 o list\u0103 de linkuri c\u0103tre componentele pe care le-am folosit pentru aceast\u0103 sarcin\u0103. Toate sunt open-source. <\/p>\n<h4>Graphite:<\/h4>\n<p><\/p>\n<ul>\n<li>go-carbon: <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/lomik\/go-carbon\">github.com\/lomik\/go-carbon<\/a><\/noindex><\/li>\n<li>whisper: <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/graphite-project\/whisper\">github.com\/graphite-project\/whisper<\/a><\/noindex> <\/li>\n<li>graphite-api: <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/brutasse\/graphite-api\">github.com\/brutasse\/graphite-api<\/a><\/noindex> <\/li>\n<\/ul>\n<h4>Carbon-c-relay: <\/h4>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/grobian\/carbon-c-relay\">github.com\/grobian\/carbon-c-relay<\/a><\/noindex> <\/p>\n<h4>Brubeck: <\/h4>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/github\/brubeck\">github.com\/github\/brubeck<\/a><\/noindex> <\/p>\n<h4>Collectd:<\/h4>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/collectd.org\/\">collectd.org<\/a><\/noindex><\/p>\n<h4>Moira: <\/h4>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/moira-alert\">github.com\/moira-alert<\/a><\/noindex> <\/p>\n<h4>Grafana: <\/h4>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/grafana.com\/\">grafana.com<\/a><\/noindex> <\/p>\n<h4>Heapster: <\/h4>\n<p>\n<noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/kubernetes\/heapster\">github.com\/kubernetes\/heapster<\/a><\/noindex><\/p>\n<h3>Statistic\u0103<\/h3>\n<p>\nIat\u0103 c\u00e2teva cifre despre cum func\u021bioneaz\u0103 sistemul nostru. <\/p>\n<h4>Aggregator (brubeck)<\/h4>\n<p>\nNum\u0103rul de metrici: ~ 300 000 \/ sec<br \/>\nIntervalul de trimitere a metricelor \u00een Graphite: 30 sec<br \/>\nUtilizarea resurselor serverului: ~ 6% CPU (ne referim la servere complete); ~ 1Gb RAM; ~ 3 Mbps LAN<\/p>\n<h4>Graphite (go-carbon)<\/h4>\n<p>\nNum\u0103rul de metrici: ~ 1 600 000 \/ min<br \/>\nIntervalul de actualizare a metricelor: 30 sec<br \/>\nSchema de stocare a metricelor: 30sec 35d, 5min 90d, 10min 365d (ofer\u0103 o \u00een\u021belegere a ceea ce se \u00eent\u00e2mpl\u0103 cu serviciul pe termen lung) <br \/>\nUtilizarea resurselor serverului: ~ 10% CPU; ~ 20Gb RAM; ~ 30 Mbps LAN<\/p>\n<h3>Flexibilitate<\/h3>\n<p>\nLa Avito, pre\u021buim foarte mult flexibilitatea \u00een serviciul nostru de monitorizare. De ce a ajuns s\u0103 fie a\u0219a? \u00cen primul r\u00e2nd, componentele sale sunt interschimbabile: at\u00e2t componentele \u00een sine, c\u00e2t \u0219i versiunile lor. \u00cen al doilea r\u00e2nd \u2014 \u00eentre\u021binerea. Deoarece \u00eentregul proiect este construit pe open-source, po\u021bi s\u0103 modifici codul, s\u0103 aduci schimb\u0103ri \u0219i s\u0103 implementezi func\u021bii care nu sunt disponibile din fabric\u0103. Se folosesc stive destul de r\u0103sp\u00e2ndite, \u00een principal, Go \u0219i Python, astfel c\u0103 acest lucru se realizeaz\u0103 destul de simplu. <\/p>\n<p>Iat\u0103 un exemplu de problem\u0103 real\u0103. Ometric\u0103 \u00een Graphite este un fi\u0219ier. Are un nume. Numele fi\u0219ierului = numele metricii. \u0218i exist\u0103 un drum p\u00e2n\u0103 la el. Numele fi\u0219ierelor \u00een Linux sunt limitate la 255 de caractere. Iar noi avem (ca \u201ecomenzi interne\u201d) oameni din departamentul de baze de date. Ne spun: \u201eVrem s\u0103 monitoriz\u0103m interog\u0103rile noastre SQL. Iar acestea \u2014 nu 255 de caractere, ci 8 MB fiecare. Vrem s\u0103 le vizualiz\u0103m \u00een Grafana, s\u0103 vedem parametrii pentru aceast\u0103 interogare, iar \u0219i mai bine, vrem s\u0103 vedem topul acestor interog\u0103ri. Ar fi grozav dac\u0103 ar ap\u0103rea \u00een timp real. \u0218i ar fi fantastic s\u0103 le integr\u0103m \u00een sistemul de alerte\u201d. <\/p>\n<p><img decoding=\"async\" alt=\"Monitorizare ca serviciu: sistem modular pentru arhitectura de microservicii\" src=\"\/wp-content\/uploads\/2019\/11\/e1136c7fd0b6156c0c3fd49bab54737e.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<i>Exemplul interog\u0103rii SQL a fost preluat ca exemplu de pe <noindex><a rel=\"nofollow\" href=\"https:\/\/postgrespro.ru\/docs\/postgrespro\/9.6\/queries-with\">site-ul postgrespro.ro<\/a><\/noindex> <\/i><\/p>\n<p>Ridic\u0103m un server Redis \u0219i cu pluginurile noastre Collectd, care se conecteaz\u0103 la Postgres \u0219i preiau toate datele, trimitem metricile \u00een Graphite. Dar \u00eenlocuim numele metricii cu hash-uri. Acest hash este trimis simultan \u00een Redis ca cheie, iar \u00eentreaga interogare SQL ca valoare. Ne-a mai r\u0103mas s\u0103 facem ca Grafana s\u0103 poat\u0103 accesa Redis \u0219i s\u0103 preia aceste informa\u021bii. Deschidem API-ul Graphite, deoarece acesta este principalul interfa\u021b\u0103 de interac\u021biune a tuturor componentelor de monitorizare cu Graphite, \u0219i scriem acolo o nou\u0103 func\u021bie, care se nume\u0219te aliasByHash() \u2014 de la Grafana primim numele metricii \u0219i \u00eel folosim \u00een interogarea la Redis ca cheie, primind ca r\u0103spuns valoarea cheii, care este \u201cinterogarea SQL\u201d a noastr\u0103. Astfel, am afi\u0219at \u00een Grafana interogarea SQL, care, teoretic, nu ar fi putut fi afi\u0219at\u0103 acolo, \u00eempreun\u0103 cu statisticile acesteia (calls, rows, total_time, ...). <\/p>\n<h3>Concluzii <\/h3>\n<p>\n<b>Disponibilitate.<\/b> Serviciul nostru de monitorizare este disponibil 24\/7 din orice aplica\u021bie \u0219i orice cod. Dac\u0103 ai acces la depozite, po\u021bi scrie date \u00een serviciu. Limbajul nu este important, solu\u021biile nu sunt importante. Trebuie doar s\u0103 \u0219tii cum s\u0103 deschizi un socket, s\u0103 trimiti acolo metrica \u0219i s\u0103 \u00eenchizi socketul. <\/p>\n<p><b>Fiabilitate.<\/b> Toate componentele sunt rezistente la defectiuni \u0219i fac fa\u021b\u0103 sarcinilor noastre foarte bine. <\/p>\n<p><b>Pragul de intrare sc\u0103zut.<\/b> Pentru a utiliza acest sistem, nu trebuie s\u0103 \u00eenve\u021bi limbaje de programare \u0219i interog\u0103ri \u00een Grafana. Pur \u0219i simplu deschizi aplica\u021bia ta, introduci socketul care va trimite metricile \u00een Graphite, \u00eel \u00eenchizi, deschizi Grafana, creezi dashboard-uri acolo \u0219i examinezi comportamentul metricilor tale, primind notific\u0103ri prin Moira.<\/p>\n<p><b>Autonomie.<\/b> Totul poate fi realizat de unul singur, f\u0103r\u0103 ajutorul inginerilor DevOps. \u0218i aceasta este o caracteristic\u0103 de supra\u00eenc\u0103rcare, deoarece po\u021bi monitoriza proiectul t\u0103u chiar acum, f\u0103r\u0103 a cere ajutor - nici pentru a \u00eencepe lucrul, nici pentru modific\u0103ri. <\/p>\n<h3>Ce ne propunem? <\/h3>\n<p>\nToate cele enumerate mai jos nu sunt doar g\u00e2nduri abstracte, ci dorin\u021ba pentru care s-au f\u0103cut deja m\u0103car primii pa\u0219i. <\/p>\n<ol>\n<li>Detector de anomalii. Vrem s\u0103 implement\u0103m un serviciu care va accesa re\u021belele noastre Graphite \u0219i va verifica fiecare metric\u0103 conform diferitelor algoritmi. Exist\u0103 deja algoritmi pe care dorim s\u0103-i examin\u0103m, avem date, \u0219tim s\u0103 lucr\u0103m cu ele.\n<\/li>\n<li>Metadatele. Avem multe servicii, care se schimb\u0103 \u00een timp, la fel ca \u0219i persoanele care lucreaz\u0103 cu ele. S\u0103 \u021binem documenta\u021bia manual nu este o op\u021biune. Din acest motiv, acum metadatele sunt integrate \u00een microserviciile noastre. Acolo este specificat cine l-a dezvoltat, limbile cu care interac\u021bioneaz\u0103, cerin\u021bele SLA, unde \u0219i cui trebuie trimise notific\u0103rile. La dezvoltarea serviciului, toate datele entit\u0103\u021bii sunt create automat. \u00cen cele din urm\u0103, ob\u021bine\u021bi dou\u0103 linkuri - unul pentru triggeri, altul pentru dashboard-uri \u00een Grafana.\n<\/li>\n<li>Monitorizare \u00een fiecare cas\u0103. Consider\u0103m c\u0103 un astfel de sistem trebuie utilizat de c\u0103tre to\u021bi dezvoltatorii. \u00cen acest fel, \u0219ti\u021bi mereu unde se afl\u0103 traficul dvs., ce se \u00eent\u00e2mpl\u0103 cu acesta, unde se \u00eentrerupe \u0219i care sunt punctele sale slabe. Dac\u0103, de exemplu, apare ceva care va pr\u0103bu\u0219i serviciul dvs., ve\u021bi afla despre asta nu chiar \u00een timpul unui apel de la manager, ci dintr-un alert, \u0219i imediat pute\u021bi deschide logurile recente \u0219i vedea ce s-a \u00eent\u00e2mplat.\n<\/li>\n<li>Performan\u021b\u0103 \u00eenalt\u0103. Proiectul nostru cre\u0219te constant, iar ast\u0103zi proceseaz\u0103 aproximativ 2.000.000 de valori de metrici pe minut. Acum un an, acest indicator era de 500.000. Iar cre\u0219terea continu\u0103, ceea ce \u00eenseamn\u0103 c\u0103, \u00eentr-un viitor apropiat, Graphite (whisper) va \u00eencepe s\u0103 suprasolicite foarte mult subsistemul de stocare. A\u0219a cum am men\u021bionat, acest sistem de monitorizare este destul de versatil datorit\u0103 interschimbabilit\u0103\u021bii componentelor. Cineva se ocup\u0103 special de Graphite \u0219i \u00ee\u0219i extinde constant infrastructura, dar noi am decis s\u0103 urm\u0103m o alt\u0103 cale: s\u0103 folosim <noindex><a rel=\"nofollow\" href=\"https:\/\/clickhouse.yandex\/\">ClickHouse<\/a><\/noindex> ca spa\u021biu de stocare pentru metricile noastre. Aceast\u0103 tranzi\u021bie este practic finalizat\u0103 \u0219i \u00een cur\u00e2nd voi povesti mai \u00een detaliu cum a fost realizat\u0103: ce dificult\u0103\u021bi au ap\u0103rut \u0219i cum au fost dep\u0103\u0219ite, cum a decurs procesul de migrare, voi descrie componentele alese ca interfa\u021b\u0103 \u0219i configura\u021biile acestora. \n<\/li>\n<\/ol>\n<p>\nMul\u021bumesc pentru aten\u021bie! \u00centreba\u021bi-v\u0103 \u00eentreb\u0103rile pe subiect, voi \u00eencerca s\u0103 r\u0103spund aici sau \u00een post\u0103rile urm\u0103toare. Poate c\u0103 cineva are experien\u021b\u0103 \u00een construirea unui astfel de sistem de monitorizare sau \u00een tranzi\u021bia c\u0103tre Clickhouse \u00een situa\u021bii similare - \u00eemp\u0103rt\u0103\u0219i\u021bi-o \u00een comentarii.<br \/>\n<br \/>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/avito\/blog\/335410\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u043d\u0430 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442\u0435, \u043f\u043e\u043c\u0438\u043c\u043e \u043c\u043e\u043d\u043e\u043b\u0438\u0442\u043d\u043e\u0433\u043e \u043a\u043e\u0434\u0430, \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0438\u0440\u0443\u044e\u0442 \u0434\u0435\u0441\u044f\u0442\u043a\u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432. \u041a\u0430\u0436\u0434\u044b\u0439 \u0438\u0437 \u043d\u0438\u0445 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0435\u0433\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043b\u0438. \u0414\u0435\u043b\u0430\u0442\u044c \u044d\u0442\u043e \u0432 \u0442\u0430\u043a\u0438\u0445 \u043e\u0431\u044a\u0435\u043c\u0430\u0445 \u0441\u0438\u043b\u0430\u043c\u0438 DevOps-\u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0442\u0438\u0447\u043d\u043e. \u041c\u044b \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0430\u043b\u0438 \u0441\u0438\u0441\u0442\u0435\u043c\u0443 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043a\u0430\u043a \u0441\u0435\u0440\u0432\u0438\u0441 \u0434\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a\u043e\u0432. \u041e\u043d\u0438 \u043c\u043e\u0433\u0443\u0442 \u0441\u0430\u043c\u043e\u0441\u0442\u043e\u044f\u0442\u0435\u043b\u044c\u043d\u043e \u043f\u0438\u0441\u0430\u0442\u044c \u043c\u0435\u0442\u0440\u0438\u043a\u0438 \u0432 \u0441\u0438\u0441\u0442\u0435\u043c\u0443 \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430, \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0438\u043c\u0438, \u0441\u0442\u0440\u043e\u0438\u0442\u044c \u043d\u0430 \u0438\u0445 \u043e\u0441\u043d\u043e\u0432\u0430\u043d\u0438\u0438 \u0434\u0430\u0448\u0431\u043e\u0440\u0434\u044b, \u043f\u0440\u0438\u043a\u0440\u0443\u0447\u0438\u0432\u0430\u0442\u044c \u043a \u043d\u0438\u043c \u0430\u043b\u0435\u0440\u0442\u044b, [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-52115","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u043d\u0430 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442\u0435, \u043f\u043e\u043c\u0438\u043c\u043e \u043c\u043e\u043d\u043e\u043b\u0438\u0442\u043d\u043e\u0433\u043e \u043a\u043e\u0434\u0430, \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0438\u0440\u0443\u044e\u0442 \u0434\u0435\u0441\u044f\u0442\u043a\u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432. \u041a\u0430\u0436\u0434\u044b\u0439 \u0438\u0437 \u043d\u0438\u0445 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0435\u0433\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043b\u0438. \u0414\u0435\u043b\u0430\u0442\u044c \u044d\u0442\u043e \u0432 \u0442\u0430\u043a\u0438\u0445 \u043e\u0431\u044a\u0435\u043c\u0430\u0445 \u0441\u0438\u043b\u0430\u043c\u0438 DevOps-\u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0442\u0438\u0447\u043d\u043e.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/monitoring-kak-servis-modulnaya-sistema-dlya-mikroservisnoj-arhitektury\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043a\u0430\u043a \u0441\u0435\u0440\u0432\u0438\u0441: \u043c\u043e\u0434\u0443\u043b\u044c\u043d\u0430\u044f \u0441\u0438\u0441\u0442\u0435\u043c\u0430 \u0434\u043b\u044f \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u043d\u0430 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442\u0435, \u043f\u043e\u043c\u0438\u043c\u043e \u043c\u043e\u043d\u043e\u043b\u0438\u0442\u043d\u043e\u0433\u043e \u043a\u043e\u0434\u0430, \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0438\u0440\u0443\u044e\u0442 \u0434\u0435\u0441\u044f\u0442\u043a\u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432. \u041a\u0430\u0436\u0434\u044b\u0439 \u0438\u0437 \u043d\u0438\u0445 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0435\u0433\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043b\u0438. \u0414\u0435\u043b\u0430\u0442\u044c \u044d\u0442\u043e \u0432 \u0442\u0430\u043a\u0438\u0445 \u043e\u0431\u044a\u0435\u043c\u0430\u0445 \u0441\u0438\u043b\u0430\u043c\u0438 DevOps-\u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0442\u0438\u0447\u043d\u043e.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/monitoring-kak-servis-modulnaya-sistema-dlya-mikroservisnoj-arhitektury\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T10:59:47+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Monitorizare ca serviciu: sistem modular pentru arhitectura microserviciilor | ProHoster","description":"Ast\u0103zi, \u00een proiectul nostru, pe l\u00e2ng\u0103 codul monolitic, func\u021bioneaz\u0103 zeci de microservicii. Fiecare dintre acestea necesit\u0103 monitorizare. A face asta \u00een asemenea volume numai cu ajutorul inginerilor DevOps este problematic.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/monitoring-kak-servis-modulnaya-sistema-dlya-mikroservisnoj-arhitektury","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433 \u043a\u0430\u043a \u0441\u0435\u0440\u0432\u0438\u0441: \u043c\u043e\u0434\u0443\u043b\u044c\u043d\u0430\u044f \u0441\u0438\u0441\u0442\u0435\u043c\u0430 \u0434\u043b\u044f \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043d\u043e\u0439 \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b | ProHoster","og:description":"\u0421\u0435\u0433\u043e\u0434\u043d\u044f \u043d\u0430 \u043d\u0430\u0448\u0435\u043c \u043f\u0440\u043e\u0435\u043a\u0442\u0435, \u043f\u043e\u043c\u0438\u043c\u043e \u043c\u043e\u043d\u043e\u043b\u0438\u0442\u043d\u043e\u0433\u043e \u043a\u043e\u0434\u0430, \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0438\u0440\u0443\u044e\u0442 \u0434\u0435\u0441\u044f\u0442\u043a\u0438 \u043c\u0438\u043a\u0440\u043e\u0441\u0435\u0440\u0432\u0438\u0441\u043e\u0432. \u041a\u0430\u0436\u0434\u044b\u0439 \u0438\u0437 \u043d\u0438\u0445 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u0435\u0433\u043e \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043b\u0438. \u0414\u0435\u043b\u0430\u0442\u044c \u044d\u0442\u043e \u0432 \u0442\u0430\u043a\u0438\u0445 \u043e\u0431\u044a\u0435\u043c\u0430\u0445 \u0441\u0438\u043b\u0430\u043c\u0438 DevOps-\u0438\u043d\u0436\u0435\u043d\u0435\u0440\u043e\u0432 \u043f\u0440\u043e\u0431\u043b\u0435\u043c\u0430\u0442\u0438\u0447\u043d\u043e.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/monitoring-kak-servis-modulnaya-sistema-dlya-mikroservisnoj-arhitektury","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T21:00:00+00:00","article:modified_time":"2020-02-18T10:59:47+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"52115","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-24 02:30:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 20:49:49","updated":"2026-01-24 02:30:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/52115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=52115"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/52115\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=52115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=52115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=52115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}