{"id":75796,"date":"2020-03-28T19:42:18","date_gmt":"2020-03-28T17:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb"},"modified":"2020-03-28T19:42:18","modified_gmt":"2020-03-28T17:42:18","slug":"klaster-elasticsearch-na-200-tb","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","title":{"rendered":"Cluster Elasticsearch de 200 TB+","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ca7a31eca0b3d4faf648dfb24ffbe215.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mul\u021bi se confrunt\u0103 cu Elasticsearch. Dar ce se \u00eent\u00e2mpl\u0103 c\u00e2nd vrei s\u0103 stochezi jurnale \"\u00eentr-un volum foarte mare\" cu ajutorul s\u0103u? \u0218i cum s\u0103 supravie\u021buie\u0219ti f\u0103r\u0103 durere \u00eentr-o situa\u021bie de e\u0219ec a unuia dintre cele c\u00e2teva centre de date? Ce arhitectur\u0103 ar trebui s\u0103 construie\u0219ti \u0219i pe ce capcane po\u021bi da peste?<\/p>\n<p><\/p>\n<p>Noi, la Odnoklassniki, am decis s\u0103 abord\u0103m problema gestion\u0103rii jurnalelelor cu ajutorul Elasticsearch \u0219i acum \u00eemp\u0103rt\u0103\u0219im cu Habr experien\u021ba noastr\u0103: at\u00e2t despre arhitectur\u0103, c\u00e2t \u0219i despre capcane.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Sunt Petr Zaitsev, lucrez ca administrator de sistem la Odnoklassniki. \u00cenainte am fost de asemenea administrator, am lucrat cu Manticore Search, Sphinx search, Elasticsearch. Posibil, dac\u0103 va ap\u0103rea un alt fel de ...search, voi lucra probabil \u0219i cu acesta. Particip de asemenea la o serie de proiecte open-source pe baz\u0103 voluntar\u0103.<\/p>\n<p><\/p>\n<p>C\u00e2nd am venit la Odnoklassniki, am spus imprudent \u00een cadrul interviului c\u0103 \u0219tiu s\u0103 lucrez cu Elasticsearch. Dup\u0103 ce m-am aclimatizat \u0219i am rezolvat c\u00e2teva sarcini simple, mi-a fost \u00eencredin\u021bat\u0103 o mare sarcin\u0103 de reformare a sistemului de gestionare a jurnalelelor, care exista la acel moment. <\/p>\n<p><\/p>\n<h2 id=\"trebovaniya\">Cerin\u021be<\/h2>\n<p><\/p>\n<p>Cerin\u021bele pentru sistem au fost formulate astfel:<\/p>\n<p><\/p>\n<ul>\n<li>Ca frontend, ar trebui s\u0103 fie utilizat Graylog. Pentru c\u0103 \u00een companie exista deja experien\u021ba utiliz\u0103rii acestui produs, programatorii \u0219i testeri \u00eel \u0219tiau, le era familiar \u0219i convenabil.<\/li>\n<li>Volumul de date: \u00een medie, 50-80 de mii de mesaje pe secund\u0103, dar dac\u0103 ceva se stric\u0103, traficul nu este restric\u021bionat, acesta poate ajunge la 2-3 milioane de r\u00e2nduri pe secund\u0103<\/li>\n<li>Discut\u00e2nd cu clien\u021bii despre cerin\u021bele de vitez\u0103 de procesare a interog\u0103rilor de c\u0103utare, am realizat c\u0103 modelul tipic de utilizare a unui astfel de sistem este urm\u0103torul: oamenii caut\u0103 \u00een jurnalele aplica\u021biei lor din ultimele dou\u0103 zile \u0219i nu doresc s\u0103 a\u0219tepte mai mult de o secund\u0103 pentru rezultatul interog\u0103rii formulate. <\/li>\n<li>Adminii au insistat ca sistemul s\u0103 fie u\u0219or scalabil, f\u0103r\u0103 a necesita o \u00een\u021belegere profund\u0103 a modului \u00een care este construit. <\/li>\n<li>Astfel, singura sarcin\u0103 de \u00eentre\u021binere care era necesar\u0103 periodic pentru aceste sisteme era de a schimba o parte de hardware.<\/li>\n<li>\u00cen plus, la Odnoklassniki exist\u0103 o tradi\u021bie tehnic\u0103 minunat\u0103: orice serviciu pe care \u00eel lans\u0103m trebuie s\u0103 supravie\u021buiasc\u0103 e\u0219ecului unui centru de date (brusc, neplanificat \u0219i \u00een orice moment).<\/li>\n<\/ul>\n<p><\/p>\n<p>Cea mai recent\u0103 cerin\u021b\u0103 \u00een realizarea acestui proiect ne-a costat cel mai mult, despre care voi mai povesti \u00een detaliu.<\/p>\n<p><\/p>\n<h2 id=\"sreda\">Mediu<\/h2>\n<p><\/p>\n<p>Oper\u0103m \u00een patru centre de date, \u00eens\u0103 nodurile de date Elasticsearch pot fi amplasate doar \u00een trei (din motive non-tehnice).<\/p>\n<p><\/p>\n<p>\u00cen aceste patru centre de date se afl\u0103 aproximativ 18.000 de surse de loguri diferite - componente hardware, containere, ma\u0219ini virtuale.<\/p>\n<p><\/p>\n<p>O caracteristic\u0103 important\u0103: clusterul este lansat \u00een containere <noindex><a rel=\"nofollow\" href=\"https:\/\/podman.io\">Podman<\/a><\/noindex> nu pe ma\u0219ini fizice, ci pe <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/346868\/\">propria noastr\u0103 solu\u021bie cloud one-cloud<\/a><\/noindex>. Containerele beneficiaz\u0103 de 2 nuclei, echivalenti cu 2.0Ghz v4, cu posibilitatea de reutilizare a celorlal\u021bi nuclei \u00een cazul \u00een care sunt neutiliza\u021bi. <\/p>\n<p><\/p>\n<p>Cu alte cuvinte:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/073bffbfc3cff761bfabe0773b7f4ebc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"topologiya\">Topologie<\/h2>\n<p><\/p>\n<p>Aspectul general al solu\u021biei mi s-a p\u0103rut ini\u021bial astfel:<\/p>\n<p><\/p>\n<ul>\n<li>3-4 VIP-uri se afl\u0103 \u00een spatele \u00eenregistr\u0103rii A a domeniului Graylog, acesta este adresa la care sunt trimise logurile.<\/li>\n<li>fiecare VIP este un balansor LVS.<\/li>\n<li>Dup\u0103 aceea, logurile ajung la bateria Graylog, o parte din date sunt \u00een format GELF, iar cealalt\u0103 parte \u00een format syslog.<\/li>\n<li>Apoi, totul este scris \u00een mari batch-uri \u00een bateria de coordonatori Elasticsearch. <\/li>\n<li>Iar ace\u0219tia, la r\u00e2ndul lor, trimit solicit\u0103ri de scriere \u0219i citire c\u0103tre nodurile de date relevante. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/21efcdf6992a07e0c5e9b477c567cca9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"terminologiya\">Terminologie<\/h2>\n<p><\/p>\n<p>Poate c\u0103 nu toat\u0103 lumea este familiarizat\u0103 cu terminologia, a\u0219a c\u0103 a\u0219 dori s\u0103 m\u0103 opresc pu\u021bin asupra acesteia.<\/p>\n<p><\/p>\n<p>\u00cen Elasticsearch exist\u0103 mai multe tipuri de noduri - master, coordinator, nod de date. Mai exist\u0103 dou\u0103 tipuri pentru diverse transform\u0103ri ale logurilor \u0219i pentru leg\u0103tura \u00eentre diferite clustere, dar noi am folosit doar cele enumerate. <\/p>\n<p><\/p>\n<p><strong>Master<\/strong><br \/>\nPingeaz\u0103 toate nodurile prezente \u00een cluster, men\u021bine o hart\u0103 actualizat\u0103 a clusterului \u0219i o distribuie \u00eentre noduri, prelucreaz\u0103 logica evenimentelor, se ocup\u0103 cu diverse activit\u0103\u021bi de \u00eentre\u021binere la nivel de cluster. <\/p>\n<p><\/p>\n<p><strong>Coordinator<\/strong><br \/>\n\u00cendepline\u0219te o singur\u0103 sarcin\u0103: accept\u0103 solicit\u0103rile clien\u021bilor pentru citire sau scriere \u0219i direc\u021bioneaz\u0103 acest trafic. \u00cen cazul \u00een care este o solicitare de scriere, va \u00eentreba probabil master-ul \u00een ce shard relevant al indexului s\u0103 fie plasat\u0103 \u0219i va redirec\u021biona cererea mai departe. <\/p>\n<p><\/p>\n<p><strong>Data node<\/strong><br \/>\nStocheaz\u0103 date, execut\u0103 solicit\u0103rile de c\u0103utare venite din exterior \u0219i opera\u021biile asupra shardurilor amplasate pe ea.<\/p>\n<p><\/p>\n<p><strong>Graylog<\/strong><br \/>\nEste ceva asem\u0103n\u0103tor cu o combina\u021bie \u00eentre Kibana \u0219i Logstash \u00een stiva ELK. Graylog \u00eembin\u0103 interfa\u021ba de utilizator \u0219i un pipeline pentru procesarea logurilor. Sub capot\u0103, Graylog folose\u0219te Kafka \u0219i Zookeeper, care asigur\u0103 conectivitatea clusterei Graylog. Graylog poate cache-ui logurile (Kafka) \u00een cazul \u00een care Elasticsearch nu este disponibil \u0219i poate repeta cererile nereu\u0219ite pentru citire \u0219i scriere, grup\u00e2nd \u0219i etichet\u00e2nd logurile conform regulilor specificate. La fel ca Logstash, Graylog are func\u021bionalitate de modificare a liniilor \u00eenainte de a le scrie \u00een Elasticsearch.<\/p>\n<p><\/p>\n<p>\u00cen plus, Graylog are o descoperire de servicii \u00eencorporat\u0103, care permite ob\u021binerea \u00eentregului harta a clusterei pe baza unei singure noduri Elasticsearch disponibile \u0219i filtrarea acesteia dup\u0103 un anumit tag, ceea ce ofer\u0103 posibilitatea de a direc\u021biona cererile c\u0103tre anumite containere.<\/p>\n<p><\/p>\n<p>Vizual, aceasta arat\u0103 cam a\u0219a:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/8673ba9c0300ea0153d34a9f98afbcf2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Aceasta este o captur\u0103 de ecran de pe o instan\u021b\u0103 specific\u0103. Aici construim un histogram\u0103 pe baza unei interog\u0103ri de c\u0103utare, afi\u0219\u00e2nd liniile relevante.<\/p>\n<p><\/p>\n<h2 id=\"indeksy\">Indec\u0219i<\/h2>\n<p><\/p>\n<p>\u00centorc\u00e2ndu-ne la arhitectura sistemului, a\u0219 dori s\u0103 m\u0103 opresc mai \u00een detaliu asupra modului \u00een care am construit modelul de indec\u0219i, astfel \u00eenc\u00e2t s\u0103 func\u021bioneze corect. <\/p>\n<p><\/p>\n<p>\u00cen schema prezentat\u0103 anterior, acesta este cel mai de jos nivel: nodurile de date Elasticsearch.<\/p>\n<p><\/p>\n<p>Un index este o entitate virtual\u0103 mare, format\u0103 din sharde Elasticsearch. Fiecare shard este, de fapt, un index Lucene. Iar fiecare index Lucene este compus din unul sau mai multe segmente. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/13b858cb102dce26b0851516b2d36c43.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Atunci c\u00e2nd am proiectat, am estimat c\u0103 pentru a \u00eendeplini cerin\u021ba de vitez\u0103 de citire la un volum mare de date trebuie s\u0103 \u201edistribuim\u201d uniform aceste date pe nodurile de date. <\/p>\n<p><\/p>\n<p>Aceasta a dus la concluzia c\u0103 num\u0103rul de sharde pe index (cu replici) trebuie s\u0103 fie strict egal cu num\u0103rul de noduri de date. \u00cen primul r\u00e2nd, pentru a asigura un factor de replicare de dou\u0103 (adic\u0103 putem pierde jum\u0103tate din cluster). \u0218i, \u00een al doilea r\u00e2nd, pentru a procesa cererile de citire \u0219i scriere, pe cel pu\u021bin jum\u0103tate din cluster.<\/p>\n<p><\/p>\n<p>Am stabilit ini\u021bial timpul de p\u0103strare ca fiind 30 de zile.<\/p>\n<p><\/p>\n<p>Distribu\u021bia shardelor poate fi reprezentat\u0103 grafic \u00een urm\u0103torul fel:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/242726053360d1a6bdf4e527edc6cae6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u00centregul dreptunghi gri \u00eenchis este indexul. Cadrul ro\u0219u din st\u00e2nga reprezint\u0103 shardul principal, primul din index. Iar cadranul albastru este shardul replicat. Ele se afl\u0103 \u00een centre de date diferite.<\/p>\n<p><\/p>\n<p>Atunci c\u00e2nd ad\u0103ug\u0103m un nou shard, acesta ajunge \u00een al treilea data center. \u0218i, \u00een cele din urm\u0103, ob\u021binem o structur\u0103 care permite pierderea DC f\u0103r\u0103 pierderea consisten\u021bei datelor:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/27cbe4d4606f5ae5f3013ea46504134b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rotirea indexurilor, adic\u0103 crearea unui nou index \u0219i \u0219tergerea celui mai vechi, a fost setat\u0103 la 48 de ore (\u00een func\u021bie de modelul de utilizare a indexului: cele mai frecvente c\u0103ut\u0103ri se fac \u00een ultimele 48 de ore).<\/p>\n<p><\/p>\n<p>Acest interval de rotire a indexurilor este legat de urm\u0103toarele motive:<\/p>\n<p><\/p>\n<p>C\u00e2nd un anumit nod de date prime\u0219te o interogare de c\u0103utare, din punct de vedere al performan\u021bei, este mai avantajos s\u0103 se interogheze un singur shard, dac\u0103 dimensiunea lui este comparabil\u0103 cu dimensiunea heap-ului nodului. Acest lucru permite p\u0103strarea p\u0103r\u021bii \u201efierbin\u021bi\u201d a indexului \u00een heap \u0219i accesarea rapid\u0103 a acesteia. C\u00e2nd devin multe \u201ep\u0103r\u021bi fierbin\u021bi\u201d, viteza de c\u0103utare \u00een index se degradeaz\u0103.<\/p>\n<p><\/p>\n<p>Atunci c\u00e2nd un nod \u00eencepe s\u0103 execute o interogare de c\u0103utare pe un shard, aloc\u0103 un num\u0103r de fire egal cu num\u0103rul nucleelor de hyperthreading ale ma\u0219inii fizice. Dac\u0103 interogarea de c\u0103utare implic\u0103 un num\u0103r mare de sharduri, num\u0103rul de fire cre\u0219te propor\u021bional. Acest lucru are un impact negativ asupra vitezei de c\u0103utare \u0219i afecteaz\u0103 negativ indexarea noilor date. <\/p>\n<p><\/p>\n<p>Pentru a asigura laten\u021ba necesar\u0103 c\u0103ut\u0103rii, am decis s\u0103 folosim SSD-uri. Pentru o procesare rapid\u0103 a cererilor, ma\u0219inile pe care erau plasate aceste containere trebuiau s\u0103 aib\u0103 cel pu\u021bin 56 de nuclee. Num\u0103rul de 56 a fost ales ca o valoare condi\u021bionat\u0103, care determin\u0103 num\u0103rul de threaduri pe care le va genera Elasticsearch \u00een timpul func\u021bion\u0103rii. \u00cen Elasticsearch, mul\u021bi parametri ai thread pool-ului depind direct de num\u0103rul de nuclee disponibile, ceea ce influen\u021beaz\u0103 \u00een mod direct num\u0103rul necesar de noduri \u00een cluster conform principiului \"mai pu\u021bine nuclee \u2014 mai multe noduri\". <\/p>\n<p><\/p>\n<p>\u00cen rezultatul final, am ob\u021binut c\u0103, \u00een medie, un shard c\u00e2nt\u0103re\u0219te cam 20 de gigabai\u021bi, iar pentru 1 index sunt 360 de sharduri. Prin urmare, dac\u0103 le rotim la fiecare 48 de ore, avem 15 dintre ele. Fiecare index con\u021bine date pentru 2 zile.<\/p>\n<p><\/p>\n<h2 id=\"shemy-zapisi-i-chteniya-dannyh\">Scheme de scriere \u0219i citire a datelor<\/h2>\n<p><\/p>\n<p>S\u0103 analiz\u0103m cum sunt scrise datele \u00een acest sistem.<\/p>\n<p><\/p>\n<p>S\u0103 presupunem c\u0103 avem o solicitare care vine de la Graylog \u00een coordonator. De exemplu, dorim s\u0103 index\u0103m 2-3 mii de r\u00e2nduri. <\/p>\n<p><\/p>\n<p>Coordonatorul, primind cererea de la Graylog, intervieveaz\u0103 masterul: \u201e\u00cen cererea de indexare, a fost specificat \u00een mod concret indexul, dar nu s-a men\u021bionat \u00een care shard trebuie s\u0103-l scrie.\u201d <\/p>\n<p><\/p>\n<p>Masterul r\u0103spunde: \u201eScrie aceast\u0103 informa\u021bie \u00een shardul num\u0103rul 71\u201d, dup\u0103 care este trimis\u0103 direct c\u0103tre nodul de date relevant, unde se afl\u0103 shardul primar num\u0103rul 71.<\/p>\n<p><\/p>\n<p>Apoi, jurnalul tranzac\u021biilor este replicat pe replica-shard, care se afl\u0103 deja \u00een alt centru de date.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/fdb6a77cb78b1eed290e440478569e3b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Din Graylog, coordonatorul prime\u0219te o solicitare de c\u0103utare. Coordonatorul o redirec\u021bioneaz\u0103 pe index, \u00een timp ce Elasticsearch distribuie cererile \u00eentre primar-shard \u0219i replica-shard pe baza principiului round-robin. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/cb32bac878e04d6f50d103ebbc395a7f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Cele 180 de noduri r\u0103spund inegal, iar, pe m\u0103sur\u0103 ce r\u0103spund, coordonatorul acumuleaz\u0103 informa\u021biile pe care nodurile de date mai rapide le-au \u201efluierat\u201d deja \u00een el. Dup\u0103 aceea, c\u00e2nd fie toat\u0103 informa\u021bia a sosit, fie cererea a atins timeout-ul, returneaz\u0103 totul direct clientului. <\/p>\n<p><\/p>\n<p>\u00centreaga acest\u0103 sistem\u0103 proceseaz\u0103, \u00een medie, cererile de c\u0103utare pentru ultimele 48 de ore \u00een 300-400ms, excluz\u00e2nd acele cereri cu wildcard de \u00eenceput.<\/p>\n<p><\/p>\n<h2 id=\"cvetochki-s-elasticsearch-nastroyka-java\">\u201eFlori\u201d cu Elasticsearch: configurarea Java<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/fc9ac4b8ee41bc35f5f01b8d9c4ff2f6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pentru ca toate acestea s\u0103 func\u021bioneze a\u0219a cum ne-am dorit ini\u021bial, am petrecut mult timp ajust\u00e2nd cele mai variate aspecte \u00een cluster. <\/p>\n<p><\/p>\n<p>Prima parte a problemelor descoperite a fost legat\u0103 de modul \u00een care Java este configurat\u0103 implicit \u00een Elasticsearch. <\/p>\n<p><\/p>\n<p><strong>Problema \u00eent\u00e2i<\/strong><br \/>\nAm observat un num\u0103r foarte mare de mesaje referitoare la faptul c\u0103, la nivelul Lucene, atunci c\u00e2nd erau pornite job-uri de fundal, fuziunile segmentelor Lucene se \u00eencheiau cu eroare. \u00cen acela\u0219i timp, \u00een jurnale s-a putut observa c\u0103 aceasta era o eroare OutOfMemoryError. Din telemetrie am v\u0103zut c\u0103 heap-ul era liber, \u0219i nu era clar de ce aceasta opera\u021bie e\u0219ueaz\u0103. <\/p>\n<p><\/p>\n<p>S-a descoperit c\u0103 fuzion\u0103rile indexurilor Lucene au loc \u00een afara heap-ului. Iar containerele sunt destul de strict limitate \u00een ceea ce prive\u0219te resursele consumate. Aceste resurse includeau doar heap-ul (valoarea heap.size era aproximativ egal\u0103 cu RAM), iar unele opera\u021biuni off-heap c\u0103deau cu eroare de alocare a memoriei, dac\u0103 dintr-un motiv oarecare nu se \u00eencadrau \u00een cele ~500MB care r\u0103m\u00e2neau p\u00e2n\u0103 la limit\u0103.<\/p>\n<p><\/p>\n<p>Fixul a fost destul de trivial: am crescut volumul de RAM disponibil pentru container, dup\u0103 care am uitat de faptul c\u0103 am avut vreo problem\u0103 de acest tip.<\/p>\n<p><\/p>\n<p><strong>Problema a doua<\/strong><br \/>\nDup\u0103 aproximativ 4-5 zile de la lansarea cluster-ului, am observat c\u0103 nodurile de date \u00eencep s\u0103 cad\u0103 periodic din cluster \u0219i se reintegreaz\u0103 \u00een el \u00een decurs de 10-20 de secunde. <\/p>\n<p><\/p>\n<p>C\u00e2nd am \u00eenceput s\u0103 investig\u0103m, am descoperit c\u0103 memoria off-heap din Elasticsearch nu este practic controlat\u0103 \u00een niciun fel. C\u00e2nd am alocat mai mult\u0103 memorie containerului, am ob\u021binut capacitatea de a umple pool-urile de buffer direct cu informa\u021bii diverse, iar acestea erau cur\u0103\u021bate doar dup\u0103 ce era lansat un GC explicit din partea Elasticsearch. <\/p>\n<p><\/p>\n<p>\u00cen unele cazuri, aceast\u0103 opera\u021biune se desf\u0103\u0219ura destul de lent, iar \u00een acest timp cluster-ul reu\u0219ea s\u0103 marcheze acest nod ca fiind deja ie\u0219it. Aceast\u0103 problem\u0103 este bine descris\u0103 <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/elasticsearch-5-6-very-quickly-increasing-direct-buffer-pools\/119561\">aici<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>Solu\u021bia a fost urm\u0103toarea: am limitat Java s\u0103 utilizeze majoritatea memoriei din afara heap-ului pentru aceste opera\u021biuni. Am limitat-o la 16 gigabai\u021bi (-XX:MaxDirectMemorySize=16g), reu\u0219ind astfel s\u0103 apel\u0103m GC explicit mult mai des, iar acesta s\u0103 func\u021bioneze semnificativ mai repede, stabiliz\u00e2nd astfel cluster-ul.<\/p>\n<p><\/p>\n<p><strong>Problema a treia<\/strong><br \/>\nDac\u0103 crede\u021bi c\u0103 problemele cu \u00abnodurile care p\u0103r\u0103sesc cluster-ul \u00een cele mai neprev\u0103zute momente\u00bb s-au oprit aici, v\u0103 \u00een\u0219ela\u021bi. <\/p>\n<p><\/p>\n<p>Atunci c\u00e2nd am configurat lucrul cu indexurile, ne-am oprit asupra mmapfs, pentru a <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/benchmarks-default-niofs-vs-memory-mapped-mmapfs\/13118\/2\">reduce timpul de c\u0103utare<\/a><\/noindex> pe shard-urile recente cu o segmentare mare. Aceasta a fost o gre\u0219eal\u0103 destul de grav\u0103, deoarece utilizarea mmapfs mapeaz\u0103 fi\u0219ierul \u00een memoria operativ\u0103, iar ulterior lucr\u0103m deja cu fi\u0219ierul mapped. Din cauza aceasta, atunci c\u00e2nd \u00eencerc\u0103m s\u0103 oprim thread-urile din aplica\u021bie, ajungem destul de greu la safepoint, iar pe drumul c\u0103tre acesta, aplica\u021bia \u00eenceteaz\u0103 s\u0103 r\u0103spund\u0103 la cererile master-ului cu privire la starea ei. Prin urmare, master-ul consider\u0103 c\u0103 nodul nu mai este prezent \u00een cluster. Dup\u0103 aceasta, dup\u0103 aproximativ 5-10 secunde, garbage collector-ul \u00ee\u0219i finalizeaz\u0103 opera\u021bia, nodul revine la via\u021b\u0103, intr\u0103 din nou \u00een cluster \u0219i \u00eencepe ini\u021bializarea shard-urilor. Totul a sem\u0103nat foarte mult cu \u201eprodu\u021bia pe care o merit\u0103m\u201d \u0219i nu era potrivit pentru nimic serios.<\/p>\n<p><\/p>\n<p>Pentru a sc\u0103pa de acest comportament, mai \u00eent\u00e2i am trecut la standardul niofs, iar apoi, c\u00e2nd ne-am migrat de la versiunile a cincea la a \u0219asea a Elastic, am \u00eencercat hybridfs, unde aceast\u0103 problem\u0103 nu s-a mai prezentat. Pute\u021bi citi mai multe despre tipurile de stocare <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/index-modules-store.html\">aici<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><strong>Problema a patra<\/strong><br \/>\nApoi a fost o alt\u0103 problem\u0103 foarte captivant\u0103, pe care am tratat-o extrem de mult timp. Am \u00eent\u00e2mpinat-o timp de 2-3 luni, deoarece modelul s\u0103u era complet neclar. <\/p>\n<p><\/p>\n<p>Uneori, coordonatorii no\u0219tri intrau \u00een Full GC, de obicei dup\u0103-amiaza, \u0219i nu se mai \u00eentorceau. \u00cen timpul log\u0103rii \u00eent\u00e2rzierilor GC, acest lucru ar\u0103ta astfel: totul mergea bine, bine, bine, apoi brusc \u2014 \u0219i totul devenea brusc r\u0103u. <\/p>\n<p><\/p>\n<p>La \u00eenceput, am crezut c\u0103 avem un utilizator r\u0103u care trimite o cerere care scoate coordonatorul din modul de lucru. Am logat cererile foarte mult, \u00eencerc\u00e2nd s\u0103 ne d\u0103m seama ce se \u00eent\u00e2mpla. <\/p>\n<p><\/p>\n<p>\u00cen cele din urm\u0103, s-a dovedit c\u0103 \u00een momentul \u00een care un utilizator trimite o cerere foarte mare \u0219i aceasta ajunge pe un coordonator Elasticsearch specific, unele noduri r\u0103spund mai lent dec\u00e2t altele. <\/p>\n<p><\/p>\n<p>Iar timpul pe care coordonatorul \u00eel a\u0219teapt\u0103 pentru a ob\u021bine r\u0103spunsul de la toate nodurile, el acumuleaz\u0103 rezultatele trimise de nodurile care au r\u0103spuns deja. Pentru GC, aceasta \u00eenseamn\u0103 c\u0103 modelul de utilizare a heap-ului se schimb\u0103 foarte repede. \u0218i GC-ul pe care l-am folosit nu se descurca cu aceast\u0103 problem\u0103. <\/p>\n<p><\/p>\n<p>Singura solu\u021bie pe care am g\u0103sit-o pentru a schimba comportamentul clusterului \u00een aceast\u0103 situa\u021bie a fost migrarea la JDK13 \u0219i utilizarea colectorului de gunoi Shenandoah. Aceasta a rezolvat problema, iar coordonatorii nu mai c\u0103deau. <\/p>\n<p><\/p>\n<p>Dup\u0103 aceasta, problemele cu Java s-au \u00eencheiat \u0219i au \u00eenceput problemele cu l\u0103\u021bimea de band\u0103. <\/p>\n<p><\/p>\n<h2 id=\"yagodki-s-elasticsearch-propusknaya-sposobnost\">\u201eCire\u0219ele\u201d cu Elasticsearch: l\u0103\u021bimea de band\u0103<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/bb5c08193d7d764515235e825cd30ce5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Problemele cu l\u0103\u021bimea de band\u0103 \u00eenseamn\u0103 c\u0103 clusterul nostru func\u021bioneaz\u0103 stabil, dar \u00een momentele de v\u00e2rf de documente indexate \u0219i \u00een momentele de manevr\u0103 performan\u021ba este insuficient\u0103.<\/p>\n<p><\/p>\n<p>Primul simptom \u00eent\u00e2lnit: la anumite \u201eexplozive\u201d \u00een produc\u021bie, c\u00e2nd se genereaz\u0103 brusc o cantitate foarte mare de loguri, \u00een Graylog \u00eencepe s\u0103 apar\u0103 frecvent eroarea de indexare es_rejected_execution. <\/p>\n<p><\/p>\n<p>Aceasta se \u00eent\u00e2mpla deoarece thread_pool.write.queue pe un nod de date, \u00eenainte ca Elasticsearch s\u0103 poat\u0103 procesa cererea de indexare \u0219i s\u0103 adauge informa\u021bia \u00een shard pe disc, \u00een mod implicit poate cache doar 200 de cereri. \u0218i \u00een <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">documenta\u021bia Elasticsearch<\/a><\/noindex> despre acest parametru se vorbe\u0219te foarte pu\u021bin. Se indic\u0103 doar num\u0103rul maxim de fire de execu\u021bie \u0219i dimensiunea implicit\u0103.<\/p>\n<p><\/p>\n<p>Desigur, am mers s\u0103 ajust\u0103m aceast\u0103 valoare \u0219i am descoperit urm\u0103toarele: \u00een configura\u021bia noastr\u0103 specific\u0103, putem cache p\u00e2n\u0103 la 300 de cereri destul de bine, dar o valoare mai mare aduce riscul de a reveni \u00een Full GC.<\/p>\n<p><\/p>\n<p>\u00cen plus, av\u00e2nd \u00een vedere c\u0103 acestea sunt pachete de mesaje care vin \u00eentr-o singur\u0103 cerere, a fost necesar s\u0103 ajust\u0103m Graylog astfel \u00eenc\u00e2t s\u0103 scrie nu frecvent \u0219i \u00een mici loturi, ci \u00een loturi mari sau o dat\u0103 la 3 secunde, dac\u0103 lotul nu este \u00eenc\u0103 plin. \u00cen acest caz, informa\u021bia pe care o scriem \u00een Elasticsearch devine disponibil\u0103 nu \u00een dou\u0103 secunde, ci \u00een cinci (ceea ce ne mul\u021bume\u0219te), dar num\u0103rul de retrageri necesare pentru a \u00eempinge un lot mare de informa\u021bii se reduce.<\/p>\n<p><\/p>\n<p>Acest lucru este deosebit de important \u00een momentele \u00een care avem ceva care a c\u0103zut undeva \u0219i anun\u021b\u0103 cu fervoare despre acest lucru, pentru a nu ob\u021bine un Elastic complet spam-uit, iar dup\u0103 un timp, nodurile Graylog care nu mai func\u021bioneaz\u0103 din cauza buffer-elor blocate.<\/p>\n<p><\/p>\n<p>\u00cen plus, atunci c\u00e2nd au avut loc aceste explozii \u00een produc\u021bie, am primit pl\u00e2ngeri de la programatori \u0219i testerii: \u00een momentul \u00een care aveau cu adev\u0103rat nevoie de aceste jurnale, acestea le erau furnizate foarte lent.<\/p>\n<p><\/p>\n<p>Am \u00eenceput s\u0103 investig\u0103m. Pe de o parte, a fost clar c\u0103 at\u00e2t cererile de c\u0103utare, c\u00e2t \u0219i cererile de indexare func\u021bioneaz\u0103, de fapt, pe acelea\u0219i ma\u0219ini fizice, \u0219i \u00eentr-un fel sau altul vor exista anumite sc\u0103deri de performan\u021b\u0103. <\/p>\n<p><\/p>\n<p>Dar acest lucru putea fi par\u021bial evitat datorit\u0103 faptului c\u0103 \u00een versiunile \u0219ase de Elasticsearch a ap\u0103rut un algoritm care permite distribuirea cererilor \u00eentre nodurile de date relevante nu pe un principiu aleatoriu round-robin (containerul, care se ocup\u0103 de indexare \u0219i men\u021bine shard-ul principal, poate fi foarte ocupat, nu va putea r\u0103spunde rapid), ci direc\u021bion\u00e2nd aceast\u0103 cerere c\u0103tre un container mai pu\u021bin aglomerat cu replica-shard, care va r\u0103spunde semnificativ mai repede. Cu alte cuvinte, am ajuns la use_adaptive_replica_selection: true.<\/p>\n<p><\/p>\n<p>Imaginea citirii \u00eencepe s\u0103 arate astfel:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ae7c1645bb63692e580b1bca5c6e5db0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Trecerea la acest algoritm a permis \u00eembun\u0103t\u0103\u021birea semnificativ\u0103 a timpului de interogare \u00een momentele \u00een care aveam un flux mare de jurnale pe scriere.<\/p>\n<p><\/p>\n<p>\u00cen cele din urm\u0103, principala problem\u0103 consta \u00een ie\u0219irea f\u0103r\u0103 durere din centrul de date.<\/p>\n<p><\/p>\n<p>Ce ne doream de la cluster imediat dup\u0103 pierderea conexiunii cu un DC: <\/p>\n<p><\/p>\n<ul>\n<li>Dac\u0103 masterul curent se afl\u0103 \u00een centrul de date c\u0103zut, acesta va fi reselectat \u0219i va muta rolul s\u0103u pe un alt nod dintr-un alt DC.<\/li>\n<li>Masterul va elimina rapid din cluster toate nodurile inaccesibile.<\/li>\n<li>Pe baza celor r\u0103mase, el va \u00een\u021belege: \u00een centrul de date pierdut am avut aceste \u0219arduri primare, va promova rapid \u0219ardurile replica complementare \u00een centrele de date r\u0103mase, \u0219i ne va continua indexarea datelor. <\/li>\n<li>Ca rezultat, capacitatea de band\u0103 a clusterei va degrada lin pentru scriere \u0219i citire, \u00eens\u0103, \u00een general, totul va func\u021biona, chiar dac\u0103 \u00eencet, dar stabil.<\/li>\n<\/ul>\n<p><\/p>\n<p>A\u0219a cum s-a dovedit, am vrut ceva de genul:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/71fa3d19a61eb58ddb93a4df55c759c5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u0218i am ob\u021binut urm\u0103toarele:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ab32be4dfbbb4bcc3279d57021707980.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Cum s-a \u00eent\u00e2mplat asta? <\/p>\n<p><\/p>\n<p>\u00cen momentul pr\u0103bu\u0219irii centrului de date, punctul nostru critic a fost masterul.<\/p>\n<p><\/p>\n<p>De ce?<\/p>\n<p><\/p>\n<p>Problema este c\u0103 \u00een master exist\u0103 TaskBatcher, care se ocup\u0103 cu distribuitul unor sarcini \u0219i evenimente specifice \u00een cluster. Orice ie\u0219ire de nod, orice promovare a unui \u0219ard din replica \u00een primar, orice sarcin\u0103 de a crea un \u0219ard undeva \u2014 toate acestea ajung mai \u00eent\u00e2i la TaskBatcher, unde sunt procesate secven\u021bial \u0219i \u00eentr-un singur fir.<\/p>\n<p><\/p>\n<p>\u00cen momentul ie\u0219irii unui centru de date, se \u00eent\u00e2mpla ca toate nodurile de date din centrele de date supravie\u021buitoare s\u0103 considere c\u0103 trebuie s\u0103 informeze masterul \u00abam pierdut astfel de \u0219arduri \u0219i astfel de noduri de date\u00bb. <\/p>\n<p><\/p>\n<p>\u00cen acela\u0219i timp, nodurile de date supravie\u021buitoare trimiteau toate aceste informa\u021bii actualului master \u0219i \u00eencercau s\u0103 a\u0219tepte confirmarea c\u0103 el le-a primit. Nu a\u0219teptau acest lucru, deoarece masterul primea sarcini mai repede dec\u00e2t reu\u0219ea s\u0103 r\u0103spund\u0103. Nodurile repetau cererile din cauza timeout-ului, iar masterul \u00een acel timp deja nu mai \u00eencerca s\u0103 le r\u0103spund\u0103, fiind complet absorbit \u00een sarcina de sortare a cererilor dup\u0103 prioritate.<\/p>\n<p><\/p>\n<p>\u00cen forma terminal\u0103, nodurile de date spaminu-l pe master at\u00e2t de mult \u00eenc\u00e2t acesta ajungea \u00een full GC. Dup\u0103 asta, rolul masterului se transfera pe un alt nod, cu care se \u00eent\u00e2mpla exact acela\u0219i lucru, \u0219i \u00een final clusterul se destr\u0103ma complet. <\/p>\n<p><\/p>\n<p>Am efectuat m\u0103sur\u0103tori, iar p\u00e2n\u0103 la versiunea 6.4.0, unde acest lucru a fost reparat, ne era suficient s\u0103 scoatem simultan doar 10 noduri de date din 360 pentru a distruge complet clusterul.<\/p>\n<p><\/p>\n<p>Asta ar\u0103ta aproximativ a\u0219a:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/3bed554b2703e7521339e9d943648ee4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dup\u0103 versiunea 6.4.0, unde s-a reparat acest bug enervant, nodurile de date au \u00eencetat s\u0103-l distrug\u0103 pe master. Dar nu a devenit mai \u201einteligent\u201d din aceasta. Adic\u0103: atunci c\u00e2nd scoatem 2, 3 sau 10 (orice num\u0103r diferit de unu) noduri de date, masterul prime\u0219te un prim mesaj, care spune c\u0103 nodul A a ie\u0219it, \u0219i \u00eencearc\u0103 s\u0103 comunice despre acest lucru nodului B, nodului C, nodului D. <\/p>\n<p><\/p>\n<p>\u00cen prezent, singura solu\u021bie pentru a gestiona acest lucru este s\u0103 set\u0103m un timeout pentru \u00eencerc\u0103rile de a comunica cu cineva, de aproximativ 20-30 de secunde, \u0219i astfel s\u0103 control\u0103m viteza de ie\u0219ire a centrului de date din cluster.<\/p>\n<p><\/p>\n<p>\u00cen principiu, acest lucru se \u00eencadreaz\u0103 \u00een cerin\u021bele ini\u021bial stabilite pentru produsul final \u00een cadrul proiectului, dar din perspectiva \u201e\u0219tiin\u021bei pure\u201d, acesta este un bug. Care, de altfel, a fost rezolvat cu succes de dezvoltatori \u00een versiunea 7.2.<\/p>\n<p><\/p>\n<p>Mai mult, atunci c\u00e2nd o anumit\u0103 dat\u0103-nod ie\u0219ea din func\u021biune, se \u00eent\u00e2mpla c\u0103 a difuza informa\u021bii despre ie\u0219irea sa era mai important dec\u00e2t a anun\u021ba \u00eentregul cluster c\u0103 pe aceasta se aflau anumite primary-shard (pentru a promova replica-shard \u00een alt centru de date la primary, unde putea s\u0103 se scrie informa\u021bie).<\/p>\n<p><\/p>\n<p>Prin urmare, c\u00e2nd totul s-a terminat, nodurile de date ie\u0219ite nu sunt marcate imediat ca stale. \u00cen consecin\u021b\u0103, suntem nevoi\u021bi s\u0103 a\u0219tept\u0103m p\u00e2n\u0103 c\u00e2nd toate ping-urile s\u0103 expire pentru nodurile de date ie\u0219ite \u0219i abia dup\u0103 aceea clusterul nostru \u00eencepe s\u0103 comunice despre faptul c\u0103 acolo, acolo \u0219i acolo trebuie continuat\u0103 \u00eenregistrarea informa\u021biilor. Despre acest subiect se poate citi mai \u00een detaliu <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/elastic\/elasticsearch\/issues\/46909\">aici<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>\u00cen cele din urm\u0103, opera\u021biunea de ie\u0219ire a centrului de date ne ocup\u0103 ast\u0103zi aproximativ 5 minute \u00een orele de v\u00e2rf. Pentru o ma\u0219in\u0103 at\u00e2t de mare \u0219i greoaie este un rezultat destul de bun.<\/p>\n<p><\/p>\n<p>\u00cen cele din urm\u0103, am ajuns la urm\u0103toarea solu\u021bie:<\/p>\n<p><\/p>\n<ul>\n<li>Avem 360 de date-noduri cu discuri de 700 de gigabytes.<\/li>\n<li>60 de coordonatori pentru rutarea traficului pe aceste date-noduri.<\/li>\n<li>40 de masteri, care ne-au r\u0103mas ca un fel de mo\u0219tenire din versiunile anterioare 6.4.0 \u2014 pentru a supravie\u021bui ie\u0219irii centrului de date, am fost moral preg\u0103ti\u021bi s\u0103 pierdem c\u00e2teva ma\u0219ini, pentru a ne asigura c\u0103, chiar \u0219i \u00een cel mai r\u0103u scenariu, s\u0103 avem cvorum de masteri.<\/li>\n<li>Orice \u00eencerc\u0103ri de a combina rolurile pe un singur container s-au lovit de faptul c\u0103, mai devreme sau mai t\u00e2rziu, nodul se strica sub sarcin\u0103. <\/li>\n<li>\u00cen \u00eentregul cluster se folose\u0219te heap.size, egal cu 31 de gigabytes: toate \u00eencerc\u0103rile de a reduce dimensiunea au dus la faptul c\u0103 la interog\u0103rile de c\u0103utare grele cu wildcard-uri de \u00eenceput fie se omorau unele noduri, fie se activa circuit breaker \u00een Elasticsearch.<\/li>\n<li>\u00cen plus, pentru a asigura performan\u021ba c\u0103ut\u0103rii, ne-am str\u0103duit s\u0103 men\u021binem num\u0103rul de obiecte \u00een cluster c\u00e2t mai redus posibil, pentru a procesa c\u00e2t mai pu\u021bine evenimente \u00een cel mai \u00eengust punct, pe care l-am avut la master.<\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"naposledok-o-monitoringe\">\u00cen final, despre monitorizare<\/h2>\n<p><\/p>\n<p>Pentru ca toate acestea s\u0103 func\u021bioneze a\u0219a cum ne-am propus, monitoriz\u0103m urm\u0103toarele:<\/p>\n<p><\/p>\n<ul>\n<li>Fiecare nod de date raporteaz\u0103 \u00een cloud-ul nostru c\u0103 exist\u0103 \u0219i c\u0103 pe el se afl\u0103 anumite fragmente. C\u00e2nd stingem ceva \u00eentr-un loc, clusterul raporteaz\u0103 \u00een 2-3 secunde c\u0103, \u00een centrul A, am stins nodurile 2, 3 \u0219i 4 \u2014 aceasta \u00eenseamn\u0103 c\u0103 \u00een alte centre de date nu putem stinge nodurile pe care mai exist\u0103 fragmente unice.<\/li>\n<li>Cunosc\u00e2nd comportamentul maestrului, ne uit\u0103m cu aten\u021bie la num\u0103rul de sarcini \u00een a\u0219teptare. Deoarece chiar \u0219i o sarcin\u0103 blocat\u0103, dac\u0103 nu este timeoutat\u0103 la timp, teoretic \u00eentr-o situa\u021bie de urgen\u021b\u0103 poate fi motivul pentru care nu vom reu\u0219i, de exemplu, s\u0103 promov\u0103m un fragment replica \u00een primar, ceea ce va duce la oprirea index\u0103rii.<\/li>\n<li>De asemenea, ne uit\u0103m foarte atent la \u00eent\u00e2rzierile garbage collector-ului, pentru c\u0103 am avut deja dificult\u0103\u021bi mari \u00een optimizare cu acest aspect.<\/li>\n<li>Rejec\u021biile pe thread-uri, pentru a \u00een\u021belege din timp unde se afl\u0103 'g\u00e2tul de sticl\u0103'.<\/li>\n<li>\u0218i metrici standard, cum ar fi heap, RAM \u0219i I\/O.<\/li>\n<\/ul>\n<p><\/p>\n<p>Atunci c\u00e2nd se construie\u0219te monitorizarea, trebuie s\u0103 se \u021bin\u0103 cont de particularit\u0103\u021bile Thread Pool din Elasticsearch. <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">Documenta\u021bia Elasticsearch<\/a><\/noindex> descrie posibilit\u0103\u021bile de configurare \u0219i valorile implicite pentru c\u0103utare, indexare, dar nu men\u021bioneaz\u0103 deloc thread_pool.management. Aceste thread-uri proceseaz\u0103, printre altele, cereri de tip _cat\/shards \u0219i alte similar, care sunt utile la scrierea monitoriz\u0103rii. Cu c\u00e2t clusterul este mai mare, cu at\u00e2t mai multe astfel de cereri sunt executate \u00eentr-o unitate de timp, iar thread_pool.management men\u021bionat mai sus nu numai c\u0103 nu este prezent \u00een documenta\u021bia oficial\u0103, dar este \u0219i limitat prin default la 5 thread-uri, ceea ce se consum\u0103 foarte repede, dup\u0103 care monitorizarea \u00eenceteaz\u0103 s\u0103 func\u021bioneze corect.<\/p>\n<p><\/p>\n<p>Ce dorim s\u0103 spunem \u00een concluzie: am reu\u0219it! Am reu\u0219it s\u0103 le oferim programatorilor \u0219i dezvoltatorilor no\u0219tri un instrument care poate oferi rapid \u0219i cu acurate\u021be informa\u021bii despre ceea ce se \u00eent\u00e2mpl\u0103 \u00een produc\u021bie, practic \u00een orice situa\u021bie.<\/p>\n<p><\/p>\n<p>Da, a fost destul de complicat, dar, cu toate acestea, dorin\u021bele noastre au reu\u0219it s\u0103 se integreze \u00een produsele deja existente, care nu au necesitat patch-uri \u0219i nici rescriere dup\u0103 nevoile noastre.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Cluster Elasticsearch de 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/1e852123ca5ec5eaec27bdc66b6c4e83.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/494260\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435. \u041d\u043e \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442, \u043a\u043e\u0433\u0434\u0430 \u0445\u043e\u0447\u0435\u0448\u044c \u0441 \u0435\u0433\u043e \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438 \u00ab\u0432 \u043e\u0441\u043e\u0431\u043e \u043a\u0440\u0443\u043f\u043d\u043e\u043c \u043e\u0431\u044a\u0451\u043c\u0435\u00bb? \u0414\u0430 \u0435\u0449\u0451 \u0438 \u0431\u0435\u0437\u0431\u043e\u043b\u0435\u0437\u043d\u0435\u043d\u043d\u043e \u043f\u0435\u0440\u0435\u0436\u0438\u0432\u0430\u0442\u044c \u043e\u0442\u043a\u0430\u0437 \u043b\u044e\u0431\u043e\u0433\u043e \u0438\u0437 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u0434\u0430\u0442\u0430-\u0446\u0435\u043d\u0442\u0440\u043e\u0432? \u041a\u0430\u043a\u043e\u0439 \u0441\u0442\u043e\u0438\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0443, \u0438 \u043d\u0430 \u043a\u0430\u043a\u0438\u0435 \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0435 \u043a\u0430\u043c\u043d\u0438 \u043d\u0430\u0442\u043a\u043d\u0451\u0448\u044c\u0441\u044f? \u041c\u044b \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 \u0440\u0435\u0448\u0438\u043b\u0438 \u043f\u0440\u0438 \u043f\u043e\u043c\u043e\u0449\u0438 elasticsearch \u0440\u0435\u0448\u0438\u0442\u044c \u0432\u043e\u043f\u0440\u043e\u0441 \u043b\u043e\u0433-\u043c\u0435\u043d\u0435\u0434\u0436\u043c\u0435\u043d\u0442\u0430, \u0430 \u0442\u0435\u043f\u0435\u0440\u044c \u0434\u0435\u043b\u0438\u043c\u0441\u044f \u0441 \u0425\u0430\u0431\u0440\u043e\u043c \u043e\u043f\u044b\u0442\u043e\u043c: \u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":75797,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-75796","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Cluster Elasticsearch de peste 200 TB | ProHoster","description":"Mul\u021bi se confrunt\u0103 cu Elasticsearch.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster","og:description":"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-28T17:42:18+00:00","article:modified_time":"2020-03-28T17:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"75796","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:49:32","updated":"2022-09-27 21:24:26","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/75796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=75796"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/75796\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/75797"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=75796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=75796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=75796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}