{"id":75796,"date":"2020-03-28T19:42:18","date_gmt":"2020-03-28T17:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb"},"modified":"2020-03-28T19:42:18","modified_gmt":"2020-03-28T17:42:18","slug":"klaster-elasticsearch-na-200-tb","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","title":{"rendered":"Elasticsearch klaster s kapacitou nad 200 TB","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/ca7a31eca0b3d4faf648dfb24ffbe215.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>S Elasticsearch sa stret\u00e1va ve\u013ea \u013eud\u00ed. Ale \u010do sa stane, ke\u010f chce\u0161 s jeho pomocou uklada\u0165 logy \"vo ve\u013emi ve\u013ekom objeme\"? A e\u0161te aj bezbolestne pre\u017ei\u0165 v\u00fdpadok ktor\u00e9hoko\u013evek z viacer\u00fdch d\u00e1tov\u00fdch centier? Ak\u00fa architekt\u00faru treba vytvori\u0165 a na ak\u00e9 pasce naraz\u00ed\u0161?<\/p>\n<p><\/p>\n<p>My v Odnoklassnikah sme sa rozhodli vyrie\u0161i\u0165 ot\u00e1zku spr\u00e1vy logov pomocou Elasticsearch a teraz sa del\u00edme s Habr o na\u0161e sk\u00fasenosti: o architekt\u00fare aj o pasciach.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Som ja, Peter Zajtsev, pracujem ako syst\u00e9mov\u00fd administr\u00e1tor v Odnoklassnik\u00e1ch. Predt\u00fdm som bol tie\u017e administr\u00e1tor, pracoval som s Manticore Search, Sphinx search, Elasticsearch. Mo\u017eno, ak sa objav\u00ed e\u0161te nejak\u00fd &#8230;search, pravdepodobne s n\u00edm budem tie\u017e pracova\u0165. Rovnako sa z\u00fa\u010dast\u0148ujem viacer\u00fdch open-source projektov na dobrovo\u013enej b\u00e1ze.<\/p>\n<p><\/p>\n<p>Ke\u010f som pri\u0161iel do Odnoklassnikah, povedal som nezmysel na pohovore, \u017ee viem pracova\u0165 s Elasticsearch. Potom, \u010do som sa usadil a vykonal nejak\u00e9 jednoduch\u00e9 \u00falohy, mi dali ve\u013ek\u00fa \u00falohu na reformu existuj\u00faceho syst\u00e9mu spr\u00e1vy logov. <\/p>\n<p><\/p>\n<h2 id=\"trebovaniya\">Po\u017eiadavky<\/h2>\n<p><\/p>\n<p>Po\u017eiadavky na syst\u00e9m boli formulovan\u00e9 nasledovne:<\/p>\n<p><\/p>\n<ul>\n<li>Ako frontend sa mal pou\u017e\u00edva\u0165 Graylog. Preto\u017ee v spolo\u010dnosti u\u017e mali sk\u00fasenosti s t\u00fdmto produktom, program\u00e1tori a testeri ho poznali, bol im pohodln\u00fd a oby\u010dajn\u00fd.<\/li>\n<li>Objem d\u00e1t: v priemere 50-80 tis\u00edc spr\u00e1v za sekundu, ale ak sa nie\u010do pokaz\u00ed, tak je prev\u00e1dzka neobmedzen\u00e1, m\u00f4\u017ee to by\u0165 2-3 mili\u00f3ny riadkov za sekundu.<\/li>\n<li>Po diskusii s objedn\u00e1vate\u013emi po\u017eiadaviek na r\u00fdchlos\u0165 spracovania vyh\u013ead\u00e1vac\u00edch dopytov sme zistili, \u017ee typick\u00fd vzor pou\u017e\u00edvania takejto syst\u00e9my je tak\u00fd: \u013eudia vyh\u013ead\u00e1vaj\u00fa logy svojej aplik\u00e1cie za posledn\u00e9 dva dni a nechc\u00fa \u010daka\u0165 na v\u00fdsledok na formulovan\u00fd dopyt dlh\u0161ie ako sekundu. <\/li>\n<li>Admini trvali na tom, aby sa syst\u00e9m v pr\u00edpade potreby \u013eahko \u0161k\u00e1loval, bez potreby hlbok\u00e9ho prenikania do toho, ako je zostaven\u00fd. <\/li>\n<li>Aby jedin\u00e1 \u00faloha \u00fadr\u017eby, ktor\u00fa tieto syst\u00e9my ob\u010das potrebovali, bola len v\u00fdmena nejak\u00e9ho hardv\u00e9ru.<\/li>\n<li>Okrem toho v Odnoklassnikah existuje vynikaj\u00faca technick\u00e1 trad\u00edcia: ka\u017ed\u00fd servis, ktor\u00fd sp\u00fa\u0161\u0165ame, mus\u00ed pre\u017ei\u0165 v\u00fdpadok d\u00e1tov\u00e9ho centra (n\u00e1hly, nepl\u00e1novan\u00fd a \u00faplne kedyko\u013evek).<\/li>\n<\/ul>\n<p><\/p>\n<p>Najnov\u0161ie po\u017eiadavky na realiz\u00e1ciu tohto projektu n\u00e1m dali najviac zabra\u0165, o \u010dom sa e\u0161te podrobnej\u0161ie zmienim.<\/p>\n<p><\/p>\n<h2 id=\"sreda\">Prostredie<\/h2>\n<p><\/p>\n<p>Pracujeme v \u0161tyroch d\u00e1tov\u00fdch centr\u00e1ch, pri\u010dom d\u00e1tov\u00e9 uzly Elasticsearch m\u00f4\u017eu by\u0165 umiestnen\u00e9 iba v troch (z r\u00f4znych netechnick\u00fdch d\u00f4vodov).<\/p>\n<p><\/p>\n<p>V t\u00fdchto \u0161tyroch d\u00e1tov\u00fdch centr\u00e1ch sa nach\u00e1dza pribli\u017ene 18 000 r\u00f4znych zdrojov logov \u2014 zariaden\u00ed, kontajnerov, virtu\u00e1lnych strojov.<\/p>\n<p><\/p>\n<p>D\u00f4le\u017eitou vlastnos\u0165ou je, \u017ee spustenie clustra prebieha v kontajneroch <noindex><a rel=\"nofollow\" href=\"https:\/\/podman.io\">Podman<\/a><\/noindex> nie na fyzick\u00fdch strojoch, ale na <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/346868\/\">na\u0161om vlastnom cloudovom produkte one-cloud<\/a><\/noindex>. Kontajnerom s\u00fa garantovan\u00e9 2 jadr\u00e1, podobn\u00e9 2.0GHz v4 s mo\u017enos\u0165ou vyu\u017eitia ostatn\u00fdch jadier v pr\u00edpade ich ne\u010dinnosti. <\/p>\n<p><\/p>\n<p>In\u00fdmi slovami:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/073bffbfc3cff761bfabe0773b7f4ebc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"topologiya\">Topol\u00f3gia<\/h2>\n<p><\/p>\n<p>Celkov\u00fd poh\u013ead na rie\u0161enie sa mi p\u00f4vodne javil nasledovne:<\/p>\n<p><\/p>\n<ul>\n<li>3-4 VIP sa nach\u00e1dzaj\u00fa za A-z\u00e1znamom dom\u00e9ny Graylog, to je adresa, na ktor\u00fa s\u00fa logy posielan\u00e9.<\/li>\n<li>ka\u017ed\u00fd VIP predstavuje vyva\u017eova\u010d LVS.<\/li>\n<li>Potom logy prech\u00e1dzaj\u00fa na bat\u00e9riu Graylog, niektor\u00e9 d\u00e1ta id\u00fa vo form\u00e1te GELF, in\u00e9 vo form\u00e1te syslog.<\/li>\n<li>Potom sa toto v\u0161etko vo ve\u013ek\u00fdch d\u00e1vkach zapisuje do bat\u00e9rie koordin\u00e1torov Elasticsearch. <\/li>\n<li>A t\u00ed zase posielaj\u00fa po\u017eiadavky na zapisovanie a \u010d\u00edtanie na relevantn\u00e9 d\u00e1tov\u00e9 uzly. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/21efcdf6992a07e0c5e9b477c567cca9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"terminologiya\">Terminol\u00f3gia<\/h2>\n<p><\/p>\n<p>Mo\u017eno nie v\u0161etci sa podrobne vyznaj\u00fa v terminol\u00f3gii, preto by som sa na nej r\u00e1d na chv\u00ed\u013eu zastavil.<\/p>\n<p><\/p>\n<p>V Elasticsearch existuje nieko\u013eko typov uzlov \u2014 master, coordinator, data node. Existuj\u00fa e\u0161te dva \u010fal\u0161ie typy pre r\u00f4zne transform\u00e1cie logov a prepojenia r\u00f4znych clustrov medzi sebou, ale my sme pou\u017eili len vymenovan\u00e9. <\/p>\n<p><\/p>\n<p><strong>Master<\/strong><br \/>\nPinguje v\u0161etky pr\u00edtomn\u00e9 uzly v clustri, udr\u017euje aktu\u00e1lnu mapu clustra a \u0161\u00edri ju medzi uzlami, spracov\u00e1va logiku udalost\u00ed, vykon\u00e1va r\u00f4zne druhy \u00fadr\u017eby clustra. <\/p>\n<p><\/p>\n<p><strong>Koordin\u00e1tor<\/strong><br \/>\nVykon\u00e1va jednu jedin\u00fa \u00falohu: prij\u00edma po\u017eiadavky od klientov na \u010d\u00edtanie alebo zapisovanie a smeruje tento pr\u00edliv. V pr\u00edpade, \u017ee ide o po\u017eiadavku na zapisovanie, pravdepodobne sa op\u00fdta mastera, do ktor\u00e9ho shardu relevantn\u00e9ho indexu by to malo da\u0165, a po\u017eiadavku prenesie \u010falej. <\/p>\n<p><\/p>\n<p><strong>D\u00e1tov\u00fd uzol<\/strong><br \/>\nUklad\u00e1 d\u00e1ta, vykon\u00e1va prich\u00e1dzaj\u00face vyh\u013ead\u00e1vacie po\u017eiadavky a oper\u00e1cie nad shardami umiestnen\u00fdmi na \u0148om.<\/p>\n<p><\/p>\n<p><strong>Graylog<\/strong><br \/>\nJe to nie\u010do ako zliatina Kibana s Logstashom v ELK-stacku. Graylog kombinuje u\u017e\u00edvate\u013esk\u00e9 rozhranie a spracovate\u013esk\u00fd kan\u00e1l pre logy. Pod kapotou Graylogu funguj\u00fa Kafka a Zookeeper, ktor\u00e9 zabezpe\u010duj\u00fa konektivitu Graylogu ako klasteru. Graylog dok\u00e1\u017ee vyrovn\u00e1va\u0165 logy (Kafka) pre pr\u00edpad nedostupnosti Elasticsearch a opakova\u0165 ne\u00faspe\u0161n\u00e9 \u010d\u00edtacie a zapisovacie po\u017eiadavky, skupinova\u0165 a ozna\u010dova\u0165 logy pod\u013ea zadan\u00fdch pravidiel. Tak ako Logstash, aj Graylog m\u00e1 funk\u010dnos\u0165 pre \u00fapravu re\u0165azcov pred z\u00e1pisom do Elasticsearch.<\/p>\n<p><\/p>\n<p>Okrem toho, Graylog obsahuje zabudovan\u00e9 objavovanie slu\u017eieb, ktor\u00e9 na z\u00e1klade jednej dostupnej nody Elasticsearch umo\u017e\u0148uje z\u00edska\u0165 cel\u00fa mapu klasteru a filtrova\u0165 ju pod\u013ea ur\u010dit\u00e9ho tagu, \u010do umo\u017e\u0148uje nasmerova\u0165 po\u017eiadavky na konkr\u00e9tne kontajnery.<\/p>\n<p><\/p>\n<p>Vizualne to vyzer\u00e1 asi takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/8673ba9c0300ea0153d34a9f98afbcf2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Toto je sn\u00edmka obrazovky z konkr\u00e9tneho in\u0161tancie. Tu vytv\u00e1rame histogram na z\u00e1klade vyh\u013ead\u00e1vacieho dopytu a zobrazujeme relevantn\u00e9 riadky.<\/p>\n<p><\/p>\n<h2 id=\"indeksy\">Indexy<\/h2>\n<p><\/p>\n<p>N\u00e1vrat k architekt\u00fare syst\u00e9mu, chcel by som podrobnej\u0161ie vysvetli\u0165, ako sme budovali model indexov, aby to v\u0161etko fungovalo spr\u00e1vne. <\/p>\n<p><\/p>\n<p>Na predch\u00e1dzaj\u00facom sch\u00e9me je to najni\u017e\u0161ia \u00farove\u0148: uzly d\u00e1t Elasticsearch.<\/p>\n<p><\/p>\n<p>Index je ve\u013ek\u00e1 virtu\u00e1lna entita pozost\u00e1vaj\u00faca z shardov Elasticsearch. Samotn\u00fd ka\u017ed\u00fd z shardov nie je ni\u010d\u00edm in\u00fdm, ako Lucene indexom. A ka\u017ed\u00fd Lucene index sa sklad\u00e1 z jedn\u00e9ho alebo viacer\u00fdch segmentov. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/13b858cb102dce26b0851516b2d36c43.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pri n\u00e1vrhu sme odhadovali, \u017ee na zabezpe\u010denie po\u017eiadaviek na r\u00fdchlos\u0165 \u010d\u00edtania pri ve\u013ekom objeme d\u00e1t, potrebujeme rovnomerne \"rozprestrie\u0165\" tieto d\u00e1ta po d\u00e1tov\u00fdch uzloch. <\/p>\n<p><\/p>\n<p>To viedlo k tomu, \u017ee po\u010det shardov na index (so replikami) by mal by\u0165 presne rovn\u00fd po\u010dtu d\u00e1tov\u00fdch uzlov. Po prv\u00e9, aby sme zabezpe\u010dili faktor replik\u00e1cie rovn\u00fd dvom (t.j. m\u00f4\u017eeme strati\u0165 polovicu klasteru). A po druh\u00e9, aby sme mohli spracov\u00e1va\u0165 \u010d\u00edtacie a zapisovacie po\u017eiadavky minim\u00e1lne na polovici klasteru.<\/p>\n<p><\/p>\n<p>\u010cas uchov\u00e1vania sme najprv ur\u010dili na 30 dn\u00ed.<\/p>\n<p><\/p>\n<p>Rozlo\u017eenie shardov si m\u00f4\u017eeme graficky predstavi\u0165 nasledovne:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/242726053360d1a6bdf4e527edc6cae6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Cel\u00fd tmavo\u0161ed\u00fd obd\u013a\u017enik celkom je index. \u013dav\u00fd \u010derven\u00fd \u0161tvorec v \u0148om je prim\u00e1rny shard, prv\u00fd v indexe. A modr\u00fd \u0161tvorec je replica shard. Nach\u00e1dzaj\u00fa sa v r\u00f4znych d\u00e1tov\u00fdch centr\u00e1ch.<\/p>\n<p><\/p>\n<p>Ke\u010f prid\u00e1me \u010fal\u0161\u00ed shard, dostane sa do tretieho d\u00e1tov\u00e9ho centra. A nakoniec z\u00edskavame tak\u00fato \u0161trukt\u00faru, ktor\u00e1 zabezpe\u010duje mo\u017enos\u0165 straty DC bez straty konzistencie d\u00e1t:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/27cbe4d4606f5ae5f3013ea46504134b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rot\u00e1ciu indexov, t.j. vytvorenie nov\u00e9ho indexu a odstr\u00e1nenie najstar\u0161ieho, sme nastavili na 48 hod\u00edn (pod\u013ea vzoru pou\u017e\u00edvania indexu: posledn\u00fdch 48 hod\u00edn sa vyh\u013ead\u00e1va naj\u010dastej\u0161ie).<\/p>\n<p><\/p>\n<p>Tak\u00fd interval rot\u00e1cie indexov je spojen\u00fd s nasleduj\u00facimi d\u00f4vodmi:<\/p>\n<p><\/p>\n<p>Ke\u010f konkr\u00e9tna d\u00e1tov\u00e1 n\u00f3d dostane vyh\u013ead\u00e1vac\u00ed dotaz, z poh\u013eadu v\u00fdkonu je v\u00fdhodnej\u0161ie, ke\u010f sa dotazuje jeden shard, ak jeho ve\u013ekos\u0165 je porovnate\u013en\u00e1 s ve\u013ekos\u0165ou heapu nody. To umo\u017e\u0148uje dr\u017ea\u0165 \"hor\u00facu\" \u010das\u0165 indexu v heape a r\u00fdchlo k nej pristupova\u0165. Ke\u010f je \"hor\u00facich \u010dast\u00ed\" ve\u013ea, r\u00fdchlos\u0165 vyh\u013ead\u00e1vania v indexe sa zhor\u0161uje.<\/p>\n<p><\/p>\n<p>Ke\u010f n\u00f3d za\u010d\u00edna vykon\u00e1va\u0165 vyh\u013ead\u00e1vac\u00ed dotaz na jednom sharde, pride\u013euje po\u010det vl\u00e1kien rovn\u00fd po\u010dtu hyper-threadingov\u00fdch jadier fyzick\u00e9ho stroja. Ak vyh\u013ead\u00e1vac\u00ed dotaz zasahuje ve\u013ek\u00e9 mno\u017estvo shardov, po\u010det vl\u00e1kien rastie proporcion\u00e1lne. To negat\u00edvne ovplyv\u0148uje r\u00fdchlos\u0165 vyh\u013ead\u00e1vania a m\u00e1 zl\u00fd dopad na index\u00e1ciu nov\u00fdch d\u00e1t. <\/p>\n<p><\/p>\n<p>Aby sme zabezpe\u010dili potrebn\u00fa latenciu vyh\u013ead\u00e1vania, rozhodli sme sa pou\u017ei\u0165 SSD. Pre r\u00fdchle spracovanie po\u017eiadaviek museli ma\u0165 stroje, na ktor\u00fdch boli umiestnen\u00e9 tieto kontejnery, aspo\u0148 56 jadier. \u010c\u00edslo 56 bolo zvolen\u00e9 ako podmiene\u010dne dostato\u010dn\u00e9 mno\u017estvo, ur\u010duj\u00face po\u010det threadov, ktor\u00e9 bude Elasticsearch vytv\u00e1ra\u0165 po\u010das prev\u00e1dzky. V Elasticsearch mnoh\u00e9 parametre thread pool priamo z\u00e1visia od mno\u017estva dostupn\u00fdch jadier, \u010do n\u00e1sledne priamo ovplyv\u0148uje potrebn\u00fd po\u010det nodov v klastri pod\u013ea princ\u00edpu 'menej jadier \u2014 viac nodov'. <\/p>\n<p><\/p>\n<p>Nakoniec sa uk\u00e1zalo, \u017ee priemern\u00fd shard v\u00e1\u017ei niekde okolo 20 gigabajtov, a na 1 index pripad\u00e1 360 shardov. Pod\u013ea toho, ak ich rotujeme raz za 48 hod\u00edn, m\u00e1me ich 15 kusov. Ka\u017ed\u00fd index obsahuje d\u00e1ta za 2 dni.<\/p>\n<p><\/p>\n<h2 id=\"shemy-zapisi-i-chteniya-dannyh\">Sch\u00e9my z\u00e1pisu a \u010d\u00edtania d\u00e1t<\/h2>\n<p><\/p>\n<p>Po\u010fme sa pozrie\u0165, ako sa v tomto syst\u00e9me zapisuj\u00fa d\u00e1ta.<\/p>\n<p><\/p>\n<p>Predpokladajme, \u017ee n\u00e1m z Graylogu prich\u00e1dza nejak\u00fd dotaz do koordin\u00e1tora. Napr\u00edklad, chceme indexova\u0165 2-3 tis\u00edce riadkov. <\/p>\n<p><\/p>\n<p>Koordin\u00e1tor, ktor\u00fd dostal po\u017eiadavku od Graylog, sa p\u00fdta mastera: \u201eV po\u017eiadavke na indexovanie bol konkr\u00e9tny index, ale do ktor\u00e9ho shardu to zap\u00edsa\u0165 \u2014 nebolo uveden\u00e9.\u201c <\/p>\n<p><\/p>\n<p>Master odpoved\u00e1: \u201eZap\u00ed\u0161 t\u00fato inform\u00e1ciu do shardu \u010d\u00edslo 71\u201c, po \u010dom sa inform\u00e1cia odosiela priamo do relevantnej d\u00e1tovej n\u00f3dy, kde sa nach\u00e1dza primary-shard \u010d\u00edslo 71.<\/p>\n<p><\/p>\n<p>Po tom sa log transakci\u00ed replikovan\u00e1 na replica-shard, ktor\u00fd sa nach\u00e1dza u\u017e v inom d\u00e1tovom centre.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/fdb6a77cb78b1eed290e440478569e3b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Z Graylogu prich\u00e1dza k koordin\u00e1torovi vyh\u013ead\u00e1vac\u00ed dotaz. Koordin\u00e1tor ho presmerov\u00e1va pod\u013ea indexu, pri\u010dom Elasticsearch na princ\u00edpe round-robin rozde\u013euje dotazy medzi primary-shard a replica-shard. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/cb32bac878e04d6f50d103ebbc395a7f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>N\u00f3dy v po\u010dte 180 kusov odpovedaj\u00fa nerovnomerne a k\u00fdm odpovedaj\u00fa, koordin\u00e1tor hromad\u00ed inform\u00e1cie, ktor\u00e9 do\u0148 u\u017e \u201evyplivli\u201c r\u00fdchlej\u0161ie d\u00e1tov\u00e9 n\u00f3dy. Potom, ke\u010f bu\u010f v\u0161etky inform\u00e1cie pri\u0161li, alebo bol dosiahnut\u00fd timeout dotazu, vracia v\u0161etko priamo klientovi. <\/p>\n<p><\/p>\n<p>Cel\u00fd tento syst\u00e9m spracov\u00e1va vyh\u013ead\u00e1vacie dotazy za posledn\u00fdch 48 hod\u00edn v priemere za 300-400ms, s v\u00fdnimkou t\u00fdch dotazov, ktor\u00e9 maj\u00fa leading wildcard.<\/p>\n<p><\/p>\n<h2 id=\"cvetochki-s-elasticsearch-nastroyka-java\">\u201eKvety\u201c s Elasticsearch: nastavenie Java<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/fc9ac4b8ee41bc35f5f01b8d9c4ff2f6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Aby to v\u0161etko fungovalo tak, ako sme p\u00f4vodne chceli, ve\u013emi sme dlho ladili najr\u00f4znej\u0161ie veci v klastroch. <\/p>\n<p><\/p>\n<p>Prv\u00e1 \u010das\u0165 zisten\u00fdch probl\u00e9mov bola spojen\u00e1 s t\u00fdm, ako je v Elasticsearch predvolene nastaven\u00e1 Java. <\/p>\n<p><\/p>\n<p><strong>Probl\u00e9m prv\u00fd<\/strong><br \/>\nPozorovali sme ve\u013emi ve\u013ek\u00e9 mno\u017estvo spr\u00e1v o tom, \u017ee na \u00farovni Lucene, ke\u010f s\u00fa spusten\u00e9 background joby, doch\u00e1dza k chyb\u00e1m pri zl\u00fa\u010den\u00ed segmentov Lucene. Z logov bolo vidie\u0165, \u017ee ide o chybu OutOfMemoryError. Pod\u013ea telemetrie sme videli, \u017ee je pam\u00e4\u0165 vo\u013en\u00e1, a nebolo jasn\u00e9, pre\u010do t\u00e1to oper\u00e1cia zlyh\u00e1va. <\/p>\n<p><\/p>\n<p>Zistilo sa, \u017ee mer\u017eovanie Lucene indexov prebieha mimo hromady. A kontajnery s\u00fa celkom pr\u00edsne obmedzen\u00e9 vo vy\u017eaduje prostriedkov. Do t\u00fdchto prostriedkov sa zmestila len hromada (hodnota heap.size bola pribli\u017ene rovn\u00e1 RAM), a nejak\u00e9 off-heap oper\u00e1cie zlyh\u00e1vali s chybou alok\u00e1cie pam\u00e4te, ak z nejak\u00e9ho d\u00f4vodu nevyhovovali t\u00fdm pribli\u017ene 500MB, ktor\u00e9 zostali do limitu.<\/p>\n<p><\/p>\n<p>Oprava bola pomerne trivi\u00e1lna: dostupn\u00fd objem RAM pre kontajner bol zv\u00fd\u0161en\u00fd, po \u010dom sme zabudli na to, \u017ee tak\u00e9to probl\u00e9my sme v\u00f4bec mali.<\/p>\n<p><\/p>\n<p><strong>Druh\u00e1 probl\u00e9m<\/strong><br \/>\nAsi 4-5 dn\u00ed po spusten\u00ed klasteru sme si v\u0161imli, \u017ee d\u00e1tov\u00e9 n\u00f3dy sa za\u010d\u00ednaj\u00fa ob\u010das vyhadzova\u0165 z klasteru a znovu do neho vstupova\u0165 za 10-20 sek\u00fand. <\/p>\n<p><\/p>\n<p>Ke\u010f sme sa za\u010dali zaobera\u0165 detailami, zistili sme, \u017ee t\u00e1to off-heap pam\u00e4\u0165 v Elasticsearch nie je prakticky nijako riaden\u00e1. Ke\u010f sme kontajneru pridelili viac pam\u00e4te, z\u00edskali sme mo\u017enos\u0165 plni\u0165 priamy buffer pool r\u00f4znymi inform\u00e1ciami, a ten bol vypr\u00e1zdnen\u00fd iba po tom, \u010do sa spustil explicitn\u00fd GC zo strany Elasticsearch. <\/p>\n<p><\/p>\n<p>V niektor\u00fdch pr\u00edpadoch t\u00e1to oper\u00e1cia trvala pomerne dlho a po\u010das tejto doby stihol klaster ozna\u010di\u0165 t\u00fato uzol ako u\u017e m\u0155tvu. Tento probl\u00e9m je dobre zdokumentovan\u00fd. <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/elasticsearch-5-6-very-quickly-increasing-direct-buffer-pools\/119561\">tu<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>Rie\u0161en\u00edm bolo obmedzi\u0165 Javu v pou\u017e\u00edvaniu v\u00e4\u010d\u0161iny pam\u00e4te mimo heap pre tieto oper\u00e1cie. Obmedzili sme ju na 16 gigabajtov (-XX:MaxDirectMemorySize=16g), \u010d\u00edm sme dosiahli to, \u017ee explicitn\u00fd GC bol vyvolan\u00fd ove\u013ea \u010dastej\u0161ie a prebiehal ove\u013ea r\u00fdchlej\u0161ie, \u010d\u00edm sme prestali destabilizova\u0165 klaster.<\/p>\n<p><\/p>\n<p><strong>Probl\u00e9m tret\u00ed<\/strong><br \/>\nAk si mysl\u00edte, \u017ee probl\u00e9my s \u201euzlami op\u00fa\u0161\u0165aj\u00facimi klaster v najneo\u010dak\u00e1vanej\u0161\u00edch momentoch\u201c t\u00fdmto skon\u010dili, m\u00fdlite sa. <\/p>\n<p><\/p>\n<p>Ke\u010f sme konfigurovali pr\u00e1cu s indexmi, zvolili sme mmapfs, aby <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/benchmarks-default-niofs-vs-memory-mapped-mmapfs\/13118\/2\">skr\u00e1tili \u010das vyh\u013ead\u00e1vania<\/a><\/noindex> vo svie\u017eich shardoch s ve\u013ekou segment\u00e1ciou. Bola to dos\u0165 hrub\u00e1 chyba, preto\u017ee pri pou\u017e\u00edvan\u00ed mmapfs sa s\u00fabor mapuje do opera\u010dnej pam\u00e4te a \u010falej pracujeme u\u017e s mapped s\u00faborom. Z toho vypl\u00fdva, \u017ee pri pokuse GC zastavi\u0165 vl\u00e1kna v aplik\u00e1cii trv\u00e1 ve\u013emi dlho, k\u00fdm sa dostaneme do safepointu, a po\u010das cesty tam aplik\u00e1cia prest\u00e1va odpoveda\u0165 na po\u017eiadavky majstra, \u010di je e\u0161te st\u00e1le akt\u00edvna. Pod\u013ea toho majster pova\u017euje, \u017ee uzol u\u017e nie je v klastri pr\u00edtomn\u00fd. Po nejakej dobe, pribli\u017ene po 5-10 sekund\u00e1ch, za\u010dne garbage collector pracova\u0165, uzol o\u017eije, znova sa pripoj\u00ed do klastra a za\u010dne inicializ\u00e1ciu shardov. To v\u0161etko ve\u013emi pripom\u00ednalo \u201eprodukciu, ktor\u00fa sme si zasl\u00fa\u017eili\u201c a nebolo to vhodn\u00e9 na ni\u010d v\u00e1\u017ene.<\/p>\n<p><\/p>\n<p>Aby sme sa zbavili tak\u00e9ho spr\u00e1vania, najprv sme pre\u0161li na \u0161tandardn\u00fd niofs, a potom, ke\u010f sme sa z piatych verzi\u00ed Elastic presunuli na \u0161ieste, sme sk\u00fa\u0161ali hybridfs, kde sa tento probl\u00e9m nevyskytoval. Viac inform\u00e1ci\u00ed o typoch \u00falo\u017eiska si m\u00f4\u017eete pre\u010d\u00edta\u0165. <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/index-modules-store.html\">tu<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><strong>Probl\u00e9m \u0161tvrt\u00fd<\/strong><br \/>\nPotom sme mali e\u0161te ve\u013emi zauj\u00edmav\u00fd probl\u00e9m, ktor\u00fd sme rie\u0161ili rekordne dlho. Chytili sme ho na 2-3 mesiace, preto\u017ee jeho vzor bol \u00faplne nejasn\u00fd. <\/p>\n<p><\/p>\n<p>Niekedy na\u0161i koordin\u00e1tori prech\u00e1dzali do Full GC, zvy\u010dajne niekde po obede, a odtia\u013e sa u\u017e nevracali. Pri logovan\u00ed oneskorenia GC to vyzeralo takto: v\u0161etko prebieha dobre, dobre, dobre, a potom bum \u2014 a v\u0161etko je n\u00e1hle zl\u00e9. <\/p>\n<p><\/p>\n<p>Spo\u010diatku sme si mysleli, \u017ee m\u00e1me zloducha, ktor\u00fd sp\u00fa\u0161\u0165a nejak\u00fd dotaz, ktor\u00fd vyhadzuje koordin\u00e1tora z pracovn\u00e9ho re\u017eimu. Dlho sme logovali dotazy, sna\u017eiac sa zisti\u0165, \u010do sa deje. <\/p>\n<p><\/p>\n<p>Nakoniec sa uk\u00e1zalo, \u017ee v momente, ke\u010f niektor\u00fd pou\u017e\u00edvate\u013e spust\u00ed obrovsk\u00fd dotaz, a ten dopadne na konkr\u00e9tneho koordin\u00e1tora Elasticsearch, niektor\u00e9 uzly reaguj\u00fa dlh\u0161ie ako ostatn\u00e9. <\/p>\n<p><\/p>\n<p>A ten \u010das, ktor\u00fd koordin\u00e1tor \u010dak\u00e1 na odpovede v\u0161etk\u00fdch uzlov, zhroma\u017e\u010fuje v\u00fdsledky, ktor\u00e9 u\u017e odoslali odpovedaj\u00face uzly. Pre GC to znamen\u00e1, \u017ee sa ve\u013emi r\u00fdchlo men\u00ed vzor vyu\u017eitia heapu. A ten GC, ktor\u00fd sme pou\u017e\u00edvali, si s touto \u00falohou neporadil. <\/p>\n<p><\/p>\n<p>Jedin\u00e9 rie\u0161enie, ktor\u00e9 sme na\u0161li, aby sme zmenili spr\u00e1vanie klastru v takejto situ\u00e1cii \u2014 migr\u00e1cia na JDK13 a pou\u017eitie garbage collectoru Shenandoah. To vyrie\u0161ilo probl\u00e9m a koordin\u00e1tori prestali pada\u0165. <\/p>\n<p><\/p>\n<p>Na tomto probl\u00e9me s Javou sa probl\u00e9my skon\u010dili a za\u010dali probl\u00e9my s priepustnos\u0165ou. <\/p>\n<p><\/p>\n<h2 id=\"yagodki-s-elasticsearch-propusknaya-sposobnost\">\u00abBobu\u013eky\u00bb s Elasticsearch: priepustnos\u0165<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/bb5c08193d7d764515235e825cd30ce5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Probl\u00e9my s priepustnos\u0165ou znamenaj\u00fa, \u017ee n\u00e1\u0161 klaster pracuje stabilne, ale pri \u0161pi\u010dk\u00e1ch po\u010dtu indexovan\u00fdch dokumentov a v momente man\u00e9vrov v\u00fdkon nie je dostato\u010dn\u00fd.<\/p>\n<p><\/p>\n<p>Prv\u00fd pozorovan\u00fd sympt\u00f3m: pri nejak\u00fdch \u00abv\u00fdbuchoch\u00bb na produkcii, ke\u010f sa n\u00e1hle generuje ve\u013emi ve\u013ek\u00e9 mno\u017estvo logov, v Graylogu sa za\u010d\u00edna \u010dasto objavova\u0165 chyba indexovania es_rejected_execution. <\/p>\n<p><\/p>\n<p>To sa dialo kv\u00f4li tomu, \u017ee thread_pool.write.queue na jednej d\u00e1tovej n\u00f3d\u011b predt\u00fdm, ne\u017e Elasticsearch stihne spracova\u0165 po\u017eiadavku na indexovanie a posla\u0165 inform\u00e1cie do shardu na disk, predvolene dok\u00e1\u017ee uklada\u0165 iba 200 po\u017eiadaviek. A v <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">dokument\u00e1cii Elasticsearch<\/a><\/noindex> o tomto parametri sa hovor\u00ed ve\u013emi m\u00e1lo. Uv\u00e1dza sa len maxim\u00e1lny po\u010det vl\u00e1kien a predvolen\u00e1 ve\u013ekos\u0165.<\/p>\n<p><\/p>\n<p>Samozrejme, i\u0161li sme nastavi\u0165 t\u00fato hodnotu a zistili sme nasledovn\u00e9: konkr\u00e9tne v na\u0161ej konfigur\u00e1cii sa celkom dobre cacheuje a\u017e 300 po\u017eiadaviek, zatia\u013e \u010do v\u00e4\u010d\u0161ie hodnoty s\u00fa nebezpe\u010dn\u00e9, lebo op\u00e4\u0165 prech\u00e1dzame do Full GC.<\/p>\n<p><\/p>\n<p>Okrem toho, ke\u010f\u017ee ide o bal\u00edky spr\u00e1v, ktor\u00e9 prich\u00e1dzaj\u00fa v r\u00e1mci jedn\u00e9ho po\u017eiadavku, bolo potrebn\u00e9 aj upravi\u0165 Graylog, aby zapisoval nie \u010dasto a mal\u00fdmi d\u00e1vkami, ale obrovsk\u00fdmi d\u00e1vkami alebo raz za 3 sekundy, ak d\u00e1vka e\u0161te nie je pln\u00e1. V takom pr\u00edpade sa inform\u00e1cie, ktor\u00e9 zapisujeme do Elasticsearch, st\u00e1vaj\u00fa dostupn\u00fdmi nie za dve sekundy, ale za p\u00e4\u0165 (\u010do n\u00e1m celkom vyhovuje), no zni\u017euje sa po\u010det opakovan\u00fdch pokusov, ktor\u00e9 je potrebn\u00e9 urobi\u0165, aby sme pretl\u00e1\u010dali ve\u013ek\u00fa d\u00e1vku inform\u00e1ci\u00ed.<\/p>\n<p><\/p>\n<p>To je obzvl\u00e1\u0161\u0165 d\u00f4le\u017eit\u00e9 v momentoch, ke\u010f sa nie\u010do niekde pokaz\u00ed a horlivo o tom informuje, aby sme nez\u00edskali \u00faplne zahlten\u00fd Elastic a nesk\u00f4r \u2014 nefunk\u010dn\u00e9 uzly Graylog z d\u00f4vodu zaplnen\u00fdch b\u00faferov.<\/p>\n<p><\/p>\n<p>Okrem toho, ke\u010f sa u n\u00e1s odohr\u00e1vali tieto v\u00fdbuchy na produkcii, prich\u00e1dzali s\u0165a\u017enosti od program\u00e1torov a testerov: v tom momente, ke\u010f ve\u013emi potrebovali tieto logy, boli im poskytovan\u00e9 ve\u013emi pomaly.<\/p>\n<p><\/p>\n<p>Za\u010dali sme sa t\u00fdm zaobera\u0165. Na jednej strane bolo jasn\u00e9, \u017ee vyh\u013ead\u00e1vacie po\u017eiadavky a po\u017eiadavky na indexovanie funguj\u00fa vlastne na rovnak\u00fdch fyzick\u00fdch strojoch, a tak \u010di onak ur\u010dit\u00e9 poklesy bud\u00fa. <\/p>\n<p><\/p>\n<p>Ale tento probl\u00e9m sa dal \u010diasto\u010dne ob\u00eds\u0165 t\u00fdm, \u017ee v \u0161iestich verzi\u00e1ch Elasticsearch sa objavil algoritmus, ktor\u00fd umo\u017e\u0148uje rozde\u013eova\u0165 po\u017eiadavky medzi relevantn\u00e9 d\u00e1tov\u00e9 uzly nie n\u00e1hodn\u00fdm sp\u00f4sobom round-robin (kontejner, ktor\u00fd sa zaober\u00e1 indexovan\u00edm a dr\u017e\u00ed primary-shard, m\u00f4\u017ee by\u0165 ve\u013emi za\u0165a\u017een\u00fd, nebude schopn\u00fd r\u00fdchlo odpoveda\u0165), ale nasmerova\u0165 tento po\u017eiadavku na menej za\u0165a\u017een\u00fd kontajner s replica-shard, ktor\u00fd odpovie ove\u013ea r\u00fdchlej\u0161ie. In\u00fdmi slovami, dostali sme sa k use_adaptive_replica_selection: true.<\/p>\n<p><\/p>\n<p>Obraz \u010d\u00edtania za\u010dne vyzera\u0165 takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/ae7c1645bb63692e580b1bca5c6e5db0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Prechod na tento algoritmus umo\u017enil v\u00fdrazne zlep\u0161i\u0165 \u010das dotazu v t\u00fdch momentoch, ke\u010f sme mali ve\u013ek\u00fd tok logov na z\u00e1pis.<\/p>\n<p><\/p>\n<p>Nakoniec hlavn\u00fd probl\u00e9m spo\u010d\u00edval v bezbolestnom vyveden\u00ed d\u00e1tov\u00e9ho centr\u00e1.<\/p>\n<p><\/p>\n<p>\u010co sme chceli od klastera hne\u010f po strate spojenia s jedn\u00fdm DC: <\/p>\n<p><\/p>\n<ul>\n<li>Ak sa v odpojenom d\u00e1tovom centre nach\u00e1dza aktu\u00e1lny master, bude prevolan\u00fd a prejde ako rola na in\u00fd uzol v inom DC.<\/li>\n<li>Master r\u00fdchlo vyhod\u00ed zo clusteru v\u0161etky nedostupn\u00e9 uzly.<\/li>\n<li>Na z\u00e1klade zost\u00e1vaj\u00facich zist\u00ed: v stratenom d\u00e1tovom centre sme mali tak\u00e9to primary shard-y, r\u00fdchlo propaguje komplement\u00e1rne replica shard-y v zost\u00e1vaj\u00facich d\u00e1tov\u00fdch centr\u00e1ch a bude pokra\u010dova\u0165 index\u00e1cia d\u00e1t. <\/li>\n<li>V\u00fdsledkom toho bude plynul\u00e1 degrad\u00e1cia priepustnosti klastru na z\u00e1pis a \u010d\u00edtanie, av\u0161ak celkovo v\u0161etko bude fungova\u0165 aj ke\u010f pomaly, ale stabilne.<\/li>\n<\/ul>\n<p><\/p>\n<p>Ako sa uk\u00e1zalo, chceli sme nie\u010do tak\u00e9to:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/71fa3d19a61eb58ddb93a4df55c759c5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A dostali sme nasledovn\u00e9:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/ab32be4dfbbb4bcc3279d57021707980.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Ako sa to stalo? <\/p>\n<p><\/p>\n<p>V momente p\u00e1du d\u00e1tov\u00e9ho centra sa na\u0161\u00edm \u00fazkym miestom stal majster.<\/p>\n<p><\/p>\n<p>Pre\u010do?<\/p>\n<p><\/p>\n<p>Ide o to, \u017ee v majsteri je TaskBatcher, ktor\u00fd zodpoved\u00e1 za \u0161\u00edrenie ur\u010dit\u00fdch \u00faloh a udalost\u00ed v klastri. Ak\u00fdko\u013evek v\u00fdstup nody, ak\u00e9ko\u013evek posunutie shardu z replica do primary, ak\u00e1ko\u013evek \u00faloha na vytvorenie niekde nejak\u00e9ho shardu \u2014 to v\u0161etko najprv prech\u00e1dza cez TaskBatcher, kde sa spracov\u00e1va sekven\u010dne a v jednom vl\u00e1kne.<\/p>\n<p><\/p>\n<p>V momente v\u00fdstupu jedn\u00e9ho d\u00e1tov\u00e9ho centra vych\u00e1dzalo, \u017ee v\u0161etky d\u00e1tov\u00e9 nody v pre\u017e\u00edvaj\u00facich d\u00e1tov\u00fdch centr\u00e1ch pova\u017eovali za svoju povinnos\u0165 ozn\u00e1mi\u0165 majsterovi \u201estratili sme tak\u00e9to shard-y a tak\u00e9to d\u00e1tov\u00e9 nody\u201c. <\/p>\n<p><\/p>\n<p>Pri tom pre\u017e\u00edvaj\u00face d\u00e1tov\u00e9 nody posielali v\u0161etky tieto inform\u00e1cie s\u00fa\u010dasn\u00e9mu majstrovi a sna\u017eili sa po\u010dka\u0165 na potvrdenie, \u017ee to prijal. Toho sa nedo\u010dkali, preto\u017ee majster prij\u00edmal \u00falohy r\u00fdchlej\u0161ie, ne\u017e stihol odpoveda\u0165. Nody opakovali po\u017eiadavky pod\u013ea timeoutu, zatia\u013e \u010do majster v tom \u010dase u\u017e ani nepok\u00fa\u0161al na ne odpoveda\u0165, ale bol \u00faplne pohlten\u00fd \u00falohou triedi\u0165 po\u017eiadavky pod\u013ea priority.<\/p>\n<p><\/p>\n<p>V termin\u00e1lnom pr\u00edklade to vyzeralo tak, \u017ee d\u00e1tov\u00e9 nody spamovali majstra tak, \u017ee pre\u0161iel do full GC. Po tom sa na\u0161a \u00faloha majstra presunula na nejak\u00fa \u010fal\u0161iu nody, s ktorou sa dialo presne to ist\u00e9, a nakoniec sa klaster \u00faplne rozpadol. <\/p>\n<p><\/p>\n<p>Robili sme merania a do verzie 6.4.0, kde to bolo opraven\u00e9, n\u00e1m sta\u010dilo vypn\u00fa\u0165 naraz len 10 d\u00e1tov\u00fdch nodov z 360 na to, aby sme \u00faplne zru\u0161ili klaster.<\/p>\n<p><\/p>\n<p>Vyzeralo to pribli\u017ene takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/3bed554b2703e7521339e9d943648ee4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Po verzii 6.4.0, kde opravili tento nepr\u00edjemn\u00fd bug, prestali d\u00e1tov\u00e9 nody zab\u00edja\u0165 majstra. Ale \u201einteligentnej\u0161\u00ed\u201c od toho sa nestal. A to konkr\u00e9tne: ke\u010f vypneme 2, 3 alebo 10 (ak\u00e9ko\u013evek mno\u017estvo, okrem jedn\u00e9ho) d\u00e1tov\u00fdch nodov, majster dostane nejak\u00fa prv\u00fa spr\u00e1vu, ktor\u00e1 hovor\u00ed, \u017ee noda A sa vypnula, a sna\u017e\u00ed sa o tom poveda\u0165 node B, node C, node D. <\/p>\n<p><\/p>\n<p>A v s\u00fa\u010dasnosti sa s t\u00fdm d\u00e1 bojova\u0165 len nastaven\u00edm \u010dasov\u00e9ho limitu na pokusy poveda\u0165 niekomu o nie\u010dom, ktor\u00e9 s\u00fa pribli\u017ene 20-30 sek\u00fand, a takto riadi\u0165 r\u00fdchlos\u0165 v\u00fdstupu d\u00e1tov\u00e9ho centra z klastru.<\/p>\n<p><\/p>\n<p>V podstate to zodpoved\u00e1 t\u00fdm po\u017eiadavk\u00e1m, ktor\u00e9 boli p\u00f4vodne stanoven\u00e9 pre kone\u010dn\u00fd produkt v r\u00e1mci projektu, ale z poh\u013eadu \u201e\u010distej vedy\u201c je to chyba. Ktor\u00e1 bola mimochodom \u00faspe\u0161ne opraven\u00e1 v\u00fdvoj\u00e1rmi vo verzii 7.2.<\/p>\n<p><\/p>\n<p>Ke\u010f nejak\u00fd d\u00e1tov\u00fd uzol vyst\u00fapil, uk\u00e1zalo sa, \u017ee roz\u0161\u00edri\u0165 inform\u00e1cie o jeho v\u00fdstupe je d\u00f4le\u017eitej\u0161ie, ne\u017e poveda\u0165 cel\u00e9mu klastru, \u017ee na \u0148om sa nach\u00e1dzali tak\u00e9to primary-shard (aby sa mohol promovova\u0165 replica-shard v inom d\u00e1tovom centre na primary, a do nich sa mohli zapisova\u0165 inform\u00e1cie).<\/p>\n<p><\/p>\n<p>Preto, ke\u010f sa u\u017e v\u0161etko \u201ena\u0165ahalo\u201c, vyst\u00fapen\u00e9 d\u00e1tov\u00e9 uzly nie s\u00fa okam\u017eite ozna\u010den\u00e9 za stale. Zodpovedaj\u00faco mus\u00edme \u010daka\u0165, k\u00fdm vypr\u0161\u00ed v\u0161etky pingi do vyst\u00fapen\u00fdch d\u00e1tov\u00fdch uzlov a a\u017e potom n\u00e1\u0161 klaster za\u010dne informova\u0165 o tom, kde treba pokra\u010dova\u0165 v zapisovan\u00ed inform\u00e1ci\u00ed. O tomto sa d\u00e1 podrobnej\u0161ie pre\u010d\u00edta\u0165 <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/elastic\/elasticsearch\/issues\/46909\">tu<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>Nakoniec oper\u00e1cia v\u00fdstupu d\u00e1tov\u00e9ho centra dnes trv\u00e1 okolo 5 min\u00fat v \u0161pi\u010dke. Pre tak ve\u013ek\u00fa a nehnutn\u00fa ma\u0161inu je to dos\u0165 dobr\u00fd v\u00fdsledok.<\/p>\n<p><\/p>\n<p>Na konci sme dospeli k nasleduj\u00facemu rie\u0161eniu:<\/p>\n<p><\/p>\n<ul>\n<li>M\u00e1me 360 d\u00e1tov\u00fdch uzlov s diskami s kapacitou 700 gigabajtov.<\/li>\n<li>60 koordin\u00e1torov na smerovanie prev\u00e1dzky medzi t\u00fdmito d\u00e1tov\u00fdmi uzlami.<\/li>\n<li>40 majstrov, ktor\u00ed u n\u00e1s zostali ako ak\u00fdsi dedi\u010dstvo z obdob\u00ed pred verziou 6.4.0 \u2014 aby sme pre\u017eili v\u00fdstup d\u00e1tov\u00e9ho centra, boli sme psychicky pripraven\u00ed strati\u0165 nieko\u013eko strojov, aby sme zaru\u010dili aj pri najhor\u0161om scen\u00e1ri ma\u0165 kvorum majstrov.<\/li>\n<li>Ak\u00e9ko\u013evek pokusy o kombinovanie rol\u00ed na jednom kontejnere narazili na to, \u017ee sk\u00f4r \u010di nesk\u00f4r sa uzol rozpadol pod z\u00e1\u0165a\u017eou. <\/li>\n<li>V celom klastri sa pou\u017e\u00edva heap.size s hodnotou 31 gigabajtov: v\u0161etky pokusy o zn\u00ed\u017eenie ve\u013ekosti viedli k tomu, \u017ee pri n\u00e1ro\u010dn\u00fdch vyh\u013ead\u00e1vac\u00edch dotazoch s leading wildcard bu\u010f zab\u00edjali niektor\u00e9 uzly, alebo sa preru\u0161\u00ed circuit breaker v samotnom Elasticsearch.<\/li>\n<li>Okrem toho, aby sme zabezpe\u010dili v\u00fdkon vyh\u013ead\u00e1vania, sna\u017eili sme sa dr\u017ea\u0165 po\u010det objektov v klastri na minim\u00e1lnej mo\u017enej \u00farovni, aby sme spracov\u00e1vali \u010do najmen\u0161ie mno\u017estvo udalost\u00ed na naj\u00fa\u017e\u0161om mieste, ktor\u00e9 sa n\u00e1m v majstrovi podarilo dosiahnu\u0165.<\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"naposledok-o-monitoringe\">Na z\u00e1ver o monitorovan\u00ed<\/h2>\n<p><\/p>\n<p>Aby to v\u0161echno fungovalo tak, jak bylo zam\u00fd\u0161leno, sledujeme n\u00e1sleduj\u00edc\u00ed:<\/p>\n<p><\/p>\n<ul>\n<li>Ka\u017ed\u00e9 datacentrum informuje na\u0161i cloudovou infrastrukturu o sv\u00e9 existenci a tom, kter\u00e9 shardov\u00e1n\u00ed na n\u011bm prob\u00edh\u00e1. Kdy\u017e n\u011bkde n\u011bco zhasneme, cluster za 2-3 sekundy hl\u00e1s\u00ed, \u017ee v datacentru A jsme zhasli uzly 2, 3 a 4 \u2014 to znamen\u00e1, \u017ee v jin\u00fdch datacentrech nem\u016f\u017eeme zhasnout ty uzly, na kter\u00fdch z\u016fstaly shardy pouze v jednom exempl\u00e1\u0159i.<\/li>\n<li>S ohledem na chov\u00e1n\u00ed hlavn\u00edho uzlu velmi pozorn\u011b sledujeme po\u010det pending \u00faloh. Proto\u017ee i jedna vis\u00edc\u00ed \u00faloha, pokud se v\u010das nezpr\u00e1vn\u00ed, m\u016f\u017ee teoreticky v n\u011bjak\u00e9 krizov\u00e9 situaci st\u00e1t p\u0159\u00ed\u010dinou, kv\u016fli n\u00ed\u017e n\u00e1m neprob\u011bhne nap\u0159\u00edklad propagace replica shardu na prim\u00e1rn\u00ed, co\u017e by zastavilo indexaci.<\/li>\n<li>Tak\u00e9 velmi pe\u010dliv\u011b sledujeme zpo\u017ed\u011bn\u00ed garbage collector, proto\u017ee s t\u00edm jsme u\u017e m\u011bli velk\u00e9 pot\u00ed\u017ee p\u0159i optimalizaci.<\/li>\n<li>Rejecty podle vl\u00e1ken, abychom p\u0159edem pochopili, kde se nach\u00e1z\u00ed \u201e\u00fazk\u00e9 hrdlo\u201c.<\/li>\n<li>A standardn\u00ed metriky, jako je heap, RAM a I\/O.<\/li>\n<\/ul>\n<p><\/p>\n<p>P\u0159i budov\u00e1n\u00ed monitoringu je nutn\u00e9 m\u00edt na pam\u011bti specifika Thread Pool v Elasticsearch. <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">Dokumentace Elasticsearch<\/a><\/noindex> popisuje mo\u017enosti nastaven\u00ed a v\u00fdchoz\u00ed hodnoty pro vyhled\u00e1v\u00e1n\u00ed, indexaci, ale zcela opom\u00edj\u00ed thread_pool.management. Tyto vl\u00e1kna zpracov\u00e1vaj\u00ed nap\u0159\u00edklad dotazy typu _cat\/shards a dal\u0161\u00ed podobn\u00e9, kter\u00e9 jsou u\u017eite\u010dn\u00e9 p\u0159i psan\u00ed monitorov\u00e1n\u00ed. \u010c\u00edm v\u011bt\u0161\u00ed je cluster, t\u00edm v\u00edce takov\u00fdch dotaz\u016f se vykon\u00e1v\u00e1 za jednotku \u010dasu, a v\u00fd\u0161e zm\u00edn\u011bn\u00fd thread_pool.management nejen\u017ee nen\u00ed uveden v ofici\u00e1ln\u00ed dokumentaci, ale je tak\u00e9 v\u00fdchoz\u00edm limitem omezeno na 5 vl\u00e1ken, co\u017e se velmi rychle vy\u010derp\u00e1, po jeho\u017e vy\u010derp\u00e1n\u00ed p\u0159est\u00e1v\u00e1 monitoring fungovat spr\u00e1vn\u011b.<\/p>\n<p><\/p>\n<p>Co bych cht\u011bl \u0159\u00edci na z\u00e1v\u011br: poda\u0159ilo se n\u00e1m! Dok\u00e1zali jsme na\u0161im program\u00e1tor\u016fm a v\u00fdvoj\u00e1\u0159\u016fm poskytnout n\u00e1stroj, kter\u00fd t\u00e9m\u011b\u0159 v ka\u017ed\u00e9 situaci dok\u00e1\u017ee rychle a spolehliv\u011b poskytnout informace o d\u011bn\u00ed v produkci.<\/p>\n<p><\/p>\n<p>Ano, bylo to pom\u011brn\u011b obt\u00ed\u017en\u00e9, ale p\u0159esto se n\u00e1m poda\u0159ilo na\u0161e po\u017eadavky vt\u011bsnat do ji\u017e existuj\u00edc\u00edch produkt\u016f, kter\u00e9 jsme nemuseli upravovat a p\u0159episovat podle sebe.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klaster s kapacitou nad 200 TB\" src=\"\/wp-content\/uploads\/2020\/03\/1e852123ca5ec5eaec27bdc66b6c4e83.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/494260\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435. \u041d\u043e \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442, \u043a\u043e\u0433\u0434\u0430 \u0445\u043e\u0447\u0435\u0448\u044c \u0441 \u0435\u0433\u043e \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438 \u00ab\u0432 \u043e\u0441\u043e\u0431\u043e \u043a\u0440\u0443\u043f\u043d\u043e\u043c \u043e\u0431\u044a\u0451\u043c\u0435\u00bb? \u0414\u0430 \u0435\u0449\u0451 \u0438 \u0431\u0435\u0437\u0431\u043e\u043b\u0435\u0437\u043d\u0435\u043d\u043d\u043e \u043f\u0435\u0440\u0435\u0436\u0438\u0432\u0430\u0442\u044c \u043e\u0442\u043a\u0430\u0437 \u043b\u044e\u0431\u043e\u0433\u043e \u0438\u0437 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u0434\u0430\u0442\u0430-\u0446\u0435\u043d\u0442\u0440\u043e\u0432? \u041a\u0430\u043a\u043e\u0439 \u0441\u0442\u043e\u0438\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0443, \u0438 \u043d\u0430 \u043a\u0430\u043a\u0438\u0435 \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0435 \u043a\u0430\u043c\u043d\u0438 \u043d\u0430\u0442\u043a\u043d\u0451\u0448\u044c\u0441\u044f? \u041c\u044b \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 \u0440\u0435\u0448\u0438\u043b\u0438 \u043f\u0440\u0438 \u043f\u043e\u043c\u043e\u0449\u0438 elasticsearch \u0440\u0435\u0448\u0438\u0442\u044c \u0432\u043e\u043f\u0440\u043e\u0441 \u043b\u043e\u0433-\u043c\u0435\u043d\u0435\u0434\u0436\u043c\u0435\u043d\u0442\u0430, \u0430 \u0442\u0435\u043f\u0435\u0440\u044c \u0434\u0435\u043b\u0438\u043c\u0441\u044f \u0441 \u0425\u0430\u0431\u0440\u043e\u043c \u043e\u043f\u044b\u0442\u043e\u043c: \u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":75797,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-75796","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.4 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.4\" \/>\n\t\t<meta property=\"og:locale\" content=\"sk_SK\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Elasticsearch cluster nad 200 TB+ | ProHoster","description":"Mnoz\u00ed se setk\u00e1vaj\u00ed s Elasticsearch.","canonical_url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sk_SK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster","og:description":"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.","og:url":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-28T17:42:18+00:00","article:modified_time":"2020-03-28T17:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"75796","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:49:32","updated":"2022-09-27 21:24:26","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/75796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/comments?post=75796"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/posts\/75796\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media\/75797"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/media?parent=75796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/categories?post=75796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sk\/wp-json\/wp\/v2\/tags?post=75796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}