{"id":75796,"date":"2020-03-28T19:42:18","date_gmt":"2020-03-28T17:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb"},"modified":"2020-03-28T19:42:18","modified_gmt":"2020-03-28T17:42:18","slug":"klaster-elasticsearch-na-200-tb","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","title":{"rendered":"Elasticsearch cluster o velikosti 200 TB+","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ca7a31eca0b3d4faf648dfb24ffbe215.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>S Elasticsearch se setk\u00e1v\u00e1 mnoho lid\u00ed. Ale co se stane, kdy\u017e chcete pomoc\u00ed n\u011bj ukl\u00e1dat logy \"v obzvl\u00e1\u0161\u0165 velk\u00e9m objemu\"? A je\u0161t\u011b bezbolestn\u011b p\u0159e\u017e\u00edt selh\u00e1n\u00ed n\u011bkter\u00e9ho z n\u011bkolika datov\u00fdch center? Jakou architekturu zvolit a na jak\u00e1 \u00faskal\u00ed naraz\u00edte?<\/p>\n<p><\/p>\n<p>V Odnoklassnik\u00e1ch jsme se rozhodli vy\u0159e\u0161it ot\u00e1zku log managementu pomoc\u00ed Elasticsearch a nyn\u00ed se d\u011bl\u00edme o na\u0161e zku\u0161enosti v Habru: jak o architektu\u0159e, tak o \u00faskal\u00edch.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Jsem Petr Zajcev, pracuji jako syst\u00e9mov\u00fd administr\u00e1tor v Odnoklassnik\u00e1ch. P\u0159edt\u00edm jsem byl tak\u00e9 administr\u00e1tor, pracoval jsem s Manticore Search, Sphinx search a Elasticsearch. Mo\u017en\u00e1, pokud se objev\u00ed n\u011bjak\u00fd dal\u0161\u00ed ...search, pravd\u011bpodobn\u011b budu pracovat i s n\u00edm. Tak\u00e9 se \u00fa\u010dastn\u00edm n\u011bkolika open-source projekt\u016f na dobrovoln\u00e9 b\u00e1zi.<\/p>\n<p><\/p>\n<p>Kdy\u017e jsem p\u0159i\u0161el do Odnoklassnik\u016f, neuv\u00e1\u017een\u011b jsem na pohovoru \u0159ekl, \u017ee um\u00edm pracovat s Elasticsearch. Pot\u00e9, co jsem se zorientoval a provedl n\u011bjak\u00e9 jednoduch\u00e9 \u00fakoly, dostal jsem velk\u00fd \u00fakol na reformu syst\u00e9mu log managementu, jak\u00fd existoval v t\u00e9 dob\u011b. <\/p>\n<p><\/p>\n<h2 id=\"trebovaniya\">Po\u017eadavky<\/h2>\n<p><\/p>\n<p>Po\u017eadavky na syst\u00e9m byly formulov\u00e1ny n\u00e1sledovn\u011b:<\/p>\n<p><\/p>\n<ul>\n<li>Jako frontend m\u011bl b\u00fdt pou\u017eit Graylog. Proto\u017ee ve firm\u011b u\u017e byly zku\u0161enosti s t\u00edmto produktem, program\u00e1to\u0159i a teste\u0159i ho znali, byl jim zn\u00e1m\u00fd a pohodln\u00fd.<\/li>\n<li>Objem dat: pr\u016fm\u011brn\u011b 50-80 tis\u00edc zpr\u00e1v za sekundu, ale pokud n\u011bco sel\u017ee, tak nen\u00ed omezen \u017e\u00e1dn\u00fdm provozem, m\u016f\u017ee to b\u00fdt 2-3 miliony \u0159\u00e1dk\u016f za sekundu.<\/li>\n<li>Po diskusi s klienty o po\u017eadavc\u00edch na rychlost zpracov\u00e1n\u00ed vyhled\u00e1vac\u00edch dotaz\u016f jsme pochopili, \u017ee typick\u00fd vzor pou\u017e\u00edv\u00e1n\u00ed takov\u00e9ho syst\u00e9mu je n\u00e1sleduj\u00edc\u00ed: lid\u00e9 hledaj\u00ed logy sv\u00e9 aplikace za posledn\u00ed dva dny a necht\u011bj\u00ed \u010dekat na v\u00fdsledky jejich dotazu d\u00e9le ne\u017e jednu sekundu. <\/li>\n<li>Administr\u00e1to\u0159i trvali na tom, aby se syst\u00e9m v p\u0159\u00edpad\u011b pot\u0159eby snadno \u0161k\u00e1loval, ani\u017e by od nich vy\u017eadoval hlubok\u00e9 porozum\u011bn\u00ed tomu, jak je uspo\u0159\u00e1d\u00e1n. <\/li>\n<li>Aby byla jedin\u00e1 \u00fadr\u017ebov\u00e1 \u00faloha, kterou tyto syst\u00e9my \u010das od \u010dasu vy\u017eadovaly - to je vym\u011bnit n\u011bjak\u00fd hardware.<\/li>\n<li>Krom\u011b toho existuje v Odnoklassnik\u00e1ch skv\u011bl\u00e1 technick\u00e1 tradice: jak\u00e1koli slu\u017eba, kterou spou\u0161t\u00edme, mus\u00ed p\u0159e\u017e\u00edt selh\u00e1n\u00ed datov\u00e9ho centra (n\u00e1hl\u00e9, nepl\u00e1novan\u00e9 a kdykoli).<\/li>\n<\/ul>\n<p><\/p>\n<p>Posledn\u00ed po\u017eadavek na realizaci tohoto projektu byl pro n\u00e1s nejt\u011b\u017e\u0161\u00ed, o \u010dem\u017e se je\u0161t\u011b podrobn\u011bji zm\u00edn\u00edm.<\/p>\n<p><\/p>\n<h2 id=\"sreda\">Prost\u0159ed\u00ed<\/h2>\n<p><\/p>\n<p>Pracujeme ve \u010dty\u0159ech datov\u00fdch centrech, p\u0159i\u010dem\u017e datov\u00e9 uzly Elasticsearch mohou b\u00fdt um\u00edst\u011bny pouze ve t\u0159ech (z n\u011bkolika netechnick\u00fdch d\u016fvod\u016f).<\/p>\n<p><\/p>\n<p>V t\u011bchto \u010dty\u0159ech datov\u00fdch centrech se nach\u00e1z\u00ed p\u0159ibli\u017en\u011b 18 tis\u00edc r\u016fzn\u00fdch zdroj\u016f log\u016f \u2014 hardware, kontejnery, virtu\u00e1ln\u00ed stroje.<\/p>\n<p><\/p>\n<p>D\u016fle\u017eit\u00e1 vlastnost: spu\u0161t\u011bn\u00ed clusteru prob\u00edh\u00e1 v kontejnerech <noindex><a rel=\"nofollow\" href=\"https:\/\/podman.io\">Podman<\/a><\/noindex> ne na fyzick\u00fdch stroj\u00edch, ale na <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/346868\/\">vlastn\u00edm cloudov\u00e9m produktu one-cloud<\/a><\/noindex>. Kontejnery maj\u00ed zaji\u0161t\u011bna 2 j\u00e1dra, obdobn\u00e1 2.0Ghz v4, s mo\u017enost\u00ed vyu\u017eit\u00ed dal\u0161\u00edch jader v p\u0159\u00edpad\u011b jejich ne\u010dinnosti. <\/p>\n<p><\/p>\n<p>Jin\u00fdmi slovy:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/073bffbfc3cff761bfabe0773b7f4ebc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"topologiya\">Topologie<\/h2>\n<p><\/p>\n<p>Celkov\u00e9 \u0159e\u0161en\u00ed mi p\u016fvodn\u011b p\u0159ipadalo n\u00e1sledovn\u011b:<\/p>\n<p><\/p>\n<ul>\n<li>3-4 VIP jsou za A-z\u00e1znamem dom\u00e9ny Graylog, co\u017e je adresa, na kterou jsou odes\u00edl\u00e1ny logy.<\/li>\n<li>ka\u017ed\u00fd VIP p\u0159edstavuje vyrovn\u00e1va\u010d z\u00e1t\u011b\u017ee LVS.<\/li>\n<li>Pot\u00e9 logy p\u0159ich\u00e1zej\u00ed do z\u00e1sobn\u00edku Graylog, \u010d\u00e1st dat jde ve form\u00e1tu GELF, \u010d\u00e1st ve form\u00e1tu syslog.<\/li>\n<li>D\u00e1le se to v\u0161echno ve velk\u00fdch d\u00e1vk\u00e1ch zapisuje do z\u00e1sobn\u00edku koordin\u00e1tor\u016f Elasticsearch. <\/li>\n<li>A ti zase odes\u00edlaj\u00ed \u017e\u00e1dosti o z\u00e1pis a \u010dten\u00ed na relevantn\u00ed datov\u00e9 uzly. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/21efcdf6992a07e0c5e9b477c567cca9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"terminologiya\">Terminologie<\/h2>\n<p><\/p>\n<p>Mo\u017en\u00e1 ne v\u0161ichni se podrobn\u011b orientuj\u00ed v terminologii, a tak bych se na ni r\u00e1da na chv\u00edli zastavila.<\/p>\n<p><\/p>\n<p>V Elasticsearch existuje n\u011bkolik typ\u016f uzl\u016f \u2014 master, coordinator, data node. Existuj\u00ed tak\u00e9 dva dal\u0161\u00ed typy pro r\u016fzn\u00e9 transformace log\u016f a propojen\u00ed r\u016fzn\u00fdch cluster\u016f mezi sebou, ale my jsme pou\u017e\u00edvali pouze uveden\u00e9. <\/p>\n<p><\/p>\n<p><strong>Master<\/strong><br \/>\nPinga v\u0161echny p\u0159\u00edtomn\u00e9 uzly v clusteru, udr\u017euje aktu\u00e1ln\u00ed mapu clusteru a roz\u0161i\u0159uje ji mezi uzly, zpracov\u00e1v\u00e1 logiku ud\u00e1lost\u00ed, zab\u00fdv\u00e1 se r\u016fzn\u00fdmi \u00fakoly t\u00fdkaj\u00edc\u00edmi se clusteru. <\/p>\n<p><\/p>\n<p><strong>Koordin\u00e1tor<\/strong><br \/>\nPln\u00ed jednu jedinou \u00falohu: p\u0159ij\u00edm\u00e1 \u017e\u00e1dosti od klient\u016f na \u010dten\u00ed nebo z\u00e1pis a sm\u011bruje tento provoz. V p\u0159\u00edpad\u011b, \u017ee jde o \u017e\u00e1dost o z\u00e1pis, pravd\u011bpodobn\u011b se zept\u00e1 u mastera, na kter\u00fd shard p\u0159\u00edslu\u0161n\u00e9ho indexu to m\u00e1 ulo\u017eit, a p\u0159esm\u011bruje \u017e\u00e1dost d\u00e1l. <\/p>\n<p><\/p>\n<p><strong>Data uzel<\/strong><br \/>\nUkl\u00e1d\u00e1 data, prov\u00e1d\u00ed vn\u011bj\u0161\u00ed p\u0159\u00edchoz\u00ed vyhled\u00e1vac\u00ed dotazy a operace s um\u00edst\u011bn\u00fdmi shardy.<\/p>\n<p><\/p>\n<p><strong>Graylog<\/strong><br \/>\nTo je n\u011bco jako slou\u010denina Kibany s Logstashem v ELK stacku. Graylog kombinuje u\u017eivatelsk\u00e9 rozhran\u00ed a pipeline pro zpracov\u00e1n\u00ed log\u016f. Pod kapotou Graylogu funguj\u00ed Kafka a Zookeeper, kter\u00e9 zaji\u0161\u0165uj\u00ed propojen\u00ed Graylogu jako clusteru. Graylog um\u00ed cacheovat logy (Kafka) pro p\u0159\u00edpad nep\u0159\u00edstupnosti Elasticsearch a opakovat ne\u00fasp\u011b\u0161n\u00e9 \u010dtec\u00ed a z\u00e1pisov\u00e9 po\u017eadavky, seskupovat a ozna\u010dovat logy podle nastaven\u00fdch pravidel. Stejn\u011b jako Logstash m\u00e1 Graylog funk\u010dnost pro modifikaci \u0159et\u011bzc\u016f p\u0159ed z\u00e1pisem do Elasticsearch.<\/p>\n<p><\/p>\n<p>Krom\u011b toho m\u00e1 Graylog vestav\u011bn\u00e9 zji\u0161\u0165ov\u00e1n\u00ed slu\u017eeb, kter\u00e9 na z\u00e1klad\u011b jedin\u00e9 dostupn\u00e9 uzlu Elasticsearch vytv\u00e1\u0159\u00ed celou mapu clusteru a filtruje ji podle konkr\u00e9tn\u00edho tagu, co\u017e umo\u017e\u0148uje sm\u011b\u0159ovat po\u017eadavky na ur\u010dit\u00e9 kontejnery.<\/p>\n<p><\/p>\n<p>Vizualn\u011b to vypad\u00e1 zhruba takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/8673ba9c0300ea0153d34a9f98afbcf2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>To je screenshot z konkr\u00e9tn\u00ed instance. Zde budujeme histogram na z\u00e1klad\u011b dotazu a zobrazujeme relevantn\u00ed \u0159et\u011bzce.<\/p>\n<p><\/p>\n<h2 id=\"indeksy\">Indexy<\/h2>\n<p><\/p>\n<p>Vr\u00e1t\u00edme-li se k architektu\u0159e syst\u00e9mu, cht\u011bl bych se podrobn\u011bji zab\u00fdvat t\u00edm, jak jsme stav\u011bli model index\u016f, aby to v\u0161echno fungovalo spr\u00e1vn\u011b. <\/p>\n<p><\/p>\n<p>Na d\u0159\u00edve uveden\u00e9m sch\u00e9matu je to nejni\u017e\u0161\u00ed \u00farove\u0148: data uzly Elasticsearch.<\/p>\n<p><\/p>\n<p>Index je velk\u00e1 virtu\u00e1ln\u00ed entita slo\u017een\u00e1 z shard\u016f Elasticsearch. Samotn\u00fd ka\u017ed\u00fd shard je nic jin\u00e9ho ne\u017e Lucene index. A ka\u017ed\u00fd Lucene index se skl\u00e1d\u00e1 z jednoho nebo v\u00edce segment\u016f. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/13b858cb102dce26b0851516b2d36c43.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u0159i n\u00e1vrhu jsme odhadovali, \u017ee pro spln\u011bn\u00ed po\u017eadavku na rychlost \u010dten\u00ed p\u0159i velk\u00e9m objemu dat je nezbytn\u00e9 rovnom\u011brn\u011b \"rozlo\u017eit\" tato data mezi data uzly. <\/p>\n<p><\/p>\n<p>To vedlo k tomu, \u017ee po\u010det shard\u016f na index (se replikami) by m\u011bl b\u00fdt p\u0159esn\u011b roven po\u010dtu data uzl\u016f. Jednak proto, aby byl zaji\u0161t\u011bn replikovan\u00fd faktor rovn\u00fd dv\u011bma (to znamen\u00e1, \u017ee m\u016f\u017eeme ztratit polovinu clusteru). A za druh\u00e9, abychom mohli zpracov\u00e1vat z\u00e1pisov\u00e9 a \u010dtec\u00ed po\u017eadavky alespo\u0148 na polovin\u011b clusteru.<\/p>\n<p><\/p>\n<p>Doba uchov\u00e1v\u00e1n\u00ed byla nejprve stanovena na 30 dn\u00ed.<\/p>\n<p><\/p>\n<p>Rozlo\u017een\u00ed shard\u016f lze graficky zn\u00e1zornit n\u00e1sleduj\u00edc\u00edm zp\u016fsobem:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/242726053360d1a6bdf4e527edc6cae6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Cel\u00fd tmav\u011b \u0161ed\u00fd obd\u00e9ln\u00edk je index. Lev\u00fd \u010derven\u00fd \u010dtverec v n\u011bm je primary shard, prvn\u00ed v indexu. A modr\u00fd \u010dtverec je replica shard. Tyto se nach\u00e1zej\u00ed v r\u016fzn\u00fdch datov\u00fdch centrech.<\/p>\n<p><\/p>\n<p>Kdy\u017e p\u0159id\u00e1me dal\u0161\u00ed shard, dostane se do t\u0159et\u00edho datacentra. A nakonec z\u00edsk\u00e1me strukturu, kter\u00e1 umo\u017e\u0148uje ztr\u00e1tu datacentra bez ztr\u00e1ty konzistence dat:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/27cbe4d4606f5ae5f3013ea46504134b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Rotaci index\u016f, tj. vytvo\u0159en\u00ed nov\u00e9ho indexu a odstran\u011bn\u00ed nejstar\u0161\u00edho, jsme nastavili na 48 hodin (podle vzoru pou\u017eit\u00ed indexu: posledn\u00edch 48 hodin se hled\u00e1 nej\u010dast\u011bji).<\/p>\n<p><\/p>\n<p>Takov\u00fd interval rotace index\u016f je spojen s n\u00e1sleduj\u00edc\u00edmi d\u016fvody:<\/p>\n<p><\/p>\n<p>Kdy\u017e na konkr\u00e9tn\u00ed datov\u00e9 uzly p\u0159ijde vyhled\u00e1vac\u00ed dotaz, z pohledu v\u00fdkonu je v\u00fdhodn\u011bj\u0161\u00ed, kdy\u017e je dotazov\u00e1na jedna shard, pokud je jej\u00ed velikost srovnateln\u00e1 s velikost\u00ed heapu uzlu. To umo\u017e\u0148uje udr\u017eovat \"horkou\" \u010d\u00e1st indexu v heapu a rychle k n\u00ed p\u0159istupovat. Kdy\u017e je \"hork\u00fdch \u010d\u00e1st\u00ed\" p\u0159\u00edli\u0161 mnoho, degraduje se rychlost vyhled\u00e1v\u00e1n\u00ed v indexu.<\/p>\n<p><\/p>\n<p>Kdy\u017e uzel za\u010d\u00edn\u00e1 zpracov\u00e1vat vyhled\u00e1vac\u00ed dotaz na jednom shardu, p\u0159id\u011bl\u00ed po\u010det vl\u00e1ken, kter\u00fd odpov\u00edd\u00e1 po\u010dtu hyperthreadingov\u00fdch jader fyzick\u00e9ho stroje. Pokud vyhled\u00e1vac\u00ed dotaz zasahuje do velk\u00e9ho po\u010dtu shard\u016f, po\u010det vl\u00e1ken roste proporcion\u00e1ln\u011b. To \u0161patn\u011b ovliv\u0148uje rychlost vyhled\u00e1v\u00e1n\u00ed a negativn\u011b se odr\u00e1\u017e\u00ed na indexaci nov\u00fdch dat. <\/p>\n<p><\/p>\n<p>Abychom zajistili pot\u0159ebnou latenci vyhled\u00e1v\u00e1n\u00ed, rozhodli jsme se pou\u017e\u00edvat SSD. Pro rychl\u00e9 zpracov\u00e1n\u00ed dotaz\u016f m\u011bly stroje, na kter\u00fdch byly um\u00edst\u011bny tyto kontejnery, m\u00edt alespo\u0148 56 jader. \u010c\u00edslo 56 bylo vybr\u00e1no jako podm\u00edn\u011bn\u011b dostate\u010dn\u00e1 hodnota, ur\u010duj\u00edc\u00ed po\u010det vl\u00e1ken, kter\u00e9 bude Elasticsearch vytv\u00e1\u0159et v pr\u016fb\u011bhu sv\u00e9 \u010dinnosti. V Elasticsearch z\u00e1vis\u00ed mnoho parametr\u016f thread poolu p\u0159\u00edmo na po\u010dtu dostupn\u00fdch jader, co\u017e p\u0159\u00edmo ovliv\u0148uje pot\u0159ebn\u00fd po\u010det uzl\u016f v klastru podle principu \"m\u00e9n\u011b jader \u2014 v\u00edce uzl\u016f\". <\/p>\n<p><\/p>\n<p>Nakonec se n\u00e1m poda\u0159ilo, \u017ee pr\u016fm\u011brn\u00fd shard v\u00e1\u017e\u00ed zhruba 20 gigabajt\u016f a na 1 index p\u0159ipad\u00e1 360 shard\u016f. P\u0159i rotaci ka\u017ed\u00fdch 48 hodin tedy m\u00e1me 15 kousk\u016f. Ka\u017ed\u00fd index obsahuje data za 2 dny.<\/p>\n<p><\/p>\n<h2 id=\"shemy-zapisi-i-chteniya-dannyh\">Sch\u00e9mata z\u00e1pisu a \u010dten\u00ed dat<\/h2>\n<p><\/p>\n<p>Poj\u010fme se pod\u00edvat, jak jsou v tomto syst\u00e9mu zapisov\u00e1na data.<\/p>\n<p><\/p>\n<p>P\u0159edstavme si, \u017ee k n\u00e1m z Graylog p\u0159ich\u00e1z\u00ed n\u011bjak\u00fd dotaz do koordin\u00e1toru. Nap\u0159\u00edklad chceme zaindexovat 2-3 tis\u00edce \u0159\u00e1dk\u016f. <\/p>\n<p><\/p>\n<p>Koordin\u00e1tor, kter\u00fd obdr\u017eel po\u017eadavek od Graylogu, se dotazuje mistra: \u201eV po\u017eadavku na indexaci byl konkr\u00e9tn\u011b uveden index, ale do kter\u00e9ho shardu to zapsat \u2014 nen\u00ed uvedeno.\u201c <\/p>\n<p><\/p>\n<p>Mistr odpov\u00edd\u00e1: \u201eZapi\u0161 tuto informaci do shardu \u010d\u00edslo 71\u201c, po \u010dem\u017e je bezprost\u0159edn\u011b sm\u011brov\u00e1na do relevantn\u00edho datov\u00e9ho uzlu, kde se nach\u00e1z\u00ed primary-shard \u010d\u00edslo 71.<\/p>\n<p><\/p>\n<p>Po tom se log transakc\u00ed replikoval na replica-shard, kter\u00fd se nach\u00e1z\u00ed ji\u017e v jin\u00e9m datov\u00e9m centru.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/fdb6a77cb78b1eed290e440478569e3b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Z Graylogu p\u0159ich\u00e1z\u00ed koordina\u010dn\u00ed po\u017eadavek. Koordin\u00e1tor ho p\u0159esm\u011brov\u00e1v\u00e1 podle indexu, p\u0159i\u010dem\u017e Elasticsearch podle principu round-robin rozd\u011bluje po\u017eadavky mezi primary-shard a replica-shard. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/cb32bac878e04d6f50d103ebbc395a7f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Uzly v po\u010dtu 180 kus\u016f odpov\u00eddaj\u00ed nerovnom\u011brn\u011b a zat\u00edmco odpov\u00eddaj\u00ed, koordin\u00e1tor shroma\u017e\u010fuje informace, kter\u00e9 do n\u011bj ji\u017e \u201evyplyvly\u201c rychlej\u0161\u00ed datov\u00e9 uzly. Pot\u00e9, kdy\u017e bu\u010f ve\u0161ker\u00e9 informace dorazily, nebo byl dosa\u017een timeout, p\u0159ed\u00e1v\u00e1 v\u0161e p\u0159\u00edmo klientovi. <\/p>\n<p><\/p>\n<p>Cel\u00fd tento syst\u00e9m pr\u016fm\u011brn\u011b zpracov\u00e1v\u00e1 vyhled\u00e1vac\u00ed po\u017eadavky za posledn\u00edch 48 hodin za 300-400ms, s v\u00fdjimkou t\u011bch po\u017eadavk\u016f, kter\u00e9 maj\u00ed leading wildcard.<\/p>\n<p><\/p>\n<h2 id=\"cvetochki-s-elasticsearch-nastroyka-java\">\u201eKv\u00edtka\u201c s Elasticsearch: konfigurace Java<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/fc9ac4b8ee41bc35f5f01b8d9c4ff2f6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Aby to v\u0161echno fungovalo tak, jak jsme p\u016fvodn\u011b cht\u011bli, dlouho jsme ladili r\u016fzn\u00e9 v\u011bci v clusteru. <\/p>\n<p><\/p>\n<p>Prvn\u00ed \u010d\u00e1st nalezen\u00fdch probl\u00e9m\u016f byla spojena s t\u00edm, jak je v Elasticsearch ve v\u00fdchoz\u00edm nastaven\u00ed p\u0159ednastaven\u00e1 Java. <\/p>\n<p><\/p>\n<p><strong>Probl\u00e9m prvn\u00ed<\/strong><br \/>\nPozorovali jsme velmi mnoho zpr\u00e1v o tom, \u017ee na \u00farovni Lucene, kdy\u017e jsou spu\u0161t\u011bny background joby, se slou\u010den\u00ed segment\u016f Lucene prov\u00e1d\u00ed s chybou. P\u0159itom v logech bylo vid\u011bt, \u017ee se jedn\u00e1 o chybu OutOfMemoryError. Podle telemetrie jsme vid\u011bli, \u017ee je pam\u011b\u0165 voln\u00e1, a nebylo jasn\u00e9, pro\u010d tato operace selh\u00e1v\u00e1. <\/p>\n<p><\/p>\n<p>Zjistilo se, \u017ee mergen\u00ed Lucene index\u016f prob\u00edh\u00e1 mimo hromadu. A kontejnery jsou pom\u011brn\u011b striktn\u011b omezeny co do spot\u0159ebovan\u00fdch zdroj\u016f. Do t\u011bchto zdroj\u016f se ve\u0161el pouze heap (hodnota heap.size byla p\u0159ibli\u017en\u011b rovna RAM), a n\u011bjak\u00e9 off-heap operace selh\u00e1valy s chybou alokace pam\u011bti, pokud z n\u011bjak\u00e9ho d\u016fvodu nep\u0159ekra\u010dovaly t\u011bch ~500MB, kter\u00e9 z\u016fst\u00e1valy do limitu.<\/p>\n<p><\/p>\n<p>Oprava byla docela trivi\u00e1ln\u00ed: zv\u00fd\u0161ili jsme dostupn\u00fd objem RAM pro kontejner, po \u010dem\u017e jsme zapomn\u011bli, \u017ee takov\u00e9 probl\u00e9my jsme m\u011bli.<\/p>\n<p><\/p>\n<p><strong>Probl\u00e9m druh\u00fd<\/strong><br \/>\nAsi po 4-5 dnech po spu\u0161t\u011bn\u00ed clusteru jsme si v\u0161imli, \u017ee datov\u00e9 uzly za\u010d\u00ednaj\u00ed periodicky vypad\u00e1vat z clusteru a znovu se do n\u011bj dost\u00e1vat po 10-20 sekund\u00e1ch. <\/p>\n<p><\/p>\n<p>Kdy\u017e jsme se za\u010dali zab\u00fdvat probl\u00e9my, zjistili jsme, \u017ee pam\u011b\u0165 off-heap v Elasticsearch nen\u00ed prakticky nijak \u0159\u00edzena. Kdy\u017e jsme kontejneru p\u0159i\u0159adili v\u00edce pam\u011bti, z\u00edskali jsme mo\u017enost napl\u0148ovat direct buffer pools r\u016fzn\u00fdmi informacemi, kter\u00e9 byly vy\u010di\u0161t\u011bny pouze po spu\u0161t\u011bn\u00ed explicitn\u00edho GC v Elasticsearch. <\/p>\n<p><\/p>\n<p>V n\u011bkter\u00fdch p\u0159\u00edpadech tato operace prob\u00edhala velmi dlouho, a mezit\u00edm stihl cluster ozna\u010dit tento uzel jako ji\u017e nedostupn\u00fd. Tento probl\u00e9m je dob\u0159e pops\u00e1n. <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/elasticsearch-5-6-very-quickly-increasing-direct-buffer-pools\/119561\">tady<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>\u0158e\u0161en\u00ed bylo n\u00e1sleduj\u00edc\u00ed: omezili jsme Java na vyu\u017e\u00edv\u00e1n\u00ed v\u011bt\u0161iny pam\u011bti mimo hromadu pro tyto operace. Omezili jsme ji na 16 gigabajt\u016f (-XX:MaxDirectMemorySize=16g), co\u017e vedlo k tomu, \u017ee se explicitn\u00ed GC spou\u0161t\u011bl v\u00fdrazn\u011b \u010dast\u011bji a b\u011b\u017eel zna\u010dn\u011b rychleji, co\u017e p\u0159estalo destabilizovat cluster.<\/p>\n<p><\/p>\n<p><strong>T\u0159et\u00ed probl\u00e9m<\/strong><br \/>\nPokud si mysl\u00edte, \u017ee probl\u00e9my s \"uzly, kter\u00e9 opou\u0161t\u011bj\u00ed cluster v nejm\u00e9n\u011b o\u010dek\u00e1vanou chv\u00edli\" t\u00edmto skon\u010dily, m\u00fdl\u00edte se. <\/p>\n<p><\/p>\n<p>Kdy\u017e jsme konfigurovali pr\u00e1ci s indexy, vybrali jsme mmapfs, abychom <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/benchmarks-default-niofs-vs-memory-mapped-mmapfs\/13118\/2\">zkr\u00e1tili dobu vyhled\u00e1v\u00e1n\u00ed<\/a><\/noindex> na \u010derstv\u00fdch shardech s velkou segmentovanost\u00ed. To byla docela hrub\u00e1 chyba, proto\u017ee p\u0159i pou\u017eit\u00ed mmapfs je soubor mapov\u00e1n do opera\u010dn\u00ed pam\u011bti, a d\u00e1le pracujeme s mapovan\u00fdm souborem. V d\u016fsledku toho se p\u0159i pokusu GC zastavit vl\u00e1kna v aplikaci velmi dlouho dost\u00e1v\u00e1me do safepoint, a po cest\u011b tam aplikace p\u0159est\u00e1v\u00e1 odpov\u00eddat na dotazy masteru ohledn\u011b toho, zda je \u017eiv\u00e1. Odpov\u00eddaj\u00edc\u00edm zp\u016fsobem master p\u0159edpokl\u00e1d\u00e1, \u017ee uzel ji\u017e nen\u00ed v clusteru p\u0159\u00edtomen. Pot\u00e9, po 5-10 sekund\u00e1ch, garbage collector dokon\u010d\u00ed svou pr\u00e1ci, uzel o\u017eije, znovu se p\u0159ipoj\u00ed k clusteru a za\u010dne inicializaci shard\u016f. To v\u0161echno siln\u011b p\u0159ipom\u00ednalo \u201eprodukci, kterou jsme si zaslou\u017eili\u201c a nebylo vhodn\u00e9 pro nic v\u00e1\u017en\u00e9ho.<\/p>\n<p><\/p>\n<p>Abychom se tohoto chov\u00e1n\u00ed zbavili, nejprve jsme p\u0159e\u0161li na standardn\u00ed niofs, a pot\u00e9, kdy\u017e jsme se z p\u00e1t\u00fdch verz\u00ed Elastic migraci na \u0161est\u00e9, vyzkou\u0161eli hybridfs, kde se tento probl\u00e9m nevyskytoval. O typech \u00falo\u017ei\u0161t\u011b si m\u016f\u017eete p\u0159e\u010d\u00edst v\u00edce. <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/index-modules-store.html\">zde<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><strong>\u010ctvrt\u00fd probl\u00e9m<\/strong><br \/>\nPot\u00e9 byla je\u0161t\u011b jedna velmi zaj\u00edmav\u00e1 probl\u00e9m, kterou jsme opravovali rekordn\u011b dlouho. Chytali jsme ji 2-3 m\u011bs\u00edce, proto\u017ee jej\u00ed vzor byl absolutn\u011b nepochopiteln\u00fd. <\/p>\n<p><\/p>\n<p>N\u011bkdy na\u0161i koordin\u00e1to\u0159i vypadli do Full GC, obvykle n\u011bkdy po ob\u011bd\u011b, a odtud se u\u017e nevr\u00e1tili. P\u0159itom, kdy\u017e jsme logovali zpo\u017ed\u011bn\u00ed GC, vypadalo to takto: v\u0161echno \u0161lo dob\u0159e, dob\u0159e, dob\u0159e, a pak najednou \u2014 a v\u0161echno se prudce zhor\u0161ilo. <\/p>\n<p><\/p>\n<p>Nejprve jsme si mysleli, \u017ee m\u00e1me zlomysln\u00e9ho u\u017eivatele, kter\u00fd spou\u0161t\u00ed n\u011bjak\u00fd dotaz, kter\u00fd vyhazuje koordin\u00e1tora z pracovn\u00edho re\u017eimu. Velmi dlouho jsme logovali dotazy a sna\u017eili jsme se zjistit, co se d\u011bje. <\/p>\n<p><\/p>\n<p>Nakonec se uk\u00e1zalo, \u017ee v okam\u017eiku, kdy n\u011bjak\u00fd u\u017eivatel spust\u00ed velmi velk\u00fd dotaz, a ten se dostane na konkr\u00e9tn\u00ed koordin\u00e1tor Elasticsearch, n\u011bkter\u00e9 uzly odpov\u00eddaj\u00ed d\u00e9le ne\u017e ostatn\u00ed. <\/p>\n<p><\/p>\n<p>A to \u010das, po kter\u00fd koordin\u00e1tor \u010dek\u00e1 na odpov\u011b\u010f v\u0161ech uzl\u016f, shroma\u017e\u010fuje v\u00fdsledky z uzl\u016f, kter\u00e9 ji\u017e odpov\u011bd\u011bly. Pro GC to znamen\u00e1, \u017ee se n\u00e1m velmi rychle m\u011bn\u00ed vzor vyu\u017eit\u00ed hromady. A ten GC, kter\u00fd jsme pou\u017e\u00edvali, si s t\u00edmto \u00fakolem neporadil. <\/p>\n<p><\/p>\n<p>Jedin\u00e9 \u0159e\u0161en\u00ed, kter\u00e9 jsme na\u0161li pro zm\u011bnu chov\u00e1n\u00ed clusteru v takov\u00e9 situaci \u2014 migrace na JDK13 a pou\u017eit\u00ed sb\u011bra\u010de odpadu Shenandoah. To vy\u0159e\u0161ilo probl\u00e9m, koordin\u00e1to\u0159i u n\u00e1s p\u0159estali padat. <\/p>\n<p><\/p>\n<p>T\u00edmto probl\u00e9my s Javou skon\u010dily a za\u010daly probl\u00e9my s propustnost\u00ed. <\/p>\n<p><\/p>\n<h2 id=\"yagodki-s-elasticsearch-propusknaya-sposobnost\">\u201eBobulky\u201c s Elasticsearch: propustnost<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/bb5c08193d7d764515235e825cd30ce5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Probl\u00e9my s propustnost\u00ed znamenaj\u00ed, \u017ee n\u00e1\u0161 cluster pracuje stabiln\u011b, ale p\u0159i vrcholech po\u010dtu indexovan\u00fdch dokument\u016f a v okam\u017eic\u00edch man\u00e9vr\u016f je v\u00fdkon nedostate\u010dn\u00fd.<\/p>\n<p><\/p>\n<p>Prvn\u00ed pozorovan\u00fd symptom: p\u0159i n\u011bjak\u00fdch \u201evzbou\u0159en\u00edch\u201c v produkci, kdy se n\u00e1hle generuje velmi velk\u00e9 mno\u017estv\u00ed log\u016f, se v Graylogu za\u010dala \u010dasto objevovat chyba indexace es_rejected_execution. <\/p>\n<p><\/p>\n<p>To se d\u011blo proto, \u017ee thread_pool.write.queue na jedn\u00e9 datov\u00e9 uzlu m\u00e1 p\u0159ed t\u00edm, ne\u017e Elasticsearch dok\u00e1\u017ee zpracovat \u017e\u00e1dost o indexaci a ulo\u017eit informace do shardu na disk, ve v\u00fdchoz\u00edm nastaven\u00ed mo\u017enost cache pouze 200 \u017e\u00e1dost\u00ed. A v <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">dokumentaci Elasticsearch<\/a><\/noindex> o tomto parametru se mluv\u00ed velmi m\u00e1lo. Uv\u00e1d\u00ed se pouze maxim\u00e1ln\u00ed po\u010det vl\u00e1ken a v\u00fdchoz\u00ed velikost.<\/p>\n<p><\/p>\n<p>Samoz\u0159ejm\u011b jsme \u0161li upravit tuto hodnotu a zjistili jsme n\u00e1sleduj\u00edc\u00ed: konkr\u00e9tn\u011b v na\u0161em nastaven\u00ed se celkem dob\u0159e cache a\u017e 300 \u017e\u00e1dost\u00ed, ale v\u011bt\u0161\u00ed hodnota s sebou nese to, \u017ee znovu let\u00edme do Full GC.<\/p>\n<p><\/p>\n<p>Krom\u011b toho, jeliko\u017e se jedn\u00e1 o d\u00e1vky zpr\u00e1v, kter\u00e9 p\u0159ich\u00e1zej\u00ed v r\u00e1mci jednoho po\u017eadavku, bylo tak\u00e9 pot\u0159eba upravit Graylog, aby zapisoval m\u00e9n\u011b \u010dasto a v mal\u00fdch d\u00e1vk\u00e1ch, ale v obrovsk\u00fdch d\u00e1vk\u00e1ch nebo ka\u017ed\u00e9 3 sekundy, pokud d\u00e1vka nen\u00ed st\u00e1le pln\u00e1. V takov\u00e9m p\u0159\u00edpad\u011b se ukazuje, \u017ee informace, kter\u00e9 zapisujeme do Elasticsearch, se st\u00e1vaj\u00ed dostupn\u00fdmi ne za dv\u011b sekundy, ale za p\u011bt (co\u017e n\u00e1m pln\u011b vyhovuje), ale sni\u017euje se po\u010det retriev\u016f, kter\u00e9 mus\u00edme prov\u00e9st, abychom protla\u010dili velkou d\u00e1vku informac\u00ed.<\/p>\n<p><\/p>\n<p>To je zvl\u00e1\u0161t\u011b d\u016fle\u017eit\u00e9 v momentech, kdy n\u00e1m n\u011bco n\u011bkde spadne a urputn\u011b o tom hl\u00e1s\u00ed, abychom nedostali \u00fapln\u011b spamatovan\u00fd Elastic, a za n\u011bjakou dobu \u2014 nefunk\u010dn\u00ed kv\u016fli ucpan\u00fdm buffer\u016fm uzly Graylog.<\/p>\n<p><\/p>\n<p>Krom\u011b toho, kdy\u017e k t\u011bmto v\u00fdbuch\u016fm doch\u00e1zelo v produkci, dost\u00e1vali jsme st\u00ed\u017enosti od program\u00e1tor\u016f a tester\u016f: v okam\u017eiku, kdy velmi pot\u0159ebovali tyto logy, byly jim poskytov\u00e1ny velmi pomalu.<\/p>\n<p><\/p>\n<p>Za\u010dali jsme se t\u00edm zab\u00fdvat. Z jedn\u00e9 strany bylo jasn\u00e9, \u017ee dotazy na vyhled\u00e1v\u00e1n\u00ed a dotazy na indexaci funguj\u00ed v podstat\u011b na stejn\u00fdch fyzick\u00fdch stroj\u00edch, a tak nebo onak budou ur\u010dit\u00e9 propady. <\/p>\n<p><\/p>\n<p>To bylo mo\u017en\u00e9 \u010d\u00e1ste\u010dn\u011b obej\u00edt d\u00edky tomu, \u017ee v \u0161esti verz\u00edch Elasticsearch vznikl algoritmus, kter\u00fd umo\u017e\u0148uje rozd\u011blovat dotazy mezi relevantn\u00ed data-uzly ne n\u00e1hodn\u011b round-robin (kontejner, kter\u00fd se zab\u00fdv\u00e1 indexac\u00ed a dr\u017e\u00ed primary-shard, m\u016f\u017ee b\u00fdt velmi vyt\u00ed\u017een, nebude m\u00edt mo\u017enost rychle odpov\u011bd\u011bt), ale nasm\u011brovat tento dotaz na m\u00e9n\u011b zat\u00ed\u017een\u00fd kontejner s replica-shard, kter\u00fd odpov\u00ed v\u00fdrazn\u011b rychleji. Jin\u00fdmi slovy, p\u0159i\u0161li jsme k use_adaptive_replica_selection: true.<\/p>\n<p><\/p>\n<p>Obraz \u010dten\u00ed za\u010d\u00edn\u00e1 vypadat takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ae7c1645bb63692e580b1bca5c6e5db0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u0159echod na tento algoritmus zna\u010dn\u011b zlep\u0161il \u010das dotazu v momentech, kdy prob\u00edhal velk\u00fd tok log\u016f na z\u00e1pis.<\/p>\n<p><\/p>\n<p>Nakonec hlavn\u00ed probl\u00e9m spo\u010d\u00edval v bezbolestn\u00e9m vyveden\u00ed datacentra.<\/p>\n<p><\/p>\n<p>Co jsme cht\u011bli od clusteru okam\u017eit\u011b po ztr\u00e1t\u011b spojen\u00ed s jedn\u00edm DC: <\/p>\n<p><\/p>\n<ul>\n<li>Pokud m\u00e1me v odpadl\u00e9m datacentru aktu\u00e1ln\u00edho mastera, bude p\u0159evolen a p\u0159esune se jako role na jin\u00fd uzel v jin\u00e9m DC.<\/li>\n<li>Master rychle vyhod\u00ed ze shluku v\u0161echny nedostupn\u00e9 uzly.<\/li>\n<li>Na z\u00e1klad\u011b zb\u00fdvaj\u00edc\u00edch bude v\u011bd\u011bt: v ztracen\u00e9m datov\u00e9m centru jsme m\u011bli takov\u00e9 primary shard, rychle propromuje komplement\u00e1rn\u00ed replica shard v zb\u00fdvaj\u00edc\u00edch datov\u00fdch centrech a na\u0161e indexov\u00e1n\u00ed dat pob\u011b\u017e\u00ed d\u00e1l. <\/li>\n<li>V d\u016fsledku toho se n\u00e1m postupn\u011b degraduje propustnost clustera pro z\u00e1pis a \u010dten\u00ed, ale celkov\u011b to bude fungovat i kdy\u017e pomalu, ale stabiln\u011b.<\/li>\n<\/ul>\n<p><\/p>\n<p>Jak se uk\u00e1zalo, cht\u011bli jsme n\u011bco takov\u00e9ho:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/71fa3d19a61eb58ddb93a4df55c759c5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>A dostali jsme n\u00e1sleduj\u00edc\u00ed:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ab32be4dfbbb4bcc3279d57021707980.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Jak se to stalo? <\/p>\n<p><\/p>\n<p>V okam\u017eiku p\u00e1du datov\u00e9ho centra se n\u00e1m \u00fazk\u00fdm m\u00edstem stal master.<\/p>\n<p><\/p>\n<p>Pro\u010d?<\/p>\n<p><\/p>\n<p>To spo\u010d\u00edv\u00e1 v tom, \u017ee v masteru je TaskBatcher, kter\u00fd zodpov\u00edd\u00e1 za roz\u0161i\u0159ov\u00e1n\u00ed ur\u010dit\u00fdch \u00faloh a ud\u00e1lost\u00ed v clusteru. Jak\u00fdkoliv v\u00fdstup uzlu, jak\u00e9koli pov\u00fd\u0161en\u00ed shardu z replica na primary, jak\u00e1koli \u00faloha na vytvo\u0159en\u00ed n\u011bjak\u00e9ho shardu \u2013 to v\u0161echno se nejprve dostane do TaskBatcheru, kde se zpracov\u00e1v\u00e1 postupn\u011b a jedn\u00edm vl\u00e1knem.<\/p>\n<p><\/p>\n<p>V okam\u017eiku v\u00fdstupu jednoho datov\u00e9ho centra se uk\u00e1zalo, \u017ee v\u0161echny datov\u00e9 uzly ve zb\u00fdvaj\u00edc\u00edch datov\u00fdch centrech c\u00edtily za povinnost informovat mastera, \u017ee \"n\u00e1s ztratily takov\u00e9 shard a takov\u00e9 datov\u00e9 uzly\". <\/p>\n<p><\/p>\n<p>Z\u00e1rove\u0148 p\u0159e\u017e\u00edvaj\u00edc\u00ed datov\u00e9 uzly pos\u00edlaly v\u0161echny tyto informace aktu\u00e1ln\u00edmu masterovi a sna\u017eily se \u010dekat na potvrzen\u00ed, \u017ee je p\u0159ijal. Toho se nedo\u010dkaly, proto\u017ee master dost\u00e1val \u00falohy rychleji, ne\u017e stihl odpov\u011bd\u011bt. Uzly opakovan\u011b poslaly \u017e\u00e1dosti s \u010dasov\u00fdm limitem, zat\u00edmco master u\u017e ani nesna\u017eil na n\u011b odpov\u011bd\u011bt, byl zcela pohlcen \u00falohou t\u0159\u00edd\u011bn\u00ed \u017e\u00e1dost\u00ed podle priority.<\/p>\n<p><\/p>\n<p>V termin\u00e1ln\u00edm pohledu to vy\u0161lo tak, \u017ee datov\u00e9 uzly spamovaly mastera a\u017e tak, \u017ee ode\u0161el do full GC. Po tom se role mastera p\u0159enesla na n\u011bjak\u00fd n\u00e1sleduj\u00edc\u00ed uzel, s n\u00edm\u017e se stalo p\u0159esn\u011b tot\u00e9\u017e, a nakonec se cluster zcela rozpadl. <\/p>\n<p><\/p>\n<p>Prov\u00e1d\u011bli jsme m\u011b\u0159en\u00ed, a do verze 6.4.0, kde to bylo opraveno, n\u00e1m sta\u010dilo vy\u0159adit najednou pouze 10 datov\u00fdch uzl\u016f ze 360, abychom zcela zni\u010dili cluster.<\/p>\n<p><\/p>\n<p>Vypadalo to zhruba takto:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/3bed554b2703e7521339e9d943648ee4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Po verzi 6.4.0, kde opravili ten \u0161patn\u00fd bug, datov\u00e9 uzly p\u0159estaly zab\u00edjet mastera. Ale \"chyt\u0159ej\u0161\u00ed\" od toho nen\u00ed. A to konkr\u00e9tn\u011b: kdy\u017e vy\u0159azujeme 2, 3 nebo 10 (jak\u00e9koli mno\u017estv\u00ed, kter\u00e9 nen\u00ed jedin\u00e9) datov\u00fdch uzl\u016f, master dost\u00e1v\u00e1 n\u011bjakou prvn\u00ed zpr\u00e1vu, kter\u00e1 \u0159\u00edk\u00e1, \u017ee uzel A ode\u0161el, a sna\u017e\u00ed se o tom \u0159\u00edct uzlu B, uzlu C, uzlu D. <\/p>\n<p><\/p>\n<p>A v sou\u010dasnosti se s t\u00edm d\u00e1 bojovat pouze nastaven\u00edm \u010dasov\u00e9ho limitu na pokusy n\u011bkomu n\u011bco \u0159\u00edct, co\u017e je zhruba 20-30 sekund, a t\u00edmto zp\u016fsobem \u0159\u00eddit rychlost v\u00fdstupu datacentra z clusteru.<\/p>\n<p><\/p>\n<p>V z\u00e1sad\u011b to spad\u00e1 do t\u011bch po\u017eadavk\u016f, kter\u00e9 byly p\u016fvodn\u011b stanoveny pro fin\u00e1ln\u00ed produkt v r\u00e1mci projektu, ale z pohledu \u201e\u010dist\u00e9 v\u011bdy\u201c je to chyba. Ta byla mimochodem \u00fasp\u011b\u0161n\u011b opravena v\u00fdvoj\u00e1\u0159i ve verzi 7.2.<\/p>\n<p><\/p>\n<p>A kdy\u017e n\u011bjak\u00fd data-uzel vy\u0161el, uk\u00e1zalo se, \u017ee \u0161\u00ed\u0159en\u00ed informace o jeho v\u00fdstupu je d\u016fle\u017eit\u011bj\u0161\u00ed ne\u017e \u0159\u00edct cel\u00e9mu clusteru, \u017ee na n\u011bm byly takov\u00e9 primary-shardy (aby se promovaly replica-shardy v jin\u00e9m datacentru na primary a bylo mo\u017en\u00e9 do nich zapisovat informace).<\/p>\n<p><\/p>\n<p>Proto, kdy\u017e u\u017e v\u0161e \u201eutichlo\u201c, vystoupl\u00e9 data-uzly nejsou okam\u017eit\u011b ozna\u010deny jako stale. D\u00e1le jsme nuceni \u010dekat, a\u017e se vypr\u0161\u00ed v\u0161echny pingy k vystoupl\u00fdm data-uzl\u016fm, a teprve pot\u00e9 n\u00e1\u0161 cluster za\u010dne vypr\u00e1v\u011bt, \u017ee tam, tam a tam je t\u0159eba pokra\u010dovat ve psan\u00ed informac\u00ed. V\u00edce o tom si m\u016f\u017eete p\u0159e\u010d\u00edst <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/elastic\/elasticsearch\/issues\/46909\">zde<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>V\u00fdsledkem je, \u017ee operace v\u00fdstupu datacentra n\u00e1m dnes trv\u00e1 p\u0159ibli\u017en\u011b 5 minut v hodin\u00e1ch \u0161pi\u010dky. Pro tak velk\u00fd a neobratn\u00fd stroj je to docela dobr\u00fd v\u00fdsledek.<\/p>\n<p><\/p>\n<p>Nakonec jsme dosp\u011bli k n\u00e1sleduj\u00edc\u00edmu \u0159e\u0161en\u00ed:<\/p>\n<p><\/p>\n<ul>\n<li>M\u00e1me 360 data-uzl\u016f s disky o kapacit\u011b 700 gigabajt\u016f.<\/li>\n<li>60 koordin\u00e1tor\u016f pro sm\u011brov\u00e1n\u00ed provozu p\u0159es tyto data-uzly.<\/li>\n<li>40 master\u016f, kter\u00e9 jsme si ponechali jako n\u011bjak\u00e9 d\u011bdictv\u00ed z dob p\u0159ed verz\u00ed 6.4.0 \u2014 abychom p\u0159e\u017eili v\u00fdstup datacentra, byli jsme mor\u00e1ln\u011b p\u0159ipraveni ztratit n\u011bkolik stroj\u016f, abychom garantovali, \u017ee i p\u0159i nejhor\u0161\u00edm sc\u00e9n\u00e1\u0159i budeme m\u00edt quorum master\u016f.<\/li>\n<li>Jak\u00e9koliv pokusy o kombinaci rol\u00ed na jednom kontejneru narazily na to, \u017ee d\u0159\u00edve nebo pozd\u011bji uzel selhal pod z\u00e1t\u011b\u017e\u00ed. <\/li>\n<li>V cel\u00e9m clusteru se pou\u017e\u00edv\u00e1 heap.size, kter\u00fd je 31 gigabajt\u016f: v\u0161echny pokusy o zmen\u0161en\u00ed velikosti vedly k tomu, \u017ee p\u0159i t\u011b\u017ek\u00fdch vyhled\u00e1vac\u00edch dotazech s leading wildcard bu\u010f zabily n\u011bjak\u00e9 uzly, nebo se v samotn\u00e9m Elasticsearch zablokoval circuit breaker.<\/li>\n<li>Krom\u011b toho, pro zaji\u0161t\u011bn\u00ed v\u00fdkonnosti vyhled\u00e1v\u00e1n\u00ed, jsme se sna\u017eili udr\u017eovat po\u010det objekt\u016f v clusteru co mo\u017en\u00e1 nejmen\u0161\u00ed, abychom zpracov\u00e1vali co nejm\u00e9n\u011b ud\u00e1lost\u00ed v nejt\u011bsn\u011bj\u0161\u00edm m\u00edst\u011b, kter\u00e9 bylo u n\u00e1s v masteru.<\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"naposledok-o-monitoringe\">Nakonec o monitorov\u00e1n\u00ed<\/h2>\n<p><\/p>\n<p>Aby to v\u0161echno fungovalo tak, jak m\u00e1, monitorujeme n\u00e1sleduj\u00edc\u00ed:<\/p>\n<p><\/p>\n<ul>\n<li>Ka\u017ed\u00e9 datov\u00e9 uzly hl\u00e1s\u00ed do na\u0161eho cloudu, \u017ee existuj\u00ed, a \u017ee na nich jsou ur\u010dit\u00e9 shard. Kdy\u017e n\u011bkde n\u011bco vypneme, cluster b\u011bhem 2-3 sekund hl\u00e1s\u00ed, \u017ee v centru A jsme vypnuli uzly 2, 3 a 4 \u2013 to znamen\u00e1, \u017ee v jin\u00fdch datov\u00fdch centrech nem\u016f\u017eeme v \u017e\u00e1dn\u00e9m p\u0159\u00edpad\u011b vyp\u00ednat ty uzly, na kter\u00fdch z\u016fstaly shardy v jedin\u00e9m exempl\u00e1\u0159i.<\/li>\n<li>Zn\u00e1me-li charakter chov\u00e1n\u00ed mastera, velmi pe\u010dliv\u011b sledujeme po\u010det pending \u00faloh. Proto\u017ee i jedna vis\u00edc\u00ed \u00faloha, pokud se v\u010das neodstran\u00ed, m\u016f\u017ee teoreticky v n\u011bjak\u00e9 krizov\u00e9 situaci zp\u016fsobit probl\u00e9m, kv\u016fli kter\u00e9mu neprob\u011bhne nap\u0159\u00edklad propagace replica shardu na primary, co\u017e zablokuje indexaci.<\/li>\n<li>Tak\u00e9 se velmi d\u016fkladn\u011b d\u00edv\u00e1me na zpo\u017ed\u011bn\u00ed garbage collector, proto\u017ee jsme s t\u00edmto m\u011bli ji\u017e velk\u00e9 pot\u00ed\u017ee p\u0159i optimalizaci.<\/li>\n<li>Rejecty podle vl\u00e1ken, abychom p\u0159edem v\u011bd\u011bli, kde se nach\u00e1z\u00ed \u201e\u00fazk\u00e9 m\u00edsto\u201c.<\/li>\n<li>A standardn\u00ed metriky, jako heap, RAM a I\/O.<\/li>\n<\/ul>\n<p><\/p>\n<p>P\u0159i budov\u00e1n\u00ed monitorov\u00e1n\u00ed je nutn\u00e9 br\u00e1t v \u00favahu specifika Thread Pool v Elasticsearch. <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">Dokumentace Elasticsearch<\/a><\/noindex> popisuje mo\u017enosti nastaven\u00ed a v\u00fdchoz\u00ed hodnoty pro vyhled\u00e1v\u00e1n\u00ed, indexaci, ale zcela ml\u010d\u00ed o thread_pool.management. Tyto vl\u00e1kna zpracov\u00e1vaj\u00ed mimo jin\u00e9 dotazy typu _cat\/shards a jin\u00e9 podobn\u00e9, kter\u00e9 jsou pohodln\u00e9 pro psan\u00ed monitorov\u00e1n\u00ed. \u010c\u00edm v\u011bt\u0161\u00ed je cluster, t\u00edm v\u00edce takov\u00fdch dotaz\u016f se vykon\u00e1v\u00e1 za jednotku \u010dasu, a v\u00fd\u0161e zm\u00edn\u011bn\u00e9 thread_pool.management nejen\u017ee nen\u00ed zastoupeno v ofici\u00e1ln\u00ed dokumentaci, ale je tak\u00e9 ve v\u00fdchoz\u00edm nastaven\u00ed limitov\u00e1no na 5 vl\u00e1ken, co\u017e se velmi rychle vyu\u017eije, po \u010dem\u017e monitorov\u00e1n\u00ed p\u0159est\u00e1v\u00e1 spr\u00e1vn\u011b fungovat.<\/p>\n<p><\/p>\n<p>Co bych cht\u011bl \u0159\u00edci na z\u00e1v\u011br: poda\u0159ilo se n\u00e1m to! Dok\u00e1zali jsme na\u0161im program\u00e1tor\u016fm a v\u00fdvoj\u00e1\u0159\u016fm poskytnout n\u00e1stroj, kter\u00fd v t\u00e9m\u011b\u0159 ka\u017ed\u00e9 situaci dok\u00e1\u017ee rychle a spolehliv\u011b poskytnout informace o tom, co se d\u011bje na produkci.<\/p>\n<p><\/p>\n<p>Ano, bylo to dost slo\u017eit\u00e9, ale p\u0159esto se n\u00e1m poda\u0159ilo na\u0161e po\u017eadavky za\u010dlenit do ji\u017e existuj\u00edc\u00edch produkt\u016f, kter\u00e9 jsme nemuseli patchovat ani p\u0159episovat pro sebe.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch cluster o velikosti 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/1e852123ca5ec5eaec27bdc66b6c4e83.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/494260\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435. \u041d\u043e \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442, \u043a\u043e\u0433\u0434\u0430 \u0445\u043e\u0447\u0435\u0448\u044c \u0441 \u0435\u0433\u043e \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438 \u00ab\u0432 \u043e\u0441\u043e\u0431\u043e \u043a\u0440\u0443\u043f\u043d\u043e\u043c \u043e\u0431\u044a\u0451\u043c\u0435\u00bb? \u0414\u0430 \u0435\u0449\u0451 \u0438 \u0431\u0435\u0437\u0431\u043e\u043b\u0435\u0437\u043d\u0435\u043d\u043d\u043e \u043f\u0435\u0440\u0435\u0436\u0438\u0432\u0430\u0442\u044c \u043e\u0442\u043a\u0430\u0437 \u043b\u044e\u0431\u043e\u0433\u043e \u0438\u0437 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u0434\u0430\u0442\u0430-\u0446\u0435\u043d\u0442\u0440\u043e\u0432? \u041a\u0430\u043a\u043e\u0439 \u0441\u0442\u043e\u0438\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0443, \u0438 \u043d\u0430 \u043a\u0430\u043a\u0438\u0435 \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0435 \u043a\u0430\u043c\u043d\u0438 \u043d\u0430\u0442\u043a\u043d\u0451\u0448\u044c\u0441\u044f? \u041c\u044b \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 \u0440\u0435\u0448\u0438\u043b\u0438 \u043f\u0440\u0438 \u043f\u043e\u043c\u043e\u0449\u0438 elasticsearch \u0440\u0435\u0448\u0438\u0442\u044c \u0432\u043e\u043f\u0440\u043e\u0441 \u043b\u043e\u0433-\u043c\u0435\u043d\u0435\u0434\u0436\u043c\u0435\u043d\u0442\u0430, \u0430 \u0442\u0435\u043f\u0435\u0440\u044c \u0434\u0435\u043b\u0438\u043c\u0441\u044f \u0441 \u0425\u0430\u0431\u0440\u043e\u043c \u043e\u043f\u044b\u0442\u043e\u043c: \u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":75797,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-75796","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Elasticsearch cluster na 200 TB+ | ProHoster","description":"S Elasticsearch se setk\u00e1v\u00e1 mnoho.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster","og:description":"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-28T17:42:18+00:00","article:modified_time":"2020-03-28T17:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"75796","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:49:32","updated":"2022-09-27 21:24:26","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/75796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=75796"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/75796\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/75797"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=75796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=75796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=75796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}