{"id":75796,"date":"2020-03-28T19:42:18","date_gmt":"2020-03-28T17:42:18","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb"},"modified":"2020-03-28T19:42:18","modified_gmt":"2020-03-28T17:42:18","slug":"klaster-elasticsearch-na-200-tb","status":"publish","type":"post","link":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","title":{"rendered":"Elasticsearch klasteris uz 200 TB+","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ca7a31eca0b3d4faf648dfb24ffbe215.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Daudzi saskaras ar Elasticsearch. Bet kas notiek, kad v\u0113lies to izmantot, lai glab\u0101tu \u017eurn\u0101lus \"\u012bpa\u0161i lielos apjomos\"? Un k\u0101 nes\u0101p\u012bgi iztur\u0113t jebkura datu centra atteik\u0161anu? K\u0101da arhitekt\u016bra b\u016btu j\u0101veido un k\u0101di sl\u0113pti riski tevi sagaida?<\/p>\n<p><\/p>\n<p>M\u0113s, Odnoklassniki, nol\u0113m\u0101m, izmantojot Elasticsearch, atrisin\u0101t \u017eurn\u0101lu p\u0101rvald\u012bbas jaut\u0101jumu un tagad dal\u0101mies ar savu pieredzi Habr\u0101: gan par arhitekt\u016bru, gan par sl\u0113ptiem riskiem.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>Es esmu P\u0113teris Zaitsevs, str\u0101d\u0101ju par sist\u0113mu administratoru Odnoklassnikos. Pirms tam ar\u012b biju administrators, str\u0101d\u0101ju ar Manticore Search, Sphinx mekl\u0113\u0161anu, Elasticsearch. Iesp\u0113jams, ja par\u0101d\u012bsies k\u0101da cita &#8230;search, dro\u0161i vien str\u0101d\u0101\u0161u ar\u012b ar to. T\u0101pat piedal\u012bjos vair\u0101kos atv\u0113rt\u0101 koda projektos, br\u012bvpr\u0101t\u012bgi.<\/p>\n<p><\/p>\n<p>Kad ierados Odnoklassniki, es nedom\u0101jot teicu intervij\u0101, ka protu str\u0101d\u0101t ar Elasticsearch. P\u0113c tam, kad iepazinos un nedaudz pabeidzu da\u017eus vienk\u0101r\u0161us uzdevumus, man tika uztic\u0113ts liels uzdevums saist\u012bb\u0101 ar eso\u0161\u0101s \u017eurn\u0101lu p\u0101rvald\u012bbas sist\u0113mas reform\u0113\u0161anu. <\/p>\n<p><\/p>\n<h2 id=\"trebovaniya\">Pras\u012bbas<\/h2>\n<p><\/p>\n<p>Sist\u0113mas pras\u012bbas tika formul\u0113tas \u0161\u0101di:<\/p>\n<p><\/p>\n<ul>\n<li>K\u0101 front-end risin\u0101jumu bija j\u0101izmanto Graylog. Jo uz\u0146\u0113mum\u0101 jau bija pieredze ar \u0161o produktu, programm\u0113t\u0101ji un test\u0113t\u0101ji to pazina, un tas vi\u0146iem bija \u0113rti un saprotami.<\/li>\n<li>Datu apjoms: vid\u0113ji 50-80 t\u016bksto\u0161i zi\u0146u sekund\u0113, bet, ja kaut kas saboj\u0101jas, tad trafiks nav ierobe\u017eots, tas var b\u016bt 2-3 miljoni rindi\u0146u sekund\u0113.<\/li>\n<li>P\u0101rrun\u0101jot ar pas\u016bt\u012bt\u0101jiem pras\u012bbas attiec\u012bb\u0101 uz mekl\u0113\u0161anas piepras\u012bjumu apstr\u0101des \u0101trumu, m\u0113s saprat\u0101m, ka tipiska \u0161\u0101das sist\u0113mas izmanto\u0161anas sh\u0113ma ir t\u0101da: cilv\u0113ki mekl\u0113 savas lietojumprogrammu \u017eurn\u0101lus p\u0113d\u0113j\u0101m div\u0101m dien\u0101m un nev\u0113las gaid\u012bt atbildi uz izteikto piepras\u012bjumu ilg\u0101k par vienu sekundi. <\/li>\n<li>Administratori uzst\u0101ja, lai sist\u0113ma, ja nepiecie\u0161ams, viegli papla\u0161in\u0101tos, neprasot no vi\u0146iem dzi\u013cus izpratnes l\u012bme\u0146us par t\u0101s darb\u012bbu. <\/li>\n<li>Lai vien\u012bg\u0101 apkalpo\u0161anas uzdevums, kas \u0161\u012bm sist\u0113m\u0101m periodiski bija nepiecie\u0161ams, bija main\u012bt k\u0101du aparat\u016bru.<\/li>\n<li>Turkl\u0101t Odnoklassniki ir lieliska tehnisk\u0101 trad\u012bcija: jebkuram pakalpojumam, ko m\u0113s uzs\u0101kam, j\u0101sp\u0113j iztur\u0113t datu centra atteik\u0161anu (neparedz\u0113ta, nepl\u0101nota un absol\u016bti jebkur\u0101 laik\u0101).<\/li>\n<\/ul>\n<p><\/p>\n<p>P\u0113d\u0113jais pras\u012bba \u0161\u012b projekta \u012bsteno\u0161an\u0101 mums izr\u0101d\u012bj\u0101s visgr\u016bt\u0101k\u0101, par ko es v\u0113l run\u0101\u0161u s\u012bk\u0101k.<\/p>\n<p><\/p>\n<h2 id=\"sreda\">Vide<\/h2>\n<p><\/p>\n<p>M\u0113s str\u0101d\u0101jam \u010detros datu centros, un Elasticsearch datu mezgli var atrasties tikai trijos (vair\u0101ku netehnisku iemeslu d\u0113\u013c).<\/p>\n<p><\/p>\n<p>\u0160ajos \u010detros datu centros atrodas aptuveni 18 t\u016bksto\u0161i da\u017e\u0101du \u017eurn\u0101lu avotu \u2014 aparat\u016bra, konteineri, virtu\u0101las ma\u0161\u012bnas.<\/p>\n<p><\/p>\n<p>Svar\u012bga iez\u012bme: klastera palai\u0161ana notiek konteineros <noindex><a rel=\"nofollow\" href=\"https:\/\/podman.io\">Podman<\/a><\/noindex> nevis uz fizisk\u0101m ma\u0161\u012bn\u0101m, bet uz <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/346868\/\">pa\u0161a m\u0101ko\u0146u produkta one-cloud<\/a><\/noindex>. Konteineriem tiek garant\u0113ti 2 kodoli, kas atbilst 2.0Ghz v4 ar iesp\u0113ju izmantot p\u0101r\u0113jos kodolus, ja tie ir d\u012bkst\u0101v\u0113. <\/p>\n<p><\/p>\n<p>Citiem v\u0101rdiem sakot:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/073bffbfc3cff761bfabe0773b7f4ebc.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"topologiya\">Topolo\u0123ija<\/h2>\n<p><\/p>\n<p>Visp\u0101r\u0113jais risin\u0101juma izskats man s\u0101kotn\u0113ji \u0161\u0137ita \u0161\u0101ds:<\/p>\n<p><\/p>\n<ul>\n<li>3-4 VIP atrodas aiz Graylog dom\u0113na A-rekorda, tas ir adrese, uz kuru tiek nos\u016bt\u012bti \u017eurn\u0101li.<\/li>\n<li>katrs VIP ir LVS l\u012bdzsvarot\u0101js.<\/li>\n<li>P\u0113c tam \u017eurn\u0101li non\u0101k Graylog baterij\u0101, da\u013ca datu tiek nos\u016bt\u012bta GELF form\u0101t\u0101, da\u013ca - syslog form\u0101t\u0101.<\/li>\n<li>T\u0101l\u0101k viss tiek lielos partiju rakst\u012bts koordin\u0101toru Elasticsearch baterij\u0101. <\/li>\n<li>Un tie, savuk\u0101rt, nos\u016bta ieraksta un las\u012b\u0161anas piepras\u012bjumus attiec\u012bgajiem datu mezgliem. <\/li>\n<\/ul>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/21efcdf6992a07e0c5e9b477c567cca9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<h2 id=\"terminologiya\">Terminolo\u0123ija<\/h2>\n<p><\/p>\n<p>Iesp\u0113jams, ka ne visi detaliz\u0113ti p\u0101rzina terminolo\u0123iju, t\u0101p\u0113c v\u0113l\u0113tos nedaudz apst\u0101ties pie t\u0101s.<\/p>\n<p><\/p>\n<p>Elasticsearch ir vair\u0101ki mezglu veidi \u2014 master, coordinator, data node. Ir ar\u012b divi citi veidi da\u017e\u0101diem \u017eurn\u0101lu p\u0101rv\u0113r\u0161anai un da\u017e\u0101du klasteru savieno\u0161anai, bet m\u0113s izmantoj\u0101m tikai uzskait\u012btos. <\/p>\n<p><\/p>\n<p><strong>Master<\/strong><br \/>\nPing visus klaster\u012b eso\u0161os mezglus, uztur aktu\u0101lu klastera karti un izplata to starp mezgliem, apstr\u0101d\u0101 notikumu lo\u0123iku, nodarbojas ar da\u017e\u0101da veida klastera m\u0101jasdarbiem. <\/p>\n<p><\/p>\n<p><strong>Coordinator<\/strong><br \/>\nVeic vienu vien\u012bgu uzdevumu: pie\u0146em piepras\u012bjumus no klientiem las\u012b\u0161anai vai rakst\u012b\u0161anai un mar\u0161rut\u0113 \u0161o trafiku. Ja piepras\u012bjums ir par rakst\u012b\u0161anu, visticam\u0101k, ka tas vaic\u0101s master, kur\u0101 shard attiec\u012bg\u0101 indeksa to novietot, un p\u0101radres\u0113s piepras\u012bjumu t\u0101l\u0101k. <\/p>\n<p><\/p>\n<p><strong>Data node<\/strong><br \/>\nGlab\u0101 datus, veic no \u0101rienes ien\u0101ko\u0161os mekl\u0113\u0161anas piepras\u012bjumus un oper\u0101cijas ar taj\u0101 novietotajiem shardiem.<\/p>\n<p><\/p>\n<p><strong>Graylog<\/strong><br \/>\nTas ir kaut kas l\u012bdz\u012bgs Kibana un Logstash sapl\u016b\u0161anai ELK kaudz\u0113. Graylog apvieno gan lietot\u0101ja saskarni, gan \u017eurn\u0101lu apstr\u0101des kan\u0101lu. Zem Graylog ir Kafka un Zookeeper, kas nodro\u0161ina Graylog sasaisti k\u0101 klusteru. Graylog var ke\u0161atmi\u0146\u0101 saglab\u0101t \u017eurn\u0101lus (Kafka), ja Elasticsearch nav pieejams, un atk\u0101rtot neveiksm\u012bgos las\u012b\u0161anas un rakst\u012b\u0161anas piepras\u012bjumus, grup\u0113t un mar\u0137\u0113t \u017eurn\u0101lus saska\u0146\u0101 ar noteiktiem noteikumiem. T\u0101pat k\u0101 Logstash, Graylog ir funkcionalit\u0101te, lai modific\u0113tu virknes pirms to ierakst\u012b\u0161anas Elasticsearch.<\/p>\n<p><\/p>\n<p>Turkl\u0101t Graylog ir ieb\u016bv\u0113ts pakalpojumu atkl\u0101\u0161anas r\u012bks, kas \u013cauj, pamatojoties uz vienu pieejamu Elasticsearch mezglu, ieg\u016bt visu klusteru karti un filtr\u0113t to p\u0113c noteikta taga, kas dod iesp\u0113ju novirz\u012bt piepras\u012bjumus uz konkr\u0113tiem konteineriem.<\/p>\n<p><\/p>\n<p>Vizualiz\u0101cija izskat\u0101s aptuveni t\u0101:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/8673ba9c0300ea0153d34a9f98afbcf2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u0160\u012b ir ekr\u0101n\u0161\u0101vi\u0146\u0161 no konkr\u0113ta instanc\u0113. \u0160eit m\u0113s veidojam histogrammu p\u0113c mekl\u0113\u0161anas piepras\u012bjuma, par\u0101dot attiec\u012bg\u0101s virknes.<\/p>\n<p><\/p>\n<h2 id=\"indeksy\">Indeksi<\/h2>\n<p><\/p>\n<p>Atgrie\u017eoties pie sist\u0113mas arhitekt\u016bras, es v\u0113l\u0113tos detaliz\u0113t\u0101k apskat\u012bt, k\u0101 m\u0113s veidoj\u0101m indeksu modeli, lai viss darbotos pareizi. <\/p>\n<p><\/p>\n<p>Iepriek\u0161 min\u0113taj\u0101 sh\u0113m\u0101 tas ir viszem\u0101kais l\u012bmenis: Elasticsearch datu mezgli.<\/p>\n<p><\/p>\n<p>Indeks ir liela virtu\u0101la b\u016btne, kas sast\u0101v no Elasticsearch shardiem. Katrs shard pats par sevi ir nekas cits k\u0101 Lucene indekss. Un katrs Lucene indekss, savuk\u0101rt, sast\u0101v no viena vai vair\u0101kiem segmentiem. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/13b858cb102dce26b0851516b2d36c43.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Projekt\u0113\u0161anas proces\u0101 m\u0113s l\u0113sa, ka, lai nodro\u0161in\u0101tu pras\u012bbu p\u0113c \u0101tras las\u012b\u0161anas lielos datu apjomos, mums ir nepiecie\u0161ams vienm\u0113r\u012bgi \"izplais\u012bt\" \u0161os datus pa datu mezgliem. <\/p>\n<p><\/p>\n<p>Tas noveda pie t\u0101, ka shardu skaitam uz indeksu (kop\u0101 ar replik\u0101m) mums j\u0101b\u016bt stingri vien\u0101dam ar datu mezglu skaitu. Pirmk\u0101rt, lai nodro\u0161in\u0101tu replik\u0101cijas faktoru divi (proti, m\u0113s varam zaud\u0113t pusi klusteru). Otrk\u0101rt, lai las\u012b\u0161anas un rakst\u012b\u0161anas piepras\u012bjumus apstr\u0101d\u0101tu vismaz uz pusi klusteru.<\/p>\n<p><\/p>\n<p>M\u0113s noteic\u0101m glab\u0101\u0161anas laiku vispirms k\u0101 30 dienas.<\/p>\n<p><\/p>\n<p>Shardu sadal\u012bjumu var att\u0113lot grafiski \u0161\u0101di:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/242726053360d1a6bdf4e527edc6cae6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Visi tum\u0161i pel\u0113kie taisnst\u016bri kopum\u0101 veido indeksu. Kreisais sarkanais kvadr\u0101ts taj\u0101 ir prim\u0101rais shard, pirmais indeks\u0101. Savuk\u0101rt gai\u0161i zils kvadr\u0101ts ir replikas shard. Tie atrodas da\u017e\u0101d\u0101s datu centr\u0101.<\/p>\n<p><\/p>\n<p>Kad pievienojam v\u0113l vienu shard, tas non\u0101k tre\u0161aj\u0101 datu centr\u0101. Un, galu gal\u0101, m\u0113s ieg\u016bstam \u0161\u0101du strukt\u016bru, kas nodro\u0161ina datu konsekvences saglab\u0101\u0161anu, pat zaud\u0113jot datu centru:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/27cbe4d4606f5ae5f3013ea46504134b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Indeksu rot\u0101cija, t.i., jaun\u0101 indeksa izveide un vec\u0101k\u0101 dz\u0113\u0161ana, ir noteikta uz 48 stund\u0101m (saska\u0146\u0101 ar indeksu izmanto\u0161anas raksturu: p\u0113d\u0113j\u0101s 48 stundas tiek mekl\u0113tas visbie\u017e\u0101k).<\/p>\n<p><\/p>\n<p>\u0160\u0101ds indeksu rot\u0101cijas interv\u0101ls ir saist\u012bts ar \u0161\u0101diem iemesliem:<\/p>\n<p><\/p>\n<p>Kad konkr\u0113t\u0101 datu noda sa\u0146em mekl\u0113\u0161anas vaic\u0101jumu, no veiktsp\u0113jas viedok\u013ca ir izdev\u012bg\u0101k, ja tiek aptauj\u0101ts viens shard, ja t\u0101 lielums ir sal\u012bdzin\u0101ms ar noda heap lielumu. Tas \u013cauj saglab\u0101t 'karsto' indeksa da\u013cu heap'\u0101 un \u0101tri pie t\u0101s piek\u013c\u016bt. Kad 'karsto da\u013cu' ir daudz, indeksa mekl\u0113\u0161anas \u0101trums samazin\u0101s.<\/p>\n<p><\/p>\n<p>Kad noda s\u0101k veikt mekl\u0113\u0161anas vaic\u0101jumu vien\u0101 shard, t\u0101 pie\u0161\u0137ir pavedienu (thread) skaitu, kas ir vien\u0101ds ar fizisk\u0101s ma\u0161\u012bnas hipertirdzniec\u012bbas kodolu skaitu. Ja mekl\u0113\u0161anas vaic\u0101jums attiecas uz lielu skaitu shard, tad pavedienu skaits pieaug proporcion\u0101li. Tas nelabv\u0113l\u012bgi ietekm\u0113 mekl\u0113\u0161anas \u0101trumu un negat\u012bvi ietekm\u0113 jauno datu indeks\u0101ciju. <\/p>\n<p><\/p>\n<p>Lai nodro\u0161in\u0101tu nepiecie\u0161amo mekl\u0113jumu latentumu, m\u0113s nol\u0113m\u0101m izmantot SSD. Lai \u0101tri apstr\u0101d\u0101tu piepras\u012bjumus, ma\u0161\u012bn\u0101m, uz kur\u0101m tika izvietoti \u0161ie konteineri, vajadz\u0113ja b\u016bt vismaz 56 kodoliem. Skaitlis 56 ir izv\u0113l\u0113ts k\u0101 nosac\u012bti pietiekams lielums, kas nosaka t\u0113mu skaitu, ko gener\u0113s Elasticsearch darba procesa laik\u0101. Elasticsearch daudzi parametri thread pool ir tie\u0161i atkar\u012bgi no pieejamo kodolu skaita, kas, savuk\u0101rt, tie\u0161i ietekm\u0113 nepiecie\u0161amo mezglu skaitu klaster\u012b saska\u0146\u0101 ar principu \"maz\u0101k kodolu \u2014 vair\u0101k mezglu.\" <\/p>\n<p><\/p>\n<p>Rezult\u0101t\u0101 m\u0113s ieguv\u0101m, ka vid\u0113ji shard sver apm\u0113ram 20 gigabaitu, un uz 1 indeksu attiecas 360 shard. T\u0101d\u0113j\u0101di, ja m\u0113s tos rot\u0113jam reizi 48 stund\u0101s, tad mums ir 15 no tiem. Katrs indekss satur datus par 2 dien\u0101m.<\/p>\n<p><\/p>\n<h2 id=\"shemy-zapisi-i-chteniya-dannyh\">Datu ieraksta un las\u012b\u0161anas sh\u0113mas<\/h2>\n<p><\/p>\n<p>Paskat\u012bsimies, k\u0101 \u0161aj\u0101 sist\u0113m\u0101 tiek ierakst\u012bti dati.<\/p>\n<p><\/p>\n<p>Pie\u0146emsim, ka no Graylog sa\u0146emam k\u0101du vaic\u0101jumu uz koordinatoru. Piem\u0113ram, m\u0113s v\u0113lamies indeks\u0113t 2-3 t\u016bksto\u0161us rindu. <\/p>\n<p><\/p>\n<p>Koordinators, sa\u0146emot piepras\u012bjumu no Graylog, uzdot meistram: \u00abM\u016bsu indeks\u0101cijas piepras\u012bjum\u0101 konkr\u0113ti tika nor\u0101d\u012bts indekss, bet kur\u0101 shard to rakst\u012bt \u2014 nav nor\u0101d\u012bts\u00bb. <\/p>\n<p><\/p>\n<p>Meistars atbild: \u00abPieraksti \u0161o inform\u0101ciju shard numur 71\u00bb, p\u0113c tam t\u0101 tiek nos\u016bt\u012bta uz attiec\u012bgo datu mezglu, kur atrodas prim\u0101rais shard numur 71.<\/p>\n<p><\/p>\n<p>P\u0113c tam dar\u012bjumu \u017eurn\u0101ls tiek replik\u0113ts uz replikas shard, kas atrodas jau cit\u0101 datu centr\u0101.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/fdb6a77cb78b1eed290e440478569e3b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>No Graylog uz koordin\u0101toru pien\u0101k mekl\u0113\u0161anas piepras\u012bjums. Koordinators to p\u0101rvirza pa indeksu, un Elasticsearch p\u0113c round-robin principa sadala piepras\u012bjumus starp prim\u0101ro shard un replikas shard. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/cb32bac878e04d6f50d103ebbc395a7f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Mezgli, kuru ir 180, atbild neviendab\u012bgi, un, kam\u0113r tie atbild, koordinators uzkr\u0101j inform\u0101ciju, ko no t\u0101 jau \u00abizsp\u013c\u0101vu\u0161as\u00bb \u0101tr\u0101kas datu mezgls. P\u0113c tam, kad vai nu visa inform\u0101cija ir sa\u0146emta, vai ir sasniegts piepras\u012bjuma laika limits, viss tiek nodots tie\u0161i klientam. <\/p>\n<p><\/p>\n<p>Visa \u0161\u012b sist\u0113ma vid\u0113ji apstr\u0101d\u0101 mekl\u0113\u0161anas piepras\u012bjumus par p\u0113d\u0113j\u0101m 48 stund\u0101m 300-400ms laik\u0101, iz\u0146emot tos piepras\u012bjumus, kuriem ir leading wildcard.<\/p>\n<p><\/p>\n<h2 id=\"cvetochki-s-elasticsearch-nastroyka-java\">\u00abZiedi\u00bb ar Elasticsearch: Java konfigur\u0101cija<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/fc9ac4b8ee41bc35f5f01b8d9c4ff2f6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Lai viss darbotu t\u0101, k\u0101 m\u0113s s\u0101kotn\u0113ji v\u0113l\u0113j\u0101mies, m\u0113s \u013coti ilgi izstr\u0101d\u0101j\u0101m da\u017e\u0101das lietas klaster\u012b. <\/p>\n<p><\/p>\n<p>Pirmaj\u0101 da\u013c\u0101 atrasto probl\u0113mu c\u0113lonis saist\u012bts ar to, k\u0101 Elasticsearch p\u0113c noklus\u0113juma ir iepriek\u0161 konfigur\u0113ta Java. <\/p>\n<p><\/p>\n<p><strong>Probl\u0113ma pirm\u0101<\/strong><br \/>\nM\u0113s nov\u0113roj\u0101m \u013coti daudz zi\u0146ojumu par to, ka m\u016bsu Lucene l\u012bmen\u012b, kad ir palaisti background job\u2019i, Lucene segmentu saska\u0146o\u0161anas procesi beidzas ar k\u013c\u016bdu. Taj\u0101 pa\u0161\u0101 laik\u0101 \u017eurn\u0101los bija redzams, ka tas ir OutOfMemoryError k\u013c\u016bda. Saska\u0146\u0101 ar telemetriju m\u0113s redz\u0113j\u0101m, ka hipes ir br\u012bvas, un nebija skaidrs, k\u0101p\u0113c \u0161\u012b oper\u0101cija neizdodas. <\/p>\n<p><\/p>\n<p>Noskaidroj\u0101s, ka Lucene indeksu sapludin\u0101\u0161anas notiek \u0101rpus hipe. Un konteineri ir diezgan stingri ierobe\u017eoti pat\u0113r\u0113to resursu zi\u0146\u0101. \u0160ajos resursos iek\u013c\u0101v\u0101s tikai hipa (v\u0113rt\u012bba heap.size bija aptuveni vien\u0101da ar RAM), bet k\u0101das off-heap oper\u0101cijas kr\u012bt ar atmi\u0146as pie\u0161\u0137ir\u0161anas k\u013c\u016bdu, ja k\u0101du iemeslu d\u0113\u013c neizdev\u0101s iek\u013cauties tajos ~500MB, kas palika l\u012bdz limita.<\/p>\n<p><\/p>\n<p>Risin\u0101jums bija diezgan vienk\u0101r\u0161s: palielin\u0101ja pieejamo RAM apjomu konteineram, p\u0113c tam aizmirs\u0101m, ka t\u0101das probl\u0113mas mums visp\u0101r bija.<\/p>\n<p><\/p>\n<p><strong>Probl\u0113ma otr\u0101<\/strong><br \/>\nP\u0113c apm\u0113ram 4-5 dien\u0101m p\u0113c klastera palai\u0161anas m\u0113s paman\u012bj\u0101m, ka datu mezgli s\u0101k periodiski izkrist no klastera un piek\u013c\u016bst tam p\u0113c 10-20 sekund\u0113m. <\/p>\n<p><\/p>\n<p>Kad nol\u0113m\u0101m analiz\u0113t, izr\u0101d\u012bj\u0101s, ka \u0161\u012b off-heap atmi\u0146a Elasticsearch praktiski nav kontrol\u0113jama. Kad m\u0113s konteineram pie\u0161\u0137\u012br\u0101m vair\u0101k atmi\u0146as, m\u0113s guv\u0101m iesp\u0113ju piepild\u012bt tie\u0161os buferu baseinus ar da\u017e\u0101du inform\u0101ciju, un t\u0101 tika izt\u012br\u012bta tikai p\u0113c tam, kad tika veikts explicit GC no Elasticsearch puses. <\/p>\n<p><\/p>\n<p>Vair\u0101kos gad\u012bjumos \u0161\u012b oper\u0101cija norisin\u0101j\u0101s diezgan ilgi, un \u0161\u0101 laik\u0101 klasters pasp\u0113ja atz\u012bm\u0113t \u0161o mezglu k\u0101 izsl\u0113gtu. \u0160\u012b probl\u0113ma ir labi aprakst\u012bta. <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/elasticsearch-5-6-very-quickly-increasing-direct-buffer-pools\/119561\">te ir<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>Risin\u0101jums bija \u0161\u0101ds: m\u0113s ierobe\u017eoj\u0101m Java iesp\u0113jas izmantot liel\u0101ko da\u013cu atmi\u0146as \u0101rpus hipa \u0161\u012bm oper\u0101cij\u0101m. M\u0113s ierobe\u017eoj\u0101m to l\u012bdz 16 gigabaitiem (-XX:MaxDirectMemorySize=16g), pan\u0101kot to, ka explicit GC tika izsaukti iev\u0113rojami bie\u017e\u0101k un darboj\u0101s iev\u0113rojami \u0101tr\u0101k, t\u0101d\u0113j\u0101di p\u0101rtraucot stabiliz\u0113t klasteri.<\/p>\n<p><\/p>\n<p><strong>Tre\u0161\u0101 probl\u0113ma.<\/strong><br \/>\nJa dom\u0101jat, ka probl\u0113mas ar \u201emezgliem, kuri negaid\u012bti pamet klasteri\u201d \u0161eit beidzas, j\u016bs mald\u0101ties. <\/p>\n<p><\/p>\n<p>Konfigur\u0113jot darbu ar indeksiem, m\u0113s izv\u0113l\u0113j\u0101mies mmapfs, lai <noindex><a rel=\"nofollow\" href=\"https:\/\/discuss.elastic.co\/t\/benchmarks-default-niofs-vs-memory-mapped-mmapfs\/13118\/2\">samazin\u0101tu mekl\u0113\u0161anas laiku<\/a><\/noindex> p\u0101r svaigiem shardiem ar lielu segment\u0101ciju. T\u0101 bija diezgan rupja k\u013c\u016bda, jo, izmantojot mmapfs, fails ir map\u0113ts operat\u012bvaj\u0101 atmi\u0146\u0101, un t\u0101l\u0101k m\u0113s str\u0101d\u0101jam jau ar mapped-failu. T\u0101 rezult\u0101t\u0101, m\u0113\u0123inot GC apst\u0101din\u0101t pavedienus lietojumprogramm\u0101, m\u0113s ilgi non\u0101kam safepoint, un ce\u013c\u0101 uz to lietojumprogramma p\u0101rst\u0101j atbild\u0113t uz meistara piepras\u012bjumiem par to, vai t\u0101 ir dz\u012bv\u0101. Attiec\u012bgi, meistars uzskata, ka mezgls m\u016bsu klaster\u012b vairs nepast\u0101v. P\u0113c tam, apm\u0113ram 5-10 sekun\u017eu laik\u0101, darbojas garbage collector, mezgls atdz\u012bvojas, atkal pievienojas klasterim un uzs\u0101k shardu inicializ\u0101ciju. Viss tas stipri atg\u0101din\u0101ja \u201cprodukciju, kuru m\u0113s izpeln\u012bj\u0101m\u201d un nebija piem\u0113rots nekam nopietnam.<\/p>\n<p><\/p>\n<p>Lai atbr\u012bvotos no \u0161\u0101da uzved\u012bbas mode\u013ca, m\u0113s vispirms p\u0101rg\u0101j\u0101m uz standarta niofs, bet p\u0113c tam, kad ar piektaj\u0101m versij\u0101m Elastic p\u0101rg\u0101j\u0101m uz sestaj\u0101m, izm\u0113\u0123in\u0101j\u0101m hybridfs, kur \u0161\u012b probl\u0113ma neizpaud\u0101s. S\u012bk\u0101k par kr\u0101tuves veidiem var izlas\u012bt. <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/index-modules-store.html\">\u0161eit<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><strong>Ceturt\u0101 probl\u0113ma.<\/strong><br \/>\nP\u0113d\u0113j\u0101 bija v\u0113l \u013coti interesanta probl\u0113ma, kuru m\u0113s risin\u0101j\u0101m rekordl\u012bdz\u012bgi ilgi. M\u0113s to nov\u0113roj\u0101m 2-3 m\u0113ne\u0161us, jo piln\u012bgi neskaidri bija t\u0101s paraugs. <\/p>\n<p><\/p>\n<p>Reiz\u0113m m\u016bsu koordinatori non\u0101ca Full GC, parasti ap pusdienlaiku, un vairs neatgriez\u0101s. Taj\u0101 pa\u0161\u0101 laik\u0101, re\u0123istr\u0113jot GC aizkaves, tas izskat\u012bj\u0101s \u0161\u0101di: viss labi, labi, labi, un tad p\u0113k\u0161\u0146i \u2013 viss strauji pasliktin\u0101j\u0101s. <\/p>\n<p><\/p>\n<p>S\u0101kum\u0101 m\u0113s dom\u0101j\u0101m, ka mums ir \u013cauns lietot\u0101js, kur\u0161 nos\u016bta k\u0101du piepras\u012bjumu, kas izsist koordinatoru no darba re\u017e\u012bma. Ilgi re\u0123istr\u0113j\u0101m piepras\u012bjumus, cen\u0161oties noskaidrot, kas notiek. <\/p>\n<p><\/p>\n<p>Galu gal\u0101 noskaidroj\u0101s, ka taj\u0101 br\u012bd\u012b, kad k\u0101ds lietot\u0101js nos\u016bta \u013coti lielu piepras\u012bjumu, un tas non\u0101k pie konkr\u0113ta Elasticsearch koordinatora, da\u017eas noder atbild ar liel\u0101ku kav\u0113\u0161anos nek\u0101 citas. <\/p>\n<p><\/p>\n<p>Un tas laiks, kam\u0113r koordinatora gaida atbildes no vis\u0101m nodo\u0161aj\u0101m nodym, vi\u0146\u0161 uzkr\u0101j sev rezult\u0101tus, kas nos\u016bt\u012bti no jau atbild\u0113ju\u0161\u0101m nodym. GC \u0161aj\u0101 gad\u012bjum\u0101 noz\u012bm\u0113, ka m\u016bsu hipa izmanto\u0161anas modelis \u0101tri main\u0101s. Un tas GC, ko m\u0113s lietoj\u0101m, ar \u0161o uzdevumu netika gal\u0101. <\/p>\n<p><\/p>\n<p>Vien\u012bgais risin\u0101jums, ko m\u0113s atrad\u0101m, lai main\u012btu klastera uzved\u012bbu \u0161\u0101d\u0101 situ\u0101cij\u0101 \u2013 p\u0101reja uz JDK13 un Shenandoah atkritumu sav\u0101c\u0113ja izmanto\u0161ana. Tas nov\u0113rsa probl\u0113mu, koordinatoriem vairs nebija kritienu. <\/p>\n<p><\/p>\n<p>Ar \u0161o probl\u0113mas ar Java beidz\u0101s un s\u0101k\u0101s probl\u0113mas ar caurlaidsp\u0113ju. <\/p>\n<p><\/p>\n<h2 id=\"yagodki-s-elasticsearch-propusknaya-sposobnost\">\u00abOgas\u00bb ar Elasticsearch: caurlaidsp\u0113ja<\/h2>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/bb5c08193d7d764515235e825cd30ce5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Probl\u0113mas ar caurlaidsp\u0113ju noz\u012bm\u0113, ka m\u016bsu klasters str\u0101d\u0101 stabil\u0101 re\u017e\u012bm\u0101, bet indeks\u0113jamo dokumentu skaita pieaugumu un manevru br\u012b\u017eos veiktsp\u0113ja ir nepietiekama.<\/p>\n<p><\/p>\n<p>Pirmais nov\u0113rojamais simptoms: pie k\u0101d\u0101m \u00abspr\u0101dzieniem\u00bb produkt\u012bvaj\u0101 vid\u0113, kad strauji \u0123ener\u0113jas \u013coti liels \u017eurn\u0101lu daudzums, Graylog\u0101 bie\u017ei par\u0101d\u0101s k\u013c\u016bda indeks\u0101cij\u0101 es_rejected_execution. <\/p>\n<p><\/p>\n<p>Tas notika t\u0101p\u0113c, ka thread_pool.write.queue vien\u0101 datu nod\u0113 pirms tam, kad Elasticsearch sp\u0113j apstr\u0101d\u0101t piepras\u012bjumu par indeks\u0113\u0161anu un nodot inform\u0101ciju shard disk\u0101, p\u0113c noklus\u0113juma var ke\u0161ot tikai 200 piepras\u012bjumus. Un <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">Elasticsearch dokument\u0101cij\u0101<\/a><\/noindex> par \u0161o parametru tiek teikts \u013coti maz. Tiek nor\u0101d\u012bts tikai maksim\u0101lais pavedienu skaits un noklus\u0113juma izm\u0113rs.<\/p>\n<p><\/p>\n<p>Protams, m\u0113s dev\u0101mies regul\u0113t \u0161o v\u0113rt\u012bbu un noskaidroj\u0101m sekojo\u0161o: konkr\u0113ti m\u016bsu uzst\u0101d\u012bjum\u0101 l\u012bdz 300 piepras\u012bjumiem ke\u0161ojas diezgan labi, bet liel\u0101ka v\u0113rt\u012bba ir b\u012bstama, jo m\u0113s atkal non\u0101kam Full GC.<\/p>\n<p><\/p>\n<p>Turkl\u0101t, \u0146emot v\u0113r\u0101, ka t\u0101s ir zi\u0146u pakas, kas tiek nos\u016bt\u012btas viena piepras\u012bjuma ietvaros, bija j\u0101uzlabo ar\u012b Graylog, lai tas nerakst\u012btu bie\u017ei un maz\u0101s partij\u0101s, bet gan liel\u0101s partij\u0101s vai ik p\u0113c 3 sekund\u0113m, ja partija v\u0113l nav pilna. \u0160\u0101d\u0101 gad\u012bjum\u0101 ieg\u016bstam, ka inform\u0101cija, ko rakst\u0101m Elasticsearch, k\u013c\u016bst pieejama ne divu, bet piecu sekun\u017eu laik\u0101 (kas m\u016bs piln\u012bb\u0101 apmierina), ta\u010du samazin\u0101s atk\u0101rtojumu skaits, kas j\u0101veic, lai nospiestu lielu inform\u0101cijas partiju.<\/p>\n<p><\/p>\n<p>Tas \u012bpa\u0161i ir svar\u012bgi br\u012b\u017eos, kad kaut kas ir izg\u0101zies un par to intens\u012bvi zi\u0146o, lai neieg\u016btu piln\u012bb\u0101 p\u0101rspamoju\u0161u Elastic, un p\u0113c k\u0101da laika \u2013 nedarb\u012bgus Graylog mezglus, jo ir aizsprostoti buferi.<\/p>\n<p><\/p>\n<p>Turkl\u0101t, kad notika \u0161ie pa\u0161i spr\u0101dzieni ra\u017eo\u0161an\u0101, m\u0113s sa\u0146\u0113m\u0101m s\u016bdz\u012bbas no programm\u0113t\u0101jiem un testeriem: taj\u0101 br\u012bd\u012b, kad vi\u0146iem \u013coti nepiecie\u0161ami \u0161ie \u017eurn\u0101li, tie tiek pieg\u0101d\u0101ti \u013coti l\u0113ni.<\/p>\n<p><\/p>\n<p>S\u0101ka analiz\u0113t. No vienas puses, bija skaidrs, ka gan mekl\u0113\u0161anas piepras\u012bjumi, gan indeks\u0113\u0161anas piepras\u012bjumi b\u016bt\u012bb\u0101 tiek apstr\u0101d\u0101ti uz vien\u0101m un t\u0101m pa\u0161\u0101m fiziskaj\u0101m ma\u0161\u012bn\u0101m, un t\u0101 vai cit\u0101di b\u016bs noteiktas novirzes. <\/p>\n<p><\/p>\n<p>Bet \u0161o da\u013c\u0113ji var\u0113ja apiet, pateicoties tam, ka sestaj\u0101s Elasticsearch versij\u0101s par\u0101d\u012bj\u0101s algoritms, kas \u013cauj sadal\u012bt piepras\u012bjumus starp attiec\u012bgajiem datu mezgliem nevis nejau\u0161\u0101 k\u0101rt\u012bb\u0101 round-robin (konteiners, kas nodarbojas ar indeks\u0113\u0161anu un tur prim\u0101ro fragmentu, var b\u016bt \u013coti noslogots, tur neb\u016bs iesp\u0113jas \u0101tri atbild\u0113t), bet nos\u016bt\u012bt \u0161o piepras\u012bjumu uz maz\u0101k noslogoto konteineru ar replikas fragmentu, kas atbild\u0113s iev\u0113rojami \u0101tr\u0101k. Citiem v\u0101rdiem sakot, m\u0113s p\u0101rg\u0101j\u0101m uz use_adaptive_replica_selection: true.<\/p>\n<p><\/p>\n<p>Las\u012b\u0161anas aina s\u0101k izskat\u012bties \u0161\u0101di:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ae7c1645bb63692e580b1bca5c6e5db0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u0101reja uz \u0161o algoritmu \u013c\u0101va iev\u0113rojami uzlabot piepras\u012bjuma laiku br\u012b\u017eos, kad mums notika liels \u017eurn\u0101lu rakst\u012b\u0161anas pl\u016bsma.<\/p>\n<p><\/p>\n<p>Visbeidzot, galven\u0101 probl\u0113ma bija s\u0101p\u012bga datu centra izsl\u0113g\u0161ana.<\/p>\n<p><\/p>\n<p>Ko m\u0113s v\u0113l\u0113j\u0101mies no klastera uzreiz p\u0113c savienojuma zaud\u0113\u0161anas ar vienu datu centru: <\/p>\n<p><\/p>\n<ul>\n<li>Ja mums atsl\u0113gtaj\u0101 datu centr\u0101 ir pa\u0161reiz\u0113jais meistars, tad tas tiks atk\u0101rtoti izv\u0113l\u0113ts un p\u0101rcelsies k\u0101 loma uz citu mezglu cit\u0101 datu centr\u0101.<\/li>\n<li>Meistars \u0101tri izmet\u012bs no klastera visus nepieejamos mezglus.<\/li>\n<li>Pamatojoties uz atliku\u0161aj\u0101m, vi\u0146\u0161 saprat\u012bs: zaud\u0113taj\u0101 datu centr\u0101 mums bija t\u0101di prim\u0101ri shardi, \u0101tri promoc\u0113sim papildino\u0161os replikas shardus atliku\u0161ajos datu centros, un mums turpin\u0101sies datu indeks\u0101cija. <\/li>\n<li>\u0160\u012b rezult\u0101t\u0101 m\u016bsu klastera caurlaidsp\u0113ja rakst\u012b\u0161anai un las\u012b\u0161anai pak\u0101peniski pasliktin\u0101sies, tom\u0113r kopum\u0101 viss darbosies, lai ar\u012b l\u0113n\u0101k, bet stabil\u0101k.<\/li>\n<\/ul>\n<p><\/p>\n<p>K\u0101 izr\u0101d\u012bj\u0101s, m\u0113s v\u0113l\u0113j\u0101mies kaut ko l\u012bdz\u012bgu tam:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/71fa3d19a61eb58ddb93a4df55c759c5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Bet sa\u0146\u0113m\u0101m sekojo\u0161o:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/ab32be4dfbbb4bcc3279d57021707980.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>K\u0101 t\u0101 notika? <\/p>\n<p><\/p>\n<p>Datu centra krituma br\u012bd\u012b m\u016bsu \u0161aurs posms bija meistars.<\/p>\n<p><\/p>\n<p>K\u0101p\u0113c?<\/p>\n<p><\/p>\n<p>Probl\u0113ma ir t\u0101, ka meistaram ir TaskBatcher, kas atbild par noteiktu uzdevumu un notikumu izplat\u012b\u0161anu klaster\u012b. Ikviens mezgla izie\u0161anas, jebkura sharda p\u0101reja no replikas uz prim\u0101ro, ikviens uzdevums k\u0101da sharda izveidei \u2014 viss tas vispirms non\u0101k TaskBatcher, kur tas tiek apstr\u0101d\u0101ts sec\u012bgi un vien\u0101 pl\u016bsm\u0101.<\/p>\n<p><\/p>\n<p>Datu centra zai\u0161anas br\u012bd\u012b izr\u0101d\u012bj\u0101s, ka visi datu mezgli izdz\u012bvoju\u0161ajos datu centros uzskat\u012bja par savu pien\u0101kumu inform\u0113t meistaru, ka \u201emums ir pazudu\u0161i t\u0101di shardi un t\u0101di datu mezgli\u201d. <\/p>\n<p><\/p>\n<p>Taj\u0101 pat laik\u0101 izdz\u012bvoju\u0161ie datu mezgli s\u016bt\u012bja visu \u0161o inform\u0101ciju pa\u0161reiz\u0113jam meistaram un cent\u0101s gaid\u012bt apstiprin\u0101jumu, ka vi\u0146\u0161 to ir sa\u0146\u0113mis. Vi\u0146i to nesa\u0146\u0113ma, jo meistars sa\u0146\u0113ma uzdevumus \u0101tr\u0101k, nek\u0101 sp\u0113ja atbild\u0113t. Mezgls p\u0113c izsnieg\u0161anas atk\u0101rtoja piepras\u012bjumus, bet meistars taj\u0101 laik\u0101 jau pat necent\u0101s atbild\u0113t uz tiem, bet piln\u012bb\u0101 bija absorb\u0113ts uzdevuma \u0161\u0137iro\u0161anai p\u0113c priorit\u0101tes.<\/p>\n<p><\/p>\n<p>Termi\u0146a zi\u0146\u0101 izr\u0101d\u012bj\u0101s, ka datu mezgli s\u016bt\u012bja meistaram tik daudz piepras\u012bjumu, ka vi\u0146\u0161 non\u0101ca piln\u012bg\u0101 GC. P\u0113c tam m\u016bsu meistara loma p\u0101rg\u0101ja uz k\u0101du n\u0101kamo mezglu, un ar to notika absol\u016bti tas pats, un galu gal\u0101 klasteris piln\u012bb\u0101 sabruka. <\/p>\n<p><\/p>\n<p>M\u0113s veic\u0101m m\u0113r\u012bjumus, un l\u012bdz versijai 6.4.0, kur \u0161\u012b probl\u0113ma tika nov\u0113rsta, mums bija pietiekami izsl\u0113gt vienlaikus tikai 10 datu mezglus no 360, lai piln\u012bb\u0101 nogalin\u0101tu klasteri.<\/p>\n<p><\/p>\n<p>Tas izskat\u012bj\u0101s apm\u0113ram \u0161\u0101di:<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/3bed554b2703e7521339e9d943648ee4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u0113c versijas 6.4.0, kur tika nov\u0113rsta \u0161\u012b bied\u0113jo\u0161\u0101 k\u013c\u016bda, datu mezgli p\u0101rst\u0101ja nogalin\u0101t meistaru. Bet no t\u0101 vi\u0146\u0161 k\u013cuva \u201egudr\u0101ks\u201d. Proti, kad m\u0113s izsl\u0113dzam 2, 3 vai 10 (jebkuru skaitu, iz\u0146emot vienu) datu mezglus, meistars sa\u0146em kaut k\u0101du pirmo zi\u0146u, kas saka, ka mezgls A ir izsl\u0113dzies, un cen\u0161as par to past\u0101st\u012bt mezglam B, mezglam C, mezglam D. <\/p>\n<p><\/p>\n<p>Pa\u0161laik ar \u0161o var c\u012bn\u012bties tikai, uzst\u0101dot laika ierobe\u017eojumu m\u0113\u0123in\u0101jumiem k\u0101dam kaut ko past\u0101st\u012bt, apm\u0113ram 20-30 sekundes, t\u0101d\u0113j\u0101di kontrol\u0113jot datu centra izsl\u0113g\u0161anas \u0101trumu no klastera.<\/p>\n<p><\/p>\n<p>Princip\u0101 tas atbilst pras\u012bb\u0101m, kas s\u0101kotn\u0113ji tika izvirz\u012btas gal\u012bgajam produktam projekta ietvaros, ta\u010du no \"t\u012br\u0101s zin\u0101tnes\" viedok\u013ca tas ir k\u013c\u016bda. Kura, starp citu, tika veiksm\u012bgi nov\u0113rsta izstr\u0101d\u0101t\u0101ju versij\u0101 7.2.<\/p>\n<p><\/p>\n<p>Turkl\u0101t, kad k\u0101da datu noda izsl\u0113dz\u0101s, izr\u0101d\u012bj\u0101s, ka svar\u012bg\u0101k ir izplat\u012bt inform\u0101ciju par t\u0101s izsl\u0113g\u0161anos nek\u0101 inform\u0113t visu klasteri, ka uz t\u0101s atrad\u0101s noteikts prim\u0101rais shards (lai reklam\u0113tu replica-shard cit\u0101 datu centr\u0101 k\u0101 prim\u0101ro, uz kuru var rakst\u012bt inform\u0101ciju).<\/p>\n<p><\/p>\n<p>T\u0101d\u0113\u013c, kad viss \"atkal ir sak\u0101rtots\", izsl\u0113gt\u0101s datu nodes nekav\u0113joties netiek apz\u012bm\u0113tas k\u0101 novecoju\u0161as. T\u0101d\u0113j\u0101di m\u0113s esam spiesti gaid\u012bt, kad beigsies visi pingus uz izsl\u0113gtaj\u0101m datu noda, un tikai p\u0113c tam m\u016bsu klasteris s\u0101k zi\u0146ot par to, ka tur, tur un tur ir j\u0101turpina rakst\u012bt inform\u0101ciju. Pla\u0161\u0101k par \u0161o var las\u012bt. <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/elastic\/elasticsearch\/issues\/46909\">\u0161eit<\/a><\/noindex>. <\/p>\n<p><\/p>\n<p>Galu gal\u0101 datu centra izsl\u0113g\u0161anas oper\u0101cija mums \u0161odien aiz\u0146em apm\u0113ram 5 min\u016btes stundas p\u012b\u0137a laik\u0101. T\u0101m lielaj\u0101m un smagn\u0113j\u0101m strukt\u016br\u0101m tas ir diezgan labs rezult\u0101ts.<\/p>\n<p><\/p>\n<p>Galu gal\u0101 m\u0113s non\u0101c\u0101m pie \u0161\u0101da risin\u0101juma:<\/p>\n<p><\/p>\n<ul>\n<li>Mums ir 360 datu nodes ar 700 gigabaitu diskiem.<\/li>\n<li>60 koordinatoru, lai mar\u0161rut\u0113tu trafiku pa \u0161\u012bm datu nod\u0113m.<\/li>\n<li>40 masteru, kas mums paliku\u0161i k\u0101 zin\u0101ms mantojums no versij\u0101m pirms 6.4.0 \u2014 lai iztur\u0113tu datu centra izsl\u0113g\u0161anu, m\u0113s mor\u0101li bij\u0101m gatavi zaud\u0113t da\u017eas ma\u0161\u012bnas, lai garant\u0113tu pat slikt\u0101k\u0101 scen\u0101rija gad\u012bjum\u0101 saglab\u0101t masteru balsu skaitu.<\/li>\n<li>Jebkuras m\u0113\u0123in\u0101jumas apvienot lomas vien\u0101 konteiner\u0101 aizvien pamat\u0101 non\u0101ca pie t\u0101, ka agr\u0101k vai v\u0113l\u0101k node saboj\u0101j\u0101s zem slodzes. <\/li>\n<li>Vis\u0101 klaster\u012b tiek izmantots heap.size, kas ir 31 gigabaits: visas m\u0113\u0123in\u0101jumi samazin\u0101t izm\u0113ru noved pie t\u0101, ka smagos mekl\u0113\u0161anas piepras\u012bjumos ar leading wildcard tika izn\u012bcin\u0101tas da\u017eas noderes vai tika iesl\u0113gts circuit breaker pa\u0161a Elasticsearch.<\/li>\n<li>Turkl\u0101t, lai nodro\u0161in\u0101tu mekl\u0113\u0161anas veiktsp\u0113ju, m\u0113s cen\u0161amies uztur\u0113t objektu skaitu klaster\u012b p\u0113c iesp\u0113jas maz\u0101ku, lai apstr\u0101d\u0101tu p\u0113c iesp\u0113jas maz\u0101k notikumu pa\u0161\u0101 \u0161aur\u0101kaj\u0101 viet\u0101, kas mums rad\u0101s master\u012b.<\/li>\n<\/ul>\n<p><\/p>\n<h2 id=\"naposledok-o-monitoringe\">Beidzot par uzraudz\u012bbu.<\/h2>\n<p><\/p>\n<p>Lai viss darbotos t\u0101, k\u0101 pl\u0101nots, m\u0113s uzraug\u0101m sekojo\u0161o:<\/p>\n<p><\/p>\n<ul>\n<li>Katrs datu centrs zi\u0146o m\u016bsu m\u0101ko\u0146os, ka tas eksist\u0113 un uz t\u0101 ir \u0161\u0101di \u0161ardu. Kad m\u0113s kaut kur k\u0101du izsl\u0113dzam, klasteris p\u0113c 2-3 sekund\u0113m zi\u0146o, ka centr\u0101 A m\u0113s izsl\u0113dz\u0101m mezglu 2, 3 un 4 \u2014 tas noz\u012bm\u0113, ka citos datu centros mums nek\u0101d\u0101 gad\u012bjum\u0101 nevar izsl\u0113gt tos mezglus, uz kuriem paliku\u0161i \u0161ardi vien\u012bg\u0101 eksempl\u0101r\u0101.<\/li>\n<li>Paz\u012bstot meistara uzved\u012bbu, m\u0113s \u013coti uzman\u012bgi sekojam pending-uzdevumu skaitam. Jo pat viens iestr\u0113dzis uzdevums, ja netiks savlaic\u012bgi izlaists, teor\u0113tiski k\u0101d\u0101 steidzam\u0101 situ\u0101cij\u0101 var k\u013c\u016bt par iemeslu, k\u0101p\u0113c mums nedarbosies, piem\u0113ram, replikas \u0161arda p\u0101reja uz prim\u0101ro, k\u0101 rezult\u0101t\u0101 apst\u0101jas indeks\u0113\u0161ana.<\/li>\n<li>M\u0113s ar\u012b \u013coti uzman\u012bgi sekojam garbage collector aizkav\u0113\u0161an\u0101s laikam, jo ar to mums jau bija lieli sare\u017e\u0123\u012bjumi optimiz\u0101cijas proces\u0101.<\/li>\n<li>Rejd\u017eekti pa pavedieniem, lai iepriek\u0161 saprastu, kur atrodas \"s\u016bkn\u0113\u0161anas piltuve\".<\/li>\n<li>Nu un standarta metriki, piem\u0113ram, heap, RAM un I\/O.<\/li>\n<\/ul>\n<p><\/p>\n<p>Izveidojot uzraudz\u012bbu, noteikti j\u0101\u0146em v\u0113r\u0101 \u012bpa\u0161\u012bbas Thread Pool Elasticsearch. <noindex><a rel=\"nofollow\" href=\"https:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/6.8\/modules-threadpool.html\">Elasticsearch dokument\u0101cija<\/a><\/noindex> apraksta iesp\u0113jas konfigur\u0113t un noklus\u0113juma v\u0113rt\u012bbas mekl\u0113\u0161anai, indeks\u0113\u0161anai, ta\u010du piln\u012bb\u0101 norobe\u017eojas no thread_pool.management. \u0160ie pavedieni apstr\u0101d\u0101, tostarp, piepras\u012bjumus, piem\u0113ram, _cat\/shards un citus l\u012bdz\u012bgus, kurus \u0113rti izmantot, rakstot uzraudz\u012bbu. Jo liel\u0101ks klasteris, jo vair\u0101k t\u0101du piepras\u012bjumu tiek izpild\u012bti vien\u0101 laika vien\u012bb\u0101, bet iepriek\u0161 min\u0113tais thread_pool.management ne tikai netiek iek\u013cauts ofici\u0101laj\u0101 dokument\u0101cij\u0101, bet ar\u012b p\u0113c noklus\u0113juma ir ierobe\u017eots ar 5 pavedieniem, kas \u013coti \u0101tri tiek izlietoti, p\u0113c k\u0101 uzraudz\u012bba p\u0101rst\u0101j darboties pareizi.<\/p>\n<p><\/p>\n<p>Ko gribas teikt nosl\u0113gum\u0101: mums izdev\u0101s! M\u0113s izveidoj\u0101m saviem programm\u0113t\u0101jiem un izstr\u0101d\u0101t\u0101jiem r\u012bku, kas praktiski jebkur\u0101 situ\u0101cij\u0101 sp\u0113j \u0101tri un prec\u012bzi sniegt inform\u0101ciju par notieko\u0161o ra\u017eo\u0161an\u0101.<\/p>\n<p><\/p>\n<p>J\u0101, tas izr\u0101d\u012bj\u0101s diezgan sare\u017e\u0123\u012bti, ta\u010du tom\u0113r m\u016bsu v\u0113lmes izdev\u0101s iek\u013caut jau eso\u0161ajos produktos, kurus taj\u0101 pa\u0161\u0101 laik\u0101 nevajadz\u0113ja labot un p\u0101rrakst\u012bt sev.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"Elasticsearch klasteris uz 200 TB+\" src=\"\/wp-content\/uploads\/2020\/03\/1e852123ca5ec5eaec27bdc66b6c4e83.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p>Avots: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/odnoklassniki\/blog\/494260\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435. \u041d\u043e \u0447\u0442\u043e \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442, \u043a\u043e\u0433\u0434\u0430 \u0445\u043e\u0447\u0435\u0448\u044c \u0441 \u0435\u0433\u043e \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u043b\u043e\u0433\u0438 \u00ab\u0432 \u043e\u0441\u043e\u0431\u043e \u043a\u0440\u0443\u043f\u043d\u043e\u043c \u043e\u0431\u044a\u0451\u043c\u0435\u00bb? \u0414\u0430 \u0435\u0449\u0451 \u0438 \u0431\u0435\u0437\u0431\u043e\u043b\u0435\u0437\u043d\u0435\u043d\u043d\u043e \u043f\u0435\u0440\u0435\u0436\u0438\u0432\u0430\u0442\u044c \u043e\u0442\u043a\u0430\u0437 \u043b\u044e\u0431\u043e\u0433\u043e \u0438\u0437 \u043d\u0435\u0441\u043a\u043e\u043b\u044c\u043a\u0438\u0445 \u0434\u0430\u0442\u0430-\u0446\u0435\u043d\u0442\u0440\u043e\u0432? \u041a\u0430\u043a\u043e\u0439 \u0441\u0442\u043e\u0438\u0442 \u0434\u0435\u043b\u0430\u0442\u044c \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u0443, \u0438 \u043d\u0430 \u043a\u0430\u043a\u0438\u0435 \u043f\u043e\u0434\u0432\u043e\u0434\u043d\u044b\u0435 \u043a\u0430\u043c\u043d\u0438 \u043d\u0430\u0442\u043a\u043d\u0451\u0448\u044c\u0441\u044f? \u041c\u044b \u0432 \u041e\u0434\u043d\u043e\u043a\u043b\u0430\u0441\u0441\u043d\u0438\u043a\u0430\u0445 \u0440\u0435\u0448\u0438\u043b\u0438 \u043f\u0440\u0438 \u043f\u043e\u043c\u043e\u0449\u0438 elasticsearch \u0440\u0435\u0448\u0438\u0442\u044c \u0432\u043e\u043f\u0440\u043e\u0441 \u043b\u043e\u0433-\u043c\u0435\u043d\u0435\u0434\u0436\u043c\u0435\u043d\u0442\u0430, \u0430 \u0442\u0435\u043f\u0435\u0440\u044c \u0434\u0435\u043b\u0438\u043c\u0441\u044f \u0441 \u0425\u0430\u0431\u0440\u043e\u043c \u043e\u043f\u044b\u0442\u043e\u043c: \u0438 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":75797,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-75796","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.5 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.5\" \/>\n\t\t<meta property=\"og:locale\" content=\"lv_LV\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-28T17:42:18+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Elasticsearch klasteris uz 200 TB+ | ProHoster","description":"Daudzi saskaras ar Elasticsearch.","canonical_url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"lv_LV","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043b\u0430\u0441\u0442\u0435\u0440 Elasticsearch \u043d\u0430 200 \u0422\u0411+ | ProHoster","og:description":"\u0421 Elasticsearch \u0441\u0442\u0430\u043b\u043a\u0438\u0432\u0430\u044e\u0442\u0441\u044f \u043c\u043d\u043e\u0433\u0438\u0435.","og:url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/klaster-elasticsearch-na-200-tb","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-28T17:42:18+00:00","article:modified_time":"2020-03-28T17:42:18+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"75796","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:49:32","updated":"2022-09-27 21:24:26","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/75796","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/comments?post=75796"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/75796\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media\/75797"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media?parent=75796"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/categories?post=75796"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/tags?post=75796"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}