Anthu ambiri amavutika ndi Elasticsearch. Koma chimachitika ndi chiyani mukafuna kuigwiritsa ntchito kusunga zipika “mu voliyumu yayikulu kwambiri”? Ndipo kodi ndizosapwetekanso kuwona kulephera kwa malo angapo a data? Ndi zomanga zotani zomwe muyenera kupanga, ndipo ndi misampha yotani yomwe mungapunthwe?
Ife ku Odnoklassniki tinaganiza zogwiritsa ntchito elasticsearch kuti tithetse vuto la kasamalidwe ka chipika, ndipo tsopano tikugawana zomwe takumana nazo ndi Habr: zonse za zomangamanga komanso zovuta.
Ndine Pyotr Zaitsev, ndimagwira ntchito ngati woyang'anira dongosolo ku Odnoklassniki. Izi zisanachitike, ndinalinso woyang'anira, ndimagwira ntchito ndi Manticore Search, kusaka kwa Sphinx, Elasticsearch. Mwina, ngati ...kufufuza kwina kukuwoneka, inenso ndigwira ntchito nayo. Ndimachita nawo ntchito zingapo zotseguka mongodzipereka.
Nditafika ku Odnoklassniki, ndinalankhula mosasamala kuti nditha kugwira ntchito ndi Elasticsearch. Nditachidziwa bwino ndi kumaliza ntchito zina zing’onozing’ono, ndinapatsidwa ntchito yaikulu yokonzanso ndondomeko ya kasamalidwe ka mitengo imene inalipo panthawiyo.
amafuna
Zofunikira zadongosolo zidapangidwa motere:
- Graylog idayenera kugwiritsidwa ntchito ngati kutsogolo. Chifukwa kampaniyo idadziwa kale kugwiritsa ntchito mankhwalawa, opanga mapulogalamu ndi oyesa amadziwa, zinali zodziwika bwino komanso zosavuta kwa iwo.
- Voliyumu ya data: pafupifupi 50-80 mauthenga zikwi pa sekondi iliyonse, koma ngati chinachake chisweka, ndiye kuti magalimoto sali ochepa, akhoza kukhala mizere 2-3 miliyoni pamphindikati.
- Titakambirana ndi makasitomala zomwe zimafunikira kuti pakhale kuthamanga kwa mafunso osakira, tidazindikira kuti njira yogwiritsira ntchito kachitidwe yotere ndi iyi: anthu akufunafuna zipika zawo kwa masiku awiri apitawa ndipo safuna kudikirira kuposa masiku awiri apitawa. chachiwiri chifukwa cha funso lopangidwa.
- Oyang'anirawo adanenetsa kuti dongosololi likhoza kuwonjezereka mosavuta ngati kuli kofunikira, popanda kuwafunsa kuti afufuze mozama momwe amagwirira ntchito.
- Kotero kuti ntchito yokhayo yokonza yomwe machitidwewa amafunikira nthawi ndi nthawi ndikusintha zida zina.
- Kuphatikiza apo, Odnoklassniki ili ndi chikhalidwe chabwino kwambiri chaukadaulo: ntchito iliyonse yomwe timayambitsa iyenera kupulumuka kulephera kwa data (mwadzidzi, mosakonzekera komanso nthawi iliyonse).
Chofunikira chomaliza pakukhazikitsa polojekitiyi chinatiwonongera ndalama zambiri, zomwe ndikambirana mwatsatanetsatane.
Lachitatu
Timagwira ntchito m'malo anayi opangira data, pomwe ma data a Elasticsearch amatha kupezeka m'magawo atatu (pazifukwa zingapo zomwe si zaukadaulo).
Malo anayi a data awa ali ndi magwero pafupifupi 18 zikwi zosiyanasiyana - zida, zotengera, makina enieni.
Chofunika kwambiri: masango amayamba m'mitsuko osati pa makina akuthupi, koma mopitirira . Zotengera ndizotsimikizika ma cores 2, ofanana ndi 2.0Ghz v4, ndikuthekera kobwezeretsanso ma cores otsala ngati alibe ntchito.
Mwanjira ina:
Topology
Poyamba ndinawona njira yothetsera vutoli motere:
- 3-4 VIPs ali kumbuyo kwa A-record ya Graylog domain, iyi ndi adiresi yomwe zipika zimatumizidwa.
- VIP iliyonse ndi LVS balancer.
- Pambuyo pake, zipika zimapita ku batire ya Graylog, zina mwazolemba zili mu mtundu wa GELF, zina mu mtundu wa syslog.
- Ndiye zonsezi zimalembedwa m'magulu akuluakulu ku batri la ogwirizanitsa a Elasticsearch.
- Ndipo iwo, nawonso, amatumiza zolemba ndikuwerenga zopempha ku ma data ofunikira.
Mawu omaliza
Mwina si aliyense amene amamvetsetsa mawuwa mwatsatanetsatane, kotero ndikufuna kukhalapo pang'ono.
Elasticsearch ili ndi mitundu ingapo ya node - master, coordinator, data node. Pali mitundu ina iwiri yosinthira zipika komanso kulumikizana pakati pamagulu osiyanasiyana, koma tidagwiritsa ntchito okhawo omwe atchulidwa.
Master
Imawongolera ma node onse omwe ali mgululi, imasunga mapu am'magulu atsopano ndikugawa pakati pa node, kukonza malingaliro a zochitika, ndikusamalira mitundu yosiyanasiyana yamagulu.
Mkonzi
Amagwira ntchito imodzi yokha: amavomereza kuwerenga kapena kulemba zopempha kuchokera kwa makasitomala ndi njira zomwe zimayendera. Ngati pali pempho lolemba, mwina, lifunsa master kuti ndi gawo liti la index lomwe liyenera kuyikamo, ndikuwongoleranso pempholo.
Node ya data
Amasunga zidziwitso, amafufuza zofufuza kuchokera kunja ndikuchita ntchito pa shards zomwe zili pamenepo.
greylog
Ichi ndi china chake ngati kuphatikiza kwa Kibana ndi Logstash mu ELK stack. Graylog imaphatikiza UI ndi payipi yopangira chipika. Pansi pa hood, Graylog imayendetsa Kafka ndi Zookeeper, zomwe zimapereka kulumikizana kwa Graylog ngati gulu. Graylog ikhoza kusunga zipika (Kafka) ngati Elasticsearch sichikupezeka ndikubwereza zopempha zomwe sizinachite bwino powerenga ndi kulemba, gulu ndikuyika zipika malinga ndi malamulo omwe atchulidwa. Monga Logstash, Graylog ili ndi magwiridwe antchito osintha mizere musanayilembe ku Elasticsearch.
Kuphatikiza apo, Graylog ili ndi chidziwitso chokhazikika chomwe chimalola, kutengera node imodzi ya Elasticsearch, kupeza mapu onse amgulu ndikusefa ndi tag inayake, zomwe zimapangitsa kuti ziwongolere zopempha kuzinthu zinazake.
Zowoneka zikuwoneka motere:
Ichi ndi chithunzi chochokera ku zochitika zinazake. Apa timapanga histogram kutengera kusaka ndikuwonetsa mizere yoyenera.
Zisonyezero
Kubwerera ku kamangidwe ka dongosolo, ndikufuna kukhala mwatsatanetsatane momwe tinapangira ndondomeko ya ndondomeko kuti zonse zigwire bwino.
Pachithunzi pamwambapa, iyi ndiye gawo lotsika kwambiri: Elasticsearch data node.
Mlozera ndi gulu lalikulu lomwe limapangidwa ndi ma Elasticsearch shards. Payokha, chilichonse mwa shards sichinthu choposa index ya Lucene. Ndipo mlozera uliwonse wa Lucene, nawonso, umakhala ndi gawo limodzi kapena angapo.
Popanga, tinaganiza kuti kuti tikwaniritse kufunikira kwa liwiro la kuwerenga pa data yambiri, tifunika "kufalitsa" detayi mofanana pamagulu a deta.
Izi zidapangitsa kuti chiwerengero cha shards pa index (yokhala ndi zobwereza) chizikhala chofanana ndi kuchuluka kwa ma data. Choyamba, pofuna kutsimikizira chinthu chobwerezabwereza chofanana ndi ziwiri (ndiko kuti, tikhoza kutaya theka la masango). Ndipo, chachiwiri, pofuna kukonza kuwerenga ndi kulemba zopempha osachepera theka la tsango.
Poyamba tidazindikira nthawi yosungira ngati masiku 30.
Kugawidwa kwa shards kumatha kuyimiridwa mwachiwonetsero motere:
Rectangle yonse ya imvi ndi index. Malo ofiira ofiira mmenemo ndi shard yoyamba, yoyamba mu ndondomeko. Ndipo bwalo la buluu ndi lofanana ndi shard. Iwo ali m'malo osiyanasiyana deta.
Tikawonjezera shard ina, imapita ku data center yachitatu. Ndipo, pamapeto pake, timapeza dongosolo ili, lomwe limapangitsa kutaya DC popanda kutaya kusasinthasintha kwa deta:
Kuzungulira kwa indexes, i.e. kupanga cholozera chatsopano ndikuchotsa chakale kwambiri, tidachipanga kukhala chofanana ndi maola 48 (malinga ndi kagwiritsidwe ntchito ka index: maola 48 omaliza amafufuzidwa nthawi zambiri).
Nthawi yozungulira iyi ndi chifukwa cha zifukwa izi:
Pamene pempho lofufuzira lifika pamtundu wina wa deta, ndiye, kuchokera ku mawonekedwe a ntchito, zimakhala zopindulitsa kwambiri pamene shard imodzi ikufunsidwa, ngati kukula kwake kukufanana ndi kukula kwa chiuno cha node. Izi zimakuthandizani kuti musunge gawo "lotentha" la index mu mulu ndikulipeza mwachangu. Pakakhala "zigawo zotentha" zambiri, kuthamanga kwakusaka kwa index kumawononga.
Node ikayamba kuchita zofufuzira pa shard imodzi, imagawa ulusi wochuluka wofanana ndi kuchuluka kwa ma hyperthreading cores a makina akuthupi. Ngati funso lofufuzira limakhudza chiwerengero chachikulu cha shards, ndiye kuti chiwerengero cha ulusi chimakula mofanana. Izi zimakhala ndi zotsatira zoyipa pa liwiro lakusaka ndipo zimakhudza molakwika kusanja kwa data yatsopano.
Kuti tipereke kuchedwa kofunikira, tinaganiza zogwiritsa ntchito SSD. Kuti akwaniritse zopempha mwachangu, makina omwe amakhala ndi zotengerazi amayenera kukhala ndi ma cores osachepera 56. Chiwerengero cha 56 chinasankhidwa ngati mtengo wokwanira womwe umatsimikizira kuchuluka kwa ulusi womwe Elasticsearch idzapanga pakagwira ntchito. Mu Elasitcsearch, magawo ambiri a dziwe la ulusi amadalira mwachindunji kuchuluka kwa ma cores omwe amapezeka, omwe amakhudza mwachindunji nambala yofunikira yamagulu mumagulu malinga ndi mfundo yakuti "ma cores ochepa - ma node ambiri".
Zotsatira zake, tapeza kuti pafupifupi shard imalemera pafupifupi 20 gigabytes, ndipo pali 1 shards pa index. Chifukwa chake, ngati tisintha kamodzi pa maola 360 aliwonse, ndiye kuti timakhala ndi 48. Mlozera uliwonse uli ndi data yamasiku awiri.
Zolemba zolembera ndi kuwerenga
Tiyeni tiwone momwe deta imalembedwera mu dongosolo lino.
Tiyerekeze kuti pempho lina lifika kuchokera ku Graylog kupita kwa wogwirizanitsa. Mwachitsanzo, tikufuna kuloza mizere 2-3 zikwi.
Wogwirizanitsa, atalandira pempho lochokera kwa Graylog, akufunsa mbuyeyo kuti: “Mu pempho lolozera, tidatchula mlozera, koma momwe shard yolembera sinatchulidwe.
Mbuyeyo akuyankha kuti: "Lembani izi ku shard nambala 71," pambuyo pake imatumizidwa mwachindunji kumalo oyenerera a deta, kumene nambala yoyamba-shard 71 ilipo.
Pambuyo pake chipikacho chimabwerezedwanso ku replica-shard, yomwe ili pamalo ena a data.
Pempho losaka lifika kuchokera ku Graylog kupita kwa wogwirizanitsa. Wogwirizanitsa amawongolera motsatira ndondomeko, pamene Elasticsearch imagawira zopempha pakati pa primary-shard ndi replica-shard pogwiritsa ntchito mfundo yozungulira.
Ma node a 180 amayankha mosagwirizana, ndipo pamene akuyankha, wogwirizanitsa akusonkhanitsa zambiri zomwe zakhala "zazilavulidwa" ndi ma data ofulumira. Pambuyo pake, pamene chidziwitso chonse chafika, kapena pempho lafika nthawi yopuma, limapereka zonse mwachindunji kwa kasitomala.
Dongosolo lonseli pafupipafupi limapanga mafunso osaka kwa maola 48 apitawa mu 300-400ms, kupatula mafunso omwe ali ndi chikwangwani chotsogola.
Maluwa okhala ndi Elasticsearch: Kukhazikitsa kwa Java
Kuti zonse zigwire ntchito momwe timafunira poyamba, tidakhala nthawi yayitali ndikuchotsa zinthu zosiyanasiyana mgululi.
Gawo loyamba lamavuto omwe adapezeka anali okhudzana ndi momwe Java imakonzedweratu mwachisawawa mu Elasticsearch.
Vuto limodzi
Tawona malipoti ochuluka kwambiri kuti pamlingo wa Lucene, ntchito zakumbuyo zikayamba, kuphatikizika kwa gawo la Lucene kumalephera ndi cholakwika. Panthawi imodzimodziyo, zinali zoonekeratu m'zipika kuti ichi chinali cholakwika cha OutOfMemoryError. Tinawona kuchokera ku telemetry kuti chiuno chinali chaulere, ndipo sizinali zomveka bwino chifukwa chake opaleshoniyi ikulephera.
Zinapezeka kuti kuphatikizika kwa index ya Lucene kumachitika kunja kwa chiuno. Ndipo zotengera ndizochepa kwambiri potengera zomwe zimagwiritsidwa ntchito. Mulu wokhawo ukhoza kulowa muzinthu izi (mtengo wa heap.size unali pafupifupi wofanana ndi RAM), ndipo ntchito zina zopanda milu zinawonongeka ndi zolakwika zogawa kukumbukira ngati pazifukwa zina sizinagwirizane ndi ~ 500MB zomwe zinatsalira malire.
Kukonzekera kunali kochepa kwambiri: kuchuluka kwa RAM komwe kunalipo pa chidebecho kunawonjezeka, pambuyo pake tinayiwala kuti tinali ndi mavuto.
Vuto lachiwiri
Masiku 4-5 pambuyo pa kukhazikitsidwa kwa tsango, tinawona kuti mfundo za deta zinayamba kugwa nthawi ndi nthawi kuchokera mgulu ndikulowa pambuyo pa masekondi 10-20.
Titayamba kuzizindikira, zidapezeka kuti kukumbukira kwa mulu wa Elasticsearch sikuyendetsedwa mwanjira iliyonse. Titakumbukira zambiri pachidebecho, tidatha kudzaza maiwewa achindunji ndi zidziwitso zosiyanasiyana, ndipo zidakonzedwa pokhapokha GC yodziwika bwino idakhazikitsidwa kuchokera ku Elasticsearch.
Nthawi zina, opareshoni iyi idatenga nthawi yayitali, ndipo panthawiyi gululi limatha kuyika chizindikiro ichi ngati chatuluka kale. Vutoli likufotokozedwa bwino .
Yankho lake linali motere: tidachepetsa kuthekera kwa Java kugwiritsa ntchito kukumbukira kochuluka kunja kwa mulu pazochita izi. Tidazichepetsa ku 16 gigabytes (-XX:MaxDirectMemorySize=16g), kuwonetsetsa kuti GC yodziwika bwino imatchedwa nthawi zambiri ndikusinthidwa mwachangu kwambiri, potero osasokonezanso gululo.
Vuto lachitatu
Ngati mukuganiza kuti mavuto omwe ali ndi "node omwe amasiya masango nthawi yosayembekezereka" atha, mukulakwitsa.
Titakonza ntchitoyo ndi ma index, tidasankha mmapfs kuti pamiyendo yatsopano yokhala ndi magawo akulu. Izi zinali zolakwika, chifukwa mukamagwiritsa ntchito mmapfs fayilo imayikidwa mu RAM, ndiyeno timagwira ntchito ndi fayilo yojambulidwa. Chifukwa cha izi, zikuwoneka kuti pamene GC ikuyesera kuyimitsa ulusi muzogwiritsira ntchito, timapita kumalo otetezeka kwa nthawi yaitali kwambiri, ndipo panjira yopitako, ntchitoyo imasiya kuyankha zopempha za mbuye wake ngati ali moyo. . Chifukwa chake, master amakhulupirira kuti node palibenso mgululi. Pambuyo pa izi, pambuyo pa masekondi 5-10, wotolera zinyalala amagwira ntchito, mfundoyo imakhala ndi moyo, imalowanso m'gululo ndikuyamba kuyambitsa shards. Zonse zinkamveka ngati "zopanga zomwe timayenera" ndipo sizinali zoyenera kuchita chilichonse chachikulu.
Kuti tichotse khalidweli, tidasinthiratu ku niofs wamba, ndiyeno, titasamuka kuchokera kumitundu yachisanu ya Elastic kupita yachisanu ndi chimodzi, tidayesa ma hybridfs, pomwe vutoli silinabwerezedwe. Mukhoza kuwerenga zambiri za mitundu yosungirako .
Vuto lachinayi
Ndiye panali vuto linanso losangalatsa lomwe tidachita kwanthawi yayitali. Tidazigwira kwa miyezi 2-3 chifukwa mawonekedwe ake anali osamvetsetseka.
Nthawi zina ogwirizanitsa athu ankapita ku Full GC, kawirikawiri nthawi ina pambuyo pa chakudya chamasana, ndipo sanabwerere kuchokera kumeneko. Panthawi imodzimodziyo, podula mitengo ya kuchedwa kwa GC, zinkawoneka ngati izi: chirichonse chikuyenda bwino, chabwino, chabwino, ndiyeno mwadzidzidzi chirichonse chikuyenda moipa kwambiri.
Poyamba tinkaganiza kuti tili ndi wogwiritsa ntchito woyipa yemwe akuyambitsa zopempha zomwe zidachotsa wogwirizanitsa ntchitoyo. Tinalemba zopempha kwa nthawi yaitali kwambiri, kuyesa kudziwa zomwe zikuchitika.
Zotsatira zake, zidapezeka kuti panthawi yomwe wogwiritsa ntchito ayambitsa pempho lalikulu, ndipo limafika kwa wogwirizanitsa wina wa Elasticsearch, ma node ena amayankha motalika kuposa ena.
Ndipo pamene wogwirizanitsa akudikirira yankho kuchokera ku mfundo zonse, amasonkhanitsa zotsatira zotumizidwa kuchokera ku mfundo zomwe zayankha kale. Kwa GC, izi zikutanthauza kuti machitidwe athu ogwiritsira ntchito milu amasintha mwachangu kwambiri. Ndipo GC yomwe tidagwiritsa ntchito sinathe kupirira ntchitoyi.
Kukonzekera kokha komwe tapeza kuti tisinthe machitidwe a gululi muzochitika izi ndikusamukira ku JDK13 ndikugwiritsa ntchito chotolera zinyalala cha Shenandoah. Izi zinathetsa vutoli, ogwirizanitsa athu anasiya kugwa.
Apa ndipamene mavuto ndi Java adatha ndipo mavuto a bandwidth adayamba.
"Zipatso" ndi Elasticsearch: throughput
Mavuto obwera chifukwa cha kuchuluka kwake amatanthauza kuti gulu lathu limagwira ntchito mokhazikika, koma pachimake pa kuchuluka kwa zolemba zomwe zili ndi indexed komanso pakuwongolera, magwiridwe antchito sakhala okwanira.
Chizindikiro choyamba chokumana nacho: pa "kuphulika" kwina pakupanga, pamene zipika zambiri zimapangidwira mwadzidzidzi, cholakwika cha indexing es_rejected_execution chimayamba kuwunikira pafupipafupi ku Graylog.
Izi zinali chifukwa chakuti thread_pool.write.queue pa node imodzi ya data, mpaka nthawi yomwe Elasticsearch imatha kukonza pempho la indexing ndikuyika zambiri ku shard pa disk, imatha kusunga zopempha 200 zokha. Ndipo mu Zochepa kwambiri zimanenedwa za parameter iyi. Kuchuluka kokha kwa ulusi ndi kukula kosasintha kumasonyezedwa.
Zachidziwikire, tidapita kukapotoza mtengowu ndikupeza zotsatirazi: makamaka, pakukhazikitsa kwathu, mpaka zopempha za 300 zimasungidwa bwino, ndipo mtengo wapamwamba umadzaza ndi mfundo yakuti timawulukiranso ku Full GC.
Kuonjezera apo, popeza awa ndi magulu a mauthenga omwe amafika mkati mwa pempho limodzi, kunali koyenera kuwongolera Graylog kuti alembe nthawi zambiri komanso m'magulu ang'onoang'ono, koma m'magulu akuluakulu kapena kamodzi pa masekondi a 3 ngati mtandawo sunathe. Pankhaniyi, zikuwonekeratu kuti zomwe timalemba mu Elasticsearch sizipezeka mumasekondi awiri, koma asanu (zomwe zimatikomera ife), koma kuchuluka kwa ma retrays omwe amayenera kupangidwa kuti adutse mulu wa zidziwitso wachepetsedwa.
Izi ndizofunikira kwambiri panthawi yomwe china chake chawonongeka kwinakwake ndikukanena mokwiya za izo, kuti musatengere Elastic ya spammed, ndipo pakapita nthawi - ma Graylog node omwe sagwira ntchito chifukwa chotsekeka.
Kuonjezera apo, pamene tinali ndi kuphulika komweku mukupanga, tinalandira madandaulo kuchokera kwa olemba mapulogalamu ndi oyesa: panthawi yomwe amafunikira kwambiri zipikazi, anapatsidwa pang'onopang'ono.
Iwo anayamba kuzilingalira izo. Kumbali imodzi, zinali zoonekeratu kuti mafunso onse osaka ndi mafunso olembera adasinthidwa, makamaka, pamakina omwewo, ndipo mwanjira ina padzakhala zovuta zina.
Koma izi zitha kuzunguliridwa pang'ono chifukwa chakuti m'matembenuzidwe achisanu ndi chimodzi a Elasticsearch, algorithm idawoneka yomwe imakulolani kugawa mafunso pakati pa ma data ofunikira osati molingana ndi mfundo yachisawawa yozungulira-robin (chidebe chomwe chimalozera ndikusunga choyambirira. -shard ikhoza kukhala yotanganidwa kwambiri, sipadzakhala njira yoyankhira mwamsanga), koma kutumiza pempholi ku chidebe chochepa chodzaza ndi replica-shard, chomwe chidzayankha mofulumira kwambiri. Mwanjira ina, tinafika pa use_adaptive_replica_selection: zoona.
Chithunzi chowerengera chimayamba kuwoneka motere:
Kusintha kwa algorithm iyi kunapangitsa kuti zitheke kukonza nthawi yamafunso munthawi zomwe tinali ndi zipika zambiri zoti tilembe.
Pomaliza, vuto lalikulu linali kuchotsedwa kopanda ululu kwa malo opangira data.
Zomwe tinkafuna pagululi titangosiya kulumikizana ndi DC imodzi:
- Ngati tili ndi mbuye wamakono mu malo osungirako deta omwe alephera, ndiye kuti adzasankhidwanso ndikusunthidwa ngati gawo ku node ina mu DC ina.
- Mbuyeyo adzachotsa mwamsanga node zonse zosafikirika kuchokera kumagulu.
- Kutengera otsalawo, amvetsetsa: mu malo otayika a data omwe tinali ndi shards zotere komanso zotere, adzalimbikitsa mwachangu ma replica shards m'malo otsala a data, ndipo tidzapitiliza kulozera deta.
- Chifukwa cha izi, zolemba ndi kuwerenga kwa masango zidzachepa pang'onopang'ono, koma zonse zidzagwira ntchito, ngakhale pang'onopang'ono, koma mokhazikika.
Monga momwe zinalili, tinkafuna zinthu monga izi:
Ndipo ife tiri nazo izi:
Kodi zimenezi zinachitika bwanji?
Pamene data center idagwa, mbuye wathu adakhala wolepheretsa.
Chifukwa chiyani?
Chowonadi ndi chakuti mbuyeyo ali ndi TaskBatcher, yomwe imayang'anira kugawa ntchito zina ndi zochitika mgululi. Kutuluka kulikonse, kukwezedwa kulikonse kwa shard kuchokera ku replica kupita ku pulayimale, ntchito iliyonse yopanga shard kwinakwake - zonsezi zimapita koyamba ku TaskBatcher, komwe zimakonzedwa motsatizana komanso mu ulusi umodzi.
Panthawi yochotsa malo amodzi a data, zidapezeka kuti ma data onse m'malo osungiramo data omwe adatsala adawona kuti ndi udindo wawo kudziwitsa mbuyeyo "tataya zing'onozing'ono zamtundu uliwonse."
Nthawi yomweyo, ma data omwe adapulumuka adatumiza chidziwitso chonse kwa mbuye wapano ndikuyesera kudikirira kuti atsimikizire kuti adavomereza. Iwo sanayembekezere zimenezi, chifukwa mbuyeyo analandira ntchito mofulumira kuposa mmene akanatha kuyankha. Ma nodewo adatha nthawi yobwereza zopempha, ndipo mbuyeyo panthawiyi sanayese kuyankha, koma adatanganidwa kwambiri ndi ntchito yosankha zopempha poyambirira.
Mu mawonekedwe omaliza, zidapezeka kuti ma data adatumizira sipamu mpaka pomwe adalowa mu GC yonse. Pambuyo pake, gawo lathu laukadaulo lidasunthira kumalo ena, zomwe zidachitikanso, ndipo tsangolo lidaphwanyidwa.
Tidayesa, ndipo tisanayambe mtundu wa 6.4.0, pomwe izi zidakhazikitsidwa, zinali zokwanira kuti titha kutulutsa nthawi imodzi ma node 10 kuchokera ku 360 kuti titseke gululo.
Zinkawoneka motere:
Pambuyo pa mtundu wa 6.4.0, pomwe cholakwika choyipachi chidakhazikitsidwa, ma data adasiya kupha mbuyeyo. Koma izi sizinamupangitse kukhala "wanzeru". Izi: tikatulutsa 2, 3 kapena 10 (nambala ina iliyonse kupatula imodzi), mbuye amalandira uthenga woyamba womwe umati node A yachoka, ndikuyesa kunena node B, node C za izi, nodi D.
Ndipo pakadali pano, izi zitha kuthetsedwa pokhazikitsa nthawi yoyesera kuwuza wina za chinthu china, chofanana ndi masekondi 20-30, ndikuwongolera liwiro la data center kuchoka pagulu.
M'malo mwake, izi zikugwirizana ndi zofunikira zomwe poyamba zinaperekedwa ku chinthu chomaliza monga gawo la polojekitiyi, koma kuchokera ku "sayansi yoyera" iyi ndi cholakwika. Zomwe, mwa njira, zidakhazikitsidwa bwino ndi opanga mu mtundu 7.2.
Komanso, pamene node ina ya deta inatuluka, zinapezeka kuti kufalitsa zambiri zokhudza kutuluka kwake kunali kofunika kwambiri kusiyana ndi kuwuza gulu lonse kuti pali zotsalira zamtundu woterezi (kuti apititse patsogolo chithunzithunzi mu deta ina. pakati pa pulayimale, ndipo muzambiri zitha kulembedwapo).
Chifukwa chake, zonse zikafa kale, ma data omwe adatulutsidwa samalembedwa nthawi yomweyo ngati akale. Chifukwa chake, timakakamizika kudikirira mpaka ma pings onse atatha ku ma data omwe adatulutsidwa, ndipo pambuyo pake gulu lathu likuyamba kutiuza kuti pamenepo, apo, ndi pamenepo tiyenera kupitiliza kujambula zambiri. Mutha kuwerenga zambiri za izi .
Zotsatira zake, ntchito yochotsa malo opangira data lero imatitengera pafupifupi mphindi 5 panthawi yothamanga. Kwa colossus yayikulu komanso yosanja, izi ndi zotsatira zabwino kwambiri.
Zotsatira zake, tinafika pachisankho chotsatirachi:
- Tili ndi ma data 360 okhala ndi ma gigabyte 700 disks.
- Ogwirizanitsa 60 oyendetsa magalimoto kudzera m'malo omwewo.
- 40 ambuye omwe tawasiya ngati cholowa kuyambira matembenuzidwe asanafike 6.4.0 - kuti tipulumuke kuchotsedwa kwa data center, tinali okonzeka m'maganizo kutaya makina angapo kuti titsimikizidwe kukhala ndi quorum of masters ngakhale mu choyipa kwambiri
- Kuyesa kulikonse kuphatikiza maudindo pachidebe chimodzi kudakumana ndi mfundo yakuti posakhalitsa nodeyo imatha kusweka.
- Gulu lonse limagwiritsa ntchito heap.size ya 31 gigabytes: kuyesa konse kuchepetsa kukula kudapangitsa kuti mwina kupha ma node ena pamafunso ovuta kwambiri ndi makadi otsogola kapena kupeza wophwanya dera mu Elasticsearch palokha.
- Kuonjezera apo, pofuna kuonetsetsa kuti kusaka kumagwira ntchito, tinayesetsa kusunga chiwerengero cha zinthu zomwe zili mumagulu ang'onoang'ono momwe tingathere, kuti tigwiritse ntchito zochitika zochepa momwe tingathere mu botolo lomwe tinapeza mbuye.
Pomaliza za kuwunika
Kuti tiwonetsetse kuti zonsezi zikuyenda monga momwe tafunira, timayang'anira zotsatirazi:
- Dongosolo lililonse la data limapereka lipoti ku mtambo wathu kuti ulipo, ndipo pali shards zotere pamenepo. Tikazimitsa chinachake kwinakwake, gululo limafotokoza pambuyo pa masekondi a 2-3 kuti pakati pa A tinazimitsa node 2, 3, ndi 4 - izi zikutanthauza kuti m'malo ena a data sitingathe kuzimitsa mfundo zomwe zili ndi shard imodzi yokha. kumanzere.
- Podziwa chikhalidwe cha khalidwe la mbuye, timayang'ana mosamala kwambiri chiwerengero cha ntchito zomwe zikuyembekezera. Chifukwa ngakhale ntchito imodzi yokhazikika, ngati sichitha nthawi, mwachidziwitso pazochitika zina zadzidzidzi zingakhale chifukwa chake, mwachitsanzo, kukwezedwa kwa replica shard ku pulayimale sikugwira ntchito, chifukwa chake kulondolera kumasiya kugwira ntchito.
- Timayang'anitsitsanso kwambiri kuchedwa kwa zinyalala, chifukwa takhala ndi zovuta zambiri ndi izi panthawi yokonza.
- Amakana ndi ulusi kuti amvetsetse pasadakhale komwe kuli botolo.
- Chabwino, ma metric wamba monga mulu, RAM ndi I/O.
Mukamapanga kuwunika, muyenera kuganizira za Thread Pool mu Elasticsearch. limafotokoza za kasinthidwe ndi makonda osakira posaka ndi indexing, koma palibe chokhudza thread_pool.management.Ulusiwu umapanga, makamaka, mafunso ngati _cat/shards ndi ena ofanana nawo, omwe ndi osavuta kugwiritsa ntchito polemba zowunikira. Kukula kwa gululo, zopempha zotere zimachitidwa pa nthawi imodzi, ndipo thread_pool.management yomwe tatchulayi siinangoperekedwa m'mabuku ovomerezeka, komanso imachepetsedwa ndi kusakhulupirika kwa ulusi 5, womwe umatayidwa mwamsanga, pambuyo pake. kuwunika komwe kumasiya kugwira ntchito moyenera.
Zomwe ndikufuna kunena pomaliza: tachita! Tinatha kupatsa opanga mapulogalamu athu ndi omanga chida chomwe, pafupifupi chilichonse, chingathe kupereka mwachangu komanso modalirika zomwe zikuchitika popanga.
Inde, zidakhala zovuta, koma, komabe, tidatha kugwirizanitsa zokhumba zathu pazinthu zomwe zidalipo, zomwe sitinayenera kuzilemba ndikudzilemberanso tokha.
Source: www.habr.com