Vanhu vazhinji vanonetsekana neElasticsearch. Asi chii chinoitika kana iwe uchida kuishandisa kuchengetedza matanda "muhuwandu hwakanyanya"? Uye hazvirwadze here kuona kutadza kweimwe yenzvimbo dzinoverengeka dzedata? Ndeupi rudzi rwezvivakwa zvaunofanira kugadzira, uye ndeapi misungo yauchagumburwa pairi?
Isu paOdnoklassniki takasarudza kushandisa elasticsearch kugadzirisa nyaya yelog manejimendi, uye ikozvino tinogovana ruzivo rwedu naHabr: zvese nezve architecture uye nezve misungo.
Ini ndiri Pyotr Zaitsev, ndinoshanda semutongi wehurongwa kuOdnoklassniki. Izvi zvisati zvaitika, ndaivewo admin, ndaishanda neManticore Search, Sphinx search, Elasticsearch. Zvichida, kana imwe ... tsvakurudzo ikaonekwa, ini ndichashanda nayo zvakare. Iniwo ndinotora chikamu mune akati wandei mapurojekiti epamhepo nekuzvidira.
Pandakauya kuOdnoklassniki, ndakataura zvisina hanya pabvunzurudzo kuti ndaigona kushanda neElasticsearch. Mushure mekunge ndazvigadzirisa uye ndapedza mamwe mabasa akareruka, ndakapihwa basa guru rekugadzirisa hurongwa hwekutarisira matanda hwaivepo panguva iyoyo.
zvinodiwa
Zvido zvehurongwa zvakaitwa sezvizvi:
- Greylog yaizoshandiswa semberi. Nekuti kambani yaitove neruzivo rwekushandisa chigadzirwa ichi, vanogadzira uye vanoedza vaizviziva, zvaive zvakajairika uye zvakavaitira.
- Dhata vhoriyamu: paavhareji 50-80 zviuru mameseji pasekondi, asi kana chimwe chinhu chaputsika, saka traffic haina kuganhurirwa nechero chinhu, inogona kuva 2-3 miriyoni mitsetse pasekondi.
- Mushure mekukurukura nevatengi zvinodikanwa zvekumhanya kwekugadzirisa mibvunzo yekutsvaga, takaona kuti iyo yakajairika maitiro ekushandisa sisitimu yakadai ndeiyi: vanhu vari kutsvaga matanda ekushandisa kwavo kwemazuva maviri apfuura uye havadi kumirira kupfuura a. chechipiri chemhedzisiro yemubvunzo wakagadzirwa.
- Mamaneja akasimbirira kuti sisitimu iyi ive nyore scalable kana zvichidikanwa, pasina kuvada kuti vaongorore zvakadzama kuti inoshanda sei.
- Zvekuti iro chete basa rekugadzirisa rinodiwa neaya masisitimu nguva nenguva ndeye kushandura imwe hardware.
- Uye zvakare, Odnoklassniki ine yakanakisa tekinoroji tsika: chero sevhisi yatinotangisa inofanirwa kurarama nekutadza kwedata data (kamwe kamwe, isina kurongeka uye zvachose chero nguva).
Chinodiwa chekupedzisira mukuitwa kwechirongwa ichi chakatidhurira zvakanyanya, izvo zvandichataura nezvazvo zvakadzama.
Chitatu
Isu tinoshanda munzvimbo ina dzedata, nepo Elasticsearch data node inogona kungowanikwa mumatatu (nekuda kwezvikonzero zvisiri zvehunyanzvi).
Aya mana data nzvimbo ane angangoita 18 zviuru zvakasiyana log masosi - Hardware, midziyo, chaiwo michina.
Chinhu chakakosha: sumbu rinotanga mumidziyo kwete pamichina yenyama, asi pairi . Containers inovimbiswa 2 cores, yakafanana ne2.0Ghz v4, paine mukana wekudzokorodza macores asara kana asina kuita basa.
Nemamwe mashoko:
Topology
Pakutanga ndakaona iyo general fomu yemhinduro seizvi:
- 3-4 VIPs iri shure kweA-rekodhi yeGreylog domain, iyi ndiyo kero iyo matanda anotumirwa.
- VIP yega yega iLVS balancer.
- Mushure mayo, matanda anoenda kuGreylog bhatiri, imwe yedata iri muGELF fomati, mamwe ari syslog fomati.
- Zvadaro zvese izvi zvakanyorwa mumabheji makuru kune bhatiri reElasticsearch varongi.
- Uye ivo, ivowo, vanotumira kunyora uye kuverenga zvikumbiro kune akakodzera data node.
Terminology
Zvichida havasi vose vanonzwisisa izwi remashoko zvakadzama, saka ndinoda kugara pairi zvishoma.
Elasticsearch ine akati wandei marudzi emanodhi - tenzi, murongi, data node. Kune mamwe marudzi maviri ekuchinja kwakasiyana kwelogi uye kutaurirana pakati pemasumbu akasiyana, asi isu takashandisa iwo akanyorwa chete.
tenzi
Iyo inonongedza node dzese dziripo musumbu, inochengetedza mepu yezvikwata zvemazuva ano uye inoiparadzira pakati pemanodhi, inogadzirisa chiitiko chechiitiko, uye inoita mhando dzakasiyana dzekuchengetedza imba.
Coordinator
Inoita basa rimwe chete: inogamuchira kuverenga kana kunyora zvikumbiro kubva kune vatengi uye nzira iyi traffic. Kana paine chikumbiro chekunyora, kazhinji, chinobvunza tenzi kuti ndechipi shard yeindekisi yakakodzera yainofanira kuisa mairi, uye inotungamira chikumbiro mberi.
Data node
Inochengeta data, inoita mibvunzo yekutsvaga ichisvika kubva kunze uye inoita mashandiro pane shards iri pairi.
Greylog
Ichi chimwe chinhu chakafanana nekusanganiswa kweKibana neLogstash mune ELK stack. Greylog inosanganisa ese UI uye pombi yekugadzira pombi. Pasi pehodhi, Graylog inomhanya Kafka uye Zookeeper, iyo inopa kubatana kuGreylog sesumbu. Graylog inogona cache matanda (Kafka) kana Elasticsearch isipo uye kudzokorora zvikumbiro zvisina kubudirira zvekuverenga nekunyora, boka uye kumaka matanda maererano nemitemo yakatarwa. Kufanana neLogstash, Graylog ine basa rekugadzirisa mitsara isati yanyora kuElasticsearch.
Pamusoro pezvo, Graylog ine yakavakirwa-mukati sevhisi kuwanikwa inobvumira, zvichibva pane imwe iripo Elasticsearch node, kuwana iyo yese cluster mepu uye kuisefa neiyo chaiyo tag, izvo zvinoita kuti zvikwanise kutungamira zvikumbiro kune chaiwo midziyo.
Zvinotaridzika zvinotaridzika seizvi:
Iyi skrini kubva kune yakatarwa muenzaniso. Pano tinovaka histogram zvichienderana nemubvunzo wekutsvaga uye kuratidza mitsara yakakodzera.
Indexes
Kudzokera kuhurongwa hwekuvaka, ndinoda kugara mune zvakadzama nezvekuvaka kwatakaita index index kuti yese ishande nemazvo.
Mumufananidzo uri pamusoro, iyi ndiyo yakaderera nhanho: Elasticsearch data node.
Indekisi inhengo hombe chaiyo inogadzirwa neElasticsearch shards. Iyo pachayo, yega yega shards haisi chinhu kunze kweLucene index. Uye imwe neimwe indekisi yeLucene, zvakare, ine chikamu chimwe kana anopfuura.
Pakuronga, takaona kuti kuitira kuti tisangane nezvinodiwa zvekumhanya kwekuverenga pane yakawanda data, taifanira "kuparadzira" iyi data zvakaenzana munzvimbo dzese dzedata.
Izvi zvakakonzera kuti nhamba ye shards per index (ine replicas) inofanira kunyatsoenzana nenhamba ye data node. Chekutanga, kuitira kuve nechokwadi chekudzokorora chinhu chakaenzana nembiri (kureva kuti, tinogona kurasikirwa nehafu yesumbu). Uye, chechipiri, kuitira kugadzirisa kuverenga nekunyora zvikumbiro pane inokwana hafu yesumbu.
Isu takatanga tasarudza nguva yekuchengetedza semazuva makumi matatu.
Kugoverwa kwe shards kunogona kumiririrwa zvakajeka sezvinotevera:
Rectangle yakasviba yegrey indekisi. Ruboshwe rutsvuku mukati maro ndiyo shard yekutanga, yekutanga mune index. Uye ruvara rwebhuruu ireplica shard. Ivo vari munzvimbo dzakasiyana dze data.
Patinowedzera imwe shard, inoenda kune yechitatu data center. Uye, pakupedzisira, tinowana chimiro ichi, chinoita kuti zvikwanise kurasikirwa neDC pasina kurasikirwa nedata kuenderana:
Kutenderera kwe indexes, i.e. kugadzira index itsva uye kubvisa yekare, takaita kuti ienzane nemaawa makumi mana nemasere (maererano nemuenzaniso wekushandiswa kwendekisi: maawa ekupedzisira e48 anotsvaga kakawanda).
Iyi index inotenderera nguva inokonzerwa nezvikonzero zvinotevera:
Kana chikumbiro chekutsvaga chasvika pane imwe data node, saka, kubva pane imwe nzvimbo yekuona, inobatsira zvikuru kana shard imwe yakabvunzwa, kana hukuru hwayo hunofananidzwa nehukuru hwehudyu ye node. Izvi zvinokutendera kuti uchengetedze chikamu che "chinopisa" cheindex mumurwi uye nekukurumidza kuiwana. Kana pane zvakawanda zve "zvinopisa zvikamu", kukurumidza kwekutsvaga index kunoderedza.
Kana node ikatanga kuita mubvunzo wekutsvaga pane imwe shard, inogovera nhamba yeshinda dzakaenzana nenhamba ye hyperthreading cores yemuchina wenyama. Kana mubvunzo wekutsvaga uchikanganisa nhamba huru ye shards, saka nhamba ye thread inokura zvakaenzana. Izvi zvine mhedzisiro yakaipa pakukurumidza kutsvaga uye zvinokanganisa kurongeka kwedata idzva.
Kuti tipe inodiwa yekutsvaga latency, takasarudza kushandisa SSD. Kukurumidza kugadzirisa zvikumbiro, michina yaibata midziyo iyi yaifanirwa kunge iine 56 cores. Huwandu hwemakumi mashanu nenhanhatu hwakasarudzwa sehukoshi hwakakwana hunotaridza huwandu hweshinda dzichagadzirwa neElasticsearch panguva yekushanda. MuElasitcsearch, akawanda tambo dziva paramita zvinoenderana nenhamba yeanowanikwa macores, ayo anobata zvakananga nhamba inodiwa yemanodhi musumbu zvinoenderana nemusimboti "mashoma macores - akawanda node".
Nekuda kweizvozvo, takaona kuti paavhareji shard inorema anenge 20 gigabytes, uye kune 1 shards per index. Saizvozvo, kana tikaatenderedza kamwe pamaawa 360 ega ega, saka tine gumi neshanu awo. Imwe neimwe index ine data kwemazuva maviri.
Data kunyora uye kuverenga matunhu
Ngationei kuti data inorekodhwa sei mune ino system.
Ngatitii chimwe chikumbiro chinosvika kubva kuGraylog kune murongi. Semuenzaniso, tinoda kuisa mitsara 2-3 zviuru.
Murongi, agamuchira chikumbiro chakabva kuna Graylog, anobvunza tenzi wacho kuti: “Muchikumbiro che indexing, takanyatsodudza indekisi, asi kuti ndeipi shard yokunyora haina kutaurwa.”
Tenzi anopindura achiti: "Nyora ruzivo urwu kune shard nhamba 71," mushure mezvo inotumirwa zvakananga kune yakakodzera data node, uko primary-shard nhamba 71 iripo.
Mushure meiyo dhizaini yekutengeserana inodzokororwa kune replica-shard, iyo iri mune imwe data data.
Chikumbiro chekutsvaga chinosvika kubva kuGreylog kune murongi. Murongi anoidzosera zvinoenderana neindex, ukuwo Elasticsearch ichigovera zvikumbiro pakati peprimary-shard uye replica-shard vachishandisa iyo round-robin musimboti.
Iyo 180 node inopindura zvisina kuenzana, uye pavanenge vachipindura, murongi ari kuunganidza ruzivo rwakatove "rakapfirwa" nekukurumidza data node. Mushure meizvi, kana ruzivo rwese rwasvika, kana chikumbiro chasvika pakupera kwenguva, chinopa zvese zvakananga kumutengi.
Iyi yese system paavhareji inoita mibvunzo yekutsvaga kwemaawa makumi mana nemasere ekupedzisira mu48-300ms, kusasanganisa iyo mibvunzo ine inotungamira wildcard.
Maruva ane Elasticsearch: Java setup
Kuita kuti zvese zvishande nenzira yataida pakutanga, takapedza nguva yakareba tichigadzirisa zvinhu zvakasiyana-siyana musumbu.
Chikamu chekutanga chematambudziko akawanikwa chaive chine chekuita nenzira iyo Java inofanogadziriswa nekusarudzika muElasticsearch.
Dambudziko rimwe
Isu takaona huwandu hukuru hwemishumo yekuti padanho reLucene, kana mabasa ekumashure achimhanya, chikamu cheLucene chinosanganiswa chinotadza nekukanganisa. Panguva imwecheteyo, zvaive pachena mumatanda kuti iyi yaive OutOfMemoryError kukanganisa. Takaona kubva patelemetry kuti chiuno chaive chemahara, uye hazvina kujeka kuti sei kuvhiya uku kwanga kukundikana.
Zvakazoitika kuti Lucene index merges inoitika kunze kwehudyu. Uye midziyo yakanyatso kuganhurirwa maererano nezviwanikwa zvinodyiwa. Murwi chete waigona kukwana mune izvi zviwanikwa (iyo heap.size kukosha yaive yakaenzana ne RAM), uye mamwe maoparesheni emurwi akapunzika nekanganiso yekugovera yekurangarira kana nekuda kwechimwe chikonzero asina kukwana mu ~ 500MB yakasara isati yasvika.
Kugadzirisa kwacho kwaive kudiki: huwandu hwe RAM huripo hwechigadziko hwakawedzerwa, mushure mezvo takakanganwa kuti isu takatombova nematambudziko akadaro.
Dambudziko rechipiri
4-5 mazuva mushure mekutangwa kwesumbu, takaona kuti data node dzakatanga kudonha nguva nenguva kubva musumbu uye kupinda mairi mushure me10-20 masekonzi.
Patakatanga kuzvifunga, zvakazoitika kuti iyi off-murwi ndangariro muElasticsearch haidzorwe chero nzira. Patakapa imwe ndangariro kumudziyo, takakwanisa kuzadza madziva akananga buffer neruzivo rwakasiyana, uye yakacheneswa chete mushure mekunge GC yakajeka yatangwa kubva kuElasticsearch.
Mune zvimwe zviitiko, kuvhiya uku kwakatora nguva yakareba, uye panguva iyi sumbu rakakwanisa kumaka iyi node seyatobuda. Dambudziko iri rinotsanangurwa zvakanaka .
Mhinduro yaive seiyi: isu takaganhurira kugona kweJava kushandisa yakawanda yendangariro kunze kwemurwi wekuita uku. Isu takaiganhura kune gumi nematanhatu gigabytes (-XX:MaxDirectMemorySize = 16g), kuve nechokwadi kuti yakajeka GC yakadanwa kakawanda uye yakagadziriswa nekukurumidza, nekudaro haichakonzeresa sumbu.
Dambudziko rechitatu
Kana iwe uchifunga kuti matambudziko ane "node anosiya sumbu panguva isingatarisirwe" apera, uri kukanganisa.
Patakagadzirisa basa nemaindex, takasarudza mmapfs kuti pamashadhi matsva ane kupatsanurwa kukuru. Uku kwaive kukanganisa, nekuti kana uchishandisa mmapfs faira rinoiswa mu RAM, tobva tashanda nemepu faira. Nekuda kweizvi, zvinoitika kuti kana iyo GC ichiedza kumisa tambo mukushandisa, tinoenda kunzvimbo yekuchengetedza kwenguva yakareba, uye munzira kuenda nayo, chikumbiro chinomira kupindura zvikumbiro zvatenzi nezve kuti mupenyu. . Saizvozvo, tenzi anotenda kuti node haisisiripo musumbu. Mushure meizvi, mushure memasekonzi 5-10, muunganidzi wemarara anoshanda, node inouya kuhupenyu, inopinda musumbu zvakare uye inotanga kutanga shards. Izvo zvese zvainzwa zvakanyanya se "kugadzirwa kwatakakodzera" uye kwakanga kusina kukodzera chero chinhu chakakomba.
Kuti tibvise maitiro aya, isu takatanga kuchinjika kune yakajairika niofs, uyezve, patatama kubva kune yechishanu shanduro yeElastic kuenda kune yechitanhatu, takaedza mahybridfs, apo dambudziko iri harina kudzokororwa. Unogona kuverenga zvakawanda pamusoro pemhando dzekuchengetedza .
Dambudziko rechina
Ipapo kwakava nerimwe dambudziko rinofadza zvikuru iro takarapa kwenguva rekodhi. Takaibata kwemwedzi 2-3 nekuti maitiro ayo aive asinganzwisisike zvachose.
Dzimwe nguva varongi vedu vaienda kuFull GC, kazhinji imwe nguva mushure mekudya kwemasikati, uye havana kuzombodzoka kubva ipapo. Panguva imwecheteyo, pakutema GC kunonoka, zvaiita sezvizvi: zvese zviri kufamba zvakanaka, zvakanaka, zvakanaka, uye ipapo ipapo zvese zviri kufamba zvakanyanya.
Pakutanga takafunga kuti isu tine mushandisi akaipa aivhura imwe mhando yekukumbira iyo yakagogodza murongi kubva pakushanda. Takanyorerana zvikumbiro kwenguva refu kwazvo, tichiedza kufunga kuti chii chaiitika.
Nekuda kweizvozvo, zvakazoitika kuti panguva iyo mushandisi paanotanga chikumbiro chakakura, uye inosvika kune yakananga Elasticsearch murongi, dzimwe node dzinopindura kwenguva refu kupfuura dzimwe.
Uye apo murongi ari kumirira mhinduro kubva kune nodes dzose, anounganidza migumisiro yakatumirwa kubva kune node dzakatopindura. Kune GC, izvi zvinoreva kuti maitiro edu ekushandisa murwi anoshanduka nekukurumidza. Uye iyo GC yatakashandisa haina kukwanisa kuita basa iri.
Kugadzirisa chete kwatakawana kushandura maitiro echikwata mune ino mamiriro kutamira kuJDK13 uye kushandiswa kweShenandoah muunganidzi wemarara. Izvi zvakagadzirisa dambudziko, varongi vedu vakamira kudonha.
Apa ndipo pakapera matambudziko neJava uye matambudziko ebandwidth akatanga.
"Mabheri" ane Elasticsearch: throughput
Matambudziko nekubuda anoreva kuti cluster yedu inoshanda zvakatsiga, asi pakakwirira muhuwandu hwemagwaro akaiswa uye panguva yekufambisa, kuita hakuna kukwana.
Chiratidzo chekutanga chakasangana: panguva yeimwe "kuputika" mukugadzirwa, apo nhamba huru kwazvo yematanda inogadzirwa kamwe kamwe, iyo indexing error es_rejected_execution inotanga kupenya kakawanda muGreylog.
Izvi zvakakonzerwa nekuti thread_pool.write.queue pane imwe data node, kusvikira nguva Elasticsearch inokwanisa kugadzirisa chikumbiro che indexing uye kuisa ruzivo kune shard pa diski, inokwanisa kuchengetedza zvikumbiro mazana maviri chete nekutadza. Uye mu Zvakanyanya zvishoma zvinotaurwa nezve iyi parameter. Chete nhamba yepamusoro yeshinda uye saizi yekusagadzika ndiyo inotaridzwa.
Ehe, takaenda kunomonyorora kukosha uku uye takaona zvinotevera: kunyanya, mukuseta kwedu, zvinosvika mazana matatu zvikumbiro zvakachengetwa zvakanaka, uye kukosha kwepamusoro kuzere nenyaya yekuti tinobhururuka zvakare muFull GC.
Pamusoro pezvo, sezvo aya ari mabheji emeseji anosvika mukati mechikumbiro chimwe chete, zvaive zvakakodzera kuti tiite Graylog kuitira kuti inyore kwete kazhinji uye mumabheji madiki, asi mumabheji makuru kana kamwe chete masekonzi matatu kana batch isati yakwana. Muchiitiko ichi, zvinoitika kuti ruzivo rwatinonyora muElasticsearch runowanika kwete mumasekonzi maviri, asi mumashanu (izvo zvinotikodzera chaizvo), asi huwandu hwemaretrays anofanirwa kuitwa kuitira kusunda kuburikidza nehombe. murwi weruzivo wakaderedzwa.
Izvi zvinonyanya kukosha munguva idzodzo kana chimwe chinhu charovera pane imwe nzvimbo uye nehasha inoshuma nezvazvo, kuitira kuti urege kuwana zvachose spammed Elastic, uye mushure menguva yakati - Graylog node isingashande nekuda kwekuvhara mabhafa.
Mukuwedzera, patakanga tine kuputika kwakafanana mukugadzirwa, takagamuchira zvichemo kubva kune vanogadzira mapurogiramu uye testers: panguva iyo ivo vaida chaizvo matanda aya, vakapiwa zvishoma nezvishoma.
Vakatanga kufunga nezvazvo. Kune rimwe divi, zvaive pachena kuti yese mibvunzo yekutsvaga uye mibvunzo ye indexing yakagadziriswa, zvakanyanya, pamakina mamwe chete emuviri, uye neimwe nzira kana imwe paizove nekumwe kudhirowa.
Asi izvi zvinogona kutenderedzwa zvishoma nekuda kwekuti mushanduro yechitanhatu yeElasticsearch, algorithm yakaoneka iyo inokutendera kuti ugovane mibvunzo pakati peakakodzera data node kwete zvinoenderana neyakajairwa kutenderera-robin musimboti (mudziyo unoisa indexing uye unobata yekutanga. -shard inogona kunge yakabatikana zvakanyanya, hapazovi nekupindura nekukurumidza), asi kutumira chikumbiro ichi kumudziyo usina kutakurwa une replica-shard, iyo ichapindura nekukurumidza. Nemamwe manzwi, tasvika pa use_adaptive_replica_selection: chokwadi.
Mufananidzo wekuverenga unotanga kutaridzika seizvi:
Shanduko kune iyi algorithm yakaita kuti zvikwanise kuvandudza zvakanyanya nguva yekubvunza munguva idzodzo patakanga tine kuyerera kukuru kwematanda ekunyora.
Pakupedzisira, dambudziko guru raive kubviswa kusingarwadzi kwe data data.
Zvataida kubva musumbu pakarepo mushure mekurasikirwa nekubatana neimwe DC:
- Kana isu tine tenzi wezvino mune yakakundikana data center, zvino ichave yakasarudzwa zvakare uye inotamiswa sechikamu kune imwe node mune imwe DC.
- Tenzi achakurumidza kubvisa node dzose dzisingasviki kubva musumbu.
- Kubva pane zvakasara, achanzwisisa: munzvimbo yakarasika yedata takanga tine shards yakadaro uye yakadaro, achakurumidza kusimudzira complementary replica shards munzvimbo dzakasara dze data, uye isu tichaenderera mberi tichinyora data.
- Somugumisiro weizvi, kunyora kwechikwata uye kuverenga kwekuverenga kuchaderera zvishoma nezvishoma, asi kazhinji zvinhu zvose zvichashanda, zvishoma nezvishoma, asi zvakagadzikana.
Sezvakazoitika, isu taida chimwe chinhu chakadai:
Uye isu tine zvinotevera:
Izvi zvakaitika sei?
Pakadonha nzvimbo yedata, tenzi wedu akave bhodhoro.
Sei?
Icho chokwadi ndechekuti tenzi ane TaskBatcher, iyo ine basa rekugovera mamwe mabasa uye zviitiko musumbu. Chero node kubuda, chero kukwidziridzwa kwe shard kubva ku replica kuenda kupuraimari, chero basa rekugadzira shard kumwe kunhu - zvese izvi zvinoenda kutanga kuTaskBatcher, uko kunogadziriswa zvakateedzana uye mune imwe tambo.
Panguva yekubviswa kweimwe nzvimbo yedata, zvakazoitika kuti ese madhata munzvimbo dzasara dzedhata dzakaona sebasa ravo kuzivisa tenzi "tarasikirwa neakadaro neakadaro uye akadai neakadaro data node."
Panguva imwecheteyo, iyo yakasara data node yakatumira ruzivo urwu kune tenzi wezvino uye akaedza kumirira kusimbiswa kuti akaigamuchira. Havana kumirira ikoku, sezvo tenzi akagamuchira mabasa nokukurumidza kupfuura aigona kupindura. Manodhi akapera nguva yekudzokorora zvikumbiro, uye tenzi panguva iyi haana kana kumboedza kuzvipindura, asi akabatikana zvizere nebasa rekugadzirisa zvikumbiro nekutanga.
Mune fomu yekupedzisira, zvakazoitika kuti data node spammed tenzi kusvika padanho rekuti yakapinda muGC yakazara. Mushure meizvozvo, basa redu ramambo rakatamira kune imwe nhanho inotevera, zvakafanana zvakaitika kwairi, uye semhedzisiro, sumbu racho rakadonha zvachose.
Takatora zviyero, uye tisati tasvika vhezheni 6.4.0, apo izvi zvakagadziriswa, zvaive zvakakwana kuti isu panguva imwe chete tibudise gumi chete data node kubva pa10 kuti tivhare zvachose chisumbu.
Zvaiita seizvi:
Mushure meshanduro 6.4.0, apo iyi bug yakashata yakagadziriswa, data node dzakamira kuuraya tenzi. Asi izvozvo hazvina kumuita "akachenjera." Zvinonzi: patinoburitsa 2, 3 kana 10 (chero nhamba isiri imwe) data node, tenzi anogamuchira imwe meseji yekutanga inoti node A yasiya, uye anoedza kuudza node B, node C nezveizvi, node D.
Uye panguva ino, izvi zvinogona kugadziriswa chete nekuisa nguva yekuedza kuudza mumwe munhu nezvechimwe chinhu, yakaenzana ne20-30 seconds, uye nokudaro kudzora kumhanya kwe data center ichibva kunze kweboka.
Muchidimbu, izvi zvinopindirana nezvinodiwa zvakatanga kuratidzwa kune chigadzirwa chekupedzisira sechikamu chepurojekiti, asi kubva pakuona kwe "pure science" iyi ibug. Izvo, nenzira, yakagadziriswa zvakabudirira nevagadziri muvhezheni 7.2.
Uyezve, kana imwe node yedata yakabuda, zvakazoitika kuti kuparadzira ruzivo nezve kubuda kwayo kwakakosha pane kuudza boka rese kuti pane akadaro uye akadai ekutanga-shards pairi (kuitira kusimudzira replica-shard mune imwe data. pakati pepuraimari, uye muruzivo rwaigona kunyorwa paari).
Naizvozvo, kana zvese zvatofa pasi, iyo data yakaburitswa node haina kubva yaiswa chiratidzo sechinyakare. Saizvozvo, isu tinomanikidzwa kumirira kusvikira pings dzese dzapera kusvika kune dzakaburitswa data node, uye chete mushure meizvozvo cluster yedu inotanga kutiudza kuti ipapo, ipapo, uye ipapo isu tinoda kuenderera nekurekodha ruzivo. Unogona kuverenga zvakawanda pamusoro peizvi .
Nekuda kweizvozvo, kushanda kwekubvisa data data nhasi kunotitora anenge maminetsi mashanu panguva yekumhanyisa nguva. Kune yakakura uye yakaoma colossus, iyi mhedzisiro yakanaka.
Nekuda kweizvozvo, takasvika kune inotevera sarudzo:
- Tine 360 data nodes ane 700 gigabyte disks.
- 60 varongi vekufambisa traffic kuburikidza neaya ma data node.
- 40 masters atakasiya serudzi rwenhaka kubvira shanduro dzisati dzasvika 6.4.0 - kuti tipone pakubviswa kwenzvimbo yedata, isu takanga takagadzirira mupfungwa kurasikirwa nemichina yakati wandei kuitira kuti tive nechokwadi chekuve nequorum yevatenzi kunyangwe mu iyo yakaipisisa mamiriro ezvinhu
- Chero kuedza kusanganisa mabasa pamudziyo mumwe kwakasangana nenyaya yekuti pasina nguva node yaizoputsika pasi pemutoro.
- Iyo cluster yese inoshandisa heap.size ye31 gigabytes: kuedza kwese kudzikisa saizi kwakaguma nekuuraya mamwe ma node pazvinorema kutsvaga mibvunzo neinotungamira wildcard kana kuwana dunhu rinotyora muElasticsearch pacharo.
- Mukuwedzera, kuti tive nechokwadi chekushanda kwekutsvaga, takaedza kuchengetedza nhamba yezvinhu musumbu zvishoma sezvinobvira, kuitira kuti tigadzirise zviitiko zvishomanana sezvinobvira mubhodhoro ratakawana muna tenzi.
Pakupedzisira nezve monitoring
Kuti tive nechokwadi chekuti zvese izvi zvinoshanda sezvakarongwa, isu tinotarisa zvinotevera:
- Imwe neimwe data node inoshuma kune gore redu kuti ririko, uye kune akadaro uye akadaro shards pairi. Patinodzima chimwe chinhu pane imwe nzvimbo, cluster inoshuma mushure memasekondi 2-3 kuti pakati A takadzima node 2, 3, uye 4 - izvi zvinoreva kuti mune dzimwe nzvimbo dze data isu pasina mamiriro ezvinhu tinogona kudzima node idzo pane shard imwe chete. ruboshwe.
- Kuziva chimiro chemaitiro atenzi, tinotarisa zvakanyanya pahuwandu hwemabasa akamirira. Nekuti kunyangwe basa rimwechete rakanamira, kana rikasapera nenguva, dzidziso mune imwe mamiriro ekukurumidzira inogona kuva chikonzero nei, semuenzaniso, kukwidziridzwa kweiyo replica shard mukutanga kusashanda, ndosaka indexing icharega kushanda.
- Isu tinotarisisawo zvakanyanya kunonoka kunhonga marara, nekuti isu takatove nematambudziko makuru neizvi panguva yekugadzirisa.
- Inoramba neshinda kuti unzwisise pachine nguva kuti bhodhoro riripi.
- Zvakanaka, zviyero zvakajairwa senge murwi, RAM uye I/O.
Paunenge uchivaka yekutarisa, iwe unofanirwa kufunga nezve maficha eThread Pool muElasticsearch. inotsanangura magadzirirwo esarudzo uye tsika dzekutanga dzekutsvaga nekuisa indexing, asi yakanyarara zvachose nezve thread_pool.management.Idzi tambo dzinogadzira, kunyanya, mibvunzo yakaita se _katsi/shards nemamwe akafanana, ayo ari nyore kushandisa pakunyora monitoring. Kukura kweboka, ndiko kunowedzera zvikumbiro zvakadaro zvinoitwa pachikamu chenguva, uye thread_pool.management yambotaurwa haisi kungoiswa mugwaro repamutemo, asi inogumirawo nekusarudzika kune shinda 5, idzo dzinokurumidza kuraswa, mushure. iyo yekutarisa inomira kushanda nemazvo.
Zvandinoda kutaura mukupedzisa: takazviita! Isu takakwanisa kupa vagadziri vedu uye vanogadzira chishandiso icho, munenge chero mamiriro ezvinhu, anogona kukurumidza uye nekuvimbika kupa ruzivo nezve zviri kuitika mukugadzira.
Ehe, zvakazove zvakaoma, asi, zvakadaro, takakwanisa kuenderana nezvishuwo zvedu muzvigadzirwa zvaivepo, izvo zvatisina kufanira kupeta nekuzvinyora pachedu.
Source: www.habr.com