Abantu abaningi banenkinga nge-Elasticsearch. Kodwa kwenzekani uma ufuna ukuyisebenzisela ukugcina izingodo “ngevolumu enkulu ngokukhethekile”? Futhi ingabe futhi akubuhlungu ukuzwa ukwehluleka kwanoma yiziphi izikhungo zedatha eziningana? Hlobo luni lwezakhiwo okufanele ulwenze, futhi yiziphi izingibe ozokhubeka kuzo?
Thina kwa-Odnoklassniki sinqume ukusebenzisa i-elasticsearch ukuze sixazulule inkinga yokuphathwa kwelogi, futhi manje sebelana ngolwazi lwethu no-Habr: kokubili mayelana nezakhiwo kanye nezingibe.
NginguPyotr Zaitsev, ngisebenza njengomphathi wesistimu e-Odnoklassniki. Ngaphambi kwalokho, ngangibuye ngingumphathi, ngisebenza ne-Manticore Search, i-Sphinx search, i-Elasticsearch. Mhlawumbe, uma olunye ...ukusesha kuvela, ngizosebenza nakho futhi. Ngiphinde ngibambe iqhaza kumaphrojekthi amaningi omthombo ovulekile ngokuzithandela.
Lapho ngifika e-Odnoklassniki, ngasho ngokunganaki engxoxweni ukuthi ngingasebenza ne-Elasticsearch. Ngemva kokuba sengiyitholile futhi ngiqede imisebenzi ethile elula, nganikezwa umsebenzi omkhulu wokulungisa uhlelo lokuphathwa kwamalogi olwalukhona ngaleso sikhathi.
izidingo
Izidingo zesistimu zakhiwe kanje:
- I-Greylog yayizosetshenziswa njenge-frontend. Ngenxa yokuthi inkampani isivele inolwazi lokusebenzisa lo mkhiqizo, abahleli bezinhlelo nabahloli babekwazi, bekujwayelekile futhi kulula kubo.
- Ivolumu yedatha: ngokwesilinganiso imiyalezo eyizinkulungwane ezingama-50-80 ngomzuzwana, kodwa uma okuthile kuphuka, khona-ke ithrafikhi ayinqunyelwe yinoma yini, ingaba imigqa eyizigidi ezingu-2-3 ngomzuzwana.
- Ngemva kokuxoxisana namakhasimende ngezidingo zejubane lokucubungula imibuzo yosesho, saqaphela ukuthi iphethini evamile yokusebenzisa uhlelo olunjalo yilena: abantu bafuna izingodo zesicelo sabo ezinsukwini ezimbili ezedlule futhi abafuni ukulinda ngaphezu kwe okwesibili ngomphumela wombuzo owenziwe.
- Abaphathi bagcizelele ukuthi uhlelo lunyuke kalula uma kunesidingo, ngaphandle kokudinga ukuthi bajule ukuthi lusebenza kanjani.
- Ukuze okuwukuphela komsebenzi wokulungisa odingwa yilezi zinhlelo ngezikhathi ezithile ukushintsha ihadiwe ethile.
- Ngaphezu kwalokho, i-Odnoklassniki inomkhuba omuhle kakhulu wezobuchwepheshe: noma iyiphi isevisi esiyiqalayo kufanele iphumelele ekuhlulekeni kwesikhungo sedatha (ngokungazelelwe, okungahleliwe futhi nganoma yisiphi isikhathi).
Isidingo sokugcina ekuqalisweni kwalo msebenzi sibize kakhulu, engizokhuluma ngakho kabanzi.
Environment
Sisebenza ezikhungweni zedatha ezine, kuyilapho amanodi edatha e-Elasticsearch angatholakala kuphela kokuthathu (ngenxa yezizathu ezimbalwa ezingezona ezobuchwepheshe).
Lezi zikhungo ezine zedatha ziqukethe cishe imithombo yelogi ehlukahlukene eyizinkulungwane eziyi-18 - ihadiwe, iziqukathi, imishini ebonakalayo.
Isici esibalulekile: iqoqo liqala ezitsheni hhayi emishinini yomzimba, kodwa kuqhubeke . Iziqukathi ziqinisekisiwe ama-cores angu-2, afana ne-2.0Ghz v4, okungenzeka ukuthi kusetshenziswe kabusha ama-cores asele uma engenzi lutho.
Ngamanye amazwi:
I-Topology
Ngiqale ngabona indlela ejwayelekile yesixazululo kanje:
- Ama-VIP angu-3-4 angemuva kwerekhodi le-A lesizinda se-Greylog, leli ikheli lapho amalogi athunyelwa khona.
- I-VIP ngayinye iyibhalansi ye-LVS.
- Ngemuva kwayo, izingodo ziya ebhethri ye-Greylog, enye idatha ifomethi ye-GELF, enye ngefomethi ye-syslog.
- Bese konke lokhu kubhalwa ngamaqoqo amakhulu kubhethri labaxhumanisi be-Elasticsearch.
- Futhi bona, bathumela ukubhala nokufunda izicelo kumanodi wedatha afanele.
I-terminology
Mhlawumbe akuwona wonke umuntu oqonda amagama ngokuningiliziwe, ngakho-ke ngingathanda ukuhlala kuwo kancane.
I-Elasticsearch inezinhlobo ezimbalwa zama-node - master, umxhumanisi, idatha yedatha. Kunezinye izinhlobo ezimbili zokuguqulwa okuhlukile kwelogi nokuxhumana phakathi kwamaqoqo ahlukene, kodwa sisebenzise lezo ezisohlwini kuphela.
Master
Icindezela wonke ama-node akhona kuqoqo, igcina imephu yeqoqo esesikhathini samanje futhi isabalalise phakathi kwama-node, icubungule ingqondo yomcimbi, futhi yenza izinhlobo ezahlukene zokugcinwa kwendlu okubanzi kweqoqo.
Umxhumanisi
Yenza umsebenzi owodwa owodwa: yamukela izicelo zokufunda noma ukubhala ezivela kumakhasimende kanye nemizila yale thrafikhi. Uma kwenzeka kunesicelo sokubhala, cishe, izobuza inkosi ukuthi iyiphi ingxenye yenkomba efanele okufanele isifake kuyo, futhi izoqondisa kabusha isicelo ngokuqhubekayo.
Inombolo yedatha
Igcina idatha, yenza imibuzo yosesho evela ngaphandle futhi yenze imisebenzi kumashadi atholakala kuwo.
Mpunga
Lokhu kufana nokuhlanganiswa kwe-Kibana ne-Logstash kusitaki se-ELK. I-Greylog ihlanganisa kokubili i-UI kanye nepayipi lokucubungula ilogi. Ngaphansi kwe-hood, i-Greylog isebenzisa i-Kafka ne-Zookeeper, ehlinzeka ngokuxhumeka ku-Greylog njengeqoqo. I-Greylog ingakwazi ukugcina izingodo (i-Kafka) uma kwenzeka i-Elasticsearch ingatholakali futhi iphinda izicelo zokufunda nokubhala ezingaphumelelanga, iqoqo futhi imake izingodo ngokuvumelana nemithetho ethile. Njenge-Logstash, i-Greylog inomsebenzi wokushintsha imigqa ngaphambi kokuyibhalela ku-Elasticsearch.
Ngaphezu kwalokho, i-Graylog inokutholwa kwesevisi eyakhelwe ngaphakathi evumela, ngokusekelwe endaweni eyodwa etholakalayo ye-Elasticsearch, ukuthola yonke imephu yeqoqo futhi ihlunge ngethegi ethile, okwenza kube nokwenzeka ukuqondisa izicelo ezitsheni ezithile.
Ngokubukeka kubukeka kanjena:
Lesi yisithombe-skrini esivela kusibonelo esithile. Lapha sakha i-histogram esekelwe embuzweni wosesho futhi sibonisa imigqa efanele.
Izinkomba
Uma sibuyela ekwakhiweni kwesistimu, ngingathanda ukuhlala ngemininingwane eminingi yokuthi sakha kanjani imodeli yenkomba ukuze yonke isebenze kahle.
Kumdwebo ongenhla, leli izinga eliphansi kunawo wonke: Elasticsearch data nodes.
Inkomba yibhizinisi elikhulu elibonakalayo elenziwe amashadi e-Elasticsearch. Ngokwayo, i-shards ngayinye ayiyona into engaphezu kwenkomba ye-Lucene. Futhi inkomba ngayinye ye-Lucene, yona, iqukethe ingxenye eyodwa noma ngaphezulu.
Lapho siklama, sicabange ukuthi ukuze sihlangabezane nemfuneko yesivinini sokufunda enanini elikhulu ledatha, kwakudingeka "sisabalalise" le datha ngokulinganayo kuwo wonke amanodi edatha.
Lokhu kuphumele ekutheni inani lamashadi kunkomba ngayinye (enezifaniso) kufanele lilingane ngokuqinile nenani lamanodi edatha. Okokuqala, ukuze kuqinisekiswe isici sokuphindaphinda esilingana nokubili (okungukuthi, singalahlekelwa uhhafu weqoqo). Futhi, okwesibili, ukuze kucutshungulwe izicelo zokufunda nokubhala okungenani engxenyeni yeqoqo.
Siqale sanquma isikhathi sesitoreji njengezinsuku ezingama-30.
Ukusatshalaliswa kwama-shards kungamelwa ngezithombe kanje:
Wonke unxande omnyama ompunga uyinkomba. Isikwele esibomvu esingakwesokunxele kuso yi-shard eyinhloko, eyokuqala kunkomba. Futhi isikwele esiluhlaza okwesibhakabhaka siyi-replica shard. Atholakala ezikhungweni zedatha ezahlukene.
Uma sengeza enye i-shard, iya esikhungweni sedatha sesithathu. Futhi, ekugcineni, sithola lesi sakhiwo, okwenza kube lula ukulahlekelwa i-DC ngaphandle kokulahlekelwa ukungaguquguquki kwedatha:
Ukujikeleza kwezinkomba, i.e. ukudala inkomba entsha futhi sisusa endala kunazo zonke, siyenze yalingana namahora angu-48 (ngokwephethini yokusetshenziswa kwenkomba: amahora angu-48 okugcina aseshwa kakhulu).
Lesi sikhawu sokuzungezisa inkomba kungenxa yezizathu ezilandelayo:
Uma isicelo sokusesha sifika endaweni ethile yedatha, ngakho-ke, kusukela ekubukeni kokusebenza, kunenzuzo enkulu uma i-shard eyodwa ibuzwa, uma ubukhulu bayo buqhathaniswa nobukhulu be-hip ye-node. Lokhu kukuvumela ukuthi ugcine ingxenye yenkomba "eshisayo" ibe yinqwaba futhi uyifinyelele ngokushesha. Uma kukhona "izingxenye ezishisayo" eziningi, isivinini sokusesha inkomba siyancipha.
Uma inodi iqala ukwenza umbuzo wosesho kushadi olulodwa, inikezela ngenani lemicu elingana nenani lama-hyperthreading cores omshini obonakalayo. Uma umbuzo wosesho uthinta inombolo enkulu yamashadi, khona-ke inani lezintambo likhula ngokulinganayo. Lokhu kunomthelela ongemuhle esivinini sokusesha futhi kuthinta kabi ukukhonjwa kwedatha entsha.
Ukuze sinikeze ukubambezeleka kokusesha okudingekayo, sinqume ukusebenzisa i-SSD. Ukuze kucutshungulwe izicelo ngokushesha, imishini eyayisingatha la mabhokisi kwakudingeka okungenani ibe nama-cores angama-56. Inani elingu-56 likhethwe njengenani elanele elinemibandela elinquma inani lemicu ezokhiqizwa yi-Elasticsearch ngesikhathi sokusebenza. Ku-Elasitcsearch, amapharamitha amaningi we-thread pool ancike ngqo enanini lama-cores atholakalayo, nawo athinta ngqo inombolo edingekayo yama-node ku-cluster ngokusho kwesimiso "ama-cores ambalwa - ama-node amaningi".
Ngenxa yalokho, sithole ukuthi ngokwesilinganiso i-shard inesisindo esingaba ngu-20 gigabytes, futhi kukhona ama-shards angu-1 inkomba ngayinye. Ngakho-ke, uma siwazungeza kanye njalo emahoreni angama-360, sizoba nayi-48. Inkomba ngayinye iqukethe idatha yezinsuku ezingu-15.
Izifunda zokubhala nokufunda idatha
Ake sithole ukuthi idatha irekhodwa kanjani kulolu hlelo.
Ake sithi isicelo esithile sifika sivela kuGreylog siya kumxhumanisi. Isibonelo, sifuna ukukhomba imigqa eyizinkulungwane ezingu-2-3.
Umxhumanisi, ngemva kokuthola isicelo esivela ku-Graylog, ubuza umphathi: “Esicelweni senkomba, siyicacisile inkomba, kodwa ukuthi i-shard okufanele siyibhale kuphi ayizange ichazwe.”
Inkosi iyaphendula: "Bhala lolu lwazi kunombolo ye-shard 71," emva kwalokho ithunyelwa ngqo ku-node yedatha efanele, lapho inombolo eyinhloko-shard 71 ikhona.
Ngemva kwalokho ilogi yokwenziwe iphindwa ku-replica-shard, etholakala kwesinye isikhungo sedatha.
Isicelo sokusesha sifika sisuka kuGreylog siya kumxhumanisi. Umxhumanisi uyiqondisa kabusha ngokwenkomba, kuyilapho i-Elasticsearch isabalalisa izicelo phakathi kwe-primary-shard kanye ne-replica-shard isebenzisa isimiso se-round-robin.
Ama-node angu-180 aphendula ngokungalingani, futhi ngenkathi ephendula, umxhumanisi uqongelela ulwazi oseluvele "luchithiwe" ngama-data node asheshayo. Ngemuva kwalokhu, lapho lonke ulwazi selufikile, noma isicelo sesifinyelele isikhathi sokuvala, sinikeza yonke into ngqo kuklayenti.
Yonke le sistimu ngokwesilinganiso icubungula imibuzo yosesho emahoreni angu-48 okugcina ku-300-400ms, ngaphandle kwaleyo mibuzo ene-wildcard ehamba phambili.
Izimbali ezine-Elasticsearch: Ukusethwa kwe-Java
Ukwenza konke kusebenze ngendlela ebesiyifuna ngayo ekuqaleni, sichithe isikhathi eside kakhulu silungisa amaphutha ezinhlobonhlobo zezinto kuqoqo.
Ingxenye yokuqala yezinkinga ezitholiwe yayihlobene nendlela i-Java ecushwa ngayo ngaphambili ngokuzenzakalelayo ku-Elasticsearch.
Inkinga eyodwa
Sibone inombolo enkulu kakhulu yemibiko yokuthi ezingeni le-Lucene, uma imisebenzi yangemuva isebenza, ukuhlangana kwengxenye ye-Lucene kwehluleka ngephutha. Ngesikhathi esifanayo, kwacaca kulogi ukuthi leli iphutha le-OutOfMemoryError. Sibonile nge-telemetry ukuthi inqulu imahhala, futhi akucaci ukuthi kungani lo msebenzi uhluleka.
Kuvele ukuthi ukuhlangana kwe-Lucene kwenzeka ngaphandle kwe-hip. Futhi iziqukathi zilinganiselwe impela ngokwezinsiza ezisetshenziswayo. Inqwaba kuphela engangena kulezi zinsiza (inani le-heap.size lalicishe lilingane ne-RAM), futhi eminye imisebenzi engaphandle kwenqwaba iphahlazeke ngephutha lokwabiwa kwememori uma ngesizathu esithile ingangeni ku-~500MB esele ngaphambi komkhawulo.
Ukulungiswa bekuyinto encane kakhulu: inani le-RAM etholakalayo esitsheni lanyuswa, ngemva kwalokho sakhohlwa ukuthi saba nezinkinga ezinjalo.
Inkinga yesibili
Ezinsukwini ezingu-4-5 ngemva kokwethulwa kweqoqo, saqaphela ukuthi ama-node edatha aqala ukuphuma ngezikhathi ezithile kuqoqo futhi ayifake ngemva kwemizuzwana engu-10-20.
Lapho siqala ukukuthola, kwavela ukuthi le nkumbulo engekho ku-Elasticsearch ayilawulwa nganoma iyiphi indlela. Lapho sinikeza inkumbulo eyengeziwe esitsheni, sakwazi ukugcwalisa amachibi ebhafa aqondile ngolwazi oluhlukahlukene, futhi yasulwa kuphela ngemva kokwethulwa kwe-GC ecacile ku-Elasticsearch.
Kwezinye izimo, lokhu kusebenza kwathatha isikhathi eside, futhi phakathi nalesi sikhathi iqoqo likwazile ukumaka le nodi njengevele isiphumile. Le nkinga ichazwe kahle .
Isixazululo sibe ngale ndlela elandelayo: silinganisele ikhono le-Java lokusebenzisa inqwaba yememori ngaphandle kwenqwaba kule misebenzi. Siyikhawulele kumagigabhayithi angu-16 (-XX:MaxDirectMemorySize=16g), siqinisekisa ukuthi i-GC ecacile ibibizwa kaningi futhi icutshungulwa ngokushesha okukhulu, ngaleyo ndlela ingabe isathunaza iqoqo.
Inkinga yesithathu
Uma ucabanga ukuthi izinkinga "ezinama-node ashiya iqoqo ngesikhathi esingalindelekile" seziphelile, unephutha.
Lapho silungiselela umsebenzi ngezinkomba, sakhetha ama-mmapfs ukwenza kanjalo kumashadi amasha anokuhlukaniswa okukhulu. Lokhu bekuyiphutha impela, ngoba uma usebenzisa i-mmapfs ifayela lifakwe kumephu ye-RAM, bese sisebenza nefayela elimephu. Ngenxa yalokhu, kuvela ukuthi lapho i-GC izama ukumisa izintambo esicelweni, siya endaweni yokuphepha isikhathi eside kakhulu, futhi endleleni eya kuyo, isicelo siyayeka ukuphendula izicelo zenkosi mayelana nokuthi iyaphila. . Ngakho-ke, u-master ukholelwa ukuthi i-node ayisekho ku-cluster. Ngemva kwalokhu, ngemva kwemizuzwana engu-5-10, umqoqi kadoti usebenza, i-node iphila, iphinde ingene eqenjini futhi iqale ukuqalisa ama-shards. Konke kuzwakala kufana “nokukhiqizwa ebesikufanele” futhi bekungafaneleki kunoma yini engathi sína.
Ukuze sisuse lokhu kuziphatha, siqale sashintshela kuma-niofs ajwayelekile, bese kuthi lapho sisuka ezinguqulweni zesihlanu ze-Elastic siye kweyesithupha, sazama ama-hybridfs, lapho le nkinga ingazange yenziwe kabusha. Ungafunda kabanzi mayelana nezinhlobo zesitoreji .
Inkinga yesine
Kwabe sekukhona enye inkinga ethakazelisa kakhulu esiye sayiphatha isikhathi esirekhodiwe. Sayibamba izinyanga ezingu-2-3 ngoba iphethini yayo yayingaqondakali nhlobo.
Ngezinye izikhathi abaxhumanisi bethu baya ku-Full GC, ngokuvamile ngemva kwesidlo sasemini, futhi abaphindanga babuya lapho. Ngesikhathi esifanayo, lapho ugawula ukubambezeleka kwe-GC, kwakubukeka kanje: konke kuhamba kahle, kahle, kahle, bese kuthi ngokuzumayo yonke into ihamba kabi kakhulu.
Ekuqaleni besicabanga ukuthi sinomsebenzisi omubi owethula uhlobo oluthile lwesicelo olwakhipha umxhumanisi kumodi yokusebenza. Safaka izicelo isikhathi eside kakhulu, sizama ukuthola ukuthi kwenzekani.
Ngenxa yalokho, kuvele ukuthi ngesikhathi lapho umsebenzisi eqala isicelo esikhulu, futhi sifika kumxhumanisi othile we-Elasticsearch, amanye ama-node aphendula isikhathi eside kunamanye.
Futhi ngenkathi umxhumanisi elinde impendulo evela kuwo wonke ama-node, uqoqa imiphumela ethunyelwe kusukela kuma-node asevele ephendulile. Ku-GC, lokhu kusho ukuthi amaphethini ethu okusebenzisa inqwaba ashintsha ngokushesha kakhulu. Futhi uGC esimsebenzisile akakwazanga ukumelana nalo msebenzi.
Ukuphela kokulungiswa esikutholile ukushintsha ukuziphatha kweqoqo kulesi simo ukuthuthela ku-JDK13 kanye nokusetshenziswa komqoqi kadoti we-Shenandoah. Lokhu kwaxazulula inkinga, abaxhumanisi bethu bayeka ukuwa.
Yilapho izinkinga nge-Java zaphela khona futhi izinkinga zomkhawulokudonsa zaqala.
"Amajikijolo" ane-Elasticsearch: throughput
Izinkinga nge-throughput zisho ukuthi iqoqo lethu lisebenza ngokuzinza, kodwa ekuphakameni kwenani lamadokhumenti anenkomba nangesikhathi sokuqondisa, ukusebenza akwanele.
Uphawu lokuqala okuhlangatshezwane nalo: ngesikhathi esithile "sokuqhuma" ekukhiqizeni, lapho inani elikhulu kakhulu lamalogi likhiqizwa ngokuzumayo, iphutha lenkomba es_rejected_execution liqala ukukhanya njalo ku-Greylog.
Lokhu kwakungenxa yokuthi i-thread_pool.write.queue endaweni eyodwa yedatha, kuze kube yilapho i-Elasticsearch ikwazi ukucubungula isicelo senkomba futhi ilayishe ulwazi ku-shard kudiski, ikwazi ukugcina izicelo ezingu-200 kuphela ngokuzenzakalelayo. Futhi ku Kuncane kakhulu okushiwoyo ngale parameter. Inombolo enkulu kuphela yezintambo kanye nosayizi omisiwe okhonjisiwe.
Vele, siye ukusonta leli nani futhi sathola okulandelayo: ikakhulukazi, ekusetheni kwethu, izicelo ezifika ku-300 zigcinwe kahle, futhi inani eliphakeme ligcwele iqiniso lokuthi siphinde sindize singene ku-GC Egcwele.
Ngaphezu kwalokho, njengoba lawa amaqoqo emilayezo efika ngesicelo esisodwa, bekudingeka ukuthi kulungiswe i-Greylog ukuze ingabhali kaningi futhi ngamaqoqo amancane, kodwa ngamaqoqo amakhulu noma kanye njalo emizuzwaneni emi-3 uma inqwaba ingakaqedi. Kulokhu, kuvela ukuthi ulwazi esilubhala ku-Elasticsearch alutholakali emizuzwaneni emibili, kodwa kwemihlanu (esifanelekela kahle), kodwa inani lamareyidi okufanele enziwe ukuze kudlule isivinini esikhulu. inqwaba yolwazi iyancipha.
Lokhu kubaluleke kakhulu kulezo zikhathi lapho okuthile kuphahlazeke ndawana thize futhi kubika ngokufutheka ngakho, ukuze ungatholi i-Elastic enogaxekile ngokuphelele, futhi ngemva kwesikhathi esithile - ama-Graylog node angasebenzi ngenxa yamabhafa avaliwe.
Ngaphezu kwalokho, lapho sineziqhumane ezifanayo ekukhiqizeni, sathola izikhalazo ezivela kubahleli bezinhlelo nabahloli: okwamanje lapho bewadinga ngempela lawa malogi, anikezwa kancane kakhulu.
Baqala ukukuqonda. Ngakolunye uhlangothi, kwaba sobala ukuthi imibuzo yosesho kanye nemibuzo yezinkomba yacutshungulwa, empeleni, emishinini efanayo yomzimba, futhi ngandlela thile kuzoba nokudonswa phansi okuthile.
Kodwa lokhu kungase kugwenywe kancane ngenxa yokuthi ezinguqulweni zesithupha ze-Elasticsearch kwavela i-algorithm ekuvumela ukuthi usabalalise imibuzo phakathi kwamanodi edatha afanelekile hhayi ngokulandela isimiso esingahleliwe se-round-robin (isiqukathi esenza inkomba futhi sibambe okuyinhloko- I-shard ingaba matasa kakhulu, ngeke ibe khona indlela yokuphendula ngokushesha), kodwa ukudlulisela lesi sicelo esitsheni esincane esilayishiwe esine-replica-shard, esizophendula ngokushesha kakhulu. Ngamanye amazwi, sifike ku-use_adaptive_replica_selection: true.
Isithombe esifundwayo siqala ukubukeka kanje:
Ukushintshela kule algorithm kwenze kwaba nokwenzeka ukuthuthukisa ngokuphawulekayo isikhathi sombuzo ngalezo zikhathi lapho sibe nokugeleza okukhulu kwamalogi okufanele sikubhale.
Ekugcineni, inkinga enkulu kwaba ukususwa okungenabuhlungu kwesikhungo sedatha.
Ebesikufuna kuqoqo ngokushesha ngemuva kokulahlekelwa ukuxhumana ne-DC eyodwa:
- Uma sinompetha wamanje esikhungweni sedatha esihlulekile, sizophinda sikhethwe futhi sihanjiswe njengendima kwenye i-node kwenye i-DC.
- Inkosi izosusa ngokushesha wonke ama-node angafinyeleleki kuqoqo.
- Ngokusekelwe kokusele, uzoqonda: esikhungweni sedatha esilahlekile sasinama-shards anjalo nanjalo ayisisekelo, uzothuthukisa ngokushesha ama-replica shards ahambisanayo ezikhungweni zedatha ezisele, futhi sizoqhubeka sikhomba idatha.
- Ngenxa yalokhu, ukubhala nokufunda kweqoqo kuzokwehla kancane kancane, kodwa ngokuvamile yonke into izosebenza, nakuba ihamba kancane, kodwa ngokuzinza.
Njengoba kwenzeka, sifuna okuthile okufana nalokhu:
Futhi sithole okulandelayo:
Kwenzeka kanjani lokhu?
Lapho isikhungo sedatha siwa, inkosi yethu yaba ibhodlela.
Kungani?
Iqiniso liwukuthi inkosi ine-TaskBatcher, enesibopho sokusabalalisa imisebenzi ethile nemicimbi kuqoqo. Noma iyiphi i-node ephumayo, noma yikuphi ukunyuswa kwe-shard kusuka ku-replica kuya kweyinhloko, noma yimuphi umsebenzi wokudala i-shard ndawana thize - konke lokhu kuya kuqala ku-TaskBatcher, lapho kucutshungulwa khona ngokulandelana nangochungechunge olulodwa.
Ngesikhathi sokuhoxiswa kwesikhungo esisodwa sedatha, kuvele ukuthi wonke ama-data node ezikhungweni zedatha ezisekhona akuthatha njengomsebenzi wawo ukwazisa inkosi ukuthi “silahlekelwe ama-shards afana nalawa kanye nama-data node.”
Ngesikhathi esifanayo, ama-node wedatha asele athumele lonke lolu lwazi kumphathi wamanje futhi azama ukulinda ukuqinisekiswa ukuthi uyakwamukela. Abazange bakulinde lokhu, ngoba inkosi yayithola imisebenzi ngokushesha kunaleyo eyayikwazi ukuyiphendula. Ama-node aphelelwe yisikhathi izicelo eziphindaphindayo, futhi inkosi ngalesi sikhathi ayizange izame nokuziphendula, kodwa yayigxile ngokuphelele emsebenzini wokuhlunga izicelo kuqala.
Efomini letheminali, kuvele ukuthi ama-data node afaka ugaxekile ku-master kwaze kwaba seqophelweni lokuthi iye ku-GC egcwele. Ngemuva kwalokho, indima yethu eyinhloko yathuthela kwenye indawo elandelayo, kwenzeka into efanayo kuyo, futhi ngenxa yalokho iqoqo labhidlika ngokuphelele.
Sithathe izilinganiso, futhi ngaphambi kwenguqulo 6.4.0, lapho lokhu kwalungiswa khona, bekwanele ngathi ukuthi sikhiphe ama-data node angu-10 kuphela kwangu-360 ukuze sivale ngokuphelele iqoqo.
Bekubukeka kanjena:
Ngemva kwenguqulo 6.4.0, lapho lesi siphazamisi esibi salungiswa khona, ama-data node ayeka ukubulala umphathi. Kodwa lokho akuzange kumenze “ahlakaniphe.” Okungukuthi: lapho sikhipha 2, 3 noma 10 (noma iyiphi inombolo ngaphandle kweyodwa) amanodi edatha, inkosi ithola umlayezo othile wokuqala othi inodi A isishiyile, bese izama ukutshela inodi B, inodi C ngalokhu, inodi D.
Futhi okwamanje, lokhu kungabhekwana nakho kuphela ngokubeka isikhathi sokuvala semizamo yokutshela othile ngokuthile, okulingana namasekhondi angaba ngu-20-30, futhi ngaleyo ndlela ulawule isivinini sesikhungo sedatha esiphuma kuqoqo.
Empeleni, lokhu kuhambisana nezidingo ezethulwa ekuqaleni kumkhiqizo wokugcina njengengxenye yephrojekthi, kodwa ngokombono “wesayensi emsulwa” lokhu kuyisiphazamiso. Okuthi, ngendlela, eyalungiswa ngempumelelo ngabathuthukisi kunguqulo 7.2.
Ngaphezu kwalokho, lapho indawo ethile yedatha iphuma, kwavela ukuthi ukusabalalisa ulwazi mayelana nokuphuma kwayo kwakubaluleke kakhulu kunokutshela lonke iqoqo ukuthi kukhona ama-primary-shards kuwo (ukuze kuthuthukiswe i-replica-shard kwenye idatha. esikhungweni sokuqala, futhi ngolwazi kungabhalwa kuzo).
Ngakho-ke, lapho yonke into isivele ifile, ama-data node akhululiwe awamakwa ngokushesha njengakudala. Ngakho-ke, siphoqeleka ukuthi silinde kuze kube yilapho wonke ama-pings esephelelwe yisikhathi kuma-node edatha akhishwe, futhi ngemva kwalokho iqoqo lethu liqala ukusitshela ukuthi lapho, lapho, futhi lapho sidinga ukuqhubeka nokurekhoda ulwazi. Ungafunda okwengeziwe ngalokhu .
Ngenxa yalokho, ukusebenza kokukhipha isikhungo sedatha namuhla kusithatha cishe amaminithi angu-5 phakathi nehora lokushesha. Ku-colossus enkulu neyinkimbinkimbi kangaka, lona umphumela omuhle kakhulu.
Ngenxa yalokho, sifinyelele esinqumweni esilandelayo:
- Sinama-data node angu-360 anamadiski angu-700 gigabyte.
- Abaxhumanisi abangu-60 bokuhambisa ithrafikhi kulawa manodi wedatha afanayo.
- Izingcweti ezingu-40 esizishiye njengohlobo lwefa kusukela ezinguqulweni zangaphambi kuka-6.4.0 - ukuze sisinde ekuhoxisweni kwesikhungo sedatha, sasikulungele ngokwengqondo ukulahlekelwa imishini eminingana ukuze siqinisekiswe ukuthi sinekhoramu yamakhosi ngisho naku isimo esibi kakhulu
- Noma yimiphi imizamo yokuhlanganisa izindima esitsheni esisodwa yahlangatshezwa neqiniso lokuthi ngokushesha noma kamuva i-node izophuka ngaphansi komthwalo.
- Iqoqo lonke lisebenzisa i-heap.size yamagigabhayithi angu-31: yonke imizamo yokunciphisa usayizi iholele ekubulaleni amanodi athile emibuzweni yokusesha esindayo nge-wildcard ehamba phambili noma ukuthola i-circuit breaker ku-Elasticsearch ngokwayo.
- Ngaphezu kwalokho, ukuze siqinisekise ukusebenza kosesho, sizame ukugcina inani lezinto kuqoqo lincane ngangokunokwenzeka, ukuze sicubungule izehlakalo ezimbalwa ngangokunokwenzeka kubhodlela esilithole ku-master.
Ekugcineni mayelana nokuqapha
Ukuqinisekisa ukuthi konke lokhu kusebenza njengoba kuhlosiwe, siqapha okulandelayo:
- I-data node ngayinye ibika efwini lethu ukuthi ikhona, futhi kukhona ama-shards anjalo nanjalo kuyo. Uma sicisha okuthile ndawana thize, iqoqo libika ngemva kwamasekhondi angu-2-3 ukuthi phakathi nendawo A sicime amanodi 2, 3, no-4 - lokhu kusho ukuthi kwezinye izikhungo zedatha asikho ngaphansi kwezimo esingakwazi ukucisha lawo ma-node okunodindo olulodwa kuphela. kwesokunxele.
- Ukwazi uhlobo lokuziphatha kwenkosi, sibheka ngokucophelela inani lemisebenzi elindile. Ngoba ngisho nomsebenzi owodwa obambekile, uma ungaphelelwa yisikhathi, ngokwethiyori esimweni esithile esiphuthumayo kungaba isizathu sokuthi kungani, ngokwesibonelo, ukukhushulwa kwe-replica shard ekuqaleni kungasebenzi, yingakho ukukhomba kuzoyeka ukusebenza.
- Siphinde sibheke kakhulu ukubambezeleka kwabaqoqi bakadoti, ngoba sesivele saba nobunzima obukhulu ngalokhu ngesikhathi sokwenza kahle.
- Inqaba ngentambo ukuqonda kusenesikhathi lapho ibhodlela likuphi.
- Yebo, amamethrikhi ajwayelekile afana ne-heap, RAM kanye ne-I/O.
Lapho wakha ukuqapha, kufanele ucabangele izici ze-Thread Pool ku-Elasticsearch. ichaza izinketho zokumisa kanye namanani azenzakalelayo okusesha kanye nenkomba, kodwa ayithuli ngokuphelele mayelana ne-thread_pool.management. Le micu icubungula, ikakhulukazi, imibuzo efana nethi _cat/shards neminye efanayo, esebenziseka kalula lapho kubhalwa ukuqapha. Uma iqoqo lilikhulu, kulapho izicelo eziningi zenziwa khona ngeyunithi ngayinye yesikhathi, futhi i-thread_pool.management eshiwo ngenhla ayivezwanga kuphela emibhalweni esemthethweni, kodwa futhi ilinganiselwe ngokuzenzakalelayo emicucu emi-5, echithwa ngokushesha, ngemva kwalokho. okuyinto yokuqapha eyeka ukusebenza ngendlela efanele.
Engifuna ukukusho ekuphetheni: sikwenzile! Sikwazile ukunikeza abahleli bezinhlelo nabathuthukisi bethu ithuluzi, cishe kunoma yisiphi isimo, elingakwazi ukuhlinzeka ngolwazi ngokushesha nangokuthembekile mayelana nokuthi kwenzekani ekukhiqizeni.
Yebo, kube nzima impela, kepha, nokho, sikwazile ukulingana nezifiso zethu emikhiqizweni ekhona, okungadingeki ukuthi siyinamathisele futhi sizibhalele kabusha.
Source: www.habr.com