Abantu abaninzi bayasokola nge-Elasticsearch. Kodwa kwenzeka ntoni xa ufuna ukuyisebenzisa ukugcina iinkuni “kwivolumu enkulu kakhulu”? Kwaye ngaba akukho buhlungu ukufumana ukusilela kwawo nawaphi na amaziko amaninzi edatha? Loluphi uhlobo lolwakhiwo omele ulwenze, kwaye yeyiphi imigibe oya kukhubeka kuyo?
Thina e-Odnoklassniki sigqibe kwelokuba sisebenzise i-elasticsearch ukusombulula umba wolawulo lwelogi, kwaye ngoku sabelana ngamava ethu noHabr: zombini malunga noyilo lwezakhiwo kunye nemigibe.
NdinguPyotr Zaitsev, ndisebenza njengomlawuli wenkqubo e-Odnoklassniki. Ngaphambi koko, ndandingumlawuli, ndisebenza kunye neManticore Search, iSphinx search, Elasticsearch. Mhlawumbi, ukuba olunye ...uphendlo luvela, nam ndiza kusebenza nalo. Ndikwathatha inxaxheba kwinani leeprojekthi zomthombo ovulekileyo ngokuzithandela.
Ukufika kwam e-Odnoklassniki, ndatsho ngokungakhathali kudliwanondlebe ukuba ndingasebenza ne-Elasticsearch. Emva kokuba ndiyifumene kwaye ndiyigqibile imisebenzi elula, ndanikwa umsebenzi omkhulu wokuguqula inkqubo yolawulo lwelog eyayikho ngelo xesha.
iimfuno
Iimfuno zenkqubo zaqulunqwa ngolu hlobo lulandelayo:
- IGreylog yayiza kusetyenziswa njengendawo engaphambili. Ngenxa yokuba inkampani sele inamava ngokusebenzisa le mveliso, abadwelisi benkqubo kunye nabavavanyi babeyazi, yayiqhelekile kwaye ifanelekile kubo.
- Umthamo wedatha: kumyinge we-50-80 amawaka emiyalezo ngomzuzwana, kodwa ukuba kukho into ephukayo, i-traffic ayikhawulelwanga nantoni na, ingaba yi-2-3 yezigidi zemigca ngomzuzwana.
- Emva kokuba sixoxe nabathengi ngeemfuno zesantya sokuphendula imibuzo yokukhangela, siye safumanisa ukuba indlela eqhelekileyo yokusebenzisa inkqubo enjalo yile: abantu bafuna iilog zesicelo sabo kwiintsuku ezimbini ezidlulileyo kwaye abafuni kulinda ngaphezu kwe okwesibini kwisiphumo sombuzo oqulunqiweyo.
- Abalawuli bagxininise ukuba inkqubo mayihlawulwe ngokulula ukuba kuyimfuneko, ngaphandle kokuba bafune ukuba baphonononge nzulu ngendlela esebenza ngayo.
- Ukuze umsebenzi wolondolozo kuphela ofunwa zezi nkqubo ngamaxesha athile kukutshintsha i-hardware ethile.
- Ukongeza, i-Odnoklassniki inesiko elibalaseleyo lobugcisa: nayiphi na inkonzo esiyisungulayo kufuneka isinde ekungaphumeleli kweziko ledatha (ngequbuliso, ingacwangciswanga kwaye nangaliphi na ixesha).
Imfuno yokugqibela ekuphunyezweni kwale projekthi ixabisa kakhulu, endiya kuthetha ngayo ngokubanzi.
NgoLwesithathu
Sisebenza kumaziko edatha amane, ngelixa i-Elasticsearch data nodes inokufumaneka kuphela kwisithathu (ngenxa yezizathu ezingezona zobuchwepheshe).
La maziko edatha amane aqulathe malunga ne-18 lamawaka emithombo eyahlukeneyo yelog - i-hardware, izikhongozeli, oomatshini benyani.
Uphawu olubalulekileyo: iqela liqala kwizikhongozeli hayi koomatshini bomzimba, kodwa kuqhubeke . Izikhongozeli ziqinisekisiwe ii-cores ezi-2, ezifana ne-2.0Ghz v4, kunye nethuba lokuphinda kusetyenziswe ii-cores eziseleyo ukuba zingasebenzi.
Ngamanye amazwi:
I-Topology
Ekuqaleni ndabona indlela eqhelekileyo yesisombululo ngolu hlobo lulandelayo:
- I-3-4 VIPs isemva kwe-A-record ye-Graylog domain, le yidilesi apho iilogi zithunyelwa khona.
- I-VIP nganye yi-balancer ye-LVS.
- Emva kwayo, iilogi ziya kwibhetri yeGreylog, enye idatha ikwifomathi ye-GELF, enye kwifomathi ye-syslog.
- Ke konke oku kubhalwe kwiibhetshi ezinkulu kwibhetri yabanxibelelanisi be-Elasticsearch.
- Kwaye bona, bathumele ukubhala kwaye bafunde izicelo kwiindawo ezifanelekileyo zedatha.
I sigama
Mhlawumbi ayinguye wonke umntu osiqondayo isigama ngokweenkcukacha, ngoko ke ndingathanda ukuhlala kulo kancinane.
I-Elasticsearch ineentlobo ezininzi zeendawo zokuhlala - inkosi, umnxibelelanisi, indawo yedatha. Kukho ezinye iindidi ezimbini zotshintsho lwelog eyahlukeneyo kunye nonxibelelwano phakathi kwamaqela ahlukeneyo, kodwa sisebenzise kuphela ezo zidwelisiweyo.
Master
Idibanisa zonke iindawo ezikhoyo kwiqela, igcina imephu yeqela esexesheni kwaye isasaze phakathi kweendawo zokuhlala, iqhuba ingqiqo yomnyhadala, kwaye yenza iindidi ezahlukeneyo zogcino lwendlu olubanzi.
UMququzeleli
Wenza umsebenzi omnye: wamkela ukufunda okanye ukubhala izicelo ezivela kubathengi kunye neendlela kule traffic. Kwimeko apho kukho isicelo sokubhala, kunokwenzeka ukuba, iya kubuza inkosi ukuba yeyiphi na inxalenye yesalathiso esifanelekileyo ekufuneka ifake kuyo, kwaye iya kuphinda ihambise isicelo phambili.
Indawo yedatha
Igcina idatha, yenza imibuzo yokukhangela evela ngaphandle kwaye yenza imisebenzi kwiishadi ezikuyo.
Ngwevu
Le yinto efana ne-fusion ye-Kibana kunye ne-Logstash kwisitaki se-ELK. I-Greylog idibanisa zombini i-UI kunye nombhobho wokucubungula ilog. Ngaphantsi kwe-hood, i-Graylog iqhuba i-Kafka kunye ne-Zookeeper, enika uqhagamshelwano kwiGreylog njengeqela. I-Graylog inokufihla iilog (Kafka) kwimeko apho i-Elasticsearch ayifumaneki kwaye iphinda izicelo zokufunda nokubhala ezingaphumelelanga, iqela kunye nokumakisha iilogi ngokwemigaqo echaziweyo. NjengoLogstash, iGreylog inomsebenzi wokuguqula imiqolo ngaphambi kokuba uyibhale kwi-Elasticsearch.
Ukongeza, i-Graylog inokufumanisa inkonzo eyakhelwe-ngaphakathi evumela, ngokusekelwe kwi-node enye ye-Elasticsearch ekhoyo, ukufumana yonke imephu ye-cluster kwaye ihluze ngethegi ethile, eyenza kube lula ukuhambisa izicelo kwiibhokisi ezithile.
Ngokubonakalayo ijongeka ngolu hlobo:
Lo ngumfanekiso wekhusi osuka kumzekelo othile. Apha sakha i-histogram esekelwe kumbuzo wokukhangela kunye nokubonisa imiqolo efanelekileyo.
Izalathiso
Ukubuyela kuyilo lwenkqubo, ndingathanda ukuhlala kwiinkcukacha ezithe vetshe malunga nendlela esakhe ngayo imodeli yesalathiso ukuze yonke isebenze ngokuchanekileyo.
Kumzobo ongentla, eli lelona nqanaba lisezantsi: Elasticsearch data nodes.
Isalathisi liqumrhu elikhulu elenziwe ngeeshadi ze-Elasticsearch. Ngokwayo, i-shards nganye ayikho into ngaphandle kwesalathisi seLucene. Kwaye isalathiso ngasinye seLucene, ngokulandelelana, sinecandelo elinye okanye ngaphezulu.
Xa siyila, sicinge ukuba ukuze sihlangabezane nemfuneko yesantya sokufunda kwisixa esikhulu sedatha, kufuneka "sisasaze" le datha ngokulinganayo kuzo zonke iindawo zedatha.
Oku kubangele ukuba inani leeshadi ngesalathiso (kunye neekopi) kufuneka lilingane ngokungqongqo nenani leedatha zedatha. Okokuqala, ukuze kuqinisekiswe into ephindaphindwayo elingana nezimbini (oko kukuthi, sinokuphulukana nesiqingatha seqela). Kwaye, okwesibini, ukwenzela ukuba kuqhutywe ukufunda nokubhala izicelo ubuncinane kwisiqingatha seqela.
Siqale sagqiba ixesha lokugcina njengeentsuku ezingama-30.
Ukuhanjiswa kweeshadi kunokubonakaliswa ngokulandelayo:
Lonke uxande olungwevu olumnyama sisalathiso. Isikwele esibomvu esisekhohlo kuyo yi-shard ephambili, eyokuqala kwisalathisi. Kwaye isikwere esibhlowu sisiqwengana sekopi. Zifumaneka kumaziko ahlukeneyo edatha.
Xa songeza enye i-shard, iya kwiziko lesithathu ledatha. Kwaye, ekugqibeleni, sifumana olu lwakhiwo, okwenza kube lula ukuphulukana ne-DC ngaphandle kokulahlekelwa ukungaguquguquki kwedatha:
Ukujikeleza kwezalathisi, okt. ukudala isalathisi esitsha kunye nokucima eyona indala, senze ukuba ilingane neeyure ze-48 (ngokomzekelo wokusetyenziswa kwesalathisi: iiyure ze-48 zokugqibela zikhangelwa rhoqo).
Eli thuba lokujikeleza kwesalathiso kungenxa yezi zizathu zilandelayo:
Xa isicelo sokukhangela sifika kwindawo ethile yedatha, ngoko ke, ukusuka kwindawo yokujonga ukusebenza, kunenzuzo ngakumbi xa i-shard enye ibuzwa, ukuba ubukhulu bayo bufaniswa nobukhulu be-hip ye-node. Oku kukuvumela ukuba ugcine inxalenye "eshushu" yesalathisi kwimfumba kwaye ufikelele kuyo ngokukhawuleza. Xa kukho "iindawo ezishushu" ezininzi, isantya sokukhangela isalathisi siyancipha.
Xa i-node iqala ukwenza umbuzo wokukhangela kwishadi enye, yabela inani lemisonto elingana nenani leecores zehyperthreading zomatshini womzimba. Ukuba umbuzo wokukhangela uchaphazela inani elikhulu lee-shards, ngoko ke inani lemicu likhula ngokulinganayo. Oku kunempembelelo embi kwisantya sokukhangela kwaye kuchaphazela kakubi isalathisi sedatha entsha.
Ukubonelela nge-latency yokukhangela eyimfuneko, sagqiba ekubeni sisebenzise i-SSD. Ukuze kuqwalaselwe izicelo ngokukhawuleza, oomatshini ababesindleke ezi zikhongozeli kwakufuneka ubuncinane babe neecores ezingama-56. Inani le-56 likhethwe njengexabiso elaneleyo ngokwemiqathango elimisela inani lemisonto eya kuveliswa yi-Elasticsearch ngexesha lokusebenza. Kwi-Elasitcsearch, iiparamitha ezininzi ze-thread pool zixhomekeke ngokuthe ngqo kwinani lee-cores ezikhoyo, ezichaphazela ngokuthe ngqo inani elifunekayo lee-nodes kwiqela ngokomgaqo othi "i-cores ezimbalwa - iindawo ezininzi".
Ngenxa yoko, sifumene ukuba umyinge we-shard inobunzima obumalunga nama-20 gigabytes, kwaye kukho i-1 shards kwisalathisi ngasinye. Ngokufanelekileyo, ukuba siyawajikelezisa kube kanye kwiiyure ezingama-360, ngoko sine-48 kuzo. Isalathisi ngasinye siqulethe idatha yeentsuku ezi-15.
Iisekethe zokubhala kunye nokufunda idatha
Makhe sibone ukuba idatha irekhodwa njani kule nkqubo.
Masithi esinye isicelo sifika sisuka eGreylog sisiya kumnxibelelanisi. Umzekelo, sifuna isalathisi 2-3 amawaka imiqolo.
Umnxibelelanisi, akuba efumene isicelo esivela kuGraylog, ubuza inkosi: “Kwisicelo sesalathiso, siye sachaza ngokukhethekileyo isalathiso, kodwa asichazwanga ukuba kubhalwe phi na.”
Inkosi iyaphendula: "Bhala olu lwazi kwinombolo ye-shard 71," emva koko ithunyelwe ngokuthe ngqo kwi-node yedatha efanelekileyo, apho i-primary-shard inombolo ye-71 ikhona.
Emva koko i-log yentengiselwano iphinda iphindwe kwi-replica-shard, ehlala kwelinye iziko ledatha.
Isicelo sokukhangela sifika sivela kuGreylog sisiya kumququzeleli. Umnxibelelanisi uyayiqondisa kwakhona ngokwesalathiso, ngelixa i-Elasticsearch isasaza izicelo phakathi kwe-primary-shard kunye ne-replica-shard isebenzisa umgaqo-robin ojikelezayo.
Ii-node ze-180 ziphendula ngokungafaniyo, kwaye ngelixa ziphendula, umququzeleli uqokelela ulwazi olusele "lutshiciwe" ngeenkcukacha ezikhawulezayo zedatha. Emva koko, xa mhlawumbi lonke ulwazi lufikile, okanye isicelo sifikelele kwixesha lokuphuma, sinika yonke into ngokuthe ngqo kumxhasi.
Le nkqubo iyonke kwi-avareji iqhuba imibuzo yokukhangela kwiiyure zokugqibela ze-48 kwi-300-400ms, ngaphandle kwaloo mibuzo kunye ne-wildcard ehamba phambili.
Iintyatyambo ezine-Elasticsearch: ukuseta iJava
Ukwenza yonke into isebenze ngendlela ebesiyifuna ngayo ekuqaleni, sichithe ixesha elide silungisa izinto ezininzi ezahlukeneyo kwiqela.
Inxalenye yokuqala yeengxaki ezifunyenweyo zazinxulumene nendlela iJava eqwalaselwe ngayo ngokungagqibekanga kwi-Elasticsearch.
Ingxaki enye
Siye sabona inani elikhulu kakhulu leengxelo ezikwinqanaba leLucene, xa imisebenzi yangasemva isebenza, icandelo leLucene lidibanisa lingaphumeleli ngempazamo. Kwangaxeshanye, kwakucacile kwiilog ukuba le yimpazamo yeOutOfMemoryError. Sabona kwi-telemetry ukuba i-hip ikhululekile, kwaye akucaci ukuba kutheni le nto ingaphumeleli.
Kwavela ukuba ukudibanisa kweLucene index kwenzeka ngaphandle kwe-hip. Kwaye izikhongozeli zinqongophele ngokungqongqo ngokwezixhobo ezisetyenziswayo. Yimfumba kuphela enokungena kwezi zixhobo (ixabiso le-heap.size laliphantse lilingane ne-RAM), kwaye ezinye izinto ezingaphandle kwemfumba zantlitheka ngempazamo yolwabiwo lwememori ukuba ngesizathu esithile azizange zingene kwi ~500MB eshiyeke phambi komda.
Ulungiso lwalungabalulekanga: inani le-RAM ekhoyo kwisikhongozeli liye landa, emva koko salibala ukuba sineengxaki ezinjalo.
Ingxaki yesibini
Kwiintsuku ezi-4-5 emva kokuqaliswa kweqela, saqaphela ukuba ii-node zedatha zaqala ukuphuma rhoqo kwi-cluster kwaye zingena emva kwemizuzwana ye-10-20.
Xa saqala ukuyiqonda, kwavela ukuba le nkumbulo ingekho kwimfumba kwi-Elasticsearch ayilawulwa nangayiphi na indlela. Xa sinike inkumbulo engakumbi kwisikhongozeli, sakwazi ukuzalisa amachibi e-buffer ngokuthe ngqo ngolwazi olwahlukeneyo, kwaye yacocwa kuphela emva kokuba i-GC ecacileyo yaziswa kwi-Elasticsearch.
Kwezinye iimeko, lo msebenzi wathatha ixesha elide kakhulu, kwaye ngeli xesha iqela likwazile ukuphawula le nodi njengesele iphumile. Le ngxaki ichazwe kakuhle .
Isisombululo saba ngolu hlobo lulandelayo: sinciphise amandla eJava okusebenzisa isambuku sememori ngaphandle kwemfumba kule misebenzi. Siyilinganisele kwi-16 gigabytes (-XX:MaxDirectMemorySize=16g), siqinisekisa ukuba i-GC ecacileyo yayibizwa ngokuphindaphindiweyo kwaye iqhutywe ngokukhawuleza, ngaloo ndlela ingasayi kutshabalalisa iqela.
Ingxaki yesithathu
Ukuba ucinga ukuba iingxaki "ezineendawo ezishiya i-cluster ngexesha elingalindelekanga" ziphelile, uyaphazama.
Xa siqwalasela umsebenzi ngezalathisi, sikhethe mmapfs ukuya kumatshadi amatsha anokwahlulwa okukhulu. Oku ibiyimpazamo enkulu, kuba xa usebenzisa mmapfs ifayile izotywe kwi RAM, kwaye ke sisebenza ngefayile ebhalwe kwimaphu. Ngenxa yoko, kuvela ukuba xa i-GC izama ukumisa imicu kwisicelo, siya kwindawo ekhuselekileyo ixesha elide, kwaye endleleni eya kuyo, isicelo siyayeka ukuphendula kwizicelo zenkosi malunga nokuba iyaphila. . Ngokufanelekileyo, inkosi ikholelwa ukuba i-node ayisekho kwiqela. Emva koko, emva kwemizuzwana emi-5 ukuya kwe-10, umqokeleli wenkunkuma usebenza, i-node iphila, iphinda ingene kwiqela kwaye iqalise ukuqalisa i-shards. Yonke le nto yayivakala ngathi “yimveliso ebesiyifanele” kwaye ibingafanelanga nantoni na enzulu.
Ukulahla le ndlela yokuziphatha, saqala satshintshela kwii-niofs eziqhelekileyo, kwaye emva koko, xa sifuduka kwiinguqulelo zesihlanu ze-Elastic ukuya kweyesithandathu, sazama i-hybridfs, apho le ngxaki ayizange iphinde iphinde yenziwe. Unokufunda ngakumbi malunga neentlobo zokugcina .
Ingxaki yesine
Emva koko kwabakho enye ingxaki enomdla kakhulu esiyinyangayo ixesha elirekhodiweyo. Sayibamba kwiinyanga ezi-2-3 ngenxa yokuba ipateni yayo yayingenakuqondakala ngokupheleleyo.
Ngamanye amaxesha abaququzeleli bethu baya kwi-GC egcweleyo, ngokuqhelekileyo emva kwesidlo sasemini, kwaye abazange babuye apho. Ngexesha elifanayo, xa ugawula ukulibaziseka kweGC, kwakubonakala ngathi: yonke into ihamba kakuhle, kakuhle, kakuhle, kwaye ngokukhawuleza yonke into ihamba kakubi kakhulu.
Ekuqaleni sasicinga ukuba sinomsebenzisi okhohlakeleyo owayesungula uhlobo oluthile lwesicelo esikhupha umnxibelelanisi kwindlela yokusebenza. Safaka izicelo ixesha elide kakhulu, sizama ukuqonda ukuba kwenzeka ntoni.
Ngenxa yoko, kuye kwavela ukuba okwangoku xa umsebenzisi eqalisa isicelo esikhulu, kwaye sifika kumnxibelelanisi othile we-Elasticsearch, ezinye iindawo ziphendula ixesha elide kunezinye.
Kwaye ngelixa umququzeleli elindele impendulo kuzo zonke iinqununu, uqokelela iziphumo ezithunyelwe kwii-nodes esele ziphendule. Kwi-GC, oku kuthetha ukuba iipatheni zethu zokusetyenziswa kweemfumba zitshintsha ngokukhawuleza. Kwaye uGC esimsebenzisileyo akakwazanga ukumelana nalo msebenzi.
Ukuphela kokulungiswa esikufumeneyo ukutshintsha ukuziphatha kweqela kule meko kukufudukela kwi-JDK13 kunye nokusetyenziswa komqokeleli wenkunkuma yaseShenandoah. Oku kwayisombulula ingxaki, abaququzeleli bethu bayeka ukuwa.
Yilapho iingxaki ngeJava zaphela kwaye iingxaki ze-bandwidth zaqala.
"Amaqunube" aneElasticsearch: throughput
Iingxaki nge-throughput ithetha ukuba iqela lethu lisebenza ngokuzinzileyo, kodwa kwincopho yenani lamaxwebhu anesalathiso nangexesha lokuqhuba, ukusebenza awonelanga.
Uphawu lokuqala oludibana nayo: ngexesha elithile "lokuqhuma" kwimveliso, xa inani elikhulu kakhulu leelogi zenziwa ngokukhawuleza, i-indexing error es_rejected_execution iqala ukukhanya rhoqo kwiGreylog.
Oku kwakungenxa yokuba i-thread_pool.write.queue kwindawo enye yedatha, kude kube ngumzuzu i-Elasticsearch ikwazi ukucubungula isicelo se-indexing kwaye ilayishe ulwazi kwi-shard kwi-disk, ikwazi ukugcina izicelo ze-200 kuphela ngokungagqibekanga. Kwaye ngaphakathi Kuncinci kakhulu okuthethwayo malunga nale parameter. Kuphela inani eliphezulu lemisonto kunye nobukhulu obumiselweyo bubonisiwe.
Ewe kunjalo, siye sajija eli xabiso kwaye safumanisa oku kulandelayo: ngokuthe ngqo, kulungiselelo lwethu, ukuya kuthi ga kwizicelo ezingama-300 zigcinwe kakuhle, kwaye ixabiso eliphezulu lizaliswe yinto yokuba siphinda sibhabhe kwi-GC egcweleyo.
Ukongeza, ekubeni ezi ziibhetshi zemiyalezo efika ngesicelo esinye, bekuyimfuneko ukuba i-Greylog ibhale rhoqo kwaye ingabhali kwiibhetshi ezincinci, kodwa kwiibhetshi ezinkulu okanye kube kanye ngemizuzwana emi-3 ukuba ibhetshi ayikagqitywa. Kule meko, kuvela ukuba ulwazi esilubhala kwi-Elasticsearch alufumaneki kwimizuzwana emibini, kodwa kwimizuzu emihlanu (esifanelekela kakuhle), kodwa inani leetreyi ekufuneka zenziwe ukuze kutyhale indawo enkulu. inkitha yolwazi iyancitshiswa.
Oku kubaluleke ngakumbi ngezo zihlandlo xa kukho into ephazamisekile kwindawo ethile kwaye ibika ngomsindo ngayo, ukuze ungafumani i-Elastic egalelwe ngokupheleleyo, kwaye emva kwexesha elithile - ii-Graylog nodes ezingasebenziyo ngenxa ye-buffers evaliweyo.
Ukongezelela, xa sineziqhushumbisi ezifanayo kwimveliso, safumana izikhalazo ezivela kubaprogram kunye nabavavanyi: okwangoku xa babezifuna ngokwenene ezi zigodo, banikwa ngokukhawuleza kakhulu.
Baqala ukuyiqonda. Ngakolunye uhlangothi, kwakucacile ukuba zombini imibuzo yokukhangela kunye nemibuzo ye-indexing yaqhutyelwa, ngokukodwa, kumatshini ofanayo womzimba, kwaye enye indlela okanye enye kuya kubakho ukutsalwa okuthile.
Kodwa oku kunokuthintelwa ngokuyinxenye ngenxa yokuba kwiinguqulelo zesithandathu ze-Elasticsearch, kuvele i-algorithm ekuvumela ukuba usasaze imibuzo phakathi kweendawo zedatha ezifanelekileyo hayi ngokomgaqo-siseko ojikelezayo ojikelezayo (isikhongozelo esenza isalathisi kwaye sibambe eyona ndlela iphambili yokujonga ubungakanani bezinto ozifunayo. -i-shard inokuxakeke kakhulu, ayiyi kubakho indlela yokuphendula ngokukhawuleza), kodwa ukuhambisa esi sicelo kwisitya esincinci esilayishiweyo kunye ne-replica-shard, eya kuphendula ngokukhawuleza. Ngamanye amazwi, sifike use_adaptive_replica_selection: true.
Umfanekiso ofundayo uqala ukujongeka ngolu hlobo:
Ukutshintshela kule algorithm kwenze ukuba kube lula ukuphucula ixesha lombuzo ngaloo maxesha apho sasinohambo olukhulu lweelogi zokubhala.
Ekugqibeleni, ingxaki enkulu yayikukususwa okungenabuhlungu kwiziko ledatha.
Into ebesiyifuna kwiqela ngokukhawuleza emva kokuphulukana nonxibelelwano ne-DC enye:
- Ukuba sinenkosi yangoku kwiziko ledatha elingaphumelelanga, ngoko liya kuphinda likhethwe kwaye lihanjiswe njengendima kwenye i-node kwenye i-DC.
- Inkosi iya kususa ngokukhawuleza zonke ii-nodes ezingafumanekiyo kwi-cluster.
- Ngokusekelwe kulawo asele, uya kuqonda: kwiziko ledatha elahlekileyo sasine-shards enjalo kunye neprayimari, uya kukhuthaza ngokukhawuleza i-replica shards kumaziko edatha asele, kwaye siya kuqhubeka ne-indexing data.
- Ngenxa yoko, ukubhala kunye nokufunda kweqela kuya kuthotywa ngokuthe ngcembe, kodwa ngokubanzi yonke into iya kusebenza, nangona kancinci, kodwa ngokuzinzileyo.
Njengoko kwavela, sasifuna into efana nale:
Kwaye sifumene oku kulandelayo:
Yenzeka njani le nto?
Xa iziko ledatha lawa, inkosi yethu yaba ngumqobo.
Kutheni?
Inyaniso kukuba inkosi ine-TaskBatcher, ejongene nokusabalalisa imisebenzi ethile kunye neziganeko kwiqela. Nayiphi na i-node ephumayo, naluphi na unyuso lwe-shard ukusuka kwi-replica ukuya kwiprayimari, nawuphi na umsebenzi wokwenza i-shard kwenye indawo - konke oku kuya kuqala kwi-TaskBatcher, apho iqhutywe ngokulandelelana kunye nentambo enye.
Ngexesha lokurhoxiswa kweziko ledatha enye, kuye kwavela ukuba zonke iindawo zedatha kumaziko edatha asele ziwuthatha njengomsebenzi wazo ukwazisa inkosi "siphulukene neeshadi ezinje kunye nezinje kunye nezo zedatha."
Ngelo xesha, i-data nodes esaphila yathumela yonke le ngcaciso kwinkosi yangoku kwaye yazama ukulinda ukuqinisekiswa ukuba uyayamkela. Abazange bakulinde oku, ekubeni inkosi yayifumana imisebenzi ngokukhawuleza kunokuba yayinokuyiphendula. Amanqaku aphelelwe lixesha izicelo eziphindaphindayo, kwaye inkosi ngeli xesha ayizange izame nokuziphendula, kodwa yayixakeke ngokupheleleyo kumsebenzi wokuhlela izicelo ngokubaluleka.
Kwifom ye-terminal, kwavela ukuba i-data nodes i-spammed inkosi ukuya kwinqanaba lokuba lingene kwi-GC epheleleyo. Emva koko, indima yethu ephambili yafudukela kwindawo elandelayo, kwenzeka into efanayo kuyo, kwaye ngenxa yoko iqela lawa ngokupheleleyo.
Sithathe imilinganiselo, kwaye ngaphambi kwenguqulo ye-6.4.0, apho oku kwalungiswa khona, kwakwanele ukuba siphume ngaxeshanye kuphela i-10 data nodes ngaphandle kwe-360 ukuze sivale ngokupheleleyo iqela.
Yayijongeka ngolu hlobo:
Emva kwenguqulo 6.4.0, apho le bug yoyikekayo yalungiswa, iindawo zedatha zayeka ukubulala inkosi. Kodwa oko akuzange kumenze abe “krelekrele.” Okukuthi: xa sikhupha u-2, 3 okanye 10 (nakweliphi na inani ngaphandle kwelinye) iindawo zedatha, inkosi ifumana umyalezo wokuqala othi inodi A imkile, kwaye izama ukuxelela indawo engu-B, inodi C ngale nto, inqaku D.
Kwaye okwangoku, oku kungaqwalaselwa kuphela ngokubeka ixesha lokuzama ukuxelela umntu ngento ethile, elingana nemizuzwana engama-20-30, kwaye ngaloo ndlela ulawule isantya seziko ledatha eliphuma kwiqela.
Ngokomgaqo, oku kuhambelana neemfuno eziye zanikezelwa ekuqaleni kwimveliso yokugqibela njengenxalenye yeprojekthi, kodwa ukusuka kwindawo yokujonga "isayensi ecocekileyo" le yimpazamo. Yiyiphi, ngendlela, yalungiswa ngempumelelo ngabaphuhlisi kwinguqulo 7.2.
Ngaphezu koko, xa i-node ethile yedatha iphumile, kwavela ukuba ukusasaza ulwazi malunga nokuphuma kwayo kwakubaluleke ngaphezu kokuxelela iqela elipheleleyo ukuba kukho i-primary-shards kuyo (ukwenzela ukukhuthaza i-replica-shard kwenye idatha. Iziko kwindawo yokuqala, kwaye kulwazi kungabhalwa kuzo).
Ke ngoko, xa yonke into sele ifile, iinodi zedatha ezikhutshiweyo aziphawulwa ngokukhawuleza njengezidala. Ngako oko, siphoqeleka ukuba silinde de kube zonke iipings ziphelelwe lixesha kwii-node zedatha ezikhutshiweyo, kwaye emva kokuba iqela lethu liqala ukusixelela ukuba apho, apho, kwaye apho kufuneka siqhubeke nokurekhoda ulwazi. Unokufunda ngakumbi malunga noku .
Ngenxa yoko, ukusebenza kokurhoxisa iziko ledatha namhlanje kusithatha malunga nemizuzu emi-5 ngexesha lokukhawuleza. Kwi-colossus enkulu kunye ne-clumsy, esi sisiphumo esihle kakhulu.
Ngenxa yoko, siye safikelela kwesi sigqibo silandelayo:
- Sineendawo zedatha ezingama-360 ezineediski ezingama-700 zegigabyte.
- Bangama-60 abaququzeleli bokuhamba kwetrafikhi kwezi ndawo zedatha.
- Iinkosi ezingama-40 esizishiyile njengohlobo lwelifa ukusukela kwiinguqulelo ngaphambi kwe-6.4.0 - ukuze sisinde ukurhoxiswa kweziko ledatha, sasilungele ngokwengqondo ukulahlekelwa ngoomatshini abaliqela ukuze siqinisekiswe ukuba sinekhoram yeenkosi nakwi. eyona meko imbi
- Naziphi na iinzame zokudibanisa iindima kwisikhongozeli esinye zidibene nenyaniso yokuba kungekudala okanye kamva i-node iya kuphuka phantsi komthwalo.
- Iqela lonke lisebenzisa i-heap.size ye-31 gigabytes: zonke iinzame zokunciphisa ubungakanani zibangele nokuba kubulawe ezinye iindawo kwimibuzo yokukhangela enzima kunye ne-wildcard ehamba phambili okanye ukufumana i-circuit breaker kwi-Elasticsearch ngokwayo.
- Ukongezelela, ukuqinisekisa ukusebenza kokukhangela, sizame ukugcina inani lezinto kwi-cluster lincinci kangangoko kunokwenzeka, ukwenzela ukuba siqhubekisele phambili iziganeko ezimbalwa kunokwenzeka kwi-bottleneck esiyifumene kwi-master.
Ekugqibeleni malunga nokubeka iliso
Ukuqinisekisa ukuba konke oku kusebenza njengoko bekucetyiwe, sibeka esweni oku kulandelayo:
- Indawo nganye yedatha inika ingxelo kwilifu lethu ukuba likhona, kwaye kukho iishadi ezinje nezinjalo kuyo. Xa sicima into ethile kwindawo ethile, i-cluster ibika emva kwemizuzwana ye-2-3 ukuba kumbindi we-A sacima ii-nodes 2, 3, kunye ne-4 - oku kuthetha ukuba kwamanye amaziko edatha thina phantsi kweemeko singakwazi ukucima ezo ndawo apho kukho i-shard enye kuphela. khohlo.
- Ukwazi uhlobo lokuziphatha kwenkosi, sijonga ngokucophelela inani lemisebenzi elindileyo. Ngenxa yokuba umsebenzi omnye obambekileyo, ukuba awupheli ngexesha, ithiyori kwimeko ethile yongxamiseko inokuba sisizathu sokuba, umzekelo, ukunyuswa kwe-replica shard kwiprayimari kungasebenzi, yiyo loo nto i-indexing iya kuyeka ukusebenza.
- Sikwajonge kakhulu ukulibaziseka komqokeleli wenkunkuma, kuba sele sineengxaki ezinkulu malunga noku ngexesha lokuphucula.
- Ikhatywa ngomsonto ukuqonda kwangaphambili apho i-bottleneck iphi.
- Ewe, iimetriki ezisemgangathweni ezifana nemfumba, i-RAM kunye ne-I/O.
Xa usakha iliso, kufuneka uthathele ingqalelo iimpawu ze-Thread Pool kwi-Elasticsearch. ichaza iinketho zoqwalaselo kunye nexabiso elingagqibekanga lokukhangela kunye nesalathisi, kodwa lithe cwaka ngokupheleleyo malunga ne-thread_pool.management Le misonto iqhubekeka, ngakumbi imibuzo efana ne-_cat/shards kunye nezinye ezifanayo, ezisebenzisekayo xa kubhalwa iliso. Iqela elikhulu, kokukhona izicelo ezinjalo ziphunyezwa kwiyunithi yexesha ngalinye, kwaye intambo_pool.management ikhankanywe ngasentla ayiboniswanga kuxwebhu olusemthethweni kuphela, kodwa ikwanqunyelwe ngokungagqibekanga kwimisonto emi-5, echithwa ngokukhawuleza, emva kokuba apho ukubeka iliso kuyeka ukusebenza ngokuchanekileyo.
Into endifuna ukuyithetha ekugqibeleni: siyenzile! Siye sakwazi ukunika abadwelisi benkqubo kunye nabaphuhlisi isixhobo esinokuthi, phantse nayiphi na imeko, inokubonelela ngokukhawuleza nangokuthembekileyo ngolwazi malunga nento eyenzekayo kwimveliso.
Ewe, kuye kwanzima kakhulu, kodwa, nangona kunjalo, sikwazile ukulungelelanisa iminqweno yethu kwiimveliso esele zikhona, ekunganyanzelekanga ukuba sizipeyishe kwaye sizibhalele thina.
umthombo: www.habr.com