Sawubona, Habr! Igama lami nginguMaxim Vasiliev, ngisebenza njengomhlaziyi kanye nomphathi wephrojekthi kwa-FINCH. Namuhla ngithanda ukukutshela ukuthi, sisebenzisa i-ElasticSearch, sikwazile kanjani ukucubungula imibuzo eyizigidi ezingu-15 ngemizuzu engu-6 futhi sandise umthwalo wansuku zonke esizeni selinye lamakhasimende ethu. Ngeshwa, kuzodingeka senze ngaphandle kwamagama, njengoba sine-NDA, sithemba ukuthi okuqukethwe yi-athikili ngeke kuhlupheke kulokhu. Asambe.
Indlela iphrojekthi esebenza ngayo
Emuva kwethu sakha izinsizakalo eziqinisekisa ukusebenza kwamawebhusayithi eklayenti lethu kanye nesicelo seselula. Isakhiwo esijwayelekile singabonakala kumdwebo:
Enqubweni yomsebenzi, sicubungula inani elikhulu lokuthengiselana: ukuthenga, izinkokhelo, ukuthengiselana ngamabhalansi omsebenzisi, lapho sigcina khona amalogi amaningi, futhi singenisa futhi sithekelisa le datha kumasistimu angaphandle.
Kukhona nezinqubo zokuhlehla lapho sithola idatha evela kuklayenti futhi siyidlulisela kumsebenzisi. Ngaphezu kwalokho, kukhona nezinqubo zokusebenza ngezinkokhelo nezinhlelo zebhonasi.
Ingemuva elifushane
Ekuqaleni, sasebenzisa i-PostgreSQL njengokuphela kwesitoreji sethu sedatha. Izinzuzo zayo ziyizinga le-DBMS: ukuba khona kokuthengiselana, ulimi oluthuthukisiwe lokubuyisa idatha, amathuluzi ahlukahlukene okuhlanganiswa; kuhlanganiswe nokusebenza okuhle kwanelisa izidingo zethu isikhathi eside impela.
Sigcine yonke idatha ku-Postgres: kusukela kokwenziwayo kuye ezindabeni. Kodwa inani labasebenzisi likhule, futhi nalo inani lezicelo.
Ukuze uqonde, inani lonyaka lamaseshini ngo-2017 kusayithi ledeskithophu kuphela laliyizigidi ezingu-131. Ngo-2018 - izigidi ezingu-125. Ngo-2019 futhi izigidi ezingu-130. Engeza enye izigidi ezingu-100-200 kusuka kunguqulo yeselula yesayithi kanye nesicelo seselula, futhi nawe uzothola inani elikhulu lezicelo.
Njengoba iphrojekthi ikhula, i-Postgres ayikwazanga ukubhekana nomthwalo; asikwazanga ukuqhubeka - kwavela inombolo enkulu yemibuzo ehlukene, esingakwazi ngayo ukudala inombolo eyanele yezinkomba.
Sabona ukuthi kunesidingo sezinye izitolo zedatha ezingahlangabezana nezidingo zethu futhi zisuse umthwalo ku-PostgreSQL. I-Elasticsearch ne-MongoDB zithathwe njengezinketho ezingenzeka. Owokugcina ulahlekelwe ngamaphuzu alandelayo:
- Isivinini sokukhomba esinensayo njengoba umthamo wedatha kumakhombo ukhula. Nge-Elastic, isivinini asincikile enanini ledatha.
- Akukho ukusesha umbhalo ogcwele
Ngakho sizikhethele i-Elastic futhi salungiselela inguquko.
Shintshela ku-Elastic
1. Siqale inguquko ngesevisi yokusesha indawo yokuthengisa. Iklayenti lethu linamaphuzu okuthengisa angaba ngu-70, futhi ngesikhathi esifanayo izinhlobo eziningi zokusesha ziyadingeka kuwebhusayithi nakuhlelo lokusebenza:
- Sesha umbhalo ngegama lendawo
- I-Geosearch ngaphakathi kwerediyasi enikeziwe ukusuka endaweni ethile. Isibonelo, uma umsebenzisi efuna ukubona ukuthi yiziphi izindawo zokuthengisa eziseduze nekhaya lakhe.
- Sesha ngesikwele esinikeziwe - umsebenzisi uveza isikwele kumephu, futhi wonke amaphuzu kule radiyasi uyaboniswa.
- Sesha ngezihlungi ezengeziwe. Amaphuzu okuthengisa ayahluka kwamanye ngezinhlobonhlobo
Uma sikhuluma ngenhlangano, kuma-Postgres sinomthombo wedatha wakho kokubili imephu nezindaba, futhi nge-Elastic senza ama-Snapshots kusukela kudatha yoqobo. Iqiniso liwukuthi ekuqaleni i-Postgres ayikwazanga ukubhekana nokufuna zonke izindlela. Akukhona nje kuphela ukuthi kwakukhona izinkomba eziningi, futhi zingadlulana, ngakho-ke umhleli we-Postgres ulahlekile futhi wayengaqondi ukuthi iyiphi inkomba okufanele isetshenziswe.
2. Okulandelayo kulayini kwakuyisigaba sezindaba. Okushicilelwe kuvela kusayithi nsuku zonke ukuze umsebenzisi angalahleki ekuhambeni kolwazi, idatha kufanele ihlungwe ngaphambi kokukhishwa. Yilokhu okushiwo ukusesha: kusayithi ungakwazi ukusesha ngokufanisa umbhalo, futhi ngesikhathi esifanayo uxhume izihlungi ezengeziwe, njengoba zenziwe nge-Elastic.
3. Sabe sesihambisa ukucubungula okwenziwe. Abasebenzisi bangathenga umkhiqizo othile kusayithi futhi babambe iqhaza ekutolweni kwemiklomelo. Ngemva kokuthenga okunjalo, sicubungula inani elikhulu ledatha, ikakhulukazi ngezimpelasonto namaholide. Uma kuqhathaniswa, uma ngezinsuku ezijwayelekile inani lokuthenga lisendaweni engaba yizigidi ezingu-1,5-2, khona-ke ngamaholide isibalo singafinyelela ku-53 million.
Ngesikhathi esifanayo, idatha kufanele icutshungulwe ngenani elincane lesikhathiβabasebenzisi abathandi ukulinda izinsuku ezimbalwa ukuze bathole imiphumela. Ayikho indlela yokufinyelela lezo zinsuku zokugcina ngama-Postgres - sasivame ukuthola amabhulokhi, futhi ngenkathi sisacubungula zonke izicelo, abasebenzisi abakwazanga ukubheka ukuthi bayitholile yini imiklomelo noma cha. Lokhu akujabulisi neze ebhizinisini, ngakho-ke sithuthele ukucutshungulwa ku-Elasticsearch.
I-periodicity
Manje izibuyekezo zilungiswa ngokususelwa kumcimbi, ngokuya ngemibandela elandelayo:
- Amaphuzu okuthengisa. Ngokushesha nje lapho sithola idatha emthonjeni wangaphandle, ngokushesha sethula isibuyekezo.
- Izindaba. Lapho nje noma yiziphi izindaba zihlelwa kusayithi, zithunyelwa ngokuzenzakalelayo ku-Elastic.
Lapha futhi kufanelekile ukusho izinzuzo ze-Elastic. Ku-Postgres, lapho uthumela isicelo, udinga ukulinda kuze kube yilapho sicubungula ngokwethembeka wonke amarekhodi. Ungathumela amarekhodi ayizinkulungwane eziyi-10 ku-Elastic futhi uqale ukusebenza ngokushesha, ngaphandle kokulinda ukuthi amarekhodi asakazwe kuwo wonke ama-Shards. Yebo, abanye abakwa-Shard noma i-Replica bangase bangayiboni idatha ngaso leso sikhathi, kodwa maduze nje konke kuzotholakala.
Izindlela zokuhlanganisa
Kunezindlela ezi-2 zokuhlanganisa ne-Elastic:
- Ngeklayenti lomdabu nge-TCP. Umshayeli womdabu uyafa kancane kancane: ayisasekelwa, futhi i-syntax yayo iphazamisa kakhulu. Ngakho-ke, asiyisebenzisi futhi sizama ukuyiyeka ngokuphelele.
- Ngesixhumi esibonakalayo se-HTTP lapho ungasebenzisa khona kokubili izicelo ze-JSON kanye ne-syntax ye-Lucene. Eyokugcina injini yombhalo esetshenziswa i-Elastic. Kule nketho, sithola ikhono Lokuhlanganisa ngezicelo ze-JSON nge-HTTP. Lena inketho esizama ukuyisebenzisa.
Ngenxa yokuxhumana kwe-HTTP, singasebenzisa amalabhulali ahlinzeka ngokusetshenziswa okuvumelanayo kweklayenti le-HTTP. Singakwazi ukusizakala nge-Batch kanye ne-asynchronous API, egcina inikeze ukusebenza okuphezulu, okusize kakhulu ngezinsuku zephromoshini enkulu (ngaphezulu kulokhu ngezansi)
Ezinye izinombolo zokuqhathanisa:
- Ukonga abasebenzisi abathole imiklomelo ku-Postgres ngemicu engama-20 ngaphandle kwamaqembu: amarekhodi angama-460713 ngemizuzwana engama-42
- Iklayenti le-Elastic + elisebenzayo lezintambo eziyi-10 + iqoqo lezinto eziyi-1000: Amarekhodi angama-596749 ngemizuzwana eyi-11
- Iklayenti le-Elastic + elisebenzayo lezintambo eziyi-10 + inqwaba yezinto eziyi-1000: Amarekhodi angu-23801684 ngemizuzu emi-4
Manje sesibhale isiphathi sesicelo se-HTTP esakha i-JSON njengeqoqo/non-Batch futhi siyithumela nganoma yiliphi iklayenti le-HTTP, kungakhathaliseki ukuthi iyiphi ilabhulali. Ungaphinda ukhethe ukuthumela izicelo ngokuvumelanisa noma ngokulinganayo.
Kokunye ukuhlanganiswa sisasebenzisa iklayenti lezokuthutha elisemthethweni, kodwa lokhu kuyindaba nje yokulungisa kabusha izinto ngokushesha. Kulokhu, iklayenti layo, elakhiwe ngesisekelo se-Spring WebClient, lisetshenziselwa ukucubungula.
Iphromoshini enkulu
Kanye ngonyaka, iphrojekthi isingatha iphromoshini enkulu kubasebenzisi - lokhu kufana kwe-Highload, njengoba ngalesi sikhathi sisebenza namashumi ezigidi zabasebenzisi ngesikhathi esisodwa.
Ngokuvamile, imithwalo ephezulu kakhulu ivela ngesikhathi samaholide, kodwa le phromoshini isezingeni elihluke ngokuphelele. Ngonyaka owandulela owedlule, ngosuku lwephromoshini, sathengisa amayunithi ezimpahla angu-27. Idatha ithathe isikhathi esingaphezu kwesigamu sehora ukucutshungulwa, okudale ukuphazamiseka kubasebenzisi. Abasebenzisi bathole imiklomelo yokubamba iqhaza, kodwa kwacaca ukuthi inqubo idinga ukusheshiswa.
Ekuqaleni kuka-2019, sanquma ukuthi i-ElasticSearch iyadingeka. Unyaka wonke, sihlele ukucutshungulwa kwedatha etholiwe ku-Elastic kanye nokuphuma kwayo ku-API yohlelo lokusebenza lweselula newebhusayithi. Njengomphumela, ngonyaka olandelayo ngesikhathi sokuphromotha sacubungula 15 imingenelo emizuzwini eyi-131.
Njengoba sinabantu abaningi abafuna ukuthenga umkhiqizo futhi babambe iqhaza ekutolweni kwemiklomelo kumaphromoshini ethu, lesi isilinganiso sesikhashana. Manje sithumela ulwazi lwamanje ku-Elastic, kodwa esikhathini esizayo sihlela ukudlulisa imininingwane egciniwe yezinyanga ezidlule ku-Postgres njengesitoreji saphakade. Ukuze ungavimbeli inkomba ye-Elastic, nayo enemikhawulo yayo.
Isiphetho/Iziphetho
Okwamanje, sidlulisele zonke izinsiza ebesizifuna ku-Elastic futhi simile okwesikhashana okwamanje. Manje sakha inkomba ku-Elastic phezu kwesitoreji esiyinhloko esiqhubekayo ku-Postgres, esithatha umthwalo womsebenzisi.
Ngokuzayo, sihlela ukudlulisa amasevisi uma siqonda ukuthi isicelo sedatha siyahlukahluka kakhulu futhi siseshwa inombolo engenamkhawulo yamakholomu. Lokhu akusewona umsebenzi we-Postgres.
Uma sidinga ukusesha okugcwele umbhalo ekusebenzeni noma uma sinemibandela eminingi yokusesha ehlukene, lapho-ke sesivele sazi ukuthi lokhu kudinga ukuhunyushelwa ku-Elastic.
βββ
Siyabonga ngokufunda. Uma inkampani yakho isebenzisa i-ElasticSearch futhi inamacala akho okusebenzisa, sicela usitshele. Kuzokujabulisa ukwazi ukuthi abanye benza kanjani :)
Source: www.habr.com