Hlo Habr! Kuv lub npe yog Maxim Vasiliev, Kuv ua haujlwm ua tus kws tshuaj ntsuam thiab tus thawj tswj haujlwm ntawm FINCH. Niaj hnub no kuv xav qhia koj li cas, siv ElasticSearch, peb muaj peev xwm ua tau 15 lab thov hauv 6 feeb thiab ua kom zoo tshaj qhov kev thauj khoom txhua hnub ntawm qhov chaw ntawm peb cov neeg siv khoom. Hmoov tsis zoo, peb yuav tau ua yam tsis muaj npe, txij li peb muaj NDA, peb cia siab tias cov ntsiab lus ntawm tsab xov xwm yuav tsis raug kev txom nyem los ntawm qhov no. Wb mus.
Qhov project ua haujlwm li cas
Ntawm peb lub backend, peb tsim cov kev pabcuam uas ua kom muaj kev ua tau zoo ntawm peb cov neeg siv khoom lub vev xaib thiab daim ntawv thov mobile. Cov qauv dav dav tuaj yeem pom hauv daim duab:
Hauv cov txheej txheem ntawm kev ua haujlwm, peb ua cov khoom lag luam ntau: kev yuav khoom, kev them nyiaj, kev ua haujlwm nrog cov neeg siv khoom sib npaug, uas peb khaws ntau cov cav, nrog rau kev xa tawm thiab xa tawm cov ntaub ntawv no mus rau lwm lub tshuab.
Kuj tseem muaj cov txheej txheem thim rov qab thaum peb tau txais cov ntaub ntawv los ntawm tus neeg siv khoom thiab xa mus rau tus neeg siv. Tsis tas li ntawd, tseem muaj cov txheej txheem rau kev ua haujlwm nrog cov nyiaj them poob haujlwm thiab cov kev pabcuam nyiaj tshwj xeeb.
Cov keeb kwm luv luv
Thaum pib, peb siv PostgreSQL ua cov ntaub ntawv khaws cia nkaus xwb. Nws cov qauv zoo rau DBMS: muaj kev hloov pauv, tsim cov ntaub ntawv qauv lus, ntau yam cuab yeej rau kev sib koom ua ke; ua ke nrog kev ua tau zoo txaus siab peb cov kev xav tau rau lub sijhawm ntev heev.
Peb khaws tag nrho cov ntaub ntawv hauv Postgres: los ntawm kev pauv mus rau xov xwm. Tab sis tus naj npawb ntawm cov neeg siv tau loj hlob, thiab nrog nws tus naj npawb ntawm kev thov.
Rau kev nkag siab, tus naj npawb txhua xyoo ntawm kev sib tham hauv 2017 tsuas yog ntawm lub vev xaib desktop yog 131 lab. Hauv 2018 - 125 lab. 2019 dua 130 lab. Ntxiv lwm 100-200 lab los ntawm lub vev xaib mobile thiab daim ntawv thov mobile, thiab koj yuav tau txais ntau qhov kev thov.
Nrog rau kev loj hlob ntawm qhov project, Postgres tau nres los ntawm kev thauj khoom, peb tsis muaj sijhawm - ntau cov lus nug tshwm sim, uas peb tsis tuaj yeem tsim cov lej txaus.
Peb nkag siab tias muaj kev xav tau rau lwm lub khw muag khoom cov ntaub ntawv uas yuav muab peb cov kev xav tau thiab tshem tawm ntawm PostgreSQL. Elasticsearch thiab MongoDB tau suav tias yog cov kev xaiv ua tau. Cov tom kawg poob rau cov ntsiab lus hauv qab no:
- qeeb indexing ceev raws li tus nqi ntawm cov ntaub ntawv nyob rau hauv indexes loj hlob. Nrog Elastic, qhov ceev tsis yog nyob ntawm tus nqi ntawm cov ntaub ntawv.
- Tsis muaj ntawv nyeem puv
Yog li peb xaiv Elastic rau peb tus kheej thiab npaj rau kev hloov pauv.
Hloov mus rau Elastic
1. Peb pib qhov kev hloov pauv los ntawm qhov chaw muag khoom nrhiav kev pabcuam. Peb cov neeg siv khoom muaj tag nrho txog 70 cov ntsiab lus ntawm kev muag khoom, thiab qhov no yuav tsum muaj ntau yam kev tshawb fawb ntawm lub xaib thiab hauv daim ntawv thov:
- Sau ntawv nrhiav los ntawm lub nroog lub npe
- Geosearch nyob rau hauv ib lub vojvoog muab los ntawm qee qhov taw tes. Piv txwv li, yog tias tus neeg siv xav pom cov ntsiab lus ntawm kev muag khoom ze tshaj plaws rau nws lub tsev.
- Tshawb nrhiav los ntawm ib lub xwmfab muab - tus neeg siv kos ib lub xwmfab rau ntawm daim ntawv qhia, thiab tag nrho cov ntsiab lus hauv lub vojvoog no tau qhia rau nws.
- Nrhiav los ntawm cov ntxaij lim dej ntxiv. Cov ntsiab lus ntawm kev muag khoom sib txawv los ntawm ib leeg hauv assortment
Yog tias peb tham txog lub koom haum, tom qab ntawd hauv Postgres peb muaj cov ntaub ntawv rau ob qho tib si hauv daim ntawv qhia thiab cov xov xwm, thiab hauv Elastic Snapshots yog muab los ntawm cov ntaub ntawv qub. Qhov tseeb yog tias pib Postgres tsis tuaj yeem tiv nrog kev tshawb fawb los ntawm txhua yam. Tsis tsuas yog muaj ntau qhov ntsuas, lawv tuaj yeem sib tshooj, yog li Postgres tus teem sijhawm tau poob thiab tsis nkag siab qhov ntsuas twg siv.
2. Tom ntej no hauv kab yog tshooj xov xwm. Cov ntawv tshaj tawm tau tshwm sim hauv lub xaib txhua hnub kom tus neeg siv tsis tau poob hauv cov ntaub ntawv ntws, cov ntaub ntawv yuav tsum tau txheeb xyuas ua ntej muab. Qhov no yog qhov kev tshawb nrhiav: koj tuaj yeem tshawb xyuas lub vev xaib los ntawm cov ntawv sib tw, thiab tib lub sijhawm txuas ntxiv cov ntxaij lim dej, vim lawv kuj tau tsim los ntawm Elastic.
3. Tom qab ntawd peb tsiv mus rau kev ua lag luam. Cov neeg siv tuaj yeem yuav ib yam khoom ntawm qhov chaw thiab koom nrog qhov khoom plig kos. Tom qab kev yuav khoom zoo li no, peb ua cov ntaub ntawv ntau, tshwj xeeb tshaj yog hnub so thiab hnub so. Rau kev sib piv, yog tias nyob rau hnub zoo tib yam tus naj npawb ntawm kev yuav khoom yog qhov chaw nruab nrab ntawm 1,5-2 lab, tom qab ntawd hnub so daim duab tuaj yeem ncav cuag 53 lab.
Tib lub sijhawm, cov ntaub ntawv yuav tsum tau ua tiav hauv lub sijhawm luv tshaj plaws - cov neeg siv tsis nyiam tos qhov tshwm sim rau ob peb hnub. Tsis muaj txoj hauv kev kom ua tiav cov sijhawm kawg no los ntawm Postgres - peb feem ntau tau txais cov xauv, thiab thaum peb tab tom ua txhua qhov kev thov, cov neeg siv tsis tuaj yeem tshawb xyuas seb lawv puas tau txais khoom plig lossis tsis tau. Qhov no tsis zoo rau kev lag luam, yog li peb tau tsiv mus rau Elasticsearch.
Periodicity
Tam sim no cov kev hloov tshiab tau teeb tsa raws li kev tshwm sim, raws li cov xwm txheej hauv qab no:
- Cov ntsiab lus muag. Thaum peb tau txais cov ntaub ntawv los ntawm ib qho chaw sab nraud, peb pib qhov hloov tshiab tam sim ntawd.
- Xov xwm. Sai li txhua cov xov xwm raug kho rau ntawm qhov chaw, nws raug xa mus rau Elastic.
Ntawm no dua nws tsim nyog hais txog qhov zoo ntawm Elastic. Hauv Postgres, thaum xa daim ntawv thov, koj yuav tsum tau tos kom txog thaum nws ncaj ncees ua txhua yam ntaub ntawv. Koj tuaj yeem xa 10 cov ntaub ntawv mus rau Elastic thiab pib ua haujlwm tam sim ntawd, tsis tas tos cov ntaub ntawv xa mus rau txhua qhov Shards. Tau kawg, qee qhov Shard lossis Replica yuav tsis pom cov ntaub ntawv tam sim ntawd, tab sis txhua yam yuav muaj sai sai.
Kev sib xyaw ua ke
Muaj 2 txoj hauv kev los ua ke nrog Elastic:
- Los ntawm tus neeg siv khoom ib txwm dhau TCP. Tus neeg tsav tsheb ib txwm tau maj mam tuag: nws tsis muaj kev txhawb nqa ntxiv lawm, nws muaj qhov tsis yooj yim heev. Yog li ntawd, peb xyaum tsis siv nws thiab sim ua kom tiav nws.
- Los ntawm HTTP interface uas tuaj yeem siv ob qho kev thov JSON thiab Lucene syntax. Qhov kawg yog lub cav ntawv uas siv Elastic. Hauv qhov version no, peb tau txais lub peev xwm Batch los ntawm JSON thov dhau HTTP. Qhov no yog qhov kev xaiv uas peb tab tom sim siv.
Ua tsaug rau HTTP interface, peb tuaj yeem siv cov tsev qiv ntawv uas muab kev siv asynchronous ntawm tus neeg siv HTTP. Peb tuaj yeem ua kom zoo dua ntawm Batch thiab asynchronous API, uas ua rau muaj kev ua haujlwm siab, uas tau pab ntau heev nyob rau hnub ntawm kev nce qib loj (ntxiv rau qhov hauv qab no)
Qee tus lej rau kev sib piv:
- Txuag Postgres bounty cov neeg siv hauv 20 xov yam tsis muaj pawg: 460713 cov ntaub ntawv hauv 42 vib nas this
- Elastic + reactive tus thov kev pab rau 10 threads + batch rau 1000 yam: 596749 cov ntaub ntawv hauv 11 vib nas this
- Elastic + reactive tus thov kev pab rau 10 threads + batch rau 1000 yam: 23801684 nkag hauv 4 feeb
Tam sim no peb tau sau tus thawj tswj hwm HTTP thov uas tsim JSON li Batch / tsis Batch thiab xa nws los ntawm HTTP tus neeg siv khoom, tsis hais lub tsev qiv ntawv. Koj tuaj yeem xaiv xa cov lus thov synchronously lossis asynchronously.
Hauv qee qhov kev sib koom ua ke, peb tseem siv cov neeg siv khoom thauj neeg ua haujlwm, tab sis qhov no tsuas yog qhov teeb meem ntawm kev rov ua dua tom ntej. Hauv qhov no, tus neeg siv khoom ib txwm ua los ntawm Spring WebClient yog siv rau kev ua haujlwm.
kev txhawb loj
Ib xyoos ib zaug, qhov project hosts kev txhawb nqa loj rau cov neeg siv - qhov no yog tib lub Highload, txij li lub sijhawm no peb ua haujlwm nrog kaum tawm lab tus neeg siv tib lub sijhawm.
Feem ntau peaks ntawm loads tshwm sim thaum hnub so, tab sis qhov kev nce qib no nyob rau theem sib txawv kiag li. Xyoo ua ntej kawg, hnub ntawm kev tshaj tawm, peb muag 27 units ntawm cov khoom. Cov ntaub ntawv tau ua tiav rau ntau tshaj li ib nrab ib teev, uas ua rau tsis yooj yim rau cov neeg siv. Cov neeg siv tau txais khoom plig rau kev koom tes, tab sis nws tau pom tseeb tias cov txheej txheem yuav tsum tau nrawm dua.
Thaum pib ntawm 2019, peb txiav txim siab tias peb xav tau ElasticSearch. Rau ib xyoos tag nrho, peb tau teeb tsa kev ua tiav ntawm cov ntaub ntawv tau txais hauv Elastic thiab lawv qhov kev tshaj tawm hauv api ntawm daim ntawv thov mobile thiab lub vev xaib. Yog li ntawd, xyoo tom ntej thaum lub sijhawm sib tw, peb tau ua tiav 15 nkag hauv 131 feeb.
Txij li thaum peb muaj ntau tus neeg xav yuav cov khoom thiab koom nrog kev kos duab ntawm cov khoom plig hauv kev txhawb nqa, qhov no yog qhov kev ntsuas ib ntus. Tam sim no peb tab tom xa cov ntaub ntawv tshiab rau Elastic, tab sis yav tom ntej peb npaj yuav hloov cov ntaub ntawv khaws tseg rau lub hli dhau los rau Postgres ua qhov chaw khaws cia. Txhawm rau kom tsis txhob txhaws qhov Elastic Performance index, uas kuj muaj nws cov kev txwv.
Cov ntsiab lus / Cov lus xaus
Tam sim no, peb tau pauv tag nrho cov kev pabcuam uas peb xav tau Elastic thiab tau nres rau qhov no rau tam sim no. Tam sim no peb tab tom tsim qhov Performance index hauv Elastic nyob rau sab saum toj ntawm lub ntsiab tseem ceeb nyob rau hauv Postgres, uas yuav siv sij hawm dhau tus neeg siv khoom thauj.
Nyob rau hauv lub neej yav tom ntej, peb npaj yuav hloov cov kev pabcuam yog tias peb nkag siab tias cov ntaub ntawv thov dhau los ua ntau haiv neeg thiab tau tshawb nrhiav cov kab tsis txwv. Qhov no tsis yog txoj haujlwm rau Postgres lawm.
Yog tias peb xav tau cov ntawv nyeem puv hauv kev ua haujlwm lossis yog tias peb muaj ntau yam kev tshawb fawb sib txawv, ces peb twb paub tias qhov no yuav tsum tau muab txhais ua Elastic.
βββ
Ua tsaug rau kev nyeem ntawv. Yog tias koj lub tuam txhab tseem siv ElasticSearch thiab muaj nws tus kheej cov teeb meem siv, qhia rau peb. Nws yuav nthuav kom paub tias lwm tus zoo li cas π
Tau qhov twg los: www.hab.com