Mhoro, Habr! Zita rangu ndiMaxim Vasiliev, ndinoshanda semuongorori uye maneja weprojekiti kuFINCH. Nhasi ndinoda kukuudza sei, tichishandisa ElasticSearch, takakwanisa kugadzirisa zvikumbiro zvemamiriyoni gumi nemashanu mumaminetsi matanhatu uye nekukwenenzvera mitoro yemazuva ese pane saiti yemumwe wevatengi vedu. Nehurombo, isu tichafanirwa kuita pasina mazita, sezvo isu tine NDA, isu tinovimba kuti zviri muchinyorwa hazvizotambura neizvi. Handeyi.
Mashandiro anoita chirongwa
Pakumashure kwedu tinogadzira masevhisi anovimbisa kushanda kwesaiti yemutengi wedu uye nharembozha. Iyo general chimiro inogona kuoneka mudhayagiramu:
Mukuita basa, isu tinogadzira nhamba huru yekutengeserana: kutenga, kubhadhara, kutengeserana nemakero evashandisi, ayo isu tinochengeta matanda akawanda, uye zvakare kupinza uye kutumira iyi data kune ekunze masisitimu.
Kune zvakare maitiro ekudzokera kumashure patinogamuchira data kubva kumutengi uye toiendesa kumushandisi. Mukuwedzera, kuchine maitiro ekushanda nemabhadharo uye mapurogiramu ebhonasi.
Mhedziso pfupi
Pakutanga, takashandisa PostgreSQL seyo chete chitoro chedata. Zvakanakira zvaro zvakajairwa neDBMS: kuvepo kwekutengesa, mutauro wakagadziridzwa wekutora data, huwandu hwakawanda hwezvishandiso zvekubatanidza; zvakasanganiswa nekuita kwakanaka zvakagutsa zvatinoda kwenguva yakati rebei.
Isu takachengeta zvachose data rese muPostgres: kubva pakutengeserana kuenda kunhau. Asi nhamba yevashandisi yakakura, uye pamwe nayo nhamba yezvikumbiro.
Kuti unzwisise, nhamba yegore yezvikamu mu 2017 padesktop site chete yaiva mamiriyoni 131. Muna 2018 - mamiriyoni 125. Muna 2019 zvakare mamiriyoni 130. Wedzera imwe 100-200 mamiriyoni kubva kune mobile version yesaiti uye mobile application, uye iwe achagamuchira nhamba yakakura kwazvo yezvikumbiro.
Sezvo purojekiti yaikura, Postgres yakanga isisakwanise kubata nemutoro; hatina kukwanisa kuenderera mberi - nhamba huru yemibvunzo yakasiyana yakaonekwa, iyo isu hatina kukwanisa kugadzira nhamba yakakwana ye indexes.
Takaona kuti paidiwa zvimwe zvitoro zvedata zvaizozadzisa zvatinoda uye kubvisa mutoro paPostgreSQL. Elasticsearch uye MongoDB zvakaonekwa sezvingangoita sarudzo. Vekupedzisira vakarasika pane zvinotevera:
- Inononoka indexing kumhanya sezvo huwandu hwe data muma indexes hunokura. NeElastic, kumhanya hakuenderane nehuwandu hwe data.
- Hapana tsvakiridzo yemavara akazara
Saka isu takasarudza Elastic isu uye takagadzirira shanduko.
Chinja kuElastic
1. Takatanga shanduko ine nzvimbo yekutsvaga-yekutengesa sevhisi. Mutengi wedu ane huwandu hwe70 mapoinzi ekutengesa, uye panguva imwe chete akati wandei ekutsvaga anodiwa pawebhusaiti uye mukushandisa:
- Tsvaga mavara nezita reguta
- Geosearch mukati meiyo radius yakapihwa kubva pane imwe nzvimbo. Semuenzaniso, kana mushandisi achida kuona kuti ndedzipi nzvimbo dzekutengesa dziri padyo nemba yake.
- Tsvaga neskweya yakapihwa - mushandisi anotaridza sikweya pamepu, uye mapoinzi ese ari muradius iyi anoratidzwa kwaari.
- Tsvaga nemamwe masefa. Mapoinzi ekutengesa anosiyana kubva kune mumwe kune akasiyana
Kutaura nezvesangano, muPostgres isu tine data sosi yemepu uye nhau, uye muElastic tinoita Snapshots kubva kune yekutanga data. Icho chokwadi ndechekuti pakutanga Postgres yaisakwanisa kubata nekutsvaga kwese maitiro. Kwete chete kwaive nemaindex akawanda, ivo vaigona zvakare kupindirana, saka Postgres scheduler yakarasika uye haina kunzwisisa kuti ndeipi index yekushandisa.
2. Chinotevera mumutsetse chaiva chikamu chenhau. Zvinyorwa zvinoonekwa pasaiti zuva rega rega kuitira kuti mushandisi asarasika mukuyerera kweruzivo, iyo data inofanirwa kurongwa isati yaburitswa. Izvi ndizvo zvinotsvakwa: pane saiti iwe unogona kutsvaga nemavara machisi, uye panguva imwechete batanidza mamwe mafirita, sezvo iwo akaitwa zvakare kuburikidza neElastic.
3. Ipapo takatamisa transaction process. Vashandisi vanogona kutenga chimwe chigadzirwa pasaiti uye kutora chikamu mukudhirowa mibairo. Mushure mekutenga kwakadaro, tinogadzira data yakawanda, kunyanya pakupera kwevhiki nezororo. Kuenzanisa, kana pamazuva akajairwa nhamba yekutenga iri pakati pe1,5-2 miriyoni, saka pamazororo nhamba yacho inogona kusvika 53 miriyoni.
Panguva imwecheteyo, iyo data inofanirwa kugadziriswa mune shoma shoma yenguva-vashandisi havafarire kumirira mazuva akati wandei kuti vawane mhinduro. Iko hakuna nzira yekuzadzisa mazuva ekupedzisira akadaro kuburikidza nePostgres - taiwanzogamuchira zvivharo, uye patakanga tichigadzirisa zvikumbiro zvese, vashandisi havana kukwanisa kutarisa kuti vakagamuchira mibairo here kana kuti kwete. Izvi hazvinakidze kubhizinesi, saka takatamisa kugadzirisa kuElasticsearch.
Periodicity
Iye zvino zvigadziriso zvinogadziriswa chiitiko-based, zvinoenderana neanotevera mamiriro:
- Points dzekutengesa. Patinongogamuchira data kubva kune yekunze sosi, isu tinobva tangotanga update.
- News. Pakangogadziriswa chero nhau pane saiti, inotumirwa otomatiki kuElastic.
Pano zvakare zvakakodzera kutaura zvakanakira Elastic. MuPostgres, kana uchitumira chikumbiro, unofanirwa kumirira kusvika yanyatsogadzirisa marekodhi ese. Iwe unogona kutumira zviuru gumi marekodhi kuElastic uye wotanga kushanda nekukasira, usingamiriri kuti marekodhi agovere maShards ese. Ehe, imwe Shard kana Replica inogona kusaona iyo data ipapo, asi zvese zvichave zviripo munguva pfupi iri kutevera.
Kubatanidza nzira
Pane nzira mbiri dzekubatanidza neElastic:
- Kuburikidza nemutengi wekuzvarwa pamusoro peTCP. Mutyairi wekuzvarwa ari kufa zvishoma nezvishoma: haichatsigirwi, ine syntax isingaite. Naizvozvo, isu hatisi kuishandisa uye tinoedza kuisiya zvachose.
- Kuburikidza neHTTP interface yaunogona kushandisa zvese JSON zvikumbiro uye Lucene syntax. Yekupedzisira injini yemavara inoshandisa Elastic. Mune iyi vhezheni, tinowana kugona kweBatch kuburikidza nezvikumbiro zveJSON pamusoro peHTTP. Iyi ndiyo sarudzo yatiri kuedza kushandisa.
Nekuda kweiyo HTTP interface, tinogona kushandisa maraibhurari anopa asynchronous kuita kweHTTP mutengi. Tinogona kutora mukana weBatch uye iyo asynchronous API, iyo inozopedzisira yapa kuita kwepamusoro, iyo yakabatsira zvakanyanya mumazuva ekusimudzira kukuru (zvimwe pane izvi pazasi)
Dzimwe nhamba dzekuenzanisa:
- Kuchengetedza vashandisi vePostgres bounty mumatambo makumi maviri pasina kuunganidza: 20 marekodhi mumasekonzi makumi mana.
- Elastic + reactive mutengi kune gumi tambo + batch yezvinhu zana: 10 marekodhi mumasekonzi gumi nerimwe.
- Elastic + reactive mutengi kune gumi shinda + batch ye10 zvinhu: 23801684 inorekodha mumaminetsi mana
Iye zvino tanyora maneja wekukumbira weHTTP anovaka JSON seBatch/non-Batch uye anoitumira kuburikidza nechero HTTP mutengi, zvisinei neraibhurari. Iwe unogona zvakare kusarudza kutumira zvikumbiro synchronously kana asynchronously.
Mune kumwe kubatanidzwa, isu tichiri kushandisa yepamutemo mutengi wekutakura, asi iyi ingori nyaya yeinotevera refactoring. Muchiitiko ichi, mutengi wetsika akavakirwa pahwaro hweSpring WebClient inoshandiswa kugadzirisa.
Kusimudzirwa kukuru
Kamwe chete pagore, purojekiti inosimudzira kukwidziridzwa kwevashandisi - iyi yakafanana Highload, sezvo panguva ino isu tinoshanda nemakumi emamiriyoni evashandisi panguva imwe chete.
Kazhinji, mitoro yepamusoro inoitika panguva yezororo, asi kukwidziridzwa uku kuri padanho rakasiyana zvachose. Gore rakapfuura rapfuura, pazuva rekusimudzira, takatengesa 27 zvikamu zvezvinhu. Iyo data yakatora inopfuura hafu yeawa kugadzirisa, izvo zvakakonzera kusagadzikana kune vashandisi. Vashandisi vakagamuchira mibairo yekutora chikamu, asi zvakava pachena kuti maitiro aifanira kukurumidzirwa.
Pakutanga kwa2019, takasarudza kuti ElasticSearch yaidiwa. Kwegore rose, isu takaronga kugadziridzwa kweiyo yakagamuchirwa data muElastic uye kuburitswa kwavo muapi ye mobile application uye webhusaiti. Somugumisiro, gore rakatevera panguva yekusimudzira takagadzirisa 15 mapindiro mumaminetsi matanhatu.
Sezvo isu tine vanhu vazhinji vanoda kutenga chigadzirwa uye kutora chikamu mukudhirowa kwemubairo mukukwidziridzwa kwedu, ichi chiyero chenguva pfupi. Iye zvino tinotumira ruzivo rwazvino kuElastic, asi mune ramangwana tinoronga kuendesa ruzivo rwakachengetwa kwemwedzi yapfuura kuPostgres sechigarire chekuchengetedza. Kuti urege kuvhara iyo Elastic index, iyo inewo miganhu yayo.
Mhedziso/ Mhedziso
Parizvino, taendesa masevhisi ese ataida kuElastic uye takambomira izvozvi. Iye zvino tiri kuvaka index muElastic pamusoro peiyo huru inopfuurira kuchengetedza muPostgres, iyo inotora pamutoro wemushandisi.
Mune ramangwana, isu tinoronga kuendesa masevhisi kana tikanzwisisa kuti chikumbiro chedata chiri kuramba chichisiyana uye chiri kutsvagwa nhamba isingaperi yemakoramu. Iri harisisiri basa rePostgres.
Kana tichida tsvakiridzo yemavara akazara mukushanda kana kana tiine akawanda akasiyana maitiro ekutsvaga, saka isu tatoziva kuti izvi zvinoda kushandurirwa muElastic.
βββ
Ndatenda nekuverenga. Kana kambani yako ichishandisawo ElasticSearch uye iine yako yekumisikidza makesi, ndapota tiudze. Zvichave zvinonakidza kuziva kuti vamwe vari sei π
Source: www.habr.com