Pa Habr! Dzina langa ndi Maxim Vasiliev, ndimagwira ntchito ngati katswiri komanso woyang'anira polojekiti ku FINCH. Lero ndikufuna ndikuuzeni momwe, pogwiritsa ntchito ElasticSearch, tinatha kukonza zopempha 15 miliyoni m'mphindi 6 ndikukhathamiritsa katundu watsiku ndi tsiku patsamba la m'modzi mwamakasitomala athu. Tsoka ilo, tikhala opanda mayina, popeza tili ndi NDA, tikukhulupirira kuti zomwe zili m'nkhaniyi sizidzavutika ndi izi. Tiyeni tizipita.
Momwe polojekitiyi imagwirira ntchito
Kumbuyo kwathu, timapanga ntchito zomwe zimatsimikizira magwiridwe antchito amasamba a kasitomala athu komanso kugwiritsa ntchito mafoni. Kamangidwe kake kangawonekere pachithunzichi:
Pogwira ntchito, timagwira ntchito zambiri: kugula, malipiro, ntchito ndi ndalama zogwiritsira ntchito, zomwe timasungira zipika zambiri, komanso kuitanitsa ndi kutumiza deta iyi ku machitidwe akunja.
Palinso njira zosinthira tikalandira deta kuchokera kwa kasitomala ndikutumiza kwa wogwiritsa ntchito. Kuphatikiza apo, pali njira zogwirira ntchito ndi malipiro ndi mapulogalamu a bonasi.
Mbiri yachidule
Poyambirira, tidagwiritsa ntchito PostgreSQL ngati malo okhawo osungira deta. Ubwino wake wokhazikika wa DBMS: kukhalapo kwa zochitika, chilankhulo chopangidwa ndi data, zida zambiri zophatikizira; kuphatikizidwa ndi magwiridwe antchito abwino adakwaniritsa zosowa zathu kwa nthawi yayitali.
Tidasunga zidziwitso zonse ku Postgres: kuchokera pazogulitsa mpaka nkhani. Koma chiwerengero cha ogwiritsa ntchito chinakula, ndipo ndi chiwerengero cha zopempha.
Kuti mumvetse, chiwerengero cha pachaka cha magawo mu 2017 kokha pa malo apakompyuta ndi 131 miliyoni. adzapeza zopempha zambiri.
Ndi kukula kwa polojekitiyi, Postgres anasiya kulimbana ndi katunduyo, tinalibe nthawi - mafunso ambiri adawonekera, omwe sitinathe kupanga chiwerengero chokwanira cha indexes.
Tidamvetsetsa kuti pakufunika masitolo ena a data omwe angatipatse zosowa zathu ndikuchotsa PostgreSQL. Elasticsearch ndi MongoDB adawonedwa ngati zotheka. Otsatirawa adataya mfundo zotsatirazi:
- Kuthamanga kwapang'onopang'ono pamene kuchuluka kwa deta muzolozera kumakula. Ndi Elastic, kuthamanga sikudalira kuchuluka kwa deta.
- Palibe mawu osakira
Chifukwa chake tidasankha Elastic tokha ndikukonzekera kusinthako.
Kusintha kwa Elastic
1. Tinayamba kusintha kuchokera kumalo ogulitsa ntchito yosaka. Makasitomala athu ali ndi zogulitsa pafupifupi 70, ndipo izi zimafuna mitundu ingapo yakusaka patsamba komanso kugwiritsa ntchito:
- Kusaka mawu ndi dzina la mzinda
- Geosearch mkati mwa utali wopatsidwa kuchokera pamalo ena. Mwachitsanzo, ngati wogwiritsa ntchito akufuna kuwona malo ogulitsa omwe ali pafupi kwambiri ndi nyumba yake.
- Sakani ndi lalikulu lomwe mwapatsidwa - wogwiritsa ntchito amajambula masikweya pamapu, ndipo mfundo zonse mu radius iyi zikuwonetsedwa kwa iye.
- Sakani ndi zosefera zina. Malo ogulitsa amasiyana mosiyanasiyana
Ngati tilankhula za bungwe, ndiye kuti mu Postgres tili ndi gwero la mapu ndi nkhani, ndipo mu Elastic Snapshots amatengedwa kuchokera ku deta yoyambirira. Chowonadi ndi chakuti poyambirira Postgres sakanatha kuthana ndi kusaka ndi njira zonse. Osati kokha kuti panali ma index ambiri, amathanso kuphatikizika, kotero kuti Postgres scheduler adatayika ndipo samamvetsetsa kuti agwiritse ntchito index iti.
2. Chotsatira pamzere chinali gawo la nkhani. Zofalitsa zimawonekera patsamba tsiku lililonse kuti wogwiritsa ntchito asatayike pakuyenda kwa chidziwitso, deta iyenera kusanjidwa isanatulutsidwe. Izi ndi zomwe kusaka kumafunikira: mutha kusaka tsambalo potengera mawu, ndipo nthawi yomweyo kulumikiza zosefera zina, popeza zimapangidwanso kudzera mu Elastic.
3. Kenako tinasuntha processing processing. Ogwiritsa ntchito amatha kugula chinthu china patsambalo ndikutenga nawo gawo pakupanga mphotho. Pambuyo pogula zoterezi, timakonza deta yambiri, makamaka kumapeto kwa sabata ndi tchuthi. Poyerekeza, ngati pamasiku wamba kuchuluka kwa zogula kuli kwinakwake pakati pa 1,5-2 miliyoni, ndiye patchuthi chiwerengerocho chikhoza kufika 53 miliyoni.
Panthawi imodzimodziyo, deta iyenera kukonzedwa mu nthawi yaifupi kwambiri - ogwiritsa ntchito sakonda kuyembekezera zotsatira kwa masiku angapo. Palibe njira yopezera masiku oterowo kudzera ku Postgres - nthawi zambiri tinkalandira zotsekera, ndipo pomwe tinkakonza zopempha zonse, ogwiritsa ntchito sakanatha kuwona ngati adalandira mphotho kapena ayi. Izi sizosangalatsa kwambiri pabizinesi, chifukwa chake tidasamutsira kukonza ku Elasticsearch.
Periodicity
Tsopano zosintha zimakonzedwa motengera zochitika, malinga ndi izi:
- Malo ogulitsa. Tikangolandira deta kuchokera ku gwero lakunja, timayamba nthawi yomweyo kusintha.
- Nkhani. Nkhani iliyonse ikangosinthidwa patsamba, imatumizidwa ku Elastic.
Apanso ndiyenera kutchula zabwino za Elastic. Ku Postgres, potumiza pempho, muyenera kudikirira mpaka itakonza zolembedwa zonse moona mtima. Mutha kutumiza ma rekodi 10 ku Elastic ndikuyamba kugwira ntchito nthawi yomweyo, osadikirira kuti zolembazo zigawidwe ku Shards zonse. Zachidziwikire, Shard kapena Replica ena sangawone nthawi yomweyo, koma zonse zipezeka posachedwa.
Njira zophatikizira
Pali njira ziwiri zophatikizira ndi Elastic:
- Kudzera mwa kasitomala mbadwa pa TCP. Dalaivala wakomweko akumwalira pang'onopang'ono: sakuthandizidwanso, ali ndi mawu ovuta kwambiri. Chifukwa chake, sitigwiritsa ntchito ndikuyesera kusiya.
- Kupyolera mu mawonekedwe a HTTP omwe angagwiritse ntchito zopempha za JSON ndi mawu a Lucene. Yomaliza ndi injini yamawu yomwe imagwiritsa ntchito Elastic. Mu mtundu uwu, timapeza kuthekera kwa Batch kudzera muzopempha za JSON pa HTTP. Iyi ndi njira yomwe tikuyesera kugwiritsa ntchito.
Chifukwa cha mawonekedwe a HTTP, titha kugwiritsa ntchito malaibulale omwe amapereka kukhazikitsidwa kosagwirizana kwa kasitomala wa HTTP. Titha kupezerapo mwayi pa Batch ndi asynchronous API, zomwe zimapangitsa kuti pakhale magwiridwe antchito apamwamba, omwe adathandizira kwambiri masiku akukwezedwa kwakukulu (zambiri pamunsimu)
Nambala zina zofananiza:
- Kupulumutsa ogwiritsa ntchito zabwino za Postgres mu ulusi 20 popanda magulu: 460713 amalemba mumasekondi 42
- Makasitomala okhazikika + okhazikika a ulusi 10 + gulu la zinthu 1000: 596749 amalemba masekondi 11
- Makasitomala okhazikika + okhazikika a ulusi 10 + batch pazinthu 1000: 23801684 zolowera m'mphindi 4
Tsopano talemba woyang'anira pempho la HTTP yemwe amamanga JSON ngati Batch/osati Batch ndikutumiza kudzera pa kasitomala aliyense wa HTTP, mosasamala za laibulale. Mukhozanso kusankha kutumiza zopempha synchronously kapena asynchronously.
Muzophatikiza zina, timagwiritsabe ntchito kasitomala woyendera, koma iyi ndi nkhani yakukonzanso kotsatira. Pankhaniyi, kasitomala wokhazikika womangidwa pamaziko a Spring WebClient amagwiritsidwa ntchito pokonza.
kukwezedwa kwakukulu
Kamodzi pachaka, pulojekitiyi imakhala ndi kukwezedwa kwakukulu kwa ogwiritsa ntchito - iyi ndi Highload yomweyi, chifukwa panthawiyi timagwira ntchito ndi mamiliyoni ambiri ogwiritsa ntchito nthawi imodzi.
Nthawi zambiri nsonga za katundu zimachitika patchuthi, koma kukwezedwa kumeneku kumakhala pamlingo wosiyana kwambiri. Chaka chatha, tsiku lotsatsa, tinagulitsa katundu wa 27. Detayo idakonzedwa kwa nthawi yopitilira theka la ola, zomwe zidayambitsa zovuta kwa ogwiritsa ntchito. Ogwiritsa ntchito adalandira mphotho zotenga nawo gawo, koma zidawonekeratu kuti ntchitoyi ikuyenera kufulumizitsidwa.
Kumayambiriro kwa 2019, tidaganiza kuti tikufuna ElasticSearch. Kwa chaka chathunthu, tidakonza zokonza zomwe zidalandilidwa mu Elastic ndi kuperekedwa kwawo mu api ya pulogalamu yam'manja ndi tsamba lawebusayiti. Zotsatira zake, chaka chotsatira cha kampeni, tinakonza 15 olowa m'mphindi 131.
Popeza tili ndi anthu ambiri amene akufuna kugula katundu ndi kutenga nawo mbali pa kujambula kwa mphoto muzotsatsa, ichi ndi chiyeso chakanthawi. Tsopano tikutumiza zidziwitso zaposachedwa ku Elastic, koma mtsogolomo tikukonzekera kusamutsa zidziwitso zakale za miyezi yapitayi ku Postgres ngati malo osungira okhazikika. Kuti musatseke Elastic index, yomwe ilinso ndi malire ake.
Mapeto/Mapeto
Pakadali pano, tasamutsa ntchito zonse zomwe timafuna ku Elastic ndipo tayimilira pa izi pakadali pano. Tsopano tikumanga cholozera mu Elastic pamwamba pa chosungira chachikulu chokhazikika ku Postgres, chomwe chimatenga kuchuluka kwa ogwiritsa ntchito.
M'tsogolomu, tikukonzekera kusamutsa mautumiki ngati timvetsetsa kuti pempho la data limakhala losiyana kwambiri ndipo limafufuzidwa ndi chiwerengero chopanda malire. Iyi sinalinso ntchito ya Postgres.
Ngati tikufuna kusaka kwathunthu pamagwiritsidwe kapena ngati tili ndi njira zambiri zofufuzira, ndiye kuti tikudziwa kale kuti izi ziyenera kumasuliridwa kukhala Elastic.
βββ
Zikomo powerenga. Ngati kampani yanu imagwiritsanso ntchito ElasticSearch ndipo ili ndi milandu yake, tiuzeni. Zidzakhala zosangalatsa kudziwa momwe ena alili π
Source: www.habr.com