I-Architecture namandla e-Tarantool Data Grid

Ngo-2017, sawina umncintiswano wokuthuthukisa umnyombo wokuthengiselana webhizinisi lokutshalwa kwezimali le-Alfa-Bank futhi saqala umsebenzi (ku-HighLoad++ 2018 ngombiko womnyombo webhizinisi lokutshala imali. wakhuluma UVladimir Drynkin, inhloko yebhizinisi lokutshalwa kwezimali kwe-Alfa Bank). Lolu hlelo bekufanele luhlanganise idatha yokwenziwe evela emithonjeni ehlukene ngamafomethi ahlukahlukene, ilethe idatha efomini elihlanganisiwe, iyigcine futhi inikeze ukufinyelela kuyo.

Phakathi nenqubo yokuthuthukiswa, isistimu yashintsha futhi yazuza ukusebenza, futhi ngesikhathi esithile saqaphela ukuthi sasihlanganisa okuthile okungaphezu nje kwesofthiwe yohlelo lokusebenza eyakhiwe ukuxazulula uhla lwemisebenzi echazwe ngokuqinile: siphumelele. uhlelo lokwakha izinhlelo zokusebenza ezisabalalisiwe ezinokugcinwa okuqhubekayo. Ulwazi esilutholile lwakha isisekelo somkhiqizo omusha - I-Tarantool Data Grid (TDG).

Ngifuna ukukhuluma ngezakhiwo ze-TDG kanye nezixazululo esize kuzo ngesikhathi senqubo yokuthuthukiswa, ngikwethule ekusebenzeni okuyinhloko futhi ngibonise ukuthi umkhiqizo wethu ungaba kanjani isisekelo sokwakha izixazululo eziphelele.

Ngokwezakhiwo, sihlukanise uhlelo ngokwehlukana izindima, ngayinye enesibopho sokuxazulula uhla oluthile lwezinkinga. Isicelo esisodwa esisebenzayo sisebenzisa uhlobo lwendima eyodwa noma ngaphezulu. Kungaba nezindima ezimbalwa zohlobo olufanayo kuqoqo:

Isixhumi

Isixhumi sinesibopho sokuxhumana nomhlaba wangaphandle; umsebenzi wayo ukwamukela isicelo, ukusihlaziya, futhi uma lokhu kuphumelela, bese uthumela idatha ukuze icutshungulwe kumprosesa wokufaka. Sisekela amafomethi we-HTTP, SOAP, Kafka, FIX. Izakhiwo zikuvumela ukuthi ungeze usekelo lwamafomethi amasha, ngokusekelwa kwe-IBM MQ iyeza maduze. Uma ukuncozulula isicelo kuhlulekile, isixhumi sizobuyisela iphutha; uma kungenjalo, izophendula ngokuthi isicelo sicutshungulwe ngempumelelo, noma ngabe kwenzeke iphutha phakathi nokucutshungulwa kwaso okuqhubekayo. Lokhu kwenziwa ngokuqondile ukuze kusebenze nezinhlelo ezingakwazi ukuphinda izicelo - noma, ngokuphambene nalokho, zenze ngokuphikelela kakhulu. Ukuze ungalahlekelwa idatha, kusetshenziswe umugqa wokulungisa: into iqala ukungena kuyo futhi kuphela ngemva kokukhishwa okuphumelelayo kokucubungula kuyo. Umlawuli angathola izexwayiso mayelana nezinto ezisele kulayini wokulungisa, futhi ngemva kokuqeda iphutha lesofthiwe noma ukwehluleka kwehadiwe, zama futhi.

Iphrosesa yokufaka

Iphrosesa ye-Input ihlukanisa idatha etholiwe ngokuya ngezici zesici futhi ibiza amaphrosesa afanelekile. Izibambi ziyikhodi ye-Lua esebenza kubhokisi lesihlabathi, ngakho azikwazi ukuphazamisa ukusebenza kwesistimu. Kulesi sigaba, idatha ingancishiswa ibe yifomu elidingekayo, futhi, uma kunesidingo, inombolo engafanele yemisebenzi ingaqaliswa engakwazi ukusebenzisa i-logic edingekayo. Isibonelo, kumkhiqizo we-MDM (Master Data Management) owakhiwe ku-Tarantool Data Grid, lapho ungeza umsebenzisi omusha, ukuze ungabambezeli ukucutshungulwa kwesicelo, siqala ukudalwa kwerekhodi legolide njengomsebenzi ohlukile. I-sandbox isekela izicelo zokufunda, ukushintsha nokwengeza idatha, ikuvumela ukuthi wenze umsebenzi othile kuzo zonke izindima zohlobo lwesitoreji kanye nokuhlanganisa umphumela (imephu/ukunciphisa).

Izibambi zingachazwa kumafayela:

sum.lua

local x, y = unpack(...)
return x + y

Bese, kumenyezelwe ekucushweni:

functions:
  sum: { __file: sum.lua }

Kungani uLua? IsiLua siwulimi olulula kakhulu. Ngokusekelwe kokuhlangenwe nakho kwethu, amahora ambalwa ngemva kokukwazi, abantu baqala ukubhala ikhodi exazulula inkinga yabo. Futhi laba akubona kuphela abathuthukisi abaqeqeshiwe, kodwa, isibonelo, abahlaziyi. Ngaphezu kwalokho, ngenxa ye-jit compiler, i-Lua igijima ngokushesha okukhulu.

Storage

Isitoreji sigcina idatha eqhubekayo. Ngaphambi kokulondoloza, idatha iqinisekiswa ngokumelene ne-schema yedatha. Ukuchaza isifunda sisebenzisa ifomethi enwetshiwe I-Apache Avro. Isibonelo:

{
    "name": "User",
    "type": "record",
    "logicalType": "Aggregate",
    "fields": [ 
        { "name": "id", "type": "string"}, 
        {"name": "first_name", "type": "string"}, 
        {"name": "last_name", "type": "string"} 
    ], 
    "indexes": ["id"] 
}

Ngokusekelwe kule ncazelo, i-DDL (Data Definition Language) ikhiqizwa ngokuzenzakalelayo i-Tarantula DBMS kanye IgrafuQL i-schema sokufinyelela idatha.

Ukuphindaphinda kwedatha ye-Asynchronous kuyasekelwa (kunezinhlelo zokwengeza eyodwa evumelanisiwe).

Iphrosesa yokuphuma

Kwesinye isikhathi kuyadingeka ukwazisa abathengi bangaphandle mayelana nokufika kwedatha entsha; ngale njongo, kunendima ye-Output processor. Ngemva kokulondoloza idatha, ingadluliselwa kusibambi esihambisanayo (isibonelo, ukuyiletha efomini elidingwa umthengi) - bese idluliselwa kusixhumi ukuze siyithumele. Ulayini wokulungisa nawo uyasetshenziswa lapha: uma kungekho muntu owamukele into, umlawuli angazama futhi ngemva kwesikhathi.

Ukukala

Isixhumi, iphrosesa yokufaka, nezindima zephrosesa azinasimo, okusivumela ukuthi sikale isistimu ngokuvundlile ngokungeza izimo ezintsha zohlelo lokusebenza ngohlobo lwendima olufunekayo olunikwe amandla. Isitoreji sisetshenziselwa ukukala okuvundlile indlela ukuhlela iqoqo kusetshenziswa amabhakede abonakalayo. Ngemva kokwengeza iseva entsha, amanye amabhakede asuka kumaseva amadala athuthelwa kuseva entsha ngemuva; lokhu kwenzeka ngokusobala kubasebenzisi futhi akuthinti ukusebenza kwalo lonke uhlelo.

Izakhiwo Zedatha

Izinto zingaba zinkulu kakhulu futhi ziqukathe ezinye izinto. Siqinisekisa i-athomu yokwengeza nokubuyekeza idatha ngokugcina into enakho konke ukuncika ebhakedeni elilodwa elibonakalayo. Lokhu kuvimbela into ekubeni "isabalale" kumaseva aphathekayo amaningana.

Ukuguqulwa kuyasekelwa: isibuyekezo ngasinye sento sidala inguqulo entsha, futhi singahlala sithatha ucezu lwesikhathi futhi sibone ukuthi umhlaba wawubukeka kanjani ngaleso sikhathi. Ngedatha engadingi umlando omude, singakhawulela inani lezinguqulo noma sigcine eyodwa kuphela - eyakamuva - okungukuthi, sikhubaze ukuguqulelwa kohlobo oluthile. Ungakwazi futhi ukukhawulela umlando ngesikhathi: isibonelo, susa zonke izinto zohlobo oluthile ezindala kunonyaka ongu-1. Ukufaka kungobo yomlando nakho kuyasekelwa: singakwazi ukulayisha izinto ezindala kunesikhathi esishiwo, sikhulule isikhala kuqoqo.

Imisebenzi

Phakathi kwezici ezithakazelisayo, kufanelekile ukuqaphela ikhono lokuqalisa imisebenzi ngeshejuli, ngesicelo somsebenzisi, noma ngokohlelo kusuka ku-sandbox:

Lapha sibona enye indima - umgijimi. Le ndima ayinaso isimo, futhi izimo ezengeziwe zohlelo lokusebenza ezinale ndima zingangezwa kuqoqo njengoba kudingeka. Isibopho somgijimi ukuqedela imisebenzi. Njengoba kushiwo, kungenzeka ukukhiqiza imisebenzi emisha kusuka kubhokisi lesihlabathi; zigcinwa emgqeni wokugcina bese zibulawa kumgijimi. Lolu hlobo lomsebenzi lubizwa ngokuthi uJobe. Siphinde sibe nohlobo lomsebenzi olubizwa ngokuthi i-Task - lena imisebenzi echazwe ngumsebenzisi esebenza ngohlelo (kusetshenziswa i-cron syntax) noma ngokufunwa. Ukuze siqalise futhi silandelele imisebenzi enjalo, sinomphathi wemisebenzi okahle. Ukuze lokhu kusebenza kutholakale, kufanele unike amandla indima yomhleli; le ndima inombuso, ngakho-ke ayilingani, okuyinto, nokho, engadingeki; ngesikhathi esifanayo, njengazo zonke ezinye izindima, ingaba ne-replica eqala ukusebenza uma inkosi inqaba ngokuzumayo.

Umgawuli

Enye indima ibizwa nge-logger. Iqoqa amalogi kuwo wonke amalungu eqoqo futhi inikeze isixhumi esibonakalayo ukuze ilayishe futhi ibukwe ngohlelo lwewebhu.

Izinsizakalo

Kuhle ukusho ukuthi uhlelo lwenza kube lula ukudala izinsizakalo. Efayeleni lokumisa, ungacacisa ukuthi yiziphi izicelo ezithunyelwa kusibambi esibhalwe umsebenzisi esisebenza ku-sandbox. Kulesi sibambi, ungakwazi, isibonelo, ukusebenzisa uhlobo oluthile lombuzo wokuhlaziya bese ubuyisela umphumela.

Isevisi ichazwe efayeleni lokumisa:

services:
   sum:
      doc: "adds two numbers"
      function: sum
      return_type: int
      args:
         x: int
         y: int

I-GraphQL API ikhiqizwa ngokuzenzakalelayo futhi isevisi itholakalela ukushayela:

query {
   sum(x: 1, y: 2) 
}

Lokhu kuzobiza isibambi sumokuzobuyisela umphumela:

3

Iphrofayela yombuzo kanye namamethrikhi

Ukuze siqonde ukusebenza kwesistimu nezicelo zokuphrofayela, sisebenzise usekelo lwephrothokholi ye-OpenTracing. Uhlelo lungathumela ulwazi mayelana nesidingo kumathuluzi asekela le phrothokholi, njenge-Zipkin, okuzokuvumela ukuthi uqonde ukuthi isicelo senziwe kanjani:

Ngokwemvelo, uhlelo luhlinzeka ngamamethrikhi angaphakathi angaqoqwa kusetshenziswa i-Prometheus futhi aboniswe kusetshenziswa i-Grafana.

Sebenzisa

I-Tarantool Data Grid ingasatshalaliswa isuka kumaphakeji e-RPM noma kungobo yomlando, kusetshenziswa insiza evela ekusabalaliseni noma i-Ansible, kukhona nokusekelwa kwe-Kubernetes (I-Tarantool Kubernetes Operator).

Uhlelo lokusebenza olusebenzisa ingqondo yebhizinisi (ukulungiselela, izibambi) lulayishwa kuqoqo le-Tarantool Data Grid esetshenzisiwe ngendlela yengobo yomlando nge-UI noma kusetshenziswa umbhalo nge-API esiyinikezwe yithi.

Izibonelo Zezicelo

Yiziphi izinhlelo zokusebenza ezingadalwa kusetshenziswa i-Tarantool Data Grid? Eqinisweni, imisebenzi eminingi yebhizinisi ngandlela thize ihlobene nokucubungula, ukugcina kanye nokufinyelela ukugeleza kwedatha. Ngakho-ke, uma unemithombo emikhulu yedatha edinga ukugcinwa ngokuphephile futhi ifinyelelwe, umkhiqizo wethu ungakongela isikhathi esiningi sokuthuthuka futhi ugxile kungqondongqondo yebhizinisi lakho.

Isibonelo, sifuna ukuqoqa ulwazi mayelana nemakethe yezindlu, ukuze esikhathini esizayo, isibonelo, sizoba nolwazi mayelana nokunikezwa okungcono kakhulu. Kulokhu, sizogqamisa imisebenzi elandelayo:

1. Amarobhothi aqoqa ulwazi emithonjeni evulekile azoba imithombo yethu yedatha. Ungakwazi ukuxazulula le nkinga usebenzisa izixazululo ezenziwe ngomumo noma ikhodi yokubhala nganoma yiluphi ulimi.
2. Okulandelayo, i-Tarantool Data Grid izokwamukela futhi ilondoloze idatha. Uma ifomethi yedatha evela emithonjeni ehlukene ihlukile, ungabhala ikhodi ngesi-Lua ezokwenza ukuguqulwa ibe yifomethi eyodwa. Esigabeni sangaphambi kokucubungula, uzokwazi, isibonelo, ukuhlunga okunikezwayo okuyimpinda noma ukwengeza ulwazi olumayelana nama-ejenti asebenza emakethe kusizindalwazi.
3. Manje usuvele unesixazululo esingakala kuqoqo elingagcwaliswa ngedatha futhi wenze ukukhetha kwedatha. Khona-ke ungakwazi ukusebenzisa ukusebenza okusha, isibonelo, bhala isevisi ezokwenza isicelo sedatha futhi unikeze isipho esizuzisa kakhulu ngosuku - lokhu kuzodinga imigqa embalwa efayeleni lokucushwa kanye nekhodi encane ye-Lua.

Yini okulandelayo?

Okuhamba phambili kwethu ukwenza ngcono ukusetshenziswa kalula kwentuthuko I-Tarantool Data Grid. Isibonelo, lena i-IDE enokwesekwa kwezibambi zokwenza iphrofayela nokususa iphutha ezisebenza kubhokisi lesihlabathi.

Futhi sinaka kakhulu izindaba zokuphepha. Njengamanje sithola isitifiketi yi-FSTEC yase-Russia ukuze siqinisekise izinga eliphezulu lokuphepha futhi sihlangabezane nezimfuneko zokuthola isitifiketi semikhiqizo yesofthiwe esetshenziswa ezinhlelweni zolwazi lwedatha yomuntu siqu kanye nezinhlelo zolwazi zikahulumeni.

Source: www.habr.com