Architecture thiab peev xwm ntawm Tarantool Data Grid

Hauv 2017, peb yeej qhov kev sib tw los tsim cov kev hloov pauv tseem ceeb ntawm Alfa-Bank kev nqis peev kev lag luam thiab pib ua haujlwm (ntawm HighLoad ++ 2018 nrog rau daim ntawv qhia txog lub hauv paus ntawm kev lag luam peev. ua Vladimir Drynkin, tus thawj coj ntawm kev lag luam tseem ceeb ntawm kev lag luam peev ntawm Alfa Bank). Qhov system no yuav tsum tau sau cov ntaub ntawv sib pauv los ntawm ntau qhov chaw hauv ntau hom, nqa cov ntaub ntawv mus rau hauv ib daim ntawv sib koom ua ke, khaws cia thiab muab kev nkag mus rau nws.

Thaum lub sij hawm txoj kev loj hlob, lub kaw lus evolved thiab nrhiav tau functionality, thiab nyob rau hauv tej lub sij hawm peb pom tau hais tias peb tau crystallizing ib yam dab tsi ntau npaum li cas tsuas yog daim ntawv thov software tsim los daws tau ib tug nruj me ntsis ntau yam ntawm cov hauj lwm: peb ua tau zoo. system rau kev tsim cov ntawv thov faib nrog kev khaws cia tsis tu ncua. Cov kev paub uas peb tau txais tau tsim lub hauv paus ntawm cov khoom tshiab - Tarantool Data Grid (TDG).

Kuv xav tham txog TDG architecture thiab cov kev daws teeb meem uas peb tau los rau hauv txoj kev txhim kho, qhia koj txog cov haujlwm tseem ceeb thiab qhia tias peb cov khoom tuaj yeem ua lub hauv paus rau kev tsim cov kev daws teeb meem.

Architecturally, peb faib lub system rau nyias luag hauj lwm, txhua tus muaj lub luag haujlwm los daws cov teeb meem qee yam. Ib daim ntawv thov ua haujlwm nkaus xwb siv ib lossis ntau hom haujlwm. Nws tuaj yeem muaj ntau lub luag haujlwm ntawm tib hom hauv ib pawg:

connector

Connector yog lub luag haujlwm rau kev sib txuas lus nrog lub ntiaj teb sab nraud; nws txoj haujlwm yog lees txais qhov kev thov, txheeb xyuas nws, thiab yog tias qhov no ua tiav, tom qab ntawd xa cov ntaub ntawv rau kev ua haujlwm rau cov khoom siv nkag. Peb txhawb HTTP, SOAP, Kafka, FIX hom. Lub architecture tso cai rau koj yooj yim ntxiv kev txhawb nqa rau cov qauv tshiab, nrog kev txhawb nqa rau IBM MQ los sai sai. Yog hais tias parsing qhov kev thov ua tsis tau tejyam, tus connector yuav rov qab ua yuam kev; txwv tsis pub, nws yuav teb tias qhov kev thov tau ua tiav, txawm tias muaj qhov yuam kev tshwm sim thaum nws ua haujlwm ntxiv. Qhov no tau ua tiav tshwj xeeb txhawm rau ua haujlwm nrog cov tshuab uas tsis paub yuav ua li cas rov thov dua - lossis, ntawm qhov tsis sib xws, ua nws tsis tu ncua. Txhawm rau kom tsis txhob poob cov ntaub ntawv, kev kho kab yog siv: cov khoom ua ntej nkag mus rau hauv nws thiab tsuas yog tom qab kev ua tiav raug tshem tawm ntawm nws. Tus thawj coj tuaj yeem tau txais kev ceeb toom txog cov khoom seem hauv cov kab kho, thiab tom qab tshem tawm cov software yuam kev lossis kho vajtse tsis ua haujlwm, rov sim dua.

Input processor

Input processor faib cov ntaub ntawv tau txais raws li cov yam ntxwv nta thiab hu rau cov txheej txheem tsim nyog. Handlers yog Lua code uas khiav hauv sandbox, yog li lawv tsis tuaj yeem cuam tshuam rau kev ua haujlwm ntawm lub cev. Nyob rau theem no, cov ntaub ntawv tuaj yeem raug txo rau hauv daim ntawv xav tau, thiab, yog tias tsim nyog, tus naj npawb ntawm cov dej num tuaj yeem raug tsim tawm uas tuaj yeem ua raws li qhov tsim nyog logic. Piv txwv li, hauv MDM (Master Data Management) cov khoom tsim los ntawm Tarantool Data Grid, thaum ntxiv tus neeg siv tshiab, txhawm rau kom tsis txhob ua kom qeeb ntawm kev thov, peb pib tsim cov ntaub ntawv golden raws li txoj haujlwm cais. Lub sandbox txhawb kev thov rau kev nyeem ntawv, hloov pauv thiab ntxiv cov ntaub ntawv, tso cai rau koj ua qee yam haujlwm ntawm txhua lub luag haujlwm ntawm hom kev khaws cia thiab kev sib sau ua ke ntawm qhov tshwm sim (daim duab qhia / txo).

Handlers tuaj yeem piav qhia hauv cov ntaub ntawv:

sum.lua

local x, y = unpack(...)
return x + y

Thiab ces, tshaj tawm nyob rau hauv lub configuration:

functions:
  sum: { __file: sum.lua }

Vim li cas Lua? Lua yog ib hom lus yooj yim heev. Raws li peb qhov kev paub dhau los, ob peb teev tom qab paub txog nws, tib neeg pib sau cov lej uas daws lawv qhov teeb meem. Thiab cov no tsis yog tsuas yog cov kws tshaj lij tsim tawm, tab sis, piv txwv li, cov kws tshuaj ntsuam. Tsis tas li ntawd, ua tsaug rau jit compiler, Lua khiav ceev heev.

cia

Kev khaws cia khaws cov ntaub ntawv tsis tu ncua. Ua ntej txuag, cov ntaub ntawv yog validated tiv thaiv cov ntaub ntawv schema. Txhawm rau piav qhia lub voj voog peb siv cov ntawv txuas ntxiv Apache Avro... Piv txwv:

{
    "name": "User",
    "type": "record",
    "logicalType": "Aggregate",
    "fields": [ 
        { "name": "id", "type": "string"}, 
        {"name": "first_name", "type": "string"}, 
        {"name": "last_name", "type": "string"} 
    ], 
    "indexes": ["id"] 
}

Raws li cov lus piav qhia no, DDL (Cov Lus Txhais Lus) tau txais kev tsim tawm rau Tarantula DBMS thiab GraphQL schema rau cov ntaub ntawv nkag.

Asynchronous cov ntaub ntawv replication yog txaus siab (muaj kev npaj yuav ntxiv synchronous ib).

Output processor

Qee lub sij hawm nws yuav tsum tau ceeb toom rau cov neeg siv khoom sab nraud txog qhov tuaj txog ntawm cov ntaub ntawv tshiab; rau lub hom phiaj no, muaj lub luag haujlwm tso tawm cov khoom siv. Tom qab txuag cov ntaub ntawv, nws tuaj yeem raug xa mus rau tus neeg tuav ntaub ntawv (piv txwv li, coj mus rau hauv daim ntawv xav tau los ntawm tus neeg siv khoom) - thiab tom qab ntawd dhau mus rau qhov txuas xa mus. Kev kho kab kuj tseem siv tau ntawm no: yog tias tsis muaj leej twg lees txais qhov khoom, tus thawj coj tuaj yeem sim dua tom qab.

Scaling

Lub connector, input processor, thiab tso zis processor lub luag hauj lwm yog stateless, tso cai rau peb mus scale lub kab rov tav los ntawm tsuas yog ntxiv tshiab daim ntawv thov piv txwv nrog rau lub hom phiaj xav tau enabled. Kev khaws cia yog siv rau kev ntsuas kab rov tav tus mus kom ze los npaj ib pawg siv cov thoob virtual. Tom qab ntxiv ib tus neeg rau zaub mov tshiab, qee lub thoob los ntawm cov servers qub raug hloov mus rau lub server tshiab hauv keeb kwm yav dhau; qhov no tshwm sim pob tshab rau cov neeg siv thiab tsis cuam tshuam rau kev ua haujlwm ntawm tag nrho cov system.

Cov Ntaub Ntawv Cov Khoom

Cov khoom tuaj yeem loj heev thiab muaj lwm yam khoom. Peb xyuas kom meej atomicity ntawm kev ntxiv thiab hloov kho cov ntaub ntawv los ntawm kev khaws cov khoom nrog txhua qhov kev vam khom hauv ib lub thoob virtual. Qhov no tiv thaiv cov khoom los ntawm kev "kis tawm" hla ntau lub cev servers.

Versioning tau txais kev txhawb nqa: txhua qhov hloov tshiab ntawm ib qho khoom tsim ib qho tshiab, thiab peb tuaj yeem siv sijhawm ib ntus thiab saib seb lub ntiaj teb zoo li cas thaum ntawd. Rau cov ntaub ntawv uas tsis xav tau cov keeb kwm ntev, peb tuaj yeem txwv tus naj npawb ntawm cov versions lossis txawm tias tsuas yog khaws ib qho - qhov tseeb tshaj plaws - uas yog, qhov tseem ceeb lov tes taw versioning rau ib hom. Koj tuaj yeem txwv cov keeb kwm los ntawm lub sijhawm: piv txwv li, tshem tawm tag nrho cov khoom ntawm ib hom laus dua 1 xyoos. Archiving kuj tau txais kev txhawb nqa: peb tuaj yeem tshem tawm cov khoom qub tshaj li lub sijhawm teev tseg, tso chaw nyob hauv pawg.

paub tab

Ntawm cov yam ntxwv nthuav, nws tsim nyog sau cia lub peev xwm los tsim cov haujlwm ntawm lub sijhawm, ntawm tus neeg siv qhov kev thov, lossis programmatically los ntawm sandbox:

Ntawm no peb pom lwm lub luag haujlwm - khiav. Lub luag haujlwm no tsis muaj lub xeev, thiab cov ntawv thov ntxiv nrog lub luag haujlwm no tuaj yeem ntxiv rau pawg raws li xav tau. Tus khiav lub luag haujlwm yog ua kom tiav cov haujlwm. Raws li tau hais, nws tuaj yeem tsim cov haujlwm tshiab los ntawm sandbox; lawv tau txais kev cawmdim nyob rau hauv ib tug queue ntawm kev cia thiab tom qab ntawd tua ntawm tus khiav. Hom haujlwm no hu ua Txoj Haujlwm. Peb kuj muaj hom haujlwm hu ua Task - cov no yog cov haujlwm uas tus neeg siv tau ua haujlwm raws sijhawm (siv cron syntax) lossis xav tau. Txhawm rau pib thiab taug qab cov haujlwm zoo li no, peb muaj tus thawj tswj haujlwm yooj yim. Txhawm rau kom qhov kev ua haujlwm no muaj, koj yuav tsum ua kom lub luag haujlwm teem sijhawm; lub luag haujlwm no muaj lub xeev, yog li nws tsis ntsuas, uas, txawm li cas los xij, tsis tas yuav; Nyob rau tib lub sijhawm, zoo li tag nrho lwm cov luag haujlwm, nws tuaj yeem muaj qhov hloov pauv uas pib ua haujlwm yog tias tus tswv dheev tsis kam.

cov loggers

Lwm lub luag haujlwm yog hu ua logger. Nws sau cov cav los ntawm txhua tus tswv cuab ntawm pawg thiab muab lub interface rau uploading thiab saib lawv los ntawm lub vev xaib interface.

Kev pabcuam

Nws yog tsim nyog hais tias qhov system ua kom yooj yim los tsim cov kev pabcuam. Nyob rau hauv cov ntaub ntawv configuration, koj tuaj yeem qhia meej tias qhov kev thov twg raug xa mus rau tus neeg siv sau ntawv uas khiav hauv sandbox. Nyob rau hauv no handler, koj muaj peev xwm, piv txwv li, khiav ib co ntawm cov analytical query thiab xa rov qab qhov tshwm sim.

Cov kev pabcuam tau piav qhia hauv cov ntaub ntawv teeb tsa:

services:
   sum:
      doc: "adds two numbers"
      function: sum
      return_type: int
      args:
         x: int
         y: int

GraphQL API yog tsim los thiab cov kev pabcuam yuav muaj rau kev hu:

query {
   sum(x: 1, y: 2) 
}

Qhov no yuav hu rau tus tuav sumuas yuav rov qab tau qhov tshwm sim:

3

Query Profileing thiab Metrics

Txhawm rau nkag siab txog kev ua haujlwm ntawm lub kaw lus thiab kev thov profile, peb siv kev txhawb nqa rau OpenTracing raws tu qauv. Lub kaw lus tuaj yeem xa cov ntaub ntawv ntawm kev thov rau cov cuab yeej uas txhawb nqa cov txheej txheem no, xws li Zipkin, uas yuav tso cai rau koj nkag siab tias qhov kev thov raug ua li cas:

Lawm, lub kaw lus muab kev ntsuas sab hauv uas tuaj yeem sau tau siv Prometheus thiab pom pom siv Grafana.

Muab tso rau

Tarantool Data Grid tuaj yeem siv los ntawm RPM pob lossis cov ntaub ntawv khaws tseg, siv cov khoom siv hluav taws xob los ntawm kev faib lossis Ansible, kuj tseem muaj kev txhawb nqa rau Kubernetes (Tarantool Kubernetes Operator).

Daim ntawv thov uas siv cov kev lag luam logic (configuration, handlers) yog loaded rau hauv lub deployed Tarantool Data Grid pawg nyob rau hauv daim ntawv ntawm ib tug archive los ntawm lub UI los yog siv ib tsab ntawv los ntawm API muab los ntawm peb.

Cov ntawv ua piv txwv

Cov ntawv thov twg tuaj yeem tsim siv Tarantool Data Grid? Qhov tseeb, feem ntau cov haujlwm ua lag luam yog qee yam cuam tshuam nrog kev ua, khaws cia thiab nkag mus rau cov ntaub ntawv ntws. Yog li ntawd, yog tias koj muaj cov kwj dej loj ntawm cov ntaub ntawv uas yuav tsum tau khaws cia kom ruaj ntseg thiab nkag mus, ces peb cov khoom tuaj yeem txuag koj ntau lub sijhawm txhim kho thiab tsom mus rau koj txoj kev lag luam logic.

Piv txwv li, peb xav sau cov ntaub ntawv hais txog kev ua lag luam vaj tsev, yog li ntawd yav tom ntej, piv txwv li, peb yuav muaj cov ntaub ntawv hais txog qhov zoo tshaj plaws. Hauv qhov no, peb yuav qhia txog cov haujlwm hauv qab no:

1. Cov neeg hlau uas sau cov ntaub ntawv los ntawm cov chaw qhib yuav yog peb cov ntaub ntawv. Koj tuaj yeem daws qhov teeb meem no siv cov kev daws teeb meem npaj txhij lossis sau cov lej ua ib hom lus.
2. Tom ntej no, Tarantool Data Grid yuav txais thiab khaws cov ntaub ntawv. Yog hais tias cov ntaub ntawv tawm tswv yim los ntawm ntau qhov chaw sib txawv, ces koj tuaj yeem sau cov lej hauv Lua uas yuav ua qhov hloov pauv mus rau ib hom ntawv. Nyob rau theem ua ntej, koj kuj yuav muaj peev xwm, piv txwv li, lim cov khoom sib xws lossis ntxiv cov ntaub ntawv hloov tshiab txog cov neeg ua haujlwm hauv kev ua lag luam hauv cov ntaub ntawv.
3. Tam sim no koj twb muaj cov kev daws teeb meem loj hauv ib pawg uas tuaj yeem sau cov ntaub ntawv thiab xaiv cov ntaub ntawv. Tom qab ntawd koj tuaj yeem siv cov haujlwm tshiab, piv txwv li, sau ib qho kev pabcuam uas yuav ua rau kev thov rau cov ntaub ntawv thiab muab qhov zoo tshaj plaws rau ib hnub - qhov no yuav xav tau ob peb kab hauv cov ntaub ntawv teeb tsa thiab me ntsis Lua code.

Yuav ua li cas yog tom ntej no?

Peb qhov tseem ceeb yog los txhim kho qhov yooj yim ntawm kev txhim kho siv Tarantool Data Grid. Piv txwv li, qhov no yog IDE nrog kev txhawb nqa rau profileing thiab debugging handlers khiav hauv sandbox.

Peb kuj ua tib zoo saib xyuas cov teeb meem kev nyab xeeb. Tam sim no peb tab tom ua ntawv pov thawj los ntawm FSTEC ntawm Russia kom paub meej tias qib siab ntawm kev ruaj ntseg thiab ua tau raws li qhov yuav tsum tau ua rau kev lees paub cov khoom siv software siv hauv cov ntaub ntawv tus kheej thiab tsoomfwv cov ntaub ntawv xov xwm.

Tau qhov twg los: www.hab.com