Nyob rau hauv kev cia siab ntawm lub community launch ntawm ib tug tshiab ntws ntawm tus nqi Peb tau npaj ib qho kev txhais cov ntaub ntawv nthuav.
txheej txheem cej luam
Peb yuav tham txog cov qauv nrov uas cov ntawv thov siv ntau lub khw muag khoom, qhov twg txhua lub khw yog siv rau nws tus kheej lub hom phiaj, piv txwv li, khaws cov ntaub ntawv canonical (MySQL, thiab lwm yam), muab cov peev xwm tshawb nrhiav qib siab (ElasticSearch, etc.), .), caching (Memcached, thiab lwm yam.) thiab lwm yam. Feem ntau, thaum siv ntau lub khw muag khoom cov ntaub ntawv, ib qho ntawm lawv ua lub khw tseem ceeb thiab lwm tus yog cov khw muag khoom derivative. Qhov teeb meem nkaus xwb yog yuav ua li cas synchronize cov ntaub ntawv khaws cia.
Peb tau saib ntau tus qauv sib txawv uas tau sim daws qhov teeb meem ntawm synchronizing ntau lub khw muag khoom, xws li sau ob npaug, faib kev lag luam, thiab lwm yam. Txawm li cas los xij, cov txheej txheem no muaj kev txwv tseem ceeb ntawm kev siv lub neej tiag tiag, kev ntseeg siab, thiab kev saib xyuas. Ntxiv nrog rau cov ntaub ntawv synchronization, qee cov ntawv thov kuj yuav tsum txhawb cov ntaub ntawv los ntawm kev hu rau cov kev pabcuam sab nraud.
Delta tau tsim los daws cov teeb meem no. Delta thaum kawg muab qhov zoo sib xws, kev tshwm sim-tsav platform rau cov ntaub ntawv synchronization thiab txhawb nqa.
Cov kev daws teeb meem uas twb muaj lawm
Ob chav nkag
Txhawm rau khaws ob lub khw muag khoom hauv sync, koj tuaj yeem siv dual sau, uas sau rau ib lub khw thiab tom qab ntawd sau mus rau lwm qhov tam sim ntawd. Thawj cov ntaub ntawv tuaj yeem rov sim dua thiab qhov thib ob tuaj yeem rho tawm yog tias thawj zaug ua tsis tiav tom qab tus naj npawb ntawm kev sim ua tiav. Txawm li cas los xij, ob lub khw muag khoom cov ntaub ntawv yuav dhau los ua tsis sib xws yog tias sau ntawv mus rau lub khw thib ob ua tsis tiav. Qhov teeb meem no feem ntau daws tau los ntawm kev tsim cov txheej txheem rov qab uas tuaj yeem hloov pauv cov ntaub ntawv los ntawm thawj qhov chaw cia mus rau qhov thib ob, lossis ua li ntawd tsuas yog pom qhov sib txawv hauv cov ntaub ntawv.
Teeb meem:
Kev ua cov txheej txheem rov qab yog ib txoj haujlwm tshwj xeeb uas tsis tuaj yeem rov qab siv dua. Tsis tas li ntawd, cov ntaub ntawv nruab nrab ntawm qhov chaw khaws cia tseem tawm ntawm sync mus txog thaum cov txheej txheem rov qab tshwm sim. Kev daws yuav nyuaj dua yog tias siv ntau tshaj ob lub khw muag khoom siv. Thaum kawg, cov txheej txheem rov qab tuaj yeem ntxiv load rau cov ntaub ntawv qub.
Hloov lub rooj log
Thaum cov kev hloov pauv tshwm sim rau ib pawg ntawm cov rooj (xws li ntxig, hloov kho, thiab tshem tawm cov ntaub ntawv), cov ntaub ntawv hloov pauv tau ntxiv rau lub rooj log uas yog ib feem ntawm tib qhov kev hloov pauv. Lwm cov xov lossis cov txheej txheem tas li thov cov xwm txheej los ntawm lub rooj log thiab sau lawv mus rau ib lossis ntau lub khw muag khoom, yog tias tsim nyog, tshem tawm cov xwm txheej ntawm lub rooj log tom qab cov ntaub ntawv tau lees paub los ntawm txhua lub khw muag khoom.
Teeb meem:
Cov qauv no yuav tsum tau ua raws li lub tsev qiv ntawv, thiab qhov zoo tshaj plaws yam tsis hloov cov cai ntawm daim ntawv thov uas siv nws. Nyob rau hauv ib puag ncig polyglot, kev siv lub tsev qiv ntawv zoo li no yuav tsum muaj nyob rau hauv txhua yam lus tsim nyog, tab sis kev ua kom muaj kev sib luag ntawm kev ua haujlwm thiab kev coj tus cwj pwm thoob plaws hom lus yog qhov nyuaj heev.
Lwm qhov teeb meem nyob hauv kev tau txais schema hloov hauv cov tshuab uas tsis txhawb kev hloov pauv hloov pauv [1][2], xws li MySQL. Yog li ntawd, tus qauv ntawm kev hloov pauv (piv txwv li, kev hloov pauv schema) thiab kev hloov pauv hauv cov lus hloov pauv yuav tsis ua haujlwm ib txwm.
Distributed Transactions
Kev faib khoom tuaj yeem siv los faib kev lag luam thoob plaws ntau lub khw muag khoom cov ntaub ntawv heterogeneous kom cov haujlwm tau cog lus rau txhua lub khw muag khoom siv, lossis tsis cog lus rau ib qho ntawm lawv.
Teeb meem:
Kev faib khoom lag luam yog qhov teeb meem loj heev rau cov khw muag ntaub ntawv heterogeneous. Los ntawm lawv qhov xwm txheej, lawv tsuas tuaj yeem tso siab rau qhov qis tshaj plaws ntawm cov kab ke koom nrog. Piv txwv li, XA kev lag luam thaiv kev ua tiav yog tias cov txheej txheem thov ua tsis tiav thaum lub sijhawm npaj. Tsis tas li ntawd, XA tsis muab kev tshawb nrhiav qhov tseeb lossis kev txhawb nqa kev tswj hwm qhov zoo sib xws. Tsis tas li ntawd, qee lub tshuab xws li ElasticSearch tsis txhawb XA lossis lwm yam qauv hloov pauv hloov pauv. Yog li, kom ntseeg tau tias kev sau atomicity nyob rau hauv ntau yam ntaub ntawv cia technologies tseem yog ib qho nyuaj heev rau kev siv [3].
Delta
Delta tau tsim los daws cov kev txwv ntawm cov kev daws teeb meem uas twb muaj lawm synchronization thiab tseem ua rau cov ntaub ntawv nthuav dav hauv-tus-fly. Peb lub hom phiaj yog kom paub daws teeb meem tag nrho cov kev nyuaj no kom deb ntawm cov neeg tsim daim ntawv thov kom lawv tuaj yeem tsom mus rau kev ua haujlwm ntawm kev lag luam. Tom ntej no peb yuav piav qhia txog "Kev Tshawb Fawb Movie", qhov kev siv tiag tiag rau Netflix's Delta.
Netflix dav siv microservice architecture, thiab txhua qhov microservice feem ntau ua haujlwm rau ib hom ntaub ntawv. Cov ntaub ntawv tseem ceeb ntawm zaj duab xis muaj nyob rau hauv microservice hu ua Movie Service, thiab cov ntaub ntawv cuam tshuam xws li cov ntaub ntawv hais txog cov neeg tsim khoom, cov neeg ua yeeb yam, cov neeg muag khoom, thiab lwm yam yog tswj hwm los ntawm ntau lwm microservices (xws li Deal Service, Talent Service thiab Vendor Service).
Cov neeg siv kev lag luam ntawm Netflix Studios feem ntau yuav tsum tau tshawb nrhiav thoob plaws ntau yam yeeb yaj kiab, uas yog vim li cas nws tseem ceeb heev rau lawv kom muaj peev xwm tshawb nrhiav thoob plaws txhua cov ntaub ntawv ntsig txog yeeb yaj kiab.
Ua ntej Delta, pab pawg nrhiav yeeb yaj kiab xav tau rub cov ntaub ntawv los ntawm ntau lub microservices ua ntej indexing cov ntaub ntawv yeeb yaj kiab. Tsis tas li ntawd, pab neeg no yuav tsum tau tsim ib lub kaw lus uas yuav hloov kho cov kev tshawb fawb ib ntus los ntawm kev thov kev hloov pauv los ntawm lwm cov microservices, txawm tias tsis muaj kev hloov pauv txhua. Qhov no system sai sai ua complex thiab nyuaj rau tswj.
Daim duab 1. Kev xaiv tsa rau Delta
Tom qab siv Delta, lub kaw lus tau yooj yim mus rau qhov kev tshwm sim uas tau tsav qhov system raws li qhia hauv daim duab hauv qab no. CDC (Change-Data-Capture) cov xwm txheej raug xa mus rau Keystone Kafka cov ncauj lus siv Delta-Connector. Daim ntawv thov Delta tau tsim los siv Delta Stream Processing Framework (raws li Flink) tau txais CDC cov xwm txheej los ntawm cov ncauj lus, txhawb lawv los ntawm kev hu rau lwm cov kev pabcuam microservices, thiab thaum kawg dhau cov ntaub ntawv muaj txiaj ntsig rau kev tshawb nrhiav hauv Elasticsearch. Tag nrho cov txheej txheem yuav siv sij hawm nyob rau hauv lub sij hawm ntawm lub sij hawm, uas yog, sai li sai tau raws li cov kev hloov pauv tau cog lus rau cov ntaub ntawv warehouses, kev tshawb fawb indexes yog tshiab.
Daim duab 2. Cov ntaub ntawv kav dej siv Delta
Nyob rau hauv cov ntu nram qab no, peb yuav piav qhia txog kev ua haujlwm ntawm Delta-Connector, uas txuas rau kev khaws cia thiab tshaj tawm CDC cov xwm txheej mus rau txheej thauj, uas yog cov ntaub ntawv xa mus rau lub sijhawm tiag tiag uas coj CDC cov xwm txheej mus rau Kafka cov ncauj lus. Thiab thaum kawg, peb yuav tham txog Delta kwj txheej txheem txheej txheem, uas cov neeg tsim khoom siv tuaj yeem siv rau kev ua cov ntaub ntawv thiab kev ua kom muaj txiaj ntsig zoo.
CDC (Hloov-Data-Capture)
Peb tau tsim ib qho kev pabcuam CDC hu ua Delta-Connector, uas tuaj yeem ntes cov kev hloov pauv ntawm cov ntaub ntawv khaws cia hauv lub sijhawm tiag tiag thiab sau rau hauv kwj. Kev hloov pauv ntawm lub sijhawm tiag tiag yog coj los ntawm cov ntawv teev kev sib pauv thiab cov khoom pov tseg. Cov pov tseg yog siv vim tias cov ntaub ntawv sib pauv feem ntau tsis khaws tag nrho cov keeb kwm ntawm kev hloov pauv. Cov kev hloov pauv feem ntau yog serialized li Delta cov xwm txheej, yog li tus neeg txais yuav tsis tas txhawj txog qhov twg qhov kev hloov pauv tuaj.
Delta-Connector txhawb nqa ntau yam ntxiv xws li:
- Muaj peev xwm sau cov ntaub ntawv tso tawm kev cai dhau los Kafka.
- Muaj peev xwm qhib phau ntawv pov tseg txhua lub sijhawm rau txhua lub rooj, ib lub rooj tshwj xeeb, lossis rau cov yuam sij tseem ceeb.
- Cov khib nyiab tuaj yeem rov qab tau rau hauv chunks, yog li tsis tas yuav tsum pib dua thaum tsis ua haujlwm.
- Tsis tas yuav tso cov xauv rau ntawm cov rooj, nws yog ib qho tseem ceeb heev los xyuas kom meej tias cov ntaub ntawv sau cov tsheb yuav tsis thaiv los ntawm peb cov kev pabcuam.
- Muaj ntau ntau vim muaj cov xwm txheej rov ua dua hauv AWS Availability Zones.
Peb tam sim no txhawb nqa MySQL thiab Postgres, suav nrog kev xa tawm ntawm AWS RDS thiab Aurora. Peb kuj txhawb Cassandra (multi-master). Koj tuaj yeem nrhiav cov ntsiab lus ntxiv txog Delta-Connector ntawm no .
Kafka thiab thauj txheej
Delta cov txheej xwm thauj khoom yog tsim los ntawm lub platform txoj kev xa xov .
Keeb kwm, tshaj tawm ntawm Netflix tau ua kom zoo rau kev nkag mus tau zoo dua li kev siv sijhawm ntev (saib hauv qab). ). Kev lag luam tawm yog qhov muaj peev xwm broker cov ntaub ntawv tsis sib xws hauv ntau qhov xwm txheej ntug. Piv txwv li, kev xaiv nom tswv tsis huv yog lub luag haujlwm rau tus neeg tau txais kev pabcuam uas muaj cov xwm txheej sib npaug lossis ploj.
Nrog Delta, peb xav tau kev ruaj ntseg ruaj khov kom ntseeg tau tias xa khoom ntawm CDC cov xwm txheej rau cov khw muag khoom. Rau lub hom phiaj no, peb tau npaj tshwj xeeb tsim Kafka pawg ua ib qho khoom siv thawj zaug. Koj tuaj yeem saib qee qhov chaw broker hauv cov lus hauv qab no:
Hauv Keystone Kafka pawg, kev xaiv nom tswv tsis huv feem ntau suav nrog los xyuas kom meej cov neeg tshaj tawm nkag mus tau. Qhov no yuav ua rau cov lus poob yog tias qhov kev hloov pauv tsis sib xws raug xaiv los ua tus thawj coj. Rau qhov muaj siab tshiab Kafka pawg, qhov kev xaiv kev xaiv nom tswv tsis huv muab tua los tiv thaiv cov lus poob.
Peb kuj nce ntxiv replication factor los ntawm 2 rau 3 thiab yam tsawg kawg insync replicas 1 txog 2. Cov neeg tshaj tawm sau rau pawg no xav tau acks los ntawm tag nrho lwm tus, kom ntseeg tau tias 2 ntawm 3 cov ntawv luam tawm muaj cov lus tam sim no xa los ntawm tus tshaj tawm.
Thaum tus broker piv txwv xaus, ib qho piv txwv tshiab hloov qhov qub. Txawm li cas los xij, tus broker tshiab yuav tsum tau ntes nrog cov khoom siv tsis sib xws, uas yuav siv sijhawm ntau teev. Txhawm rau txo lub sijhawm rov qab los rau qhov xwm txheej no, peb pib siv cov ntaub ntawv thaiv cov ntaub ntawv (Amazon Elastic Block Store) es tsis txhob siv cov khoom lag luam hauv zos. Thaum ib qho piv txwv tshiab hloov qhov piv txwv tus broker uas tau txiav tawm, nws txuas nrog EBS ntim uas qhov kev txiav tawm tau muaj thiab pib txais cov lus tshiab. Cov txheej txheem no txo ββcov sij hawm rov qab los ntawm cov xuab moos mus rau feeb vim tias qhov piv txwv tshiab tsis tas yuav rov ua dua los ntawm lub xeev khoob. Feem ntau, cais cia thiab broker lifecycles txo qhov cuam tshuam ntawm tus broker hloov.
Txhawm rau ntxiv cov ntaub ntawv xa tuaj, peb siv txhawm rau txheeb xyuas cov lus poob hauv qhov xwm txheej hnyav (piv txwv li, moos desynchronization hauv tus thawj coj muab faib).
kwj txheej txheej txheem
Delta cov txheej txheem ua tiav yog ua rau saum Netflix SPaaS platform, uas muab Apache Flink kev koom ua ke nrog Netflix ecosystem. Lub platform muab cov neeg siv interface uas tswj hwm kev xa tawm ntawm Flink txoj haujlwm thiab kev ua haujlwm ntawm Flink pawg nyob rau sab saum toj ntawm peb Titus lub thawv tswj hwm lub platform. Lub interface tseem tswj hwm txoj haujlwm teeb tsa thiab tso cai rau cov neeg siv los hloov kho qhov hloov pauv yam tsis tas yuav rov ua haujlwm Flink.
Delta muab cov txheej txheem ua haujlwm raws li Flink thiab SPaaS uas siv raws li annotation DSL (Domain Specific Language) kom paub meej cov ntsiab lus. Piv txwv li, txhawm rau txheeb xyuas cov kauj ruam ntawm cov xwm txheej yuav ua kom muaj txiaj ntsig los ntawm kev hu xov tooj rau cov kev pabcuam sab nraud, cov neeg siv yuav tsum tau sau cov DSL hauv qab no, thiab lub moj khaum yuav tsim cov qauv raws li nws, uas yuav raug tua los ntawm Flink.
Daim duab 3. Piv txwv ntawm kev txhawb nqa ntawm DSL hauv Delta
Cov txheej txheem kev ua haujlwm tsis yog tsuas yog txo cov kev kawm nkhaus xwb, tab sis kuj tseem muab cov kev ua haujlwm sib xws xws li deduplication, schematization, thiab yoog raws thiab muaj peev xwm los daws cov teeb meem kev ua haujlwm.
Delta Stream Processing Framework muaj ob lub ntsiab lus tseem ceeb, DSL & API module thiab Runtime module. Lub DSL & API module muab DSL thiab UDF (User-Defined-Function) APIs kom cov neeg siv tuaj yeem sau lawv tus kheej ua cov logic (xws li lim lossis hloov pauv). Lub Runtime module muab kev siv ntawm DSL parser uas tsim ib qho kev sawv cev sab hauv ntawm kev ua cov kauj ruam hauv DAG qauv. Qhov Kev Ua Txhaum Cai txhais cov qauv DAG los pib qhov tseeb Flink nqe lus thiab thaum kawg khiav Flink daim ntawv thov. Lub architecture ntawm lub moj khaum yog piav qhia hauv daim duab hauv qab no.
Daim duab 4. Delta Stream Processing Framework architecture
Txoj kev no muaj ntau yam zoo:
- Cov neeg siv tuaj yeem tsom mus rau lawv cov laj thawj kev lag luam yam tsis tas yuav delve rau hauv qhov tshwj xeeb ntawm Flink lossis SPaaS qauv.
- Kev ua kom zoo tuaj yeem ua tiav hauv txoj hauv kev uas muaj pob tshab rau cov neeg siv, thiab qhov yuam kev tuaj yeem kho yam tsis tas yuav muaj kev hloov pauv rau tus neeg siv code (UDF).
- Delta daim ntawv thov kev paub tau yooj yim rau cov neeg siv vim tias lub platform muab kev yooj yim thiab rov ua haujlwm tawm ntawm lub thawv thiab sau ntau yam kev ntsuas ntxaws uas tuaj yeem siv rau kev ceeb toom.
Kev siv ntau lawm
Delta tau tsim tawm ntau tshaj ib xyoos thiab ua lub luag haujlwm tseem ceeb hauv ntau daim ntawv thov Netflix Studio. Nws tau pab pawg siv cov xwm txheej xws li kev tshawb nrhiav indexing, khaws cov ntaub ntawv, thiab cov xwm txheej uas tau tsav los ntawm kev ua haujlwm. Hauv qab no yog cov ntsiab lus ntawm cov qib siab architecture ntawm Delta platform.
Daim duab 5. Delta's high-level architecture.
Kev lees paub
Peb xav ua tsaug rau cov neeg hauv qab no uas tau koom nrog hauv kev tsim thiab kev loj hlob ntawm Delta ntawm Netflix: Allen Wang, Charles Zhao, Jaebin Yoon, Josh Snyder, Kasturi Chatterjee, Mark Cho, Olof Johansson, Piyush Goyal, Prashanth Ramdas, Raghuram Onti Srinivasan, Sandeep Gupta, Steven Wu, Tharanga Gamaethige, Yun Wang thiab Zhenzhong Xu.
Cov chaw
- Martin Kleppmann, Alastair R. Beresford, Boerge Svingen: Kev ua haujlwm online. Pawg. ACM 62(5): 43β49 (2019). DOI:
: "Data Build Tool for Amazon Redshift Storage."
Tau qhov twg los: www.hab.com