Ngokulindele ukwethulwa kokugeleza okusha ngenani Silungiselele ukuhunyushwa kokwaziswa okuthakazelisayo.
Uhlolojikelele
Sizokhuluma ngephethini ethandwa kakhulu lapho izinhlelo zokusebenza zisebenzisa izitolo eziningi zedatha, lapho isitolo ngasinye sisetshenziselwa izinjongo zaso, isibonelo, ukugcina uhlobo lwe-canonical lwedatha (MySQL, njll.), ukunikeza amakhono okusesha athuthukile (ElasticSearch, njll.) .), ukulondoloza isikhashana (Memcached, njll.) nokunye. Ngokuvamile, uma usebenzisa izitolo eziningi zedatha, esinye sazo sisebenza njengesitolo esiyinhloko kanti ezinye njengezitolo eziphuma kokunye. Inkinga nje ukuthi ungavumelanisa kanjani lezi zitolo zedatha.
Sibheke amaphethini amaningi ahlukene azame ukuxazulula inkinga yokuvumelanisa izitolo eziningi, ezifana nokubhala kabili, ukuthengiselana okusabalalisiwe, njll. Kodwa-ke, lezi zindlela zinemikhawulo ebalulekile mayelana nokusetshenziswa kwangempela kwempilo, ukwethembeka, nokugcinwa. Ngaphezu kokuvumelanisa idatha, ezinye izinhlelo zokusebenza nazo zidinga ukucebisa idatha ngokushayela izinsizakalo zangaphandle.
I-Delta yenzelwe ukuxazulula lezi zinkinga. I-Delta ekugcineni inikeza inkundla engaguquki, eqhutshwa umcimbi yokuvumelanisa idatha nokunothisa.
Izixazululo ezikhona
Ukufaka kabili
Ukuze ugcine izitolo zedatha ezimbili zivumelanisiwe, ungasebenzisa ukubhala okumbaxambili, okubhalela esitolo esisodwa bese ubhalela kwesinye ngokushesha ngemva kwalokho. Ukurekhoda kokuqala kungazanywa futhi futhi okwesibili kungahoxiswa uma eyokuqala yehluleka ngemva kokuba inani lemizamo seliphelile. Nokho, izitolo ezimbili zedatha zingase zingasavumelanisi uma ukubhalela isitolo sesibili kwehluleka. Le nkinga ivame ukuxazululwa ngokudala inqubo yokutakula engaphinda idlulise ngezikhathi ezithile idatha isuka endaweni yokugcina iye kweyesibili, noma ikwenze lokho kuphela uma kutholwa umehluko kudatha.
Izinkinga:
Ukwenza inqubo yokutakula kuwumsebenzi othile ongakwazi ukuphinda usetshenziswe. Ngaphezu kwalokho, idatha phakathi kwezindawo zokugcina ihlala ingavumelaniswa kuze kube yilapho kwenziwa inqubo yokubuyisela. Isixazululo siba nzima kakhulu uma kusetshenziswa izitolo zedatha ezingaphezu kwezimbili. Ekugcineni, inqubo yokubuyisela ingangeza umthwalo kumthombo wedatha wangempela.
Shintsha ithebula lokungena
Uma izinguquko zenzeka kusethi yamathebula (njengokufaka, ukuvuselela, nokususa irekhodi), amarekhodi oshintsho engezwa kuthebula lokungena njengengxenye yomsebenzi ofanayo. Enye intambo noma inqubo ihlale icela izehlakalo etafuleni lokungena futhi izibhalela esitolo sedatha esisodwa noma ngaphezulu, uma kunesidingo, isusa imicimbi etafuleni lokungena ngemva kokuqinisekiswa kwerekhodi yizo zonke izitolo.
Izinkinga:
Le phethini kufanele isetshenziswe njengomtapo wezincwadi, futhi ngaphandle kokushintsha ikhodi yohlelo oluyisebenzisayo. Esimweni se-polyglot, ukusetshenziswa komtapo onjalo kufanele kube nganoma yiluphi ulimi oludingekayo, kodwa ukuqinisekisa ukuvumelana kokusebenza nokuziphatha kuzo zonke izilimi kunzima kakhulu.
Enye inkinga isekutholeni izinguquko ze-schema ezinhlelweni ezingasekeli izinguquko ze-schema yokwenziwayo [1][2], njenge-MySQL. Ngakho-ke, iphethini yokwenza ushintsho (isibonelo, ukuguqulwa kwe-schema) nokurekhoda ngokuthengiselana kuthebula lefayela lokungena ngeke isebenze njalo.
Okwenziwayo Okusabalalisiwe
Okwenziwayo okusabalalisiwe kungasetshenziswa ukuhlukanisa okwenziwayo kuzo zonke izitolo zedatha eziningi ezihlukahlukene ukuze umsebenzi uzinikele kuzo zonke izitolo zedatha ezisetshenzisiwe, noma ungazibophezeli kunoma yisiphi sazo.
Izinkinga:
Ukuthengiselana okusabalalisiwe kuyinkinga enkulu kakhulu ezitolo zedatha ezinhlobonhlobo. Ngokwemvelo yabo, bangathembela kuphela entweni evamile ephansi yezinhlelo ezihilelekile. Isibonelo, imisebenzi ye-XA ivimba ukwenziwa uma inqubo yesicelo yehluleka ngesikhathi sesigaba sokulungiselela. Ukwengeza, i-XA ayihlinzeki ngokutholwa kwe-deadlock noma isekele izikimu zokulawula ukuhambisana kwemali okunethemba. Ngaphezu kwalokho, amanye amasistimu afana ne-ElasticSearch awasekeli i-XA nanoma iyiphi enye imodeli yokwenziwayo ehlukahlukene. Ngakho-ke, ukuqinisekisa ukubhala i-atomicity kubuchwepheshe obuhlukahlukene bokugcinwa kwedatha kuhlala kuwumsebenzi oyinselele kakhulu wezinhlelo zokusebenza [3].
Delta
I-Delta yakhelwe ukubhekana nemikhawulo yezisombululo ezikhona zokuvumelanisa idatha futhi ivumela nokunothisa idatha lapho undiza. Umgomo wethu bekuwukukhipha zonke lezi zinkimbinkimbi kude nabathuthukisi bohlelo lokusebenza ukuze bakwazi ukugxila ngokugcwele ekusebenziseni ukusebenza kwebhizinisi. Okulandelayo sizobe sichaza "Usesho Lwefilimu", isimo sangempela sokusebenzisa i-Delta ye-Netflix.
I-Netflix isebenzisa kabanzi i-microservice architecture, futhi i-microservice ngayinye ngokuvamile isebenzisa uhlobo olulodwa lwedatha. Ulwazi oluyisisekelo mayelana nefilimu luqukethwe kusevisi encane ebizwa nge-Movie Service, kanye nedatha ehlobene, efana nolwazi olumayelana nabakhiqizi, abalingisi, abathengisi, njalo njalo, iphethwe ezinye izinsiza ezincane ezimbalwa (okungukuthi Isevisi Yokuthengiselana, Isevisi Yamathalente kanye Nesevisi Yomthengisi).
Abasebenzisi bebhizinisi ku-Netflix Studios bavame ukudinga ukusesha kuzo zonke izici ezihlukahlukene zama-movie, yingakho kubaluleke kakhulu ukuthi bakwazi ukusesha kuyo yonke idatha ehlobene ne-movie.
Ngaphambi kwe-Delta, ithimba lokusesha ama-movie lalidinga ukudonsa idatha kuma-microservices amaningi ngaphambi kokukhomba idatha ye-movie. Ngaphezu kwalokho, ithimba bekufanele lenze uhlelo oluzobuyekeza inkomba yokusesha ngezikhathi ezithile ngokucela izinguquko kwamanye ama-microservices, noma ngabe bezingekho nhlobo izinguquko. Lesi simiso ngokushesha saba yinkimbinkimbi futhi kwaba nzima ukusigcina.
Umfanekiso 1. Uhlelo lokuvota oluya eDelta
Ngemva kokusebenzisa i-Delta, uhlelo lwenziwe lwaba lula ohlelweni oluqhutshwa umcimbi njengoba kuboniswe esithombeni esilandelayo. Imicimbi ye-CDC (Change-Data-Capture) ithunyelwa kuzihloko ze-Keystone Kafka kusetshenziswa i-Delta-Connector. Uhlelo lokusebenza lwe-Delta olwakhiwe kusetshenziswa i-Delta Stream Processing Framework (esekelwe ku-Flink) luthola imicimbi ye-CDC esihlokweni, luyicebise ngokubiza amanye ama-microservices, futhi ekugcineni ludlulisele idatha ethuthukisiwe enkombeni yosesho ku-Elasticsearch. Yonke inqubo yenzeka cishe ngesikhathi sangempela, okungukuthi, ngokushesha nje lapho izinguquko zizibophezele endaweni yokugcina idatha, izinkomba zokusesha ziyabuyekezwa.
Umfanekiso 2. Ipayipi ledatha lisebenzisa i-Delta
Ezigabeni ezilandelayo, sizochaza ukusebenza kwe-Delta-Connector, exhuma kwisitoreji futhi ishicilele imicimbi ye-CDC kungqimba lwezokuthutha, okuyingqalasizinda yokudlulisa idatha yesikhathi sangempela ehambisa imicimbi ye-CDC ezihlokweni ze-Kafka. Futhi ekugcineni, sizokhuluma ngohlaka lokucubungula umfudlana we-Delta, abathuthukisi bohlelo lokusebenza abangalusebenzisela ukucubungula idatha kanye nengqondo yokucebisa.
I-CDC (Shintsha-Idatha-Capture)
Senze isevisi ye-CDC ebizwa ngokuthi i-Delta-Connector, engathwebula izinguquko ezizibophezele esitolo sedatha ngesikhathi sangempela futhi izibhale ekusakazweni. Izinguquko zesikhathi sangempela zithathwa kulogi yokwenziwe kanye nokulahlwa kwesitoreji. Ukulahla kuyasetshenziswa ngoba amalogi okwenziwayo ngokuvamile awagcini wonke umlando wezinguquko. Izinguquko zijwayele ukuhlelwa njengemicimbi ye-Delta, ngakho-ke umamukeli akudingeki akhathazeke ngokuthi ushintsho luvelaphi.
I-Delta-Connector isekela izici ezimbalwa ezengeziwe ezifana nalezi:
- Ikhono lokubhala idatha yokuphuma ngokwezifiso usebenzisa i-Kafka.
- Ikhono lokuvula ukulahlwa kwezandla nganoma yisiphi isikhathi kuwo wonke amathebula, ithebula elithile, noma kokhiye abathile abayinhloko.
- Okulahlwayo kungabuyiswa kube yizicucu, ngakho-ke asikho isidingo sokuqala kabusha uma kwenzeka ukwehluleka.
- Asikho isidingo sokubeka izingidi ematafuleni, okubaluleke kakhulu ukuqinisekisa ukuthi ithrafikhi yokubhala egciniwe ayivinjwa neze yinkonzo yethu.
- Ukutholakala okuphezulu ngenxa yezehlakalo ezingadingekile Ezindaweni Ezitholakalayo ze-AWS.
Okwamanje sisekela i-MySQL ne-Postgres, okuhlanganisa ukuthunyelwa ku-AWS RDS ne-Aurora. Siphinde sisekele i-Cassandra (i-multi-master). Ungathola imininingwane eyengeziwe mayelana ne-Delta-Connector lapha .
I-Kafka kanye nesendlalelo sezokuthutha
Isendlalelo sezokuthutha zomcimbi we-Delta sakhelwe kusevisi yemiyalezo yesikhulumi .
Ngokomlando, ukuthumela ku-Netflix kulungiselelwe ukufinyeleleka kunokuhlala isikhathi eside (bona ngezansi). ). Ukuhwebelana kube ukungahambisani kwedatha yomthengisi okungenzeka ezimeni ezehlukene. Ngokwesibonelo, ukhetho lomholi ongcolile unesibopho sokuthi umamukeli angaba nemicimbi eyimpinda noma elahlekile.
Nge-Delta, besifuna iziqinisekiso eziqinile zokuqinisekisa ukulethwa kwemicimbi ye-CDC ezitolo ezitholiwe. Ngale njongo, siphakamise iqoqo le-Kafka eliklanywe ngokukhethekile njengento esezingeni lokuqala. Ungabheka ezinye izilungiselelo zomthengisi kuthebula elingezansi:
Kumaqoqo e-Keystone Kafka, ukhetho lomholi ongcolile ngokuvamile kufakwe ukuqinisekisa ukufinyeleleka komshicileli. Lokhu kungase kuphumele emilayezweni elahlekile uma ikhophi engavumelanisiwe ikhethwa njengomholi. Ngokutholakala okuphezulu kweqoqo le-Kafka, inketho ukhetho lomholi ongcolile icishiwe ukuvimbela ukulahleka komlayezo.
Sandisa futhi isici sokuphindaphinda kusuka ku-2 kuya ku-3 futhi ubuncane bokukopisha obungavumelanisiwe 1 kuya ku-2. Abashicileli ababhalela leli qoqo badinga ama-acks avela kubo bonke abanye, baqinisekise ukuthi izifaniso ezi-2 kwezingu-3 zinemilayezo yamanje kakhulu ethunyelwa umshicileli.
Uma isenzakalo somdayiseli sinqanyulwa, isenzakalo esisha singena esikhundleni sakudala. Kodwa-ke, umthengisi omusha uzodinga ukuhlangana nezifaniso ezingavumelanisiwe, ezingathatha amahora ambalwa. Ukunciphisa isikhathi sokuthola kabusha lesi simo, siqale ukusebenzisa i-block data storage (Amazon Elastic Block Store) esikhundleni samadiski omthengisi wendawo. Uma isenzakalo esisha singena esikhundleni sesenzakalo somdayiseli esinqanyuliwe, sinamathisela ivolomu ye-EBS isenzakalo esinqanyuliwe sibe nayo futhi siqala ukuthola imilayezo emisha. Le nqubo inciphisa isikhathi sokususa ukusilela emuva kusukela emahoreni kuye emaminithini ngoba isenzakalo esisha asisadingi ukuphindaphinda sisuka esimweni esingenalutho. Sekukonke, isitoreji esihlukene kanye nemijikelezo yokuphila yomthengisi kunciphisa kakhulu umthelela wokushintsha umthengisi.
Ukwandisa isiqinisekiso sokulethwa kwedatha, sasebenzisa ukuthola noma yikuphi ukulahleka komlayezo ngaphansi kwezimo ezimbi kakhulu (isibonelo, ukunqanyulwa kwewashi kumholi wokuhlukanisa).
Uhlaka Lokucubungula Ukusakaza
Isendlalelo sokucutshungulwa kweDelta sakhelwe phezulu kwenkundla yeNetflix SPaaS, ehlinzeka ngokuhlanganiswa kwe-Apache Flink ne-Netflix ecosystem. Inkundla inikeza ukusebenzelana komsebenzisi olawula ukuthunyelwa kwemisebenzi ye-Flink kanye nokuhlelwa kwamaqoqo e-Flink phezu kwenkundla yethu yokuphatha iziqukathi ze-Titus. I-interface iphinde ilawule ukucushwa kwemisebenzi futhi ivumela abasebenzisi ukuthi benze izinguquko zokumisa ngamandla ngaphandle kokuthi baphinde bahlanganise imisebenzi ye-Flink.
I-Delta ihlinzeka ngohlaka lokucubungula ukusakaza okusekelwe ku-Flink ne-SPaaS esebenzisa okusekelwe esichasiselweni I-DSL (Ulimi Oluthile Lwesizinda) ukuze uthole imininingwane yobuchwepheshe efingqiwe. Isibonelo, ukuze uchaze isinyathelo lapho imicimbi izothuthukiswa khona ngokubiza izinsizakalo zangaphandle, abasebenzisi badinga ukubhala i-DSL elandelayo, futhi uhlaka luzodala imodeli esekelwe kulo, ezokwenziwa yi-Flink.
Umfanekiso 3. Isibonelo sokunothisa ku-DSL e-Delta
Uhlaka lokucubungula alunciphisi kuphela ijika lokufunda, kodwa futhi luhlinzeka ngezici ezivamile zokucubungula ukusakaza okufana nokuphindaphinda, ukuhlela, nokuguquguquka nokuqina ukuze kuxazululwe izinkinga ezivamile zokusebenza.
I-Delta Stream Processing Framework inamamojula amabili ayisihluthulelo, imojula ye-DSL & API kanye nemojula ye-Runtime. Imojula ye-DSL & API inikeza i-DSL ne-UDF (User-Defined-Function) APIs ukuze abasebenzisi bakwazi ukubhala eyabo ingqondo yokucubungula (njengokuhlunga noma ukuguqulwa). Imojula Yesikhathi Sokusebenza ihlinzeka ngokusetshenziswa komhlaziyi we-DSL owakha ukumelwa kwangaphakathi kwezinyathelo zokucubungula kumamodeli e-DAG. Ingxenye Yokwenza ihumusha amamodeli e-DAG ukuze kuqaliswe izitatimende zangempela ze-Flink futhi ekugcineni iqalise uhlelo lwe-Flink. Ukwakheka kohlaka kuboniswe esithombeni esilandelayo.
Umfanekiso 4. Isakhiwo se-Delta Stream Processing Framework
Le ndlela inezinzuzo ezimbalwa:
- Abasebenzisi bangagxila kungqondongqondo yebhizinisi labo ngaphandle kokuthi bahlole imininingwane ye-Flink noma ukwakheka kwe-SPaaS.
- Ukuthuthukisa kungenziwa ngendlela esobala kubasebenzisi, futhi amaphutha angalungiswa ngaphandle kokudinga izinguquko kukhodi yomsebenzisi (UDF).
- Umuzwa wohlelo lwe-Delta wenziwa lula kubasebenzisi ngoba inkundla inikeza ukuguquguquka nokuqina ngaphandle kwebhokisi futhi iqoqa izinhlobonhlobo zamamethrikhi anemininingwane engasetshenziselwa izexwayiso.
Ukusetshenziswa kokukhiqiza
I-Delta isikhiqize isikhathi esingaphezu konyaka futhi idlala indima ebalulekile ezinhlelweni eziningi zeNetflix Studio. Usize amaqembu asebenzise izimo zokusetshenziswa ezifana nokukhonjwa kosesho, ukugcinwa kwedatha, nokugeleza komsebenzi okuqhutshwa umcimbi. Ngezansi ukubuka kabanzi kwesakhiwo sezinga eliphezulu seplatifomu ye-Delta.
Umfanekiso 5. Isakhiwo sezinga eliphezulu seDelta.
Ukubonga
Sithanda ukubonga abantu abalandelayo ababambe iqhaza ekwakhiweni nasekuthuthukisweni kwe-Delta e-Netflix: Allen Wang, Charles Zhao, Jaebin Yoon, Josh Snyder, Kasturi Chatterjee, Mark Cho, Olof Johansson, Piyush Goyal, Prashanth Ramdas, Raghuram Onti Srinivasan, Sandeep Gupta , Steven Wu, Tharanga Gamaethige, Yun Wang kanye noZhenzhong Xu.
Imithombo
- UMartin Kleppmann, Alastair R. Beresford, Boerge Svingen: Ukucubungula umcimbi ku-inthanethi. Umphakathi. I-ACM 62(5): 43β49 (2019). I-DOI:
: "Ithuluzi Lokwakha Idatha le-Amazon Redshift Storage."
Source: www.habr.com