Noma yikuphi ukusebenza kwedatha enkulu kudinga amandla amaningi ekhompyutha. Ukuhanjiswa okuvamile kwedatha kusuka kusizindalwazi kuya ku-Hadoop kungathatha amaviki noma kubize njengephiko lendiza. Awufuni ukulinda futhi usebenzise imali? Linganisa umthwalo kuwo wonke amapulatifomu ahlukene. Enye indlela i-pushdown optimization.
Ngicele umqeqeshi ohamba phambili waseRussia wokuthuthukiswa nokuphathwa kwemikhiqizo ye-Informatica, u-Alexey Ananyev, ukuthi akhulume ngomsebenzi wokwenza ngcono ukwehla ku-Informatica Big Data Management (BDM). Uke wafunda ukusebenza ngemikhiqizo ye-Informatica? Kungenzeka ukuthi kwakungu-Alexey owakutshela izisekelo ze-PowerCenter futhi wachaza indlela yokwakha imephu.
U-Alexey Ananyev, inhloko yokuqeqeshwa kwa-DIS Group
Yini i-pushdown?
Abaningi benu sebejwayelene ne-Informatica Big Data Management (BDM). Umkhiqizo ungahlanganisa idatha enkulu evela emithonjeni ehlukene, uyihambise phakathi kwamasistimu ahlukene, unikeze ukufinyelela okulula kuwo, ikuvumela ukuthi uyiphrofayili, nokunye okuningi.
Ezandleni ezifanele, i-BDM ingenza izimanga: imisebenzi izoqedwa ngokushesha nangezinsiza ezincane zekhompyutha.
Uyakufuna nawe lokho? Funda ukusebenzisa isici sokwehlisa ku-BDM ukuze usabalalise umthwalo wekhompuyutha ezinkundleni ezahlukene. Ubuchwepheshe be-Pushdown bukuvumela ukuthi uguqule imephu ibe iskripthi bese ukhetha indawo lapho lesi skripthi sizosebenza khona. Lokhu kukhetha kukuvumela ukuthi uhlanganise amandla amapulatifomu ahlukene futhi uzuze ukusebenza kwawo okuphezulu.
Ukuze ulungiselele imvelo yokusebenzisa iskripthi, udinga ukukhetha uhlobo lokusunduza. Iskripthi singasebenza ngokuphelele ku-Hadoop noma sisatshalaliswe kancane phakathi komthombo nosinki. Kunezinhlobo ezi-4 ezingase ziphushelwe phansi. Ukwenza imephu akudingeki ukuthi kuguqulwe kube iskripthi (somdabu). Ukwenza imephu kungenziwa ngangokunokwenzeka emthonjeni (umthombo) noma ngokuphelele emthonjeni (okugcwele). Ukwenza imephu kungabuye kuguqulwe kube iskripthi se-Hadoop (akukho).
Ukuthuthukisa ukusunduza phansi
Izinhlobo ezi-4 ezisohlwini zingahlanganiswa ngezindlela ezihlukene - i-pushdown ingenziwa elungiselelwe izidingo ezithile zohlelo. Isibonelo, ngokuvamile kufaneleka kakhulu ukukhipha idatha kusizindalwazi kusetshenziswa amakhono ayo. Futhi idatha izoguqulwa kusetshenziswa i-Hadoop, ukuze ingalayishi kakhulu i-database ngokwayo.
Ake sicabangele icala lapho kokubili umthombo nendawo okuyiwa kuyo ku-database, futhi isiteji sokwenza uguquko singakhethwa: kuye ngezilungiselelo, kuzoba i-Informatica, iseva yedathabhesi, noma i-Hadoop. Isibonelo esinjalo sizokuvumela ukuthi uqonde ngokunembe kakhulu uhlangothi lobuchwepheshe lokusebenza kwalo mshini. Ngokwemvelo, empilweni yangempela, lesi simo asiveli, kodwa sifaneleka kakhulu ekuboniseni ukusebenza.
Masithathe imephu ukuze sifunde amathebula amabili kusizindalwazi esisodwa se-Oracle. Futhi vumela imiphumela yokufunda irekhodwe kuthebula kusizindalwazi esifanayo. Uhlelo lwemephu luzoba kanje:
Ngendlela yokwenza imephu ku-Informatica BDM 10.2.1 kubukeka kanje:
Uhlobo lwe-Pushdown - lwendabuko
Uma sikhetha uhlobo lomdabu lokuphushela phansi, khona-ke ukumepha kuzokwenziwa kuseva ye-Informatica. Idatha izofundwa kuseva ye-Oracle, idluliselwe kuseva ye-Informatica, iguqulwe lapho futhi idluliselwe ku-Hadoop. Ngamanye amazwi, sizothola inqubo ye-ETL evamile.
Uhlobo lwe-Pushdown - umthombo
Lapho sikhetha uhlobo lomthombo, sithola ithuba lokusabalalisa inqubo yethu phakathi kweseva yedathabhesi (DB) ne-Hadoop. Uma inqubo yenziwa nalokhu kulungiselelwa, izicelo zokubuyisa idatha kumathebula zizothunyelwa kusizindalwazi. Futhi okunye kuzokwenziwa ngendlela yezinyathelo ku-Hadoop.
Umdwebo wokwenza uzobukeka kanje:
Ngezansi isibonelo sokusetha indawo yesikhathi sokusebenza.
Kulokhu, ukwenza imephu kuzokwenziwa ngezinyathelo ezimbili. Ezilungiselelweni zayo sizobona ukuthi isiphenduke iskripthi esizothunyelwa kumthombo. Ngaphezu kwalokho, ukuhlanganisa amathebula nokuguqula idatha kuzokwenziwa ngendlela yombuzo obhalwe ngaphezulu emthonjeni.
Esithombeni esingezansi, sibona imephu ethuthukisiwe ku-BDM, kanye nombuzo ochazwe kabusha emthonjeni.
Indima ye-Hadoop kulokhu kucushwa izoncishiswa ekulawuleni ukugeleza kwedatha - ukuyihlela. Umphumela wombuzo uzothunyelwa ku-Hadoop. Lapho ukufundwa sekuqediwe, ifayela elivela ku-Hadoop lizobhalwa kusinki.
Uhlobo lokusunduza - lugcwele
Uma ukhetha uhlobo olugcwele, ukwenza imephu kuzoshintsha ngokuphelele kube umbuzo wesizindalwazi. Futhi umphumela wesicelo uzothunyelwa eHadoop. Umdwebo wenqubo enjalo uvezwa ngezansi.
Ukusethwa kwesibonelo kuboniswe ngezansi.
Njengomphumela, sizothola imephu ethuthukisiwe efana neyangaphambilini. Umehluko kuphela ukuthi yonke i-logic idluliselwa kumamukeli ngendlela yokweqa ukufakwa kwayo. Isibonelo semephu ethuthukisiwe yethulwa ngezansi.
Lapha, njengasekuqaleni, uHadoop udlala indima yomqhubi. Kodwa lapha umthombo ufundwa ngokuphelele, bese i-logic yokucubungula idatha yenziwa ezingeni lomamukeli.
Uhlobo lwe-Pushdown alunalutho
Hhayi-ke, inketho yokugcina uhlobo lokusunduza, lapho imephu yethu izophenduka ibe iskripthi se-Hadoop.
Imephu ethuthukisiwe manje izobukeka kanje:
Lapha idatha evela kumafayela omthombo izoqale ifundwe ku-Hadoop. Khona-ke, esebenzisa indlela yakhe siqu, lawa mafayela amabili azohlanganiswa. Ngemva kwalokhu, idatha izoguqulwa futhi ilayishwe ku-database.
Ngokuqonda izimiso zokuphushela phansi, ungakwazi ukuhlela ngempumelelo izinqubo eziningi zokusebenza ngedatha enkulu. Ngakho-ke, muva nje, inkampani enkulu, emasontweni ambalwa nje, ilande idatha enkulu isuka kwisitoreji iye ku-Hadoop, eyayiyiqoqile iminyaka eminingana.
Source: www.habr.com