Chero kushanda kukuru kwedata kunoda simba rakawanda rekombuta. Kufamba kwe data kubva kudhatabhesi kuenda kuHadoop kunogona kutora mavhiki kana kudhura zvakanyanya sebapiro rendege. Haudi kumirira uye kushandisa mari? Balance mutoro pamapuratifomu akasiyana. Imwe nzira ndeye pushdown optimization.
Ndakakumbira mudzidzisi weRussia anotungamira wekusimudzira uye kutonga kweInformatica zvigadzirwa, Alexey Ananyev, kuti ataure nezve pushdown optimization basa muInformatica Big Data Management (BDM). Wakambodzidza kushanda neInformatica zvigadzirwa? Zvingangodaro, aive Alexey akakuudza izvo zvekutanga zvePowerCenter uye akatsanangura nzira yekuvaka mepu.
Alexey Ananyev, mukuru wekudzidziswa kuDIS Group
Chii chinonzi pushdown?
Vazhinji venyu munotoziva Informatica Big Data Management (BDM). Chigadzirwa chacho chinogona kubatanidza data hombe kubva kwakasiyana masosi, kuifambisa pakati pemasisitimu akasiyana, inopa nyore kuwana kwairi, inobvumidza iwe kuti uinyore, uye zvimwe zvakawanda.
Mumaoko ekurudyi, BDM inogona kuita zvishamiso: mabasa anopedzwa nekukurumidza uye neashoma komputa zviwanikwa.
Unodawo izvozvo here? Dzidza kushandisa iyo yekusundira pasi muBDM kugovera iyo komputa mutoro pamapuratifomu akasiyana. Pushdown tekinoroji inokutendera iwe kushandura mepu kuita script uye wosarudza nharaunda iyo script inomhanya. Iyi sarudzo inokubvumira kuti ubatanidze masimba emapuratifomu akasiyana uye kuzadzisa kuita kwavo kwakanyanya.
Kuti ugadzirise iyo script execution nharaunda, unofanirwa kusarudza iyo pushdown mhando. Iyo script inogona kumhanya zvachose paHadoop kana kugoverwa zvishoma pakati pekwakabva uye kunyura. Pane 4 inokwanisika pushdown marudzi. Kugadzira mepu hakufanire kushandurwa kuita script (yekuzvarwa). Mepu inogona kuitwa zvakanyanya sezvinobvira pane sosi (mabviro) kana zvachose pane sosi (yakazara). Mepu inogona zvakare kushandurwa kuita Hadoop script (hapana).
Pushdown optimization
Iwo akanyorwa marudzi mana anogona kusanganiswa nenzira dzakasiyana - pushdown inogona kugadzirwa kune chaiyo inodiwa yehurongwa. Semuyenzaniso, inowanzova yakanyatsokodzera kubvisa data kubva mudura rekushandisa uchishandisa kugona kwayo. Uye iyo data ichashandurwa uchishandisa Hadoop, kuitira kuti isaremedza dhatabhesi pachayo.
Ngatitarisei nyaya yacho kana zvese kwainobva uye kwainoenda zviri mudhatabhesi, uye shanduko yekuuraya chikuva inogona kusarudzwa: zvichienderana nezvirongwa, ichava Informatica, dhatabhesi server, kana Hadoop. Muenzaniso wakadaro uchakubvumidza kuti unyatso nzwisisa iyo tekinoroji rutivi rwekushanda kweiyi michina. Nomuzvarirwo, muhupenyu chaihwo, mamiriro aya haasimuke, asi akanyatsokodzera kuratidzira kushanda.
Ngatitore mepu kuti tiverenge matafura maviri mune imwechete Oracle dhatabhesi. Uye rega mhedzisiro yekuverenga inyorwe mutafura mune imwechete database. Iyo mepu chirongwa chichaita seizvi:
Muchimiro chemepu paInformatica BDM 10.2.1 inoita seizvi:
Pushdown type - yekuzvarwa
Kana tikasarudza pushdown yerudzi rwechinyakare, ipapo mepu ichaitwa paInformatica server. Iyo data ichaverengwa kubva kuOracle server, inoendeswa kune Informatica server, yakashandurwa ipapo uye ichiendeswa kuHadoop. Mune mamwe mazwi, isu tichawana yakajairika ETL maitiro.
Pushdown mhando - tsime
Pakusarudza mhando yemhando, tinowana mukana wekugovera maitiro edu pakati pesevha yedatabase (DB) neHadoop. Kana maitirwo aitwa negadziriro iyi, zvikumbiro zvekutora data kubva kumatafura zvinotumirwa kudhatabhesi. Uye zvakasara zvichaitwa nenzira yematanho paHadoop.
Iyo dhizaini yekuuraya ichaita seizvi:
Pazasi pane muenzaniso wekumisikidza nharaunda yekumhanya.
Muchiitiko ichi, mepu ichaitwa mumatanho maviri. Muzvigadziro zvayo tichaona kuti yashanduka kuva script iyo inotumirwa kune iyo tsime. Uyezve, kusanganisa matafura uye kushandura data kuchaitwa muchimiro chemubvunzo wakawandisa pane sosi.
Mumufananidzo uri pazasi, tinoona mepu yakagadziridzwa paBDM, uye mubvunzo wakatsanangurwa patsva.
Basa reHadoop mukugadzirisa uku richaderedzwa kugadzirisa kuyerera kwedata - kuironga. Mhedzisiro yemubvunzo ichatumirwa kuHadoop. Kana kuverenga kwapera, faira kubva kuHadoop ichanyorwa kune sink.
Pushdown type - yakazara
Paunosarudza izere mhando, mepu inoshanduka zvachose kuita dhatabhesi mubvunzo. Uye mhedzisiro yechikumbiro ichaendeswa kuHadoop. Mufananidzo wemaitiro akadaro unoratidzwa pasi apa.
Muenzaniso wekugadzirisa unoratidzwa pasi apa.
Nekuda kweizvozvo, tichawana mepu yakagadziridzwa yakafanana neyakare. Musiyano chete ndewekuti pfungwa dzese dzinoendeswa kune anogamuchira nenzira yekupfuura kuiswa kwayo. Muenzaniso wemepu yakagadziridzwa unoratidzwa pazasi.
Pano, sezvakaitika kare, Hadoop inotamba basa remuitisi. Asi pano sosi inoverengwa yakazara, uye ipapo iyo data yekugadzirisa logic inoitwa padanho rekugamuchira.
Pushdown type is null
Zvakanaka, sarudzo yekupedzisira ndeye pushdown mhando, mukati memepu yedu inoshanduka kuita Hadoop script.
Iyo mepu yakagadziridzwa ikozvino yave kutaridzika seizvi:
Pano iyo data kubva kune mafaira ekutanga inotanga kuverengwa paHadoop. Zvadaro, achishandisa nzira yake pachake, mafaira maviri aya achabatanidzwa. Mushure meizvi, data ichashandurwa uye kuiswa kune database.
Nekunzwisisa misimboti yekusundira pasi optimization, unogona kunyatso kuronga akawanda maitiro ekushanda nedata hombe. Saka, nguva pfupi yadarika, imwe kambani hombe, mumavhiki mashoma, yakadhawunirodha data hombe kubva mudura muHadoop, iyo yayakange yaunganidza kwemakore akati wandei.
Source: www.habr.com