Txhua cov ntaub ntawv loj ua haujlwm yuav tsum muaj kev suav ntau lub zog. Ib qho kev txav ntawm cov ntaub ntawv los ntawm cov ntaub ntawv mus rau Hadoop tuaj yeem siv sijhawm ntau lub lis piam lossis raug nqi ntau npaum li lub dav hlau tis. Tsis xav tos thiab siv nyiaj? Ntsuas cov load hla ntau lub platform. Ib txoj hauv kev yog pushdown optimization.
Kuv nug Russia tus kws qhia ua thawj coj rau kev txhim kho thiab kev tswj hwm ntawm Informatica cov khoom, Alexey Ananyev, tham txog kev ua haujlwm zoo tshaj plaws hauv Informatica Big Data Management (BDM). Koj puas tau kawm ua haujlwm nrog Informatica cov khoom? Feem ntau, nws yog Alexey uas tau qhia koj txog cov hauv paus ntawm PowerCenter thiab piav qhia yuav ua li cas tsim cov duab kos.
Alexey Ananyev, tus thawj coj ntawm kev cob qhia ntawm DIS Group
Pushdown yog dab tsi?
Ntau tus ntawm koj twb paub txog Informatica Big Data Management (BDM). Cov khoom lag luam tuaj yeem sib xyaw cov ntaub ntawv loj los ntawm ntau qhov chaw, txav mus los ntawm cov tshuab sib txawv, muab kev nkag tau yooj yim rau nws, tso cai rau koj kom profile nws, thiab ntau ntxiv.
Nyob rau hauv txoj cai tes, BDM tuaj yeem ua haujlwm tau zoo: cov haujlwm yuav ua tiav sai thiab nrog cov khoom siv suav tsawg tsawg.
Koj puas xav tau qhov ntawd thiab? Kawm siv lub zog thawb tawm hauv BDM los faib cov kev suav nrog hla ntau lub platform. Pushdown thev naus laus zis tso cai rau koj tig daim ntawv qhia ua ib tsab ntawv thiab xaiv ib puag ncig uas tsab ntawv no yuav khiav. Qhov kev xaiv no tso cai rau koj los ua ke cov kev ua tau zoo ntawm cov platform sib txawv thiab ua tiav lawv qhov kev ua tau zoo tshaj plaws.
Txhawm rau teeb tsa cov ntawv ua tiav ib puag ncig, koj yuav tsum xaiv hom pushdown. Tsab ntawv tuaj yeem khiav tag nrho ntawm Hadoop lossis ib feem faib ntawm qhov chaw thiab dab dej. Muaj 4 hom kev thawb tawm. Daim ntawv qhia tsis tas yuav hloov mus rau hauv tsab ntawv (native). Daim ntawv qhia tuaj yeem ua tau ntau npaum li qhov ua tau ntawm qhov chaw (qhov chaw) lossis tag nrho ntawm qhov chaw (tag nrho). Daim ntawv qhia kuj tuaj yeem hloov mus rau hauv Hadoop tsab ntawv (tsis muaj).
Pushdown optimization
Cov npe 4 hom tuaj yeem ua ke hauv ntau txoj hauv kev - kev thawb tawm tuaj yeem ua kom zoo rau qhov tshwj xeeb xav tau ntawm lub kaw lus. Piv txwv li, nws yog feem ntau tsim nyog los rho tawm cov ntaub ntawv los ntawm ib lub database siv nws tus kheej muaj peev xwm. Thiab cov ntaub ntawv yuav raug hloov dua siab tshiab siv Hadoop, thiaj li tsis mus overload lub database nws tus kheej.
Cia peb xav txog qhov xwm txheej thaum ob lub hauv paus thiab qhov chaw nyob hauv cov ntaub ntawv, thiab kev hloov pauv kev ua haujlwm tuaj yeem xaiv tau: nyob ntawm qhov chaw, nws yuav yog Informatica, database server, lossis Hadoop. Xws li ib qho piv txwv yuav tso cai rau koj kom nkag siab qhov tseeb tshaj plaws ntawm kev ua haujlwm ntawm lub tshuab no. Lawm, hauv lub neej tiag tiag, qhov teeb meem no tsis tshwm sim, tab sis nws yog qhov zoo tshaj plaws rau kev ua kom pom kev ua haujlwm.
Cia peb ua daim ntawv qhia los nyeem ob lub rooj hauv ib qho Oracle database. Thiab cia cov ntawv nyeem tau muab sau rau hauv ib lub rooj hauv tib lub database. Cov txheej txheem mapping yuav zoo li no:
Hauv daim ntawv qhia ntawm Informatica BDM 10.2.1 nws zoo li no:
Pushdown type - haiv neeg
Yog tias peb xaiv cov hom pushdown ib txwm, ces daim ntawv qhia yuav ua rau ntawm Informatica server. Cov ntaub ntawv yuav raug nyeem los ntawm Oracle neeg rau zaub mov, xa mus rau Informatica server, hloov pauv muaj thiab xa mus rau Hadoop. Hauv lwm lo lus, peb yuav tau txais cov txheej txheem ETL ib txwm.
Pushdown hom - qhov chaw
Thaum xaiv qhov chaw, peb tau txais lub sijhawm los faib peb cov txheej txheem ntawm database server (DB) thiab Hadoop. Thaum cov txheej txheem raug ua tiav nrog qhov teeb tsa no, thov kom khaws cov ntaub ntawv los ntawm cov ntxhuav yuav raug xa mus rau cov ntaub ntawv. Thiab tus so yuav tau ua nyob rau hauv daim ntawv ntawm cov kauj ruam ntawm Hadoop.
Daim duab execution yuav zoo li no:
Hauv qab no yog ib qho piv txwv ntawm kev teeb tsa lub sijhawm ua haujlwm ib puag ncig.
Hauv qhov no, daim ntawv qhia yuav ua nyob rau hauv ob kauj ruam. Hauv nws qhov chaw peb yuav pom tias nws tau hloov mus rau hauv tsab ntawv uas yuav raug xa mus rau qhov chaw. Tsis tas li ntawd, kev sib txuas cov ntxhuav thiab cov ntaub ntawv hloov pauv yuav raug ua nyob rau hauv daim ntawv ntawm cov lus nug overridden ntawm qhov chaw.
Hauv daim duab hauv qab no, peb pom qhov ua kom zoo tshaj plaws ntawm BDM, thiab cov lus nug rov hais dua ntawm qhov chaw.
Lub luag hauj lwm ntawm Hadoop nyob rau hauv no configuration yuav raug txo kom tswj cov ndlwg ntawm cov ntaub ntawv - orchestrating nws. Qhov tshwm sim ntawm cov lus nug yuav raug xa mus rau Hadoop. Thaum nyeem ntawv tiav, cov ntaub ntawv los ntawm Hadoop yuav raug sau rau hauv lub dab dej.
Pushdown type - tag nrho
Thaum koj xaiv tag nrho hom, daim ntawv qhia yuav ua tiav rau hauv cov lus nug database. Thiab qhov tshwm sim ntawm qhov kev thov yuav raug xa mus rau Hadoop. Ib daim duab ntawm cov txheej txheem zoo li no tau nthuav tawm hauv qab no.
Ib qho piv txwv teeb yog qhia hauv qab no.
Raws li qhov tshwm sim, peb yuav tau txais daim ntawv qhia kev ua kom zoo zoo ib yam li yav dhau los. Qhov sib txawv tsuas yog tias tag nrho cov logic raug xa mus rau tus txais hauv daim ntawv ntawm overriding nws insertion. Ib qho piv txwv ntawm optimized mapping yog nthuav tawm hauv qab no.
Ntawm no, zoo li yav dhau los, Hadoop plays lub luag haujlwm ntawm tus neeg xyuas pib. Tab sis ntawm no lub hauv paus yog nyeem nyob rau hauv nws tag nrho, thiab ces cov ntaub ntawv ua logic yog ua nyob rau hauv lub receiver theem.
Pushdown hom yog null
Zoo, qhov kev xaiv kawg yog hom pushdown, nyob rau hauv uas peb daim ntawv qhia yuav tig mus rau hauv tsab ntawv Hadoop.
Tam sim no optimized mapping yuav zoo li no:
Ntawm no cov ntaub ntawv los ntawm lub hauv paus ntaub ntawv yuav xub nyeem rau Hadoop. Tom qab ntawd, siv nws tus kheej txhais tau tias, ob cov ntaub ntawv no yuav ua ke. Tom qab ntawd, cov ntaub ntawv yuav raug hloov dua siab tshiab thiab upload rau hauv database.
Los ntawm kev nkag siab txog cov hauv paus ntsiab lus ntawm kev ua kom zoo dua qub, koj tuaj yeem npaj ntau cov txheej txheem rau kev ua haujlwm nrog cov ntaub ntawv loj. Yog li, tsis ntev los no, ib lub tuam txhab loj, tsuas yog ob peb lub lis piam, rub tawm cov ntaub ntawv loj los ntawm kev khaws cia rau hauv Hadoop, uas nws tau sau yav dhau los rau ntau xyoo.
Tau qhov twg los: www.hab.com