ETL feem ntawm cov ntaub ntawv warehouse feem ntau overshadowed los ntawm lub warehouse nws tus kheej thiab tau txais kev saib xyuas tsawg tshaj li lub ntsiab database los yog pem hauv ntej-kawg tivthaiv, BI, thiab qhia. Nyob rau tib lub sijhawm, los ntawm qhov pom ntawm cov neeg kho tshuab ntawm kev sau cov khoom lag luam nrog cov ntaub ntawv, ETL ua lub luag haujlwm tseem ceeb thiab yuav tsum tsis muaj kev saib xyuas tsawg dua los ntawm cov thawj coj dua li lwm cov khoom. Kuv lub npe yog Alexander, tam sim no kuv tswj hwm ETL ntawm Rostelecom, thiab hauv tsab xov xwm no kuv yuav sim qhia me ntsis txog dab tsi tus thawj coj ntawm ib lub npe nrov tshaj plaws ETL hauv lub tsev khaws ntaub ntawv loj ntawm Rostelecom yuav tsum tau nrog.
Yog tias cov neeg nyeem tau paub ntau dhau ntawm peb cov ntaub ntawv warehouse project thiab nrog Informatica PowerCenter cov khoom, ces koj tuaj yeem txav mus rau ntu tom ntej.
Ob peb xyoos dhau los, lub tswv yim ntawm ib lub tuam txhab cov ntaub ntawv warehouse matured thiab pib siv nyob rau hauv Rostelecom. Ib tug xov tooj ntawm repositories uas daws cov teeb meem ntawm ib tug neeg twb tau tsim, tab sis tus naj npawb ntawm scenarios loj hlob, kev them nyiaj yug nqi kuj nce, thiab nws tau pom tseeb tias yav tom ntej nteg nyob rau hauv centralization. Architecturally, qhov no yog qhov chaw cia nws tus kheej, muaj ob peb txheej, siv rau Hadoop thiab GreenPlum, pab pawg databases, ETL mechanisms thiab BI.
Nyob rau tib lub sijhawm, vim muaj coob tus ntawm thaj chaw faib khoom, cov ntaub ntawv sib txawv, cov ntaub ntawv tshwj xeeb upload mechanism tau tsim, kev ua haujlwm uas yog tswj hwm los ntawm Informatica. Raws li qhov tshwm sim, cov ntaub ntawv pob xaus rau hauv thaj chaw Hadoop interface, tom qab ntawd cov txheej txheem ntawm kev thauj cov ntaub ntawv los ntawm cov txheej txheem cia, Hadoop thiab GreenPlum pib, thiab lawv tau tswj hwm los ntawm lub npe hu ua ETL tswj mechanism siv hauv Informatica. Yog li, Informatica system yog ib qho ntawm cov ntsiab lus tseem ceeb uas ua kom muaj kev ua haujlwm ntawm lub tsev khaws khoom.
Peb qhov chaw khaws cia yuav piav qhia ntau ntxiv hauv ib qho ntawm cov ntawv hauv qab no.
Informatica PowerCenter/Big Data Management yog tam sim no suav hais tias yog tus thawj coj software hauv kev sib koom ua ke cov ntaub ntawv. Qhov no yog ib yam khoom ntawm American tuam txhab Informatica, uas yog ib tug ntawm cov muaj zog players nyob rau hauv ETL (Extract Transform Load), cov ntaub ntawv zoo tswj, MDM (Master Data Management), ILM (Information Lifecycle Management) thiab ntau dua.
Lub PowerCenter peb siv yog ib qho kev sib koom ua ke Tomcat daim ntawv thov server uas cov ntawv thov Informatica lawv tus kheej khiav, siv nws cov kev pabcuam:
sau, qhov tseeb, qhov no yog lub hauv paus rau txhua yam; cov kev pabcuam, cov neeg siv, thiab GRID cov khoom ua haujlwm nyob rau hauv lub npe.
Administrator Console, lub vev xaib tswj hwm thiab saib xyuas cov cuab yeej, ntxiv rau Informatica Developer tus neeg siv khoom, cov cuab yeej tseem ceeb rau kev cuam tshuam nrog cov khoom
MRS, Model Repository Service, metadata repository, yog ib txheej ntawm cov ntaub ntawv uas metadata tau khaws cia hauv lub cev thiab Informatica Developer tus neeg siv khoom uas muaj kev txhim kho. Repositories khaws cov ntaub ntawv piav qhia thiab lwm yam ntaub ntawv, suav nrog rau ntau qhov kev pabcuam Infromatica, piv txwv li, lub sijhawm rau kev khiav haujlwm (Schedules) lossis kev soj ntsuam cov ntaub ntawv, nrog rau cov ntawv thov tsis, tshwj xeeb, tso cai rau siv tib daim ntawv thov ua haujlwm nrog. ntau yam ntaub ntawv qhov chaw thiab receivers.
DIS, Data Integration Service, qhov no yog ib qho kev pabcuam uas cov txheej txheem ua haujlwm tseem ceeb tshwm sim, cov ntawv thov khiav hauv nws thiab qhov tseeb tso tawm ntawm Workflows (cov lus piav qhia ntawm qhov sib lawv liag ntawm daim ntawv qhia thiab lawv cov kev sib cuam tshuam) thiab Mappings (kev hloov pauv, thaiv qhov kev hloov pauv lawv tus kheej tshwm sim, ua cov ntaub ntawv. ) coj qhov chaw.
GRID configuration - qhov tseem ceeb, ib qho kev xaiv rau kev tsim lub complex siv ob peb servers, thaum lub load launched los ntawm DIS yog faib ntawm cov nodes (uas yog, servers uas yog ib feem ntawm lub sau). Nyob rau hauv cov ntaub ntawv ntawm qhov kev xaiv no, ntxiv rau kev faib cov load hauv DIS los ntawm ib qho ntxiv GRID abstraction txheej uas sib sau ua ke ob peb lub nodes, uas DIS khiav es tsis txhob ua hauj lwm ntawm ib qho ntawm ib qho, ntxiv backup MRS piv txwv kuj tuaj yeem tsim. Koj tuaj yeem txawm siv tau siab, qhov twg hu xov tooj sab nraud tuaj yeem ua los ntawm cov thaub qab yog tias lub ntsiab tsis ua haujlwm. Peb tau tso tseg qhov kev xaiv tsim kho rau tam sim no.
Informatica PowerCenter, schematic
Nyob rau theem pib ntawm kev ua haujlwm raws li ib feem ntawm cov ntaub ntawv xa khoom, cov teeb meem tsis tu ncua tshwm sim, qee qhov ntawm lawv vim kev ua haujlwm tsis ruaj khov ntawm Informatica thaum lub sijhawm ntawd. Kuv yuav qhia qee lub sijhawm nco txog ntawm zaj dabneeg no - mastering Informatica 10 .
Yav dhau los Informatica logo
Peb cheeb tsam ntawm lub luag haujlwm tseem suav nrog lwm qhov chaw Informatica, lawv muaj lawv tus kheej tshwj xeeb vim muaj qhov sib txawv, tab sis rau tam sim no kuv yuav nco ntsoov raws nraim li cas Informatica tsim los ua ETL tivthaiv ntawm cov ntaub ntawv warehouse nws tus kheej.
Qhov no tshwm sim li cas
Xyoo 2016, thaum peb tau los ua lub luag haujlwm rau kev ua haujlwm ntawm Informatica, nws twb tau mus txog version 10.0, thiab rau cov neeg ua haujlwm zoo uas tau txiav txim siab siv cov khoom me me version .0 hauv kev daws teeb meem loj, txhua yam zoo li pom tseeb - peb yuav tsum siv. tus tshiab version! Los ntawm qhov pom ntawm cov khoom siv kho vajtse, txhua yam zoo nyob rau lub sijhawm ntawd.
Txij li thaum lub caij nplooj ntoo hlav xyoo 2016, tus neeg cog lus tau ua lub luag haujlwm rau kev ua haujlwm ntawm Informatica, thiab raws li ob peb tus neeg siv ntawm lub kaw lus, "nws ua haujlwm ob peb zaug hauv ib lub lis piam." Ntawm no nws yog ib qho tsim nyog los qhia meej tias qhov chaw cia khoom yog qhov tseeb ntawm PoC theem, tsis muaj cov thawj coj hauv pab pawg thiab lub kaw lus tsis tu ncua cuam tshuam rau ntau yam laj thawj, tom qab ntawd tus neeg cog lus tus engineer tau khaws nws dua.
Thaum lub caij nplooj zeeg, peb tus thawj coj tau koom nrog pab pawg, faib lawv lub luag haujlwm ntawm lawv tus kheej, thiab kev ua haujlwm ib txwm pib los txhim kho kev ua haujlwm ntawm cov tshuab hauv qhov project, suav nrog Informatica. Cais, nws yuav tsum tau hais tias cov khoom no tsis yog dav thiab muaj lub zej zog loj uas koj tuaj yeem nrhiav cov lus teb rau cov lus nug thiab daws txhua yam teeb meem. Yog li ntawd, tag nrho cov kev txhawb nqa los ntawm Lavxias tus khub Informatica yog ib qho tseem ceeb heev, nrog kev pab los ntawm tag nrho peb cov kev ua yuam kev thiab kev ua yuam kev ntawm cov tub ntxhais hluas Informatica 10 tau raug kho.
Thawj qhov uas peb yuav tsum tau ua rau cov neeg tsim tawm ntawm peb pab neeg thiab cov neeg cog lus yog kom ruaj khov rau kev ua haujlwm ntawm Informatica nws tus kheej, kom ntseeg tau tias kev ua haujlwm ntawm lub vev xaib tswj hwm console (Informatica Administrator).
Nov yog qhov peb feem ntau ntsib Informatica developers
Tawm ntawm cov txheej txheem ntawm kev tshawb nrhiav cov laj thawj, qhov laj thawj tseem ceeb ntawm kev sib tsoo yog cov qauv kev sib cuam tshuam ntawm Informatica software nrog cov chaw khaws ntaub ntawv khaws cia, uas nyob rau ntawm tus neeg rau zaub mov nyob deb, los ntawm qhov pom ntawm thaj chaw network. Qhov no ua rau qeeb thiab cuam tshuam cov txheej txheem uas saib xyuas lub xeev ntawm Informatica sau. Tom qab qee qhov kev hloov kho ntawm cov ntaub ntawv, hloov qhov tsis sib xws ntawm Informatica, uas ua rau nws ua siab ntev dua ntawm cov ntaub ntawv qeeb, thiab thaum kawg hloov kho Informatica version rau 10.1 thiab hloov cov ntaub ntawv los ntawm cov neeg rau zaub mov dhau los mus rau lub server nyob ze rau Informatica, qhov teeb meem poob nws. qhov tseeb, thiab txij thaum ntawd los muaj kev sib tsoo ntawm hom no peb tsis saib.
Ib qho kev sim ua kom tau Informatica Monitor ua haujlwm
Qhov xwm txheej nrog rau kev tswj hwm console kuj tseem ceeb heev. Txij li kev txhim kho nquag tau ua tiav ncaj qha rau ntawm qhov chaw tsim tau zoo, cov npoj yaig xav tau los soj ntsuam kev ua haujlwm ntawm daim ntawv qhia thiab kev ua haujlwm "ntawm kev mus." Hauv Informatica tshiab, Cov Kev Pabcuam Cov Ntaub Ntawv tsis muaj cov cuab yeej sib cais rau kev saib xyuas zoo li no, tab sis ib ntu kev saib xyuas tau tshwm sim hauv kev tswj hwm lub vev xaib (Informatica Administrator Monitor), uas koj tuaj yeem saib xyuas kev ua haujlwm ntawm kev siv, kev ua haujlwm thiab kev ua haujlwm, launches, log. Ib ntus, lub console tau dhau los ua qhov tsis muaj, lossis cov ntaub ntawv hais txog cov txheej txheem tam sim no hauv DIS tau tso tseg tsis hloov kho, lossis qhov yuam kev tshwm sim thaum thauj cov nplooj ntawv.
Kev xaiv ntawm java parameters kom ruaj khov ua haujlwm
Qhov teeb meem tau raug kho nyob rau hauv ntau txoj kev, kev sim tau ua tiav los hloov qhov tsis, cav thiab jstack tau sau, xa mus rau kev txhawb nqa, tib lub sijhawm muaj kev ua haujlwm googling thiab kev soj ntsuam yooj yim.
Ua ntej tshaj plaws, ib qho MRS cais tau tsim los saib xyuas; raws li nws tau muab tawm tom qab, qhov no yog ib qho ntawm cov neeg siv khoom tseem ceeb ntawm cov khoom siv hauv peb ib puag ncig, txij li cov phiaj xwm tau nthuav tawm ntau heev. Parameters hais txog java heap thiab lwm tus tau hloov pauv.
Raws li qhov tshwm sim, los ntawm qhov hloov tshiab tom ntej Informatica 10.1.1, kev ua haujlwm ntawm lub console thiab saib xyuas tau ruaj khov, cov neeg tsim khoom tau pib ua haujlwm zoo dua, thiab cov txheej txheem tsis tu ncua tau dhau los ua ntau dua.
Kev paub txog kev sib cuam tshuam ntawm kev txhim kho thiab kev tswj hwm yuav nthuav. Qhov teeb meem ntawm kev nkag siab dav dav ntawm yuav ua li cas ua haujlwm, dab tsi tuaj yeem ua tau thiab ua tsis tau, yog ib qho tseem ceeb thaum siv cov txheej txheem nyuaj. Yog li ntawd, peb tuaj yeem pom zoo kom koj ua ntej cob qhia pawg thawj coj saib xyuas yuav ua li cas tswj hwm software, thiab pab pawg txhim kho yuav ua li cas sau cov lej thiab kos cov txheej txheem hauv qhov system, thiab tsuas yog tom qab ntawd xa thawj thiab thib ob los ua haujlwm ntawm qhov tshwm sim. Qhov no tseem ceeb heev thaum lub sij hawm tsis yog ib qho kev pab tsis kawg. Ntau qhov teeb meem tuaj yeem daws tau txawm tias los ntawm kev tshawb nrhiav random ntawm kev xaiv, tab sis qee zaum qee qhov xav tau kev paub ua ntej - peb cov ntaub ntawv lees paub qhov tseem ceeb ntawm kev nkag siab qhov axiom no.
Piv txwv li, thaum peb sim ua kom muaj kev hloov pauv hauv MRS (raws li nws tau tawm thaum kawg, xav tau ib qho txawv ntawm SVN), tom qab qee lub sij hawm peb tau ceeb toom kom pom tias lub sijhawm rov pib dua system tau nce mus rau ntau kaum feeb. Tau pom tias yog vim li cas rau qhov ncua sij hawm pib thiab disabling versioning, peb ua tau zoo dua.
Cov teeb meem tseem ceeb cuam tshuam nrog Informatica suav nrog kev sib ntaus sib tua epic nrog loj hlob java threads. Qee lub sijhawm, lub sijhawm tau los rau kev rov ua dua, uas yog, txuas ntxiv cov txheej txheem tsim los rau ntau lub hauv paus. Nws tau muab tawm tias tsis yog tag nrho cov txheej txheem hauv 10.1.1 ua haujlwm zoo, thiab tom qab qee lub sijhawm DIS ua haujlwm tsis tau. Kaum tawm txhiab tus xov tau raug kuaj pom, lawv cov lej loj hlob tshwj xeeb tshaj yog thaum lub sijhawm thov kev xa mus. Qee zaum kuv yuav tsum rov pib dua ob peb zaug hauv ib hnub kom rov ua haujlwm dua.
Ntawm no peb yuav tsum ua tsaug rau qhov kev txhawb nqa; cov teeb meem tau nyob hauv cheeb tsam thiab kho sai sai siv EBF (Kev Kho Kab Mob Kub Kub Kub Kub) - tom qab ntawd, txhua tus tau txais kev xav tias lub cuab yeej ua haujlwm tiag tiag.
Nws tseem ua haujlwm!
Los ntawm lub sijhawm peb pib ua haujlwm hauv hom phiaj, Informatica zoo li qhov no. Version of Informatica 10.1.1HF1 (HF1 yog HotFix1, cov neeg muag khoom sib dhos los ntawm ib qho nyuaj ntawm EBFs) nrog rau ntxiv EBF, uas kho peb cov teeb meem nrog scaling thiab lwm yam, ntawm ib tus neeg rau zaub mov ntawm peb uas yog ib feem ntawm GRID, 20 x86_64 cores thiab cia, ntawm qhov loj loj qeeb ntawm cov disks hauv zos - qhov no yog tus neeg rau zaub mov teeb tsa rau Hadoop pawg. Ntawm lwm tus neeg rau zaub mov zoo sib xws - Oracle DBMS uas ob qho tib si Informatica sau thiab ETL tswj mechanism ua haujlwm. Tag nrho cov no yog saib xyuas los ntawm tus qauv saib xyuas cov cuab yeej siv hauv pab pawg (Zabbix + Grafana) ntawm ob sab - Informatica nws tus kheej nrog nws cov kev pabcuam, thiab cov txheej txheem thauj khoom mus rau hauv nws. Tam sim no ob qho tib si kev ua tau zoo thiab kev ruaj ntseg, tsis tas yuav mus rau hauv tus account lwm yam, tam sim no nyob ntawm qhov chaw uas txwv cov load.
Cais, peb tuaj yeem hais txog GRID. Ib puag ncig tau tsim los ntawm peb cov nodes, nrog rau qhov muaj peev xwm ntawm kev sib npaug ntawm kev thauj khoom. Txawm li cas los xij, thaum lub sij hawm sim, nws tau pom tias vim muaj teeb meem kev sib cuam tshuam ntawm kev ua haujlwm ntawm peb cov ntawv thov, qhov kev teeb tsa no tsis ua haujlwm raws li qhov xav tau, thiab lawv tau txiav txim siab tso tseg qhov kev tsim kho no ib ntus, tshem tawm ob ntawm peb qhov ntawm lub npe. Nyob rau tib lub sijhawm, lub tswv yim nws tus kheej tau zoo ib yam, thiab tam sim no nws yog qhov tseeb ntawm GRID kev pabcuam, tab sis degenerate rau ib ntawm.
Tam sim no, qhov nyuaj tseem cuam tshuam nrog kev poob qis hauv kev ua haujlwm thaum tu lub ntsuas hluav taws xob tsis tu ncua - nrog rau cov txheej txheem ib txhij hauv CNN thiab khiav tu, ua haujlwm tsis zoo hauv kev ua haujlwm ntawm ETL tswj mechanism yuav tshwm sim. Qhov no yog tam sim no tau daws "raws li ib tug crutch" - manually clearing lub monitor Circuit Court, nrog poob ntawm tag nrho nws cov ntaub ntawv yav dhau los. Qhov no tsis yog qhov tseem ceeb heev rau kev tsim khoom, thaum lub sijhawm ua haujlwm ib txwm ua, tab sis tam sim no kev tshawb nrhiav kev daws teeb meem ib txwm tab tom pib.
Lwm qhov teeb meem tshwm sim los ntawm tib qhov xwm txheej no - qee zaum muaj ntau qhov kev tso tawm ntawm peb cov txheej txheem tswj tau tshwm sim.
Ntau daim ntawv thov launches ua rau kev ua tsis tiav
Thaum khiav raws li lub sij hawm, thaum lub sij hawm ntawm hnyav load ntawm lub system, tej zaum muaj tej yam tshwm sim uas ua rau kev puas tsuaj ntawm lub mechanism. Qhov teeb meem tseem raug kho manually, thiab nrhiav kev daws teeb meem mus tas li.
Feem ntau, peb tuaj yeem xaus lus tias thaum muaj kev thauj khoom hnyav, nws yog ib qho tseem ceeb heev los muab cov peev txheej txaus rau nws, qhov no kuj siv rau cov khoom siv kho vajtse rau Informatica nws tus kheej, thiab tib yam rau nws cov chaw khaws ntaub ntawv, nrog rau muab kev pom zoo. rau lawv. Tsis tas li ntawd, cov lus nug tseem qhib rau qhov twg cov txheej txheem kev tso kawm database zoo dua - ntawm ib tus tswv tsev sib cais, lossis ntawm tib qhov uas Informatica software khiav. Ntawm qhov tod tes, nws yuav pheej yig dua ntawm ib tus neeg rau zaub mov, thiab thaum sib xyaw ua ke, qhov teeb meem tshwm sim nrog kev sib cuam tshuam hauv lub network tau raug tshem tawm; ntawm qhov tod tes, cov khoom thauj ntawm tus tswv tsev los ntawm cov ntaub ntawv yog ntxiv los ntawm kev thauj khoom los ntawm Informatica.
Ib yam li txhua yam khoom loj, Informatica kuj muaj lub sijhawm lom zem.
Ib zaug, thaum txheeb xyuas qee yam kev huam yuaj, kuv pom tias MRS cov ntawv teev lus txawv txawv qhia lub sijhawm ntawm cov xwm txheej.
Temporal dualism hauv MRS cav "los ntawm kev tsim"
Nws muab tawm tias cov ntawv teev sijhawm tau sau ua 12 teev, yam tsis tau qhia AM / PM, uas yog, ua ntej tav su lossis tom qab. Ib daim ntawv thov tau qhib txawm hais txog qhov teeb meem no, thiab tau txais cov lus teb raug cai - qhov no yog qhov nws tau npaj, cov qhab nia tau sau rau hauv MRS teev nyob rau hauv raws nraim hom ntawv no. Ntawd yog, qee zaum tseem muaj qee qhov kev xav txog lub sijhawm tshwm sim ntawm qee qhov yuam kev ...
Siv zog rau qhov zoo tshaj plaws
Niaj hnub no, Informatica yog lub cuab yeej ruaj khov, yooj yim rau cov thawj coj thiab cov neeg siv, muaj zog heev ntawm nws cov peev xwm tam sim no thiab muaj peev xwm. Nws tshaj peb cov kev xav tau ntau zaus dhau los thiab tam sim no de facto yog siv rau hauv qhov project nyob rau hauv ib txoj kev uas tsis yog feem ntau raug thiab raug. Cov teeb meem yog ib feem cuam tshuam nrog txoj kev ua haujlwm ntawm cov txheej txheem - qhov tshwj xeeb yog tias nyob rau lub sijhawm luv luv ntawm ntau cov xov tau nthuav tawm uas siv cov kev hloov kho tshiab thiab ua haujlwm nrog cov chaw khaws ntaub ntawv khaws cia, thaum lub server hardware cov peev txheej tau siv yuav luag tag. los ntawm CPU.
Peb tam sim no nyob ze rau kev tsiv mus rau Informatica 10.2.1 lossis 10.2.2, uas tau rov ua haujlwm qee yam ntawm cov txheej txheem sab hauv thiab kev txhawb nqa cov lus cog tseg kom tshem tawm qee qhov kev ua haujlwm thiab kev ua haujlwm uas peb muaj tam sim no. Thiab los ntawm kev xav txog kev kho vajtse, peb cia siab tias cov servers nrog kev pom zoo rau peb, suav nrog kev cia siab rau yav tom ntej vim kev loj hlob thiab kev loj hlob ntawm kev cia.
Tau kawg, yuav muaj kev sim, kev tshuaj xyuas kev sib raug zoo, thiab tej zaum yuav muaj kev hloov kho vaj tse hauv HA GRID feem. Kev loj hlob hauv Informatica yuav txuas ntxiv mus, txij li lub sijhawm luv luv peb tsis tuaj yeem muab ib yam dab tsi los hloov lub kaw lus.
Thiab cov neeg uas yuav ua lub luag haujlwm rau qhov system yav tom ntej yuav muaj peev xwm coj nws mus rau qhov yuav tsum tau muaj kev ntseeg siab thiab kev ua tau zoo ntawm cov neeg siv khoom.
Tsab ntawv tau npaj los ntawm Rostelecom pab pawg tswj xyuas cov ntaub ntawv
Tam sim no Informatica logo
Tau qhov twg los: www.hab.com