Chikamu cheETL chedura re data chinowanzo kufukidzwa neimba yekuchengetera pachezvayo uye inogamuchira kushomeka kune iyo huru dhatabhesi kana yekumberi-yekupedzisira chikamu, BI, uye kushuma. Panguva imwecheteyo, kubva pakuona kwemakanika ekuzadza imba yekuchengetera data, ETL inoita basa rakakosha uye inoda kutariswa kushoma kubva kune vatariri pane zvimwe zvikamu. Zita rangu ndiAlexander, ini iye zvino ndinotonga ETL kuRostelecom, uye munyaya ino ndichaedza kugovera zvishoma izvo mutariri weimwe yeETL inonyanya kuzivikanwa masisitimu mune hombe data warehouse paRostelecom inofanirwa kubata nayo.
Kana vaverengi vanodiwa vatojairana nepurojekiti yedu yekuchengetera data uye neInformatica PowerCenter chigadzirwa, saka iwe unogona nekukasira kuenda kuchikamu chinotevera.
Makore akati wandei apfuura, iyo pfungwa yeimwe yekambani data warehouse yakakura uye yakatanga kuitwa muRostelecom. Nhamba yezvinyorwa zvakagadzirisa matambudziko ega ega zvakange zvatogadzirwa, asi nhamba yezviitiko zvakakura, mari yekutsigira yakawedzerawo, uye zvakava pachena kuti ramangwana riri mu centralization. Architectural, iyi ndiyo yekuchengetera pachayo, ine akati wandei akaturikidzana, akaiswa paHadoop neGreenPlum, anobatsira dhatabhesi, ETL maitiro uye BI.
Panguva imwecheteyo, nekuda kwenhamba huru yenzvimbo yakagovaniswa, heterogeneous data masosi, yakakosha data upload mechanism yakagadzirwa, mashandiro ayo anodzorwa neInformatica. Nekuda kweizvozvo, mapakeji edata anoguma ari munzvimbo yeHadoop interface, mushure mezvo maitiro ekurodha data kuburikidza nekuchengetedza zvidimbu, Hadoop neGreenPlum anotanga, uye anotungamirwa neinonzi ETL control mechanism inoshandiswa muInformatica. Nekudaro, iyo Informatica sisitimu ndechimwe chezvinhu zvakakosha zvinovimbisa kushanda kweimba yekuchengetera zvinhu.
Kuchengetedza kwedu kuchatsanangurwa zvakadzama mune imwe yeanotevera mapositi.
Informatica PowerCenter/Big Data Management parizvino inoonekwa seyo inotungamira software mumunda wezvekubatanidza data maturusi. Ichi chigadzirwa cheAmerican company Informatica, inova imwe yevatambi vakasimba muETL (Extract Transform Load), data quality management, MDM (Master Data Management), ILM (Information Lifecycle Management) nezvimwe.
Iyo PowerCenter yatinoshandisa ndeye yakabatanidzwa Tomcat application server umo iyo Informatica maapplication pachayo anomhanya, ichiita masevhisi ayo:
Domain, kutaura zvazviri, iyi ndiyo hwaro hwezvimwe zvese; masevhisi, vashandisi, uye zvikamu zveGRID zvinoshanda mukati medura.
Administrator Console, webhu-yakavakirwa manejimendi uye yekutarisa chishandiso, kuwedzera kune Informatica Developer mutengi, chishandiso chikuru chekudyidzana nechigadzirwa.
MRS, Model Repository Service, metadata repository, inzvimbo pakati pedhatabhesi umo metadata inochengetwa nemuviri uye Informatica Developer mutengi umo budiriro iri kuitika. Repositories chengetedza tsananguro yedata uye rumwe ruzivo, kusanganisira kune akati wandei mamwe masevhisi eInfromatica, semuenzaniso, marongero ekuita mabasa (Mashedhi) kana data rekutarisa, pamwe nemaparamita ekushandisa, kunyanya, kubvumira kushandiswa kweiyo imwe application yekushanda nayo. zvakasiyana-siyana data zvinyorwa uye vanogamuchira.
DIS, Data Integration Service, iyi ibasa umo maitiro makuru ekushanda anoitika, zvikumbiro zvinomhanya mairi uye kutanga chaiko kweWorkflows (tsanangudzo dzekutevedzana kwemepu uye kupindirana kwavo) uye Mappings (shanduko, zvivharo umo shanduko pachayo inoitika, data data ) tora nzvimbo.
GRID kugadzirisa - chaizvo, sarudzo yekuvaka yakaoma uchishandisa akati wandei maseva, kana mutoro wakatangwa neDIS uchigoverwa pakati pemanodhi (kureva, maseva ari chikamu chedura). Panyaya yeiyi sarudzo, pamusoro pekugovera mutoro muDIS kuburikidza nekuwedzera GRID yekubvisa layer inobatanidza node akati wandei, pane iyo DIS inomhanya pachinzvimbo chekushanda pane imwe chete node, imwe backup yeMRS zviitiko zvinogona zvakare kugadzirwa. Iwe unogona kutoisa kuwanikwa kwepamusoro, uko kufona kwekunze kunogona kuitwa kuburikidza ne backup node kana iyo huru ikatadza. Isu takasiya iyi sarudzo yekuvaka parizvino.
Informatica PowerCenter, schematic
Mumatanho ekutanga ebasa sechikamu chekugovera data, matambudziko aigara achimuka, mamwe acho nekuda kwekusagadzikana kushanda kweInformatica panguva iyoyo. Ini ndichagovera dzimwe dzenguva dzisingakanganwike dzeiyi saga - mastering Informatica 10.
Yaimbova Informatica logo
Nzvimbo yedu yebasa inosanganisirawo dzimwe nzvimbo dzeInformatica, ivo vane yavo chaiyo nekuda kwemutoro wakasiyana, asi parizvino ndicharangarira chaizvo kuti Informatica yakagadziridzwa sei sechikamu cheETL chedura re data pachayo.
Izvi zvakaitika sei
Muna 2016, patakazove nebasa reInformatica, yakanga yatosvika pavhezheni 10.0, uye kune vaaishanda navo vane tariro vakanga vachisarudza kushandisa chigadzirwa chine diki vhezheni .0 mune yakakomba mhinduro, zvese zvaiita sezviri pachena - isu tinofanirwa kushandisa. shanduro itsva! Kubva pakuona kwezviwanikwa zvehardware, zvese zvaive zvakanaka panguva iyoyo.
Kubva muchirimo cha2016, kondirakiti yanga ichitarisira basa reInformatica, uye sekureva kwevashoma vashandisi vehurongwa, "yakashanda kanoverengeka pasvondo." Pano zvakakosha kujekesa kuti repository yaive de facto paPoC nhanho, pakanga pasina vatariri pachikwata uye sisitimu yaigara ichipunzika nekuda kwezvikonzero zvakasiyana, mushure maizvozvo injiniya wekontrakiti akaitora zvakare.
Mukudonha, vatariri vatatu vakabatana nechikwata, vachigovana nzvimbo dzebasa pakati pavo, uye basa rakajairwa rakatanga kuronga kushanda kwehurongwa muprojekiti, kusanganisira Informatica. Kusiyana, zvinofanirwa kutaurwa kuti chigadzirwa ichi hachina kupararira uye chine nharaunda yakakura maunogona kuwana mhinduro dzemibvunzo chero ipi neipi uye kugadzirisa chero dambudziko. Naizvozvo, kutsigirwa kwakazara kwehunyanzvi kubva kuRussia mudiwa Informatica kwaive kwakakosha, nerubatsiro rwekuti zvikanganiso zvedu zvese uye zvikanganiso zveiyo yechidiki Informatica 10 zvakagadziriswa.
Chinhu chekutanga chataifanira kuitira vagadziri vechikwata chedu uye kondirakiti yaive yekudzikamisa basa reInformatica pachayo, kuve nechokwadi chekushanda kwewebhu manejimendi console (Informatica Administrator).
Aya ndiwo maitiro ataigara tichisangana nevagadziri veInformatica
Tichisiya nzira yekutsvaga zvikonzero, chikonzero chikuru chekuputsika yaive nzira yekudyidzana yeInformatica software ine repository dhatabhesi, iyo yaive pane imwe iri kure server, kubva pakuona kweiyo network landscape. Izvi zvakakonzera kunonoka uye kukanganisa nzira dzinoongorora mamiriro eInformatica domain. Mushure mekugadzirisa dhatabhesi, kushandura maparamendi eInformatica, izvo zvakaita kuti iwedzere kushivirira kunonoka kwedhatabhesi, uye pakupedzisira kugadzirisa iyo Informatica vhezheni kusvika 10.1 uye kuendesa dhatabhesi kubva kune yapfuura server kune server iri pedyo neInformatica, dambudziko rakarasikirwa naro. relevance, uye kubva ipapo kwave nekuputsika kwerudzi urwu isu haticherechedze.
Imwe yekuedza kuwana Informatica Monitor kushanda
Mamiriro ezvinhu neadministration console aive akaomawo. Sezvo budiriro inoshanda yaiitika yakananga munzvimbo ine zvibereko, vamwe vashandi vaigara vachida kuongorora basa remepu uye mafambiro ebasa "pakufamba." MuInformatica itsva, iyo Data Integration Service haina chishandiso chakasiyana chekutarisa kwakadaro, asi chikamu chekutarisa chakaonekwa mune manejimendi web console (Informatica Administrator Monitor), maunogona kutarisa kushanda kwezvikumbiro, mafambiro uye mappings, kutanga, matanda. Nguva nenguva, iyo koni yakave isingawanikwe zvachose, kana ruzivo nezve zvazvino maitiro muDIS yakamira kugadzirisa, kana kukanganisa kwakaitika pakurodha mapeji.
Kusarudzwa kwejava parameters kudzikamisa mashandiro
Dambudziko rakagadziriswa nenzira dzakawanda, zviedzo zvakaitwa kushandura maparameter, matanda uye jstack zvakaunganidzwa, zvakatumirwa kutsigira, panguva imwechete kwaive nekushanda googling uye kungoona chete.
Chekutanga, MRS yakaparadzana yakagadzirwa kuti itariswe; sezvazvakazoitika gare gare, uyu ndiye mumwe wevatengi vezviwanikwa munzvimbo dzedu, sezvo mepu inotangwa zvakanyanya. Maparamita ane chekuita nejava murwi uye akati wandei mamwe akachinjwa.
Somugumisiro, nekuvandudzwa kunotevera Informatica 10.1.1, kushanda kwekoni uye kutarisa kwakagadzikana, vagadziri vakatanga kushanda zvakanyanya, uye maitiro enguva dzose akawedzera uye akawedzera.
Chiitiko chekudyidzana pakati pebudiriro nekutonga chingave chinonakidza. Nyaya yekunzwisisa kwakawanda kuti zvinhu zvinoshanda sei, zvingaitwe uye zvisingagoni kuitwa, nguva dzose inokosha kana uchishandisa maitiro akaoma. Naizvozvo, isu tinokwanisa kukurudzira zvakachengeteka kuti utange wadzidzisa timu yekutonga nezve maitirwo esoftware, uye timu yekuvandudza maitiro ekunyora kodhi uye kudhirowa maitiro muhurongwa, uye chete wozotumira yekutanga neyechipiri kushanda pane mhedzisiro. Izvi zvakakosha chaizvo kana nguva isiri iyo isingaperi sosi. Matambudziko mazhinji anogona kugadziriswa kunyangwe nekutsvaga kusarudzika kwesarudzo, asi dzimwe nguva zvimwe zvinoda ruzivo rwekutanga - nyaya yedu inosimbisa kukosha kwekunzwisisa iyi axiom.
Semuenzaniso, patakaedza kugonesa shanduro muMRS (sezvazvakazoitika pakupedzisira, imwe vhezheni yeSVN yaidiwa), mushure menguva yakati takavhundutswa nekuona kuti iyo system yekutanga nguva yakawedzera kusvika makumi akati wandei emaminetsi. Tawana chikonzero chekunonoka kwekutanga uye kudzima vhezheni, takaita zvakanaka zvakare.
Zvipingamupinyi zvinozivikanwa zvine chekuita neInformatica zvinosanganisira iyo epic kurwa nekukura java tambo. Pane imwe nguva, nguva yasvika yekudzokorora, ndiko kuti, kuwedzera maitiro akaiswa kune nhamba huru yemasisitimu ekutanga. Zvakazoitika kuti haasi maitiro ose mu10.1.1 akashanda zvakanaka, uye mushure mechinguva DIS yakava isingashande. Makumi ezviuru zveshinda akaonekwa, nhamba yavo ichikura zvakanyanya panguva yekutumirwa kwechikumbiro. Dzimwe nguva ndaifanira kutangazve kakawanda pazuva kuti ndidzorere kushanda.
Pano tinoda kutenda rutsigiro; matambudziko akaiswa munharaunda uye akagadziriswa nekukurumidza uchishandisa EBF (Emergency Bug Fix) - mushure meizvozvo, munhu wese akawana kunzwa kuti chishandiso chinoshanda chaizvo.
Zvichiri kushanda!
Pakazosvika nguva yatakatanga kushanda muchinangwa, Informatica yakaita seizvi. Shanduro yeInformatica 10.1.1HF1 (HF1 is HotFix1, musangano wevatengesi kubva kune yakaoma yeEBFs) ine yakawedzerwa yakaiswa EBF, iyo inogadzirisa matambudziko edu nekuyera uye mamwe mamwe, pane imwe server kubva kune matatu aive chikamu cheGRID, 20 x86_64 cores. uye kuchengetedza, pane yakakura inononoka dhisiki yemadhisiki emunharaunda - iyi ndiyo sevha yekumisikidzwa kweHadoop cluster. Pane imwe sevha yakafanana - iyo Oracle DBMS iyo ese ari maviri Informatica domain uye ETL control mechanism inoshanda. Zvese izvi zvinotariswa neakajairwa ekutarisa maturusi anoshandiswa muchikwata (Zabbix + Grafana) pamativi ese - Informatica pachayo nemasevhisi ayo, uye maitiro ekurodha achipinda mairi. Iye zvino zvose zviri zviviri kushanda uye kugadzikana, pasina kufunga nezvekunze zvinhu, ikozvino zvinoenderana nezvirongwa zvinoderedza mutoro.
Takaparadzana, tinogona kutaura nezve GRID. Nzvimbo yacho yakavakirwa pamanode matatu, paine mukana wekuenzanisa mitoro. Nekudaro, panguva yekuyedzwa, zvakaonekwa kuti nekuda kwezvinetso zvekudyidzana pakati pezvinoitika zvemaapplication edu, gadziriso iyi haina kushanda sezvaitarisirwa, uye vakafunga kusiya chirongwa ichi chekuvaka, vachibvisa maviri ematatu manodhi kubva mudura. Panguva imwecheteyo, chirongwa chacho pachacho chakaramba chakafanana, uye ikozvino ibasa reGRID, asi rinopera kune imwe node.
Parizvino, kuomerwa kunoramba kwakabatana nekudonha kwekuita kana uchigara uchichenesa dunhu rekutarisa - nemaitiro panguva imwe chete muCNN uye kumhanya kuchenesa, kukanganisa mukushanda kweiyo ETL control mechanic inogona kuitika. Izvi pari zvino zviri kugadziriswa "semudonzvo" - nekubvisa nemaoko dunhu rekutarisa, nekurasikirwa kwese data rayo rekare. Izvi hazvinyanyi kukosha pakubudirira, panguva yenguva dzose kushanda, asi ikozvino kutsvaga mhinduro yakajairwa kuri kuitika.
Rimwe dambudziko rinomuka kubva mumamiriro ezvinhu akafanana - dzimwe nguva kutangwa kwakawanda kwemaitiro edu ekutonga kunoitika.
Multiple application inotangwa zvinotungamira kutadza kwemeshini
Paunenge uchimhanya maererano nehurongwa, pane dzimwe nguva mutoro unorema pane system, mamiriro ezvinhu dzimwe nguva anoitika anokonzera kuparara kwemaitiro. Dambudziko richiri kugadziriswa nemaoko, uye mhinduro yekusingaperi iri kutsvakwa.
Kazhinji, tinogona kupfupisa kuti kana paine mutoro unorema, zvakakosha kupa zviwanikwa zvakaringana nazvo, izvi zvinoshandawo kune Hardware zviwanikwa zveInformatica pachayo, uye zvakafanana kune yayo dhatabhesi repository, pamwe nekupa yakakwana marongero. kwavari. Uye zvakare, mubvunzo unoramba wakavhurika wekuti ndeupi dhatabhesi yekuisa dhatabheti iri nani - pane yakaparadzana mugadziri, kana pane imwechete iyo iyo Informatica software inomhanya. Kune rimwe divi, ichave yakachipa pane imwe sevha, uye kana yasanganiswa, dambudziko rinogoneka nekubatana kwetiweki rinobviswa zvachose; kune rimwe divi, mutoro uri pamugamuchiri kubva kudhatabhesi unowedzerwa nemutoro kubva kuInformatica.
Sezvinoita chero chigadzirwa chakakomba, Informatica inewo nguva dzinosetsa.
Pane imwe nguva, ndichironga imwe tsaona, ndakaona kuti matanda eMRS airatidza zvinoshamisa nguva yezviitiko.
Temporal dualism muMRS matanda "nekugadzira"
Zvakazoitika kuti zvitambi zvenguva zvakanyorwa mufomati yeawa gumi nembiri, pasina kutsanangura AM/PM, ndiko kuti, masikati kana mushure. Chikumbiro chakatozarurwa maererano nenyaya iyi, uye mhinduro yakagamuchirwa - ndiwo magadzirirwo ayakaitwa, mamakisi anonyorwa mulog yeMRS nenzira chaiyo iyi. Kureva kuti, dzimwe nguva panoramba paine kunetsana maererano nenguva yekuitika kweimwe ERROR...
Edza kuita zvakanakisisa
Nhasi, Informatica chishandiso chakanyatsogadzikana, chakanakira vatariri nevashandisi, chine simba zvakanyanya maererano nekwazvino kugona uye kugona. Iyo inodarika zvatinoda zvinoshanda kakawanda pamusoro uye de facto ikozvino iri kushandiswa mupurojekiti nenzira isiri iyo yakajairika uye yakajairika. Matambudziko acho ane hukama nemashandiro anoita mashandiro acho - chinhu chaicho ndechekuti munguva pfupi nhamba yakakura yetambo inotangwa iyo yakanyatso gadziridza paramita uye inoshanda neiyo repository dhatabhesi, nepo server hardware zviwanikwa zvichishandiswa zvakazara. kubudikidza CPU.
Iye zvino tava pedyo nekuenda kuInformatica 10.2.1 kana 10.2.2, iyo yakagadzirisa zvakare mamwe maitiro emukati uye kutsigira kuvimbisa kubvisa zvimwe zvekushanda uye kushanda kwezvinhu zvatinazvo iye zvino. Uye kubva pakuona kwehardware, isu tinotarisira maseva ane dhizaini yakakwana kwatiri, tichifunga nezvekuchengetwa kweramangwana riri pedyo nekuda kwekukura nekusimudzira kwekuchengetedza.
Ehe, pachave nekuyedzwa, kuenderana kutarisa, uye pamwe shanduko yekuvaka muHA GRID chikamu. Budiriro mukati meInformatica ichaenderera mberi, sezvo munguva pfupi isu hatigone kupa chero chinhu kutsiva sisitimu.
Uye avo vanozove nemutoro weiyi sisitimu mune ramangwana vachanyatsokwanisa kuiunza kune inodiwa kuvimbika uye zviratidzo zvekuita zvinoiswa pamberi nevatengi.
Chinyorwa chakagadzirwa neRostelecom data management timu
Ikozvino Informatica logo
Source: www.habr.com