A e maua le fua Sberov. Faʻaaogaina le Ab Initio ma Hive ma GreenPlum

I se taimi ua mavae, sa matou feagai ma le fesili o le filifilia o se meafaigaluega ETL mo le galulue faatasi ma Big Data. Ole fa'aogaina muamua ole Informatica BDM solution e le'i fetaui ma i matou ona o le fa'atapula'aina o galuega. O lona fa'aogaina ua fa'aitiitia i se fa'ava'a mo le fa'alauiloaina o fa'atonuga tu'uina atu. E le tele ni analogues i luga o le maketi na, i le mataupu faavae, e mafai ona galulue ma le tele o faʻamaumauga matou te feagai i aso uma. I le faaiuga na matou filifilia Ab Initio. I le taimi o faʻataʻitaʻiga pailate, na faʻaalia e le oloa le maualuga tele o faʻamaumauga o faʻamaumauga. Toeitiiti lava leai se faʻamatalaga e uiga ia Ab Initio i le gagana Rusia, o lea na matou filifili ai e talanoa e uiga i lo matou poto masani i Habré.

O le Ab Initio e tele suiga masani ma e le masani ai, o le tulafono e mafai ona faʻalauteleina e faʻaaoga ai lana lava gagana PDL. Mo se pisinisi laʻititi, o sea meafaigaluega mamana e foliga mai e sili atu, ma o le tele o ana gafatia atonu e taugata ma le faʻaaogaina. Ae afai o lau fua e latalata i le Sberov's, o le Ab Initio atonu e manaia ia te oe.

E fesoasoani i se pisinisi e faʻaputuina le malamalama i le lalolagi atoa ma atiina ae se faʻalapotopotoga, ma se tagata faʻaleleia e faʻaleleia ona tomai i le ETL, faʻaleleia lona malamalama i totonu o le atigi, maua le avanoa e aʻoaʻo ai le gagana PDL, tuʻuina atu se ata vaʻaia o faʻagasologa o uta, ma faʻafaigofie le atinaʻe. ona o le tele o vaega aoga.

I lenei pou o le a ou talanoa e uiga i le gafatia o Ab Initio ma tuʻuina atu uiga faʻatusatusa o lana galuega ma Hive ma GreenPlum.

• Fa'amatalaga o le fa'avae MDW ma galue i lona fa'avasegaga mo GreenPlum
• Ab Initio faʻatusatusaga faʻatinoga i le va o Hive ma GreenPlum
• Galulue Ab Initio ma GreenPlum ile Fa'ata'ita'iga Taimi Moni

O le faʻatinoga o lenei oloa e matua lautele ma e manaʻomia le tele o le taimi e suʻesuʻe ai. Ae ui i lea, faʻatasi ai ma tomai faʻapitoa galuega ma faʻatulagaga saʻo o faʻatinoga, o taunuʻuga o faʻamaumauga faʻamaumauga e matua manaia lava. O le fa'aaogaina o le Ab Initio mo se fa'atupuina e mafai ona maua ai se mea manaia. Ole mea fou lea ile atina'e o le ETL, ole va'aiga ile va o se si'osi'omaga va'aia ma le fa'aleleia o le la'uina i se gagana fa'atusa.

Pisinisi o loʻo atiaʻe a latou faʻalapotopotoga faʻanatura ma o lenei meafaigaluega e sili atu ona aoga nai lo se isi lava taimi. Faatasi ai ma Ab Initio, e mafai ona e faʻaputuina le malamalama e uiga i lau pisinisi o loʻo iai nei ma faʻaoga lenei malamalama e faʻalautele ai pisinisi tuai ma tatala pisinisi fou. Su'ega i Ab Initio e aofia ai siosiomaga atina'e va'aia Informatica BDM ma siosiomaga atina'e e le va'aia Apache Spark.

Fa'amatalaga o Ab Initio

Ab Initio, pei o isi meafaigaluega ETL, o se aofaʻiga o oloa.

Ab Initio GDE (Graphical Development Environment) o se siosiomaga mo le atinaʻe lea na te faʻapipiʻiina ai suiga o faʻamatalaga ma faʻafesoʻotaʻi i latou i faʻamatalaga tafe i foliga o aū. I lenei tulaga, o se seti o suiga ua taʻua o le kalafi:

O feso'ota'iga fa'aoga ma mea fa'atino o vaega fa'atino o ports ma o lo'o iai fanua fa'atatau i totonu o suiga. O le tele o kalafi e feso'ota'i i tafega i foliga o aū i le fa'asologa o latou fa'atinoga ua ta'ua o se fuafuaga.

E tele selau vaega aoga, lea e tele. O le tele oi latou e matua fa'apitoa. Ole gafatia ole suiga masani ile Ab Initio e lautele atu nai lo isi meafaigaluega ETL. Mo se faʻataʻitaʻiga, o le Join e tele mea e fai. E le gata i le i'uga o le feso'ota'iina o fa'amaumauga, e mafai ona e maua fa'amaumauga o fa'amaumauga o fa'amaumauga tu'ufa'atasi e le mafai ona feso'ota'i ona ki. E mafai foi ona e maua ni teena, mea sese ma se ogalaau o le suiga o le gaioiga, lea e mafai ona faitau i le koluma tutusa o se faila faila ma faʻatautaia ma isi suiga:

Pe, mo se faʻataʻitaʻiga, e mafai ona e faʻatinoina se faʻamatalaga faʻamatalaga i foliga o se laulau ma faitau faʻamatalaga mai ai i le koluma tutusa.

E iai suiga muamua. Mo se fa'ata'ita'iga, o le Su'iga Fa'ata'ita'i e iai fa'atinoga tutusa ma galuega fa'atatau. O loʻo i ai suiga faʻatasi ma igoa faʻamalamalamaga a le tagata lava ia: Fausia Faʻamaumauga, Faitau Excel, Faʻatonu, Faʻavasega i totonu o Vaega, Polokalama Taʻavale, Run SQL, Faʻatasi ma le DB, ma isi. le faiga fa'aoga . O faila o lo'o i ai se seti o tapula'a ua uma ona tu'uina atu i le kalafi e ta'ua o seti fa'amau (psets).

E pei ona faʻamoemoeina, o Ab Initio GDE o loʻo i ai lana lava fale teu oloa e taʻua o le EME (Enterprise Meta Environment). E maua e le au atiaʻe le avanoa e galulue ai ma faʻalapotopotoga faʻapitonuʻu o code ma siaki i totonu o latou atinaʻe i totonu o le faleoloa tutotonu.

E mafai, i le taimi o le faʻatinoga poʻo le maeʻa ona faʻatinoina o le kalafi, e kiliki i luga o soʻo se tafe e fesoʻotaʻi ai le suiga ma vaʻai i faʻamatalaga na pasia i le va o nei suiga:

E mafai foi ona kiliki i luga o soʻo se vaitafe ma vaʻai i faʻamatalaga suʻesuʻe - pe fia le tutusa o le suiga na galue ai, pe fia laina ma paita na utaina i totonu o fea o tutusa:

E mafai ona vaevaeina le faʻatinoga o le kalafi i vaega ma faʻailogaina o nisi suiga e manaʻomia ona faia muamua (i le zero phase), o le isi i le vaega muamua, o le isi i le vaega lona lua, ma isi.

Mo suiga taʻitasi, e mafai ona e filifilia le mea e taʻua o le faʻatulagaga (lea o le a faia ai): e aunoa ma ni faʻatusatusaga poʻo ni filo tutusa, o le numera e mafai ona faʻamaonia. I le taimi lava e tasi, faila le tumau na faia e Ab Initio pe a faʻagasolo suiga e mafai ona tuʻu uma i le faila faila faila ma i le HDFS.

I suiga taʻitasi, faʻavae i luga o le faʻataʻitaʻiga le lelei, e mafai ona e fatuina lau lava tusitusiga i le PDL, e pei o se atigi.

Faatasi ai ma le PDL e mafai ona e faʻalauteleina le faʻatinoga o suiga ma, aemaise lava, e mafai ona e faʻamalosi (i le taimi taʻavale) faʻatupuina faʻailoga faʻavae faʻavae e faʻatatau i taʻaloga taʻavale.

O le Ab Initio o loʻo iai foʻi le atinaʻeina lelei o le tuʻufaʻatasia ma le OS e ala i atigi. Aemaise lava, Sberbank faʻaaogaina linux ksh. E mafai ona e fa'afesuia'i fesuiaiga ma le atigi ma fa'aogaina e fai ma fa'asologa o kalafi. E mafai ona e vala'au le fa'atinoina o le Ab Initio kalafi mai le atigi ma fa'atonu le Ab Initio.

I le faaopoopo atu ia Ab Initio GDE, e tele isi oloa o loʻo aofia i le tuʻuina atu. E iai lana lava Co> Operation System ma se tagi e taʻua o se faiga faʻaogaina. O loʻo i ai le Pulea> Nofoaga e mafai ai ona e faʻatulagaina ma mataʻituina tafega o le download. O lo'o iai oloa mo le fa'atinoina o atina'e i se tulaga e sili atu ona fa'atumauina nai lo le fa'atagaina e Ab Initio GDE.

Fa'amatalaga o le fa'avae MDW ma galue i lona fa'avasegaga mo GreenPlum

Faʻatasi ai ma ana oloa, o loʻo tuʻuina atu e le tagata faʻatau oloa le MDW (Metadata Driven Warehouse), o se faʻasologa o kalafi ua fuafuaina e fesoasoani i galuega masani o le faʻaputuina o faleteuoloa faʻamaumauga poʻo faʻamaumauga faʻamaumauga.

O lo'o i ai fa'ata'ita'iga metadata fa'apitoa (poloketi-fa'apitoa) ma fa'aputu fa'ailoga ua saunia mai le pusa.

I le avea ai ma faʻaoga, e maua e le MDW se faʻataʻitaʻiga faʻamatalaga, se faila faʻatulagaina mo le faʻatulagaina o se fesoʻotaʻiga i se faʻamaumauga (Oracle, Teradata poʻo le Hive) ma isi faʻatulagaga. O le vaega faʻapitoa o le poloketi, mo se faʻataʻitaʻiga, faʻapipiʻi le faʻataʻitaʻiga i se faʻamaumauga. O le vaega i fafo o le pusa o le oloa e gaosia ai kalafi ma faila faila mo i latou e ala i le utaina o faʻamatalaga i laulau faʻataʻitaʻi. I lenei tulaga, o kalafi (ma seti) e faia mo le tele o auala o le amataina ma le faʻaopoopoga o galuega i le faʻafouina o vaega.

I tulaga o Hive ma RDBMS, o kalafi eseese e gaosia mo le amataina ma faʻaopoopoga faʻamatalaga faʻamatalaga.

I le tulaga o Hive, o faʻamatalaga o loʻo oʻo mai e fesoʻotaʻi e ala i le Ab Initio Join faʻatasi ma faʻamatalaga sa i luga o le laulau aʻo leʻi faʻafouina. Faʻamaumauga o loʻo i totonu o le MDW (i totonu o le Hive ma le RDBMS) e le gata ina faʻapipiʻi faʻamatalaga fou mai le delta, ae faʻapea foʻi ona tapunia taimi talafeagai o faʻamaumauga o latou ki autu na mauaina le delta. E le gata i lea, e tatau ona e toe tusia le vaega e le suia o faʻamaumauga. Ae e tatau ona faia lenei mea ona e leai se faʻamalo pe faʻafouina e Hive.

I le tulaga o le RDBMS, o kalafi mo le faʻafouina o faʻamaumauga faʻaopoopo e foliga sili atu ona lelei, aua o le RDBMS o loʻo i ai le faʻafouina moni.

O le delta na maua e utaina i totonu o se laulau vaeluaga i totonu o le database. A maeʻa lenei, e fesoʻotaʻi le delta i faʻamatalaga sa i le laulau aʻo leʻi faʻafouina. Ma e faia lenei mea i le faʻaaogaina o le SQL e faʻaaoga ai se fesili SQL. O le isi, faʻaaogaina le SQL commands delete + insert, o faʻamatalaga fou mai le delta e tuʻuina i totonu o le laulau faʻatatau ma taimi talafeagai o faʻamaumauga o latou ki autu na mauaina le delta ua tapunia.
E leai se mana'oga e toe tusia fa'amaumauga e le'i suia.

O lea na matou oʻo mai ai i le faaiuga e faapea i le tulaga o Hive, MDW e tatau ona alu i le tulaga o le toe tusia o le laulau atoa ona o le Hive e leai se galuega faʻafouina. Ma e leai se mea e sili atu nai lo le toe tusia atoa o faʻamaumauga pe a faʻafouina le faʻafouina. I le tulaga o le RDBMS, i se isi itu, na iloa e tagata na faia le oloa e tatau ona tuʻuina atu le fesoʻotaʻiga ma le faʻafouina o laulau i le faʻaogaina o le SQL.

Mo se poloketi i Sberbank, na matou faia se faʻatinoga fou, toe faʻaaogaina o se uta faʻamaumauga mo GreenPlum. Na faia lenei mea e faʻavae i luga o le faʻasologa o loʻo gaosia e MDW mo Teradata. O Teradata, ae le o Oracle, na sili ona latalata ma sili ona lelei mo lenei mea, aua ... ose faiga MPP foi. O auala galue, faʻapea foʻi ma le syntax, o Teradata ma GreenPlum na foliga tutusa.

Fa'ata'ita'iga o MDW-tulaga ese'esega i le va o RDBMS eseese e fa'apea. I GreenPlum, e le pei o Teradata, pe a fatuina laulau e te manaʻomia e tusi ai se fuaiupu

distributed by

Na tusia e Teradata:

delete <table> all

, ma i GreenPlum latou te tusia

delete from <table>

I totonu o Oracle, mo faʻamoemoega sili ona lelei latou te tusia

delete from t where rowid in (<соединение t с дельтой>)

, ma Teradata ma GreenPlum tusi

delete from t where exists (select * from delta where delta.pk=t.pk)

Matou te maitauina foi mo Ab Initio e galulue ma GreenPlum, sa tatau ona faʻapipiʻi le GreenPlum client i luga o nodes uma o le Ab Initio cluster. E mafua ona ua matou fesoʻotaʻi i GreenPlum i le taimi e tasi mai nodes uma i la matou fuifui. Ma ina ia mafai ona tutusa le faitau mai le GreenPlum ma e tutusa uma le Ab Initio filo e faitau lana lava vaega o faʻamatalaga mai GreenPlum, e tatau ona matou tuʻuina se fausaga e malamalama ai Ab Initio i le "mea" vaega o fesili SQL.

where ABLOCAL()

ma fuafua le taua o lenei fausiaina e ala i le faʻamaonia o le faitau faʻamaufaʻailoga mai le faʻamaumauga o suiga

ablocal_expr=«string_concat("mod(t.", string_filter_out("{$TABLE_KEY}","{}"), ",", (decimal(3))(number_of_partitions()),")=", (decimal(3))(this_partition()))»

, lea e tuufaatasia i se mea faapena

mod(sk,10)=3

, i.e. e tatau ona e fa'aosofia GreenPlum ma se faamama manino mo vaeluaga ta'itasi. Mo isi fa'amaumauga (Teradata, Oracle), e mafai e Ab Initio ona fa'atinoina otometi lenei fa'atusa.

Ab Initio faʻatusatusaga faʻatinoga i le va o Hive ma GreenPlum

Sa faia e le Sberbank se faʻataʻitaʻiga e faʻatusatusa ai le faʻatinoga o kalafi e gaosia e le MDW e faʻatatau i Hive ma faʻatatau i GreenPlum. I le avea ai o se vaega o le faʻataʻitaʻiga, i le tulaga o Hive sa i ai 5 nodes i luga o le fuifui tutusa e pei o Ab Initio, ma i le tulaga o GreenPlum sa i ai 4 nodes i luga o se fuifui ese. O na. O Hive na i ai se mea faʻaogaina i luga ole GreenPlum.

Na matou mafaufauina ni paga e lua o kalafi o loʻo faʻatinoina le galuega lava e tasi o le faʻafouina o faʻamaumauga i Hive ma GreenPlum. I le taimi lava e tasi, o kalafi na gaosia e le MDW configurator na faʻalauiloaina:

• uta muamua + uta fa'aopoopo o fa'amaumauga fa'atupu fa'afuase'i i totonu ole laulau Hive
• uta muamua + uta fa'aopoopo o fa'amaumauga fa'atupu fa'afuase'i ile laulau GreenPlum tutusa

I tulaga uma e lua (Hive ma GreenPlum) na latou faʻapipiʻiina i luga o le 10 filo tutusa i luga o le fuifui Ab Initio tutusa. Na fa'asaoina e Ab Initio fa'amatalaga vavalo mo fa'atatau ile HDFS (e tusa ai ma le Ab Initio, fa'aoga MFS fa'aoga HDFS). E tasi le laina o fa'amaumauga fa'atupu fa'afuase'i e nofoia le 200 paita i tulaga uma e lua.

O le taunuuga e pei o lenei:

ofaga:

Uluai uta i Hive

Ua faaofi mai laina
6 000 000
60 000 000
600 000 000

Taimi amata
download i sekone
41
203
1 601

Faʻaopoopo le utaina i totonu o le Hive

Numera o laina avanoa i totonu
laulau fa'atatau ile amataga ole su'ega
6 000 000
60 000 000
600 000 000

Numera o laina delta e fa'aoga i ai
laulau fa'atatau ile taimi ole su'ega
6 000 000
6 000 000
6 000 000

Umi ole fa'aopoopoga
download i sekone
88
299
2 541

GreenPlum:

Uluai uta i GreenPlum

Ua faaofi mai laina
6 000 000
60 000 000
600 000 000

Taimi amata
download i sekone
72
360
3 631

Faʻateleina uta i GreenPlum

Numera o laina avanoa i totonu
laulau fa'atatau ile amataga ole su'ega
6 000 000
60 000 000
600 000 000

Numera o laina delta e fa'aoga i ai
laulau fa'atatau ile taimi ole su'ega
6 000 000
6 000 000
6 000 000

Umi ole fa'aopoopoga
download i sekone
159
199
321

Matou te vaʻaia o le saoasaoa o le uta muamua i Hive ma GreenPlum laina faʻalagolago i le aofaʻi o faʻamaumauga ma, mo mafuaʻaga o meafaigaluega sili atu, e sili atu le vave mo Hive nai lo GreenPlum.

O le fa'aopoopoina o le utaina i totonu o le Hive e fa'alagolago fo'i ile tele o fa'amaumauga sa uta muamua o lo'o maua ile laulau fa'atatau ma fa'agasolo malie a'o fa'atuputeleina le voluma. E mafua lea ona o le mana'omia ona toe tusi atoa le laulau fa'atatau. O lona uiga o le faʻaogaina o suiga laiti i laulau tetele e le o se faʻaoga lelei mo Hive.

O le faʻaopoopoga o le utaina i GreenPlum e faʻalagolago lava i le tele o faʻamaumauga na faʻapipiʻiina muamua o loʻo maua i le laulau faʻatatau ma faʻagasolo vave. Na tupu lenei mea faʻafetai i le SQL Joins ma le GreenPlum architecture, lea e mafai ai ona tape le gaioiga.

O lea la, GreenPlum faʻaopoopo le delta e faʻaaoga ai le tape + insert method, ae o Hive e leai ni faʻamalo pe faʻafouina gaioiga, o lea na faʻamalosia ai le faʻamaumauga atoa o faʻamaumauga ina ia toe tusia atoa i le taimi o se faʻafouga faʻaopoopo. O le faʻatusatusaga o sela o loʻo faʻamamafaina i le mataʻutia e sili ona faʻaalia, talu ai e fetaui ma le filifiliga sili ona taatele mo le faʻaogaina o punaoa-malosi tele download. Matou te vaʻaia na taia e GreenPlum Hive i lenei suʻega e 8 taimi.

Galulue Ab Initio ma GreenPlum ile Fa'ata'ita'iga Taimi Moni

I lenei fa'ata'ita'iga, o le a matou fa'ata'ita'iina le agava'a o Ab Initio e fa'afou le laulau GreenPlum ma fa'atupu fa'afuase'i pusi o fa'amaumauga i se taimi lata mai. Sei o tatou mafaufau i le laulau GreenPlum dev42_1_db_usl.TESTING_SUBJ_org_finval, lea o le a tatou galulue ai.

O le a matou faʻaogaina kalafi e tolu Ab Initio e galulue ai:

1) Graph Create_test_data.mp - fatuina faila faila i le HDFS ma 10 laina i 6 filo tutusa. O faʻamaumauga e faʻafuaseʻi, o lona fausaga ua faʻatulagaina mo le faʻaofiina i totonu o la matou laulau

2) Kalafi mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset – MDW gaosia kalafi e ala i le amataina o le faaofiina o faamatalaga i totonu o la tatou laulau i filo e 10 tutusa (faʻamatalaga suʻega na gaosia e le kalafi (1) faʻaaogaina)

3) Kalafi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset – o se kalafi na gaosia e le MDW mo le fa'afou fa'aopoopoina o la tatou laulau i filo e 10 tutusa e fa'aaoga ai se vaega o fa'amaumauga fou (delta) fa'atupuina e le kalafi (1)

Sei o tatou faʻataʻitaʻiina le tusitusiga i lalo ile NRT mode:

• fa'atupu 6 laina su'ega
• faia se uta muamua fa'aofi 6 laina su'ega i totonu o se laulau avanoa
• toe fai fa'aopoopo fa'aopoopo 5 taimi
  • fa'atupu 6 laina su'ega
  • faia se faʻaopoopoga faʻapipiʻi o 6 laina suʻega i totonu o le laulau (i lenei tulaga, o le valid_to_ts taimi faʻamutaina ua seti i faʻamaumauga tuai ma faʻamatalaga lata mai faʻatasi ai ma le ki autu tutusa e faʻaofiina)

O lenei faʻataʻitaʻiga e faʻataʻitaʻiina le faiga o le faʻaogaina moni o se pisinisi faʻapitoa - o se vaega tele o faʻamatalaga fou e aliali mai i le taimi moni ma vave ona sasaa atu i GreenPlum.

Se'i o tatou va'ai i le fa'amaumauga a le tusitusiga:

Amata Create_test_data.input.pset i le 2020-06-04 11:49:11
Faauma Create_test_data.input.pset i le 2020-06-04 11:49:37
Amata mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:49:37
Faauma mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:50:42
Amata Create_test_data.input.pset i le 2020-06-04 11:50:42
Faauma Create_test_data.input.pset i le 2020-06-04 11:51:06
Amata mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:51:06
Faauma mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:53:41
Amata Create_test_data.input.pset i le 2020-06-04 11:53:41
Faauma Create_test_data.input.pset i le 2020-06-04 11:54:04
Amata mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:54:04
Faauma mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:56:51
Amata Create_test_data.input.pset i le 2020-06-04 11:56:51
Faauma Create_test_data.input.pset i le 2020-06-04 11:57:14
Amata mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:57:14
Faauma mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 11:59:55
Amata Create_test_data.input.pset i le 2020-06-04 11:59:55
Faauma Create_test_data.input.pset i le 2020-06-04 12:00:23
Amata mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 12:00:23
Faauma mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 12:03:23
Amata Create_test_data.input.pset i le 2020-06-04 12:03:23
Faauma Create_test_data.input.pset i le 2020-06-04 12:03:49
Amata mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 12:03:49
Faauma mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset i le 2020-06-04 12:06:46

Ua aliali mai le ata lea:

Ata
Amata le taimi
Faauma le taimi
umi

Create_test_data.input.pset
04.06.2020 11: 49: 11
04.06.2020 11: 49: 37
00:00:26

mdw_load.day_one.current.
dev42_1_db_usl_testing_subj_org_finval.pset
04.06.2020 11: 49: 37
04.06.2020 11: 50: 42
00:01:05

Create_test_data.input.pset
04.06.2020 11: 50: 42
04.06.2020 11: 51: 06
00:00:24

mdw_load.regular.current.
dev42_1_db_usl_testing_subj_org_finval.pset
04.06.2020 11: 51: 06
04.06.2020 11: 53: 41
00:02:35

Create_test_data.input.pset
04.06.2020 11: 53: 41
04.06.2020 11: 54: 04
00:00:23

mdw_load.regular.current.
dev42_1_db_usl_testing_subj_org_finval.pset
04.06.2020 11: 54: 04
04.06.2020 11: 56: 51
00:02:47

Create_test_data.input.pset
04.06.2020 11: 56: 51
04.06.2020 11: 57: 14
00:00:23

mdw_load.regular.current.
dev42_1_db_usl_testing_subj_org_finval.pset
04.06.2020 11: 57: 14
04.06.2020 11: 59: 55
00:02:41

Create_test_data.input.pset
04.06.2020 11: 59: 55
04.06.2020 12: 00: 23
00:00:28

mdw_load.regular.current.
dev42_1_db_usl_testing_subj_org_finval.pset
04.06.2020 12: 00: 23
04.06.2020 12: 03: 23
00:03:00

Create_test_data.input.pset
04.06.2020 12: 03: 23
04.06.2020 12: 03: 49
00:00:26

mdw_load.regular.current.
dev42_1_db_usl_testing_subj_org_finval.pset
04.06.2020 12: 03: 49
04.06.2020 12: 06: 46
00:02:57

Matou te vaʻaia e 6 laina faʻaopoopo o loʻo faʻatautaia i le 000 minute, lea e fai si vave.
O faʻamaumauga i le laulau faʻamoemoe na faʻaalia e tufatufaina e pei ona taua i lalo:

select valid_from_ts, valid_to_ts, count(1), min(sk), max(sk) from dev42_1_db_usl.TESTING_SUBJ_org_finval group by valid_from_ts, valid_to_ts order by 1,2;

E mafai ona e vaʻaia le fesoʻotaʻiga o faʻamaumauga tuʻuina i taimi na faʻalauiloa ai kalafi.
O lona uiga e mafai ona e fa'asolo le fa'atuputeleina o le utaina o fa'amaumauga i totonu o le GreenPlum i Ab Initio fa'atasi ma le maualuga tele o taimi ma mata'ituina le saoasaoa maualuga o le fa'aofiina o nei fa'amatalaga i totonu o le GreenPlum. Ioe, o le a le mafai ona faʻalauiloa tasi i le lona lua, talu ai o Ab Initio, pei o soʻo se meafaigaluega ETL, e manaʻomia le taimi e "amata ai" pe a faʻalauiloa.

iʻuga

O lo'o fa'aoga nei le Ab Initio i le Sberbank e fausia ai se Unified Semantic Data Layer (ESS). O lenei poloketi e aofia ai le fausiaina o se faʻasalalauga tuʻufaʻatasia o le setete o faʻalapotopotoga faʻapisinisi eseese. O faʻamatalaga e sau mai punaoa eseese, o faʻataʻitaʻiga o loʻo saunia i Hadoop. Faʻavae i luga o manaʻoga pisinisi, ua saunia se faʻataʻitaʻiga faʻamaumauga ma faʻamatalaina suiga o faʻamatalaga. O le Ab Initio e utaina faʻamatalaga i totonu o le ESN ma o faʻamaumauga na sii mai e le gata o le fiafia i le pisinisi lava ia, ae o loʻo avea foi ma punaoa mo le fausiaina o faʻatauga faʻamatalaga. I le taimi lava e tasi, o le gaioiga o le oloa e mafai ai ona e faʻaogaina faiga eseese e pei o se tagata e taliaina (Hive, Greenplum, Teradata, Oracle), lea e mafai ai ona faigofie ona saunia faʻamatalaga mo se pisinisi i ituaiga eseese e manaʻomia.

O agava'a a Ab Initio e lautele mo se fa'ata'ita'iga, o le auivi MDW e aofia ai e mafai ai ona fausia fa'amatalaga fa'atekinisi ma pisinisi fa'asolopito mai le pusa. Mo tagata atiaʻe, Ab Initio e mafai ai ona le toe faʻafouina le uili, ae faʻaaoga le tele o vaega o loʻo i ai nei, e manaʻomia tele faletusi pe a galue ma faʻamaumauga.

O le tusitala o se tagata poto faapitoa i le sosaiete faʻapolofesa a Sberbank SberProfi DWH/BigData. O le SberProfi DWH/BigData faʻalapotopotoga faʻapitoa e nafa ma le atinaʻeina o agavaa i vaega e pei o le Hadoop ecosystem, Teradata, Oracle DB, GreenPlum, faʻapea foʻi ma meafaigaluega BI Qlik, SAP BO, Tableau, ma isi.

puna: www.habr.com