{"id":87609,"date":"2020-07-09T01:42:11","date_gmt":"2020-07-08T23:42:11","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum"},"modified":"2020-07-09T01:42:11","modified_gmt":"2020-07-08T23:42:11","slug":"kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","status":"publish","type":"post","link":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","title":{"rendered":"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Ned\u00e1vno jsme \u010delili ot\u00e1zce v\u00fdb\u011bru ETL n\u00e1stroje pro pr\u00e1ci s BigData. P\u0159edchoz\u00ed \u0159e\u0161en\u00ed Informatica BDM n\u00e1m nevyhovovalo kv\u016fli omezen\u00e9 funk\u010dnosti. Jeho pou\u017eit\u00ed se z\u00fa\u017eilo na r\u00e1mec pro spou\u0161t\u011bn\u00ed p\u0159\u00edkaz\u016f spark-submit. Na trhu nebylo p\u0159\u00edli\u0161 mnoho analog\u016f, schopn\u00fdch pracovat s objemem dat, se kter\u00fdmi se setk\u00e1v\u00e1me ka\u017ed\u00fd den. Nakonec jsme zvolili Ab Initio. B\u011bhem pilotn\u00edch demonstrac\u00ed produkt prok\u00e1zal velmi vysokou rychlost zpracov\u00e1n\u00ed dat. Informac\u00ed o Ab Initio v ru\u0161tin\u011b je t\u00e9m\u011b\u0159 nedostatek, a proto jsme se rozhodli pod\u011blit se o na\u0161e zku\u0161enosti na Habru.<\/p>\n<p>Ab Initio obsahuje \u0159adu klasick\u00fdch a neobvykl\u00fdch transformac\u00ed, jejich\u017e k\u00f3d m\u016f\u017ee b\u00fdt roz\u0161\u00ed\u0159en pomoc\u00ed vlastn\u00edho jazyka PDL. Pro mal\u00e9 podniky m\u016f\u017ee tento mocn\u00fd n\u00e1stroj b\u00fdt pravd\u011bpodobn\u011b nadbyte\u010dn\u00fd, a v\u011bt\u0161ina jeho funkc\u00ed se m\u016f\u017ee zd\u00e1t drah\u00e1 a nevyu\u017eit\u00e1. Ale pokud se va\u0161e rozsahy bl\u00ed\u017e\u00ed t\u011bm z Sberbank, pak by v\u00e1s Ab Initio mohl zaj\u00edmat. <\/p>\n<p>Pom\u00e1h\u00e1 podnik\u016fm glob\u00e1ln\u011b shroma\u017e\u010fovat znalosti a rozv\u00edjet ekosyst\u00e9m, a v\u00fdvoj\u00e1\u0159\u016fm umo\u017e\u0148uje zlep\u0161ovat sv\u00e9 dovednosti v ETL, zdokonalovat znalosti v shellu, poskytuje mo\u017enost osvojen\u00ed jazyka PDL, d\u00e1v\u00e1 vizu\u00e1ln\u00ed obraz na\u010d\u00edtac\u00edch proces\u016f a usnad\u0148uje v\u00fdvoj d\u00edky bohatstv\u00ed funk\u010dn\u00edch komponent\u016f.<\/p>\n<p>V p\u0159\u00edsp\u011bvku se pod\u011bl\u00edm o mo\u017enosti Ab Initio a uvedu srovn\u00e1vac\u00ed charakteristiky jeho pr\u00e1ce s Hive a GreenPlum.<\/p>\n<ul>\n<li>Popis r\u00e1mce MDW a pr\u00e1ce na jeho dolad\u011bn\u00ed pro GreenPlum<\/li>\n<li>Srovn\u00e1vac\u00ed charakteristiky v\u00fdkonu Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum<\/li>\n<li>Pr\u00e1ce Ab Initio s GreenPlum v re\u017eimu Near Real Time<\/li>\n<\/ul>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nFunk\u010dnost tohoto produktu je velmi \u0161irok\u00e1 a vy\u017eaduje zna\u010dn\u00e9 mno\u017estv\u00ed \u010dasu na sv\u00e9 osvojov\u00e1n\u00ed. Nicm\u00e9n\u011b, p\u0159i odpov\u00eddaj\u00edc\u00edch dovednostech a spr\u00e1vn\u00fdch nastaven\u00edch v\u00fdkonu jsou v\u00fdsledky zpracov\u00e1n\u00ed dat velmi p\u016fsobiv\u00e9. Pou\u017eit\u00ed Ab Initio pro v\u00fdvoj\u00e1\u0159e m\u016f\u017ee poskytnout zaj\u00edmav\u00e9 zku\u0161enosti. Je to nov\u00fd pohled na ETL v\u00fdvoj, hybrid mezi vizu\u00e1ln\u00edm prost\u0159ed\u00edm a v\u00fdvojem na\u010d\u00edtac\u00edch skript\u016f na jazykov\u00e9 b\u00e1zi.<\/p>\n<p>Obchody rozv\u00edjej\u00ed sv\u00e9 ekosyst\u00e9my a tento n\u00e1stroj se jim hod\u00ed jako nikdy p\u0159edt\u00edm. S pomoc\u00ed Ab Initio lze shroma\u017e\u010fovat znalosti o aktu\u00e1ln\u00edm podnik\u00e1n\u00ed a vyu\u017e\u00edvat tyto poznatky k roz\u0161i\u0159ov\u00e1n\u00ed st\u00e1vaj\u00edc\u00edho a otev\u00edr\u00e1n\u00ed nov\u00fdch podnik\u016f. Alternativami k Ab Initio jsou vizu\u00e1ln\u00ed v\u00fdvojov\u00e1 prost\u0159ed\u00ed jako Informatica BDM a nevizualiza\u010dn\u00ed prost\u0159ed\u00ed \u2013 Apache Spark.<\/p>\n<h3>Popis Ab Initio<\/h3>\n<p>\nAb Initio, jako ostatn\u00ed ETL n\u00e1stroje, p\u0159edstavuje sadu produkt\u016f.<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/c467fe1e8455805075856967b577f8b7.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAb Initio GDE (Graphical Development Environment) je prost\u0159ed\u00ed pro v\u00fdvoj\u00e1\u0159e, ve kter\u00e9m nastavuje transformace dat a spojuje je do datov\u00fdch tok\u016f ve form\u011b \u0161ipek. Takov\u00e1 sada transformac\u00ed se naz\u00fdv\u00e1 graf:<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/7b0c19a82ae8b461a9bd85117555e106.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nVstupn\u00ed a v\u00fdstupn\u00ed spojen\u00ed funk\u010dn\u00edch komponent\u016f jsou porty a obsahuj\u00ed pole, kter\u00e1 jsou vypo\u010d\u00edt\u00e1na uvnit\u0159 transformac\u00ed. N\u011bkolik graf\u016f, propojen\u00fdch toky ve form\u011b \u0161ipek v po\u0159ad\u00ed jejich vykon\u00e1n\u00ed, se naz\u00fdv\u00e1 pl\u00e1n.<\/p>\n<p>Existuje n\u011bkolik stovek funk\u010dn\u00edch komponent\u016f, co\u017e je velmi mnoho. Mnoho z nich je vysoce specializovan\u00fdch. Mo\u017enosti klasick\u00fdch transformac\u00ed v Ab Initio jsou \u0161ir\u0161\u00ed ne\u017e v jin\u00fdch ETL n\u00e1stroj\u00edch. Nap\u0159\u00edklad Join m\u00e1 n\u011bkolik v\u00fdstup\u016f. Krom\u011b v\u00fdsledku spojen\u00ed datov\u00fdch sad lze na v\u00fdstupu z\u00edskat z\u00e1znamy vstupn\u00edch datov\u00fdch sad, podle kl\u00ed\u010d\u016f, podle kter\u00fdch se nepoda\u0159ilo spojit. Tak\u00e9 lze z\u00edskat rejects, errors a log pr\u00e1ce transformace, kter\u00fd lze ve stejn\u00e9m grafu p\u0159e\u010d\u00edst jako textov\u00fd soubor a zpracovat dal\u0161\u00edmi transformacemi:<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/c7d70879daad364a3d4aa41a0a1b738d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNebo nap\u0159\u00edklad lze materializovat p\u0159\u00edjemce dat ve form\u011b tabulky a ve stejn\u00e9m grafu z n\u011bj \u010d\u00edst data.<\/p>\n<p>Existuj\u00ed origin\u00e1ln\u00ed transformace. Nap\u0159\u00edklad transformace Scan m\u00e1 funk\u010dnost, jako maj\u00ed analytick\u00e9 funkce. Existuj\u00ed transformace s v\u00fdmluvn\u00fdmi n\u00e1zvy: Create Data, Read Excel, Normalize, Sort within Groups, Run Program, Run SQL, Join with DB a dal\u0161\u00ed. Grafy mohou pou\u017e\u00edvat parametry spou\u0161t\u011bn\u00ed, v\u010detn\u011b mo\u017enosti p\u0159en\u00e1\u0161et parametry z opera\u010dn\u00edho syst\u00e9mu nebo do opera\u010dn\u00edho syst\u00e9mu. Soubory s hotov\u00fdm souborem p\u0159ed\u00e1van\u00fdch parametr\u016f grafu se naz\u00fdvaj\u00ed parameter sets (psets).<\/p>\n<p>Jak se pat\u0159\u00ed, Ab Initio GDE m\u00e1 sv\u016fj repozit\u00e1\u0159, naz\u00fdvan\u00fd EME (Enterprise Meta Environment). V\u00fdvoj\u00e1\u0159i maj\u00ed mo\u017enost pracovat s lok\u00e1ln\u00edmi verzemi k\u00f3du a prov\u00e1d\u011bt check in sv\u00fdch v\u00fdvoj\u016f do centr\u00e1ln\u00edho repozit\u00e1\u0159e.<\/p>\n<p>B\u011bhem vykon\u00e1v\u00e1n\u00ed nebo po vyhotoven\u00ed grafu je mo\u017en\u00e9 kliknout na jak\u00e9koli spojen\u00ed transformace toku a zobrazit data, kter\u00e1 pro\u0161la mezi t\u011bmito transformacemi:<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/a674507b2cd25b980cb862bda2e10cbe.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJe tak\u00e9 mo\u017en\u00e9 kliknout na jak\u00fdkoli tok a zobrazit podrobnosti sledov\u00e1n\u00ed \u2013 kolik paralel bylo pou\u017eito p\u0159i transformaci, kolik \u0159\u00e1dk\u016f a bajt\u016f se na\u010detlo v kter\u00e9 paraleln\u00ed v\u011btvi:<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/6188d17bebe756ee59e6956f9a40ec43.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nExistuje mo\u017enost rozd\u011blit vykon\u00e1n\u00ed grafu do f\u00e1z\u00ed a ozna\u010dit, kter\u00e9 transformace se maj\u00ed vykon\u00e1vat nejd\u0159\u00edve (v nult\u00e9 f\u00e1zi), n\u00e1sleduj\u00edc\u00ed v prvn\u00ed f\u00e1zi, dal\u0161\u00ed ve druh\u00e9 f\u00e1zi atd.<\/p>\n<p>U ka\u017ed\u00e9 transformace si m\u016f\u017eete vybrat takzvan\u00fd layout (kde se provede): bez paralel nebo v paraleln\u00edch toc\u00edch, jejich\u017e po\u010det m\u016f\u017eete ur\u010dit. P\u0159itom do\u010dasn\u00e9 soubory, kter\u00e9 Ab Initio vytv\u00e1\u0159\u00ed p\u0159i pr\u00e1ci s transformacemi, lze um\u00edstit jak do souborov\u00e9ho syst\u00e9mu <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/cs\/server\/dts-los-angeles\/\"   title=\"serveru\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"3635\">serveru<\/a>, tak do HDFS.<\/p>\n<p>V ka\u017ed\u00e9 transformaci na z\u00e1klad\u011b \u0161ablony lze defaultn\u011b vytvo\u0159it vlastn\u00ed skript v jazyce PDL, kter\u00fd pon\u011bkud p\u0159ipom\u00edn\u00e1 shell. <\/p>\n<p>Pomoc\u00ed jazyka PDL m\u016f\u017eete roz\u0161i\u0159ovat funk\u010dnost transformac\u00ed a zejm\u00e9na m\u016f\u017eete dynamicky (b\u011bhem vykon\u00e1v\u00e1n\u00ed) generovat libovoln\u00e9 fragmenty k\u00f3du v z\u00e1vislosti na parametrech vykon\u00e1n\u00ed.<\/p>\n<p>Tak\u00e9 v Ab Initio je siln\u011b rozvinuta integrace s opera\u010dn\u00edmi syst\u00e9my prost\u0159ednictv\u00edm shellu. Konkr\u00e9tn\u011b v Sberbank se pou\u017e\u00edv\u00e1 linux ksh. M\u016f\u017eete si vym\u011b\u0148ovat s shell prom\u011bnn\u00e9 a pou\u017e\u00edvat je jako parametry graf\u016f. M\u016f\u017eete p\u0159es shell spou\u0161t\u011bt vykon\u00e1n\u00ed graf\u016f Ab Initio a spravovat Ab Initio.<\/p>\n<p>Krom\u011b Ab Initio GDE se dod\u00e1v\u00e1 mnoho dal\u0161\u00edch produkt\u016f. Existuje vlastn\u00ed Co&gt;Operation System, kter\u00e1 si klade za c\u00edl b\u00fdt opera\u010dn\u00edm syst\u00e9mem. Existuje Control&gt;Center, ve kter\u00e9m lze napl\u00e1novat a monitorovat toky na\u010d\u00edt\u00e1n\u00ed. Existuj\u00ed produkty pro v\u00fdvoj na primitivn\u011bj\u0161\u00ed \u00farovni, ne\u017e jakou umo\u017e\u0148uje Ab Initio GDE.<\/p>\n<h3>Popis r\u00e1mce MDW a pr\u00e1ce na jeho dolad\u011bn\u00ed pro GreenPlum<\/h3>\n<p>\nSpole\u010dn\u011b se sv\u00fdmi produkty dod\u00e1v\u00e1 dodavatel i produkt MDW (Metadata Driven Warehouse), kter\u00fd p\u0159edstavuje konfigur\u00e1tor graf\u016f ur\u010den\u00fd k podpo\u0159e typick\u00fdch \u00faloh pro napl\u0148ov\u00e1n\u00ed datov\u00fdch sklad\u016f nebo data vaults.<\/p>\n<p>Obsahuje u\u017eivatelsk\u00e9 (specifick\u00e9 pro projekt) parsery metadat a hotov\u00e9 gener\u00e1tory k\u00f3du \u201ez krabice\u201c.<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/0842ecb9550f369875103ccb23cdf121.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nNa vstupu MDW obdr\u017e\u00ed model dat, konfigura\u010dn\u00ed soubor pro nastaven\u00ed p\u0159ipojen\u00ed k datab\u00e1zi (Oracle, Teradata nebo Hive) a n\u011bkter\u00e1 dal\u0161\u00ed nastaven\u00ed. Specifick\u00e1 \u010d\u00e1st pro projekt nap\u0159\u00edklad nasazuje model do datab\u00e1ze. Standardn\u00ed \u010d\u00e1st produktu generuje grafy a konfigura\u010dn\u00ed soubory pro n\u011b na z\u00e1klad\u011b na\u010d\u00edt\u00e1n\u00ed dat do tabulek modelu. P\u0159i tom jsou vytv\u00e1\u0159eny grafy (a psets) pro n\u011bkolik re\u017eim\u016f inicia\u010dn\u00ed a inkrement\u00e1ln\u00ed pr\u00e1ce na aktualizaci entit.<\/p>\n<p>V p\u0159\u00edpadech Hive a RDBMS se generuj\u00ed r\u016fznorod\u00e9 grafy pro inicia\u010dn\u00ed a inkrement\u00e1ln\u00ed aktualizaci dat.<\/p>\n<p>V p\u0159\u00edpad\u011b Hive jsou p\u0159ijat\u00e1 data delta spojena prost\u0159ednictv\u00edm Ab Initio Join s daty, kter\u00e1 byla v tabulce p\u0159ed aktualizac\u00ed. Na\u010d\u00edtac\u00ed aplikace v MDW (jak v Hive, tak v RDBMS) nejen vkl\u00e1daj\u00ed nov\u00e1 data z delty, ale tak\u00e9 uzav\u00edraj\u00ed obdob\u00ed platnosti dat podle prim\u00e1rn\u00edch kl\u00ed\u010d\u016f, ke kter\u00fdm byla delta p\u0159ijata. Krom\u011b toho je nutn\u00e9 znovu p\u0159epsat nezm\u011bn\u011bnou \u010d\u00e1st dat. Ale to je nezbytn\u00e9, proto\u017ee v Hive neexistuj\u00ed operace delete nebo update.<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/36b6dd7c4d45727ad7d7aa1183fdd5fd.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nV p\u0159\u00edpad\u011b RDBMS vypadaj\u00ed grafy pro inkrement\u00e1ln\u00ed aktualizaci dat optimalizovan\u011bji, proto\u017ee RDBMS maj\u00ed skute\u010dn\u00e9 mo\u017enosti aktualizace.<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/b62ac222bed575a60b632b13e012f603.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nP\u0159ijat\u00e1 delta je na\u010dtena do pomocn\u00e9 tabulky v datab\u00e1zi. Pot\u00e9 doch\u00e1z\u00ed ke spojen\u00ed delty s daty, kter\u00e1 byla v tabulce p\u0159ed aktualizac\u00ed. Tohoto se dosahuje pomoc\u00ed SQL prost\u0159ednictv\u00edm vygenerovan\u00e9ho SQL dotazu. D\u00e1le pomoc\u00ed SQL p\u0159\u00edkaz\u016f delete+insert doch\u00e1z\u00ed k vlo\u017een\u00ed nov\u00fdch dat z delty do c\u00edlov\u00e9 tabulky a uzav\u0159en\u00ed obdob\u00ed platnosti dat podle prim\u00e1rn\u00edch kl\u00ed\u010d\u016f, ke kter\u00fdm byla delta p\u0159ijata. <br \/>\nNen\u00ed nutn\u00e9 p\u0159episovat nezm\u011bn\u011bn\u00e1 data.<\/p>\n<p>Takto jsme dosp\u011bli k z\u00e1v\u011bru, \u017ee v p\u0159\u00edpad\u011b Hive mus\u00ed MDW p\u0159istoupit k p\u0159eps\u00e1n\u00ed cel\u00e9 tabulky, proto\u017ee Hive nem\u00e1 funkci aktualizace. A nebylo vymysleno nic lep\u0161\u00edho ne\u017e \u00fapln\u00e9 p\u0159eps\u00e1n\u00ed dat p\u0159i aktualizaci. V p\u0159\u00edpad\u011b RDBMS v\u0161ak tv\u016frci produktu pova\u017eovali za pot\u0159ebn\u00e9 sv\u011b\u0159it spojen\u00ed a aktualizaci tabulek pou\u017e\u00edv\u00e1n\u00ed SQL.<\/p>\n<p>Pro projekt ve Sberbanku jsme vytvo\u0159ili novou opakovan\u011b pou\u017eitelnou implementaci datab\u00e1zov\u00e9ho loaderu pro GreenPlum. To bylo provedeno na z\u00e1klad\u011b verze, kterou MDW generuje pro Teradata. Pr\u00e1v\u011b Teradata, a ne Oracle, byla pro toto lep\u0161\u00ed a nejbl\u00ed\u017ee, proto\u017ee je tak\u00e9 MPP syst\u00e9mem. Zp\u016fsoby pr\u00e1ce a tak\u00e9 syntaxe Teradata a GreenPlum se uk\u00e1zaly jako podobn\u00e9.<\/p>\n<p>P\u0159\u00edklady kritick\u00fdch rozd\u00edl\u016f mezi r\u016fzn\u00fdmi RDBMS pro MDW jsou n\u00e1sleduj\u00edc\u00ed. V GreenPlum, na rozd\u00edl od Teradata, p\u0159i vytv\u00e1\u0159en\u00ed tabulek je t\u0159eba napsat klauzuli<\/p>\n<pre><code class=\"sql\">distributed by<\/code><\/pre>\n<p>\nV Teradata se p\u00ed\u0161e<\/p>\n<pre><code class=\"sql\">odstranit &lt;table&gt; all<\/code><\/pre>\n<p>\n, a v GreenPlum se p\u00ed\u0161e<\/p>\n<pre><code class=\"sql\">smazat z &lt;table&gt;<\/code><\/pre>\n<p>\nV Oracle se z d\u016fvod\u016f optimalizace p\u00ed\u0161e<\/p>\n<pre><code class=\"sql\">delete from t where rowid in (<p>)<\/code><\/pre>\n<p>\n, a v Teradata a GreenPlum se p\u00ed\u0161e<\/p>\n<pre><code class=\"sql\">delete from t where exists (select * from delta where delta.pk=t.pk)<\/code><\/pre>\n<p>\nJe\u0161t\u011b bychom m\u011bli poznamenat, \u017ee pro pr\u00e1ci Ab Initio s GreenPlum bylo nutn\u00e9 nainstalovat klienta GreenPlum na v\u0161echny uzly clusteru Ab Initio. To je proto, \u017ee jsme se p\u0159ipojili k GreenPlum sou\u010dasn\u011b ze v\u0161ech uzl\u016f na\u0161eho clusteru. A aby \u010dten\u00ed z GreenPlum bylo paraleln\u00ed a ka\u017ed\u00fd paraleln\u00ed tok Ab Initio \u010detl svou \u010d\u00e1st dat z GreenPlum, bylo t\u0159eba do sekce \u201ewhere\u201c SQL dotaz\u016f um\u00edstit pro Ab Initio srozumiteln\u00e9 konstrukce<\/p>\n<pre><code class=\"sql\">where ABLOCAL()<\/code><\/pre>\n<p>\na definovat hodnotu t\u00e9to konstrukce, p\u0159i\u010dem\u017e transformaci \u010dtouc\u00ed z DB se zad\u00e1v\u00e1 parametr<\/p>\n<pre><code class=\"sql\">ablocal_expr=\u00abstring_concat(&quot;mod(t.&quot;, string_filter_out(&quot;{$TABLE_KEY}&quot;,&quot;{}&quot;), &quot;,&quot;, (decimal(3))(number_of_partitions()),&quot;)=&quot;, (decimal(3))(this_partition()))\u00bb<\/code><\/pre>\n<p>\n, kter\u00e1 se zkompiluje na n\u011bco jako<\/p>\n<pre><code class=\"sql\">mod(sk,10)=3<\/code><\/pre>\n<p>\n, tedy je t\u0159eba poskytnout GreenPlum explicitn\u00ed filtr pro ka\u017edou partition. Pro jin\u00e9 datab\u00e1ze (Teradata, Oracle) m\u016f\u017ee Ab Initio tuto paralelizaci prov\u00e9st automaticky.<\/p>\n<h3>Srovn\u00e1vac\u00ed charakteristiky v\u00fdkonu Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum<\/h3>\n<p>\nVe Sberbank byl proveden experiment porovn\u00e1n\u00ed v\u00fdkonu generovan\u00fdch MDW graf\u016f aplikovateln\u00fdch na Hive a GreenPlum. V r\u00e1mci experimentu m\u011bl Hive 5 uzl\u016f na tom sam\u00e9m clusteru jako Ab Initio, zat\u00edmco GreenPlum m\u011bl 4 uzly na samostatn\u00e9m clusteru. Tedy Hive m\u011bl ur\u010ditou v\u00fdhodu nad GreenPlum \u201ez hlediska hardwaru\u201c.<\/p>\n<p>Byly zv\u00e1\u017eeny dva p\u00e1ry graf\u016f, kter\u00e9 vykon\u00e1valy stejn\u00fd \u00fakol aktualizace dat v Hive a GreenPlum. Spou\u0161t\u011bly se grafy generovan\u00e9 konfigur\u00e1torem MDW:<\/p>\n<ul>\n<li>inicializa\u010dn\u00ed na\u010d\u00edt\u00e1n\u00ed + inkrement\u00e1ln\u00ed na\u010d\u00edt\u00e1n\u00ed n\u00e1hodn\u011b generovan\u00fdch dat do tabulky Hive<\/li>\n<li>inicializa\u010dn\u00ed na\u010d\u00edt\u00e1n\u00ed + inkrement\u00e1ln\u00ed na\u010d\u00edt\u00e1n\u00ed n\u00e1hodn\u011b generovan\u00fdch dat do stejn\u00e9 tabulky GreenPlum<\/li>\n<\/ul>\n<p>\nVe v\u0161ech p\u0159\u00edpadech (Hive a GreenPlum) byly spu\u0161t\u011bny na\u010d\u00edt\u00e1n\u00ed ve 10 paraleln\u00edch vl\u00e1knech na stejn\u00e9 klastru Ab Initio. Meziprodukty pro v\u00fdpo\u010dty byly ulo\u017eeny v HDFS (v term\u00ednech Ab Initio byl pou\u017eit MFS layout using HDFS). Jedna \u0159\u00e1dka n\u00e1hodn\u011b generovan\u00fdch dat zab\u00edrala v obou p\u0159\u00edpadech 200 byt\u016f.<\/p>\n<p>V\u00fdsledek byl n\u00e1sleduj\u00edc\u00ed:<\/p>\n<p><b>Hive:<\/b><\/p>\n<p><b>Inicia\u010dn\u00ed na\u010d\u00edt\u00e1n\u00ed v Hive<\/b><\/p>\n<p>Vlo\u017eeno \u0159\u00e1dk\u016f<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>D\u00e9lka inicia\u010dn\u00edho<br \/>\nna\u010d\u00edt\u00e1n\u00ed v sekund\u00e1ch<br \/>\n41<br \/>\n203<br \/>\n1 601<\/p>\n<p><b>Inkrement\u00e1ln\u00ed na\u010d\u00edt\u00e1n\u00ed v Hive<\/b><\/p>\n<p>Po\u010det \u0159\u00e1dk\u016f, kter\u00e9 byly v<br \/>\nc\u00edlov\u00e9 tabulce na za\u010d\u00e1tku experimentu<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Po\u010det \u0159\u00e1dk\u016f delta, kter\u00e9 byly pou\u017eity v<br \/>\nc\u00edlov\u00e9 tabulce b\u011bhem experimentu<br \/>\n6 000 000<br \/>\n6 000 000<br \/>\n6 000 000<\/p>\n<p>D\u00e9lka inkrement\u00e1ln\u00edho<br \/>\nna\u010d\u00edt\u00e1n\u00ed v sekund\u00e1ch<br \/>\n88<br \/>\n299<br \/>\n<b>2 541<\/b><\/p>\n<p>\n<b>GreenPlum:<\/b><\/p>\n<p><b>Inicia\u010dn\u00ed na\u010d\u00edt\u00e1n\u00ed v GreenPlum<\/b><\/p>\n<p>Vlo\u017eeno \u0159\u00e1dk\u016f<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>D\u00e9lka inicia\u010dn\u00edho<br \/>\nna\u010d\u00edt\u00e1n\u00ed v sekund\u00e1ch<br \/>\n72<br \/>\n360<br \/>\n3 631<\/p>\n<p><b>Inkrement\u00e1ln\u00ed na\u010d\u00edt\u00e1n\u00ed v GreenPlum<\/b><\/p>\n<p>Po\u010det \u0159\u00e1dk\u016f, kter\u00e9 byly v<br \/>\nc\u00edlov\u00e9 tabulce na za\u010d\u00e1tku experimentu<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Po\u010det \u0159\u00e1dk\u016f delta, kter\u00e9 byly pou\u017eity v<br \/>\nc\u00edlov\u00e9 tabulce b\u011bhem experimentu<br \/>\n6 000 000<br \/>\n6 000 000<br \/>\n6 000 000<\/p>\n<p>D\u00e9lka inkrement\u00e1ln\u00edho<br \/>\nna\u010d\u00edt\u00e1n\u00ed v sekund\u00e1ch<br \/>\n159<br \/>\n199<br \/>\n<b>321<\/b><\/p>\n<p>\nVid\u00edme, \u017ee rychlost inicia\u010dn\u00edho na\u010d\u00edt\u00e1n\u00ed jak v Hive, tak v GreenPlum line\u00e1rn\u011b z\u00e1vis\u00ed na objemu dat, a z d\u016fvod\u016f lep\u0161\u00edho hardwaru je m\u00edrn\u011b rychlej\u0161\u00ed pro Hive ne\u017e pro GreenPlum.<\/p>\n<p>Inkrement\u00e1ln\u00ed na\u010d\u00edt\u00e1n\u00ed v Hive tak\u00e9 line\u00e1rn\u011b z\u00e1vis\u00ed na objemu p\u0159edem nahran\u00fdch dat v c\u00edlov\u00e9 tabulce a prob\u00edh\u00e1 pom\u011brn\u011b pomalu se zvy\u0161uj\u00edc\u00edmi se objemy. To je zp\u016fsobeno nutnost\u00ed kompletn\u011b p\u0159eepisovat c\u00edlovou tabulku. To znamen\u00e1, \u017ee uplatn\u011bn\u00ed mal\u00fdch zm\u011bn na obrovsk\u00fdch tabulk\u00e1ch nen\u00ed pro Hive p\u0159\u00edli\u0161 dobr\u00e1 praxe.<\/p>\n<p>Inkrement\u00e1ln\u00ed na\u010d\u00edt\u00e1n\u00ed v GreenPlum m\u00e1 slabou z\u00e1vislost na objemu p\u0159edem nahran\u00fdch dat v c\u00edlov\u00e9 tabulce a prob\u00edh\u00e1 dostate\u010dn\u011b rychle. K tomu do\u0161lo d\u00edky SQL Joins a architektu\u0159e GreenPlum, kter\u00e1 umo\u017e\u0148uje operaci delete.<\/p>\n<p>Tak\u017ee GreenPlum vkl\u00e1d\u00e1 delta metodou delete+insert, a v Hive nejsou operace delete ani update, a proto byl cel\u00fd datov\u00fd blok p\u0159i inkrement\u00e1ln\u00ed aktualizaci nucen b\u00fdt p\u0159eps\u00e1n. Nejv\u00fdznamn\u011bj\u0161\u00ed je srovn\u00e1n\u00ed tu\u010dn\u011b vyzna\u010den\u00fdch bun\u011bk, proto\u017ee odpov\u00eddaj\u00ed nej\u010dast\u011bj\u0161\u00edmu zp\u016fsobu vyu\u017eit\u00ed n\u00e1ro\u010dn\u00fdch na\u010d\u00edt\u00e1n\u00ed. Vid\u00edme, \u017ee GreenPlum v tomto testu vyhr\u00e1l nad Hive 8x.<\/p>\n<h3>Pr\u00e1ce Ab Initio s GreenPlum v re\u017eimu Near Real Time<\/h3>\n<p>\nV tomto experimentu ov\u011b\u0159\u00edme, zda Ab Initio m\u016f\u017ee prov\u00e1d\u011bt aktualizaci tabulky GreenPlum n\u00e1hodn\u011b generovan\u00fdmi d\u00e1vkami dat v re\u017eimu bl\u00edzk\u00e9m k re\u00e1ln\u00e9mu \u010dasu. Pod\u00edv\u00e1me se na tabulku GreenPlum dev42_1_db_usl.TESTING_SUBJ_org_finval, se kterou se bude pracovat.<\/p>\n<p>Budeme pou\u017e\u00edvat t\u0159i grafy Ab Initio k pr\u00e1ci s n\u00ed:<\/p>\n<p>1) Graf Create_test_data.mp \u2013 vytv\u00e1\u0159\u00ed v 10 paraleln\u00edch vl\u00e1knech soubory s daty v HDFS s 6 000 000 \u0159\u00e1dky. Data jsou n\u00e1hodn\u00e1, jejich struktura je organizov\u00e1na pro vlo\u017een\u00ed do na\u0161\u00ed tabulky.<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/8cac605c1c5931313bd016a48fb6384e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/b86988bf1722329bde60013ba025d600.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n2) Graf mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset \u2013 generovan\u00fd MDW graf pro inicializa\u010dn\u00ed vlo\u017een\u00ed dat do na\u0161\u00ed tabulky v 10 paraleln\u00edch vl\u00e1knech (pou\u017e\u00edvaj\u00ed se testovac\u00ed data, generovan\u00e1 grafem (1)).<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/b4228a13c088c6b96abba35e9e58fe8e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n3) Graf mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset \u2013 generovan\u00fd MDW graf pro inkrement\u00e1ln\u00ed aktualizaci na\u0161\u00ed tabulky v 10 paraleln\u00edch vl\u00e1knech s pou\u017eit\u00edm porce \u010derstv\u00fdch p\u0159ich\u00e1zej\u00edc\u00edch dat (delta), generovan\u00fdch grafem (1).<\/p>\n<p><img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/714a901616cf84e5a1fd7a764373e30f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSpust\u00edme n\u00ed\u017ee uveden\u00fd sc\u00e9n\u00e1\u0159 v re\u017eimu NRT:<\/p>\n<ul>\n<li>vygenerovat 6 000 000 testovac\u00edch \u0159\u00e1dk\u016f.<\/li>\n<li>prov\u00e9st inicializa\u010dn\u00ed na\u010dten\u00ed vlo\u017een\u00edm 6 000 000 testovac\u00edch \u0159\u00e1dk\u016f do pr\u00e1zdn\u00e9 tabulky.<\/li>\n<li>opakovat 5kr\u00e1t inkrement\u00e1ln\u00ed na\u010dten\u00ed.\n<ul>\n<li>vygenerovat 6 000 000 testovac\u00edch \u0159\u00e1dk\u016f.<\/li>\n<li>prov\u00e9st inkrement\u00e1ln\u00ed vlo\u017een\u00ed 6 000 000 testovac\u00edch \u0159\u00e1dk\u016f do tabulky (star\u00fdm dat\u016fm se nastav\u00ed \u010das vypr\u0161en\u00ed platnosti valid_to_ts a vkl\u00e1daj\u00ed se nov\u011bj\u0161\u00ed data se stejn\u00fdm prim\u00e1rn\u00edm kl\u00ed\u010dem).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\nTakov\u00fd sc\u00e9n\u00e1\u0159 emuluje re\u017eim skute\u010dn\u00e9 pr\u00e1ce n\u011bjak\u00e9ho podnikov\u00e9 syst\u00e9mu \u2013 v re\u00e1ln\u00e9m \u010dase se objev\u00ed dostate\u010dn\u011b objemn\u00e1 porce nov\u00fdch dat a okam\u017eit\u011b se integruje do GreenPlum.<\/p>\n<p>Nyn\u00ed se pod\u00edvejme na log pr\u00e1ce sc\u00e9n\u00e1\u0159e:<\/p>\n<p><i>Start Create_test_data.input.pset at 2020-06-04 11:49:11<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:49:37<br \/>\nStart mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:49:37<br \/>\nFinish mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:50:42<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:50:42<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:51:06<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:51:06<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:53:41<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:53:41<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:54:04<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:54:04<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:56:51<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:56:51<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:57:14<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:57:14<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:59:55<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:59:55<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 12:00:23<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 12:00:23<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 12:03:23<br \/>\nStart Create_test_data.input.pset at 2020-06-04 12:03:23<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 12:03:49<br \/>\nSpus\u0165te mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset dne 2020-06-04 12:03:49<br \/>\nDokon\u010dete mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset dne 2020-06-04 12:06:46<\/i><\/p>\n<p>Vypad\u00e1 to takto:<\/p>\n<p>Graf<br \/>\n\u010cas zah\u00e1jen\u00ed<br \/>\n\u010cas dokon\u010den\u00ed<br \/>\nD\u00e9lka<\/p>\n<p>Vytvo\u0159te_test_data.input.pset<br \/>\n04.06.2020 11:49:11<br \/>\n04.06.2020 11:49:37<br \/>\n00:00:26<\/p>\n<p>mdw_load.day_one.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:49:37<br \/>\n04.06.2020 11:50:42<br \/>\n00:01:05<\/p>\n<p>Vytvo\u0159te_test_data.input.pset<br \/>\n04.06.2020 11:50:42<br \/>\n04.06.2020 11:51:06<br \/>\n00:00:24<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:51:06<br \/>\n04.06.2020 11:53:41<br \/>\n00:02:35<\/p>\n<p>Vytvo\u0159te_test_data.input.pset<br \/>\n04.06.2020 11:53:41<br \/>\n04.06.2020 11:54:04<br \/>\n00:00:23<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:54:04<br \/>\n04.06.2020 11:56:51<br \/>\n00:02:47<\/p>\n<p>Vytvo\u0159te_test_data.input.pset<br \/>\n04.06.2020 11:56:51<br \/>\n04.06.2020 11:57:14<br \/>\n00:00:23<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:57:14<br \/>\n04.06.2020 11:59:55<br \/>\n00:02:41<\/p>\n<p>Vytvo\u0159te_test_data.input.pset<br \/>\n04.06.2020 11:59:55<br \/>\n04.06.2020 12:00:23<br \/>\n00:00:28<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 12:00:23<br \/>\n04.06.2020 12:03:23<br \/>\n00:03:00<\/p>\n<p>Vytvo\u0159te_test_data.input.pset<br \/>\n04.06.2020 12:03:23<br \/>\n04.06.2020 12:03:49<br \/>\n00:00:26<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 12:03:49<br \/>\n04.06.2020 12:06:46<br \/>\n00:02:57<\/p>\n<p>\nVid\u00edme, \u017ee 6 000 000 \u0159\u00e1dk\u016f inkrementu je zpracov\u00e1no za 3 minuty, co\u017e je dost rychl\u00e9.<br \/>\nData v c\u00edlov\u00e9 tabulce byla rozd\u011blena n\u00e1sleduj\u00edc\u00edm zp\u016fsobem:<\/p>\n<pre><code class=\"sql\">select valid_from_ts, valid_to_ts, count(1), min(sk), max(sk) from dev42_1_db_usl.TESTING_SUBJ_org_finval group by valid_from_ts, valid_to_ts order by 1,2;<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"Kdy\u017e m\u00e1te m\u011b\u0159\u00edtko jako Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/baefcfd8142f4ad6275333e3dcfccf94.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nM\u016f\u017eeme vid\u011bt odpov\u00eddaj\u00edc\u00ed vztah mezi vlo\u017een\u00fdmi daty a okam\u017eiky spu\u0161t\u011bn\u00ed graf\u016f.<br \/>\nTo znamen\u00e1, \u017ee m\u016f\u017eeme spou\u0161t\u011bt v Ab Initio inkrement\u00e1ln\u00ed nahr\u00e1v\u00e1n\u00ed dat do GreenPlum s velmi vysokou frekvenc\u00ed a sledovat vysokou rychlost vkl\u00e1d\u00e1n\u00ed t\u011bchto dat do GreenPlum. Samoz\u0159ejm\u011b, spu\u0161t\u011bn\u00ed ka\u017edou sekundu nebude mo\u017en\u00e9, proto\u017ee Ab Initio, jako ka\u017ed\u00fd ETL n\u00e1stroj, vy\u017eaduje p\u0159i spu\u0161t\u011bn\u00ed \u010das na \"rozb\u011bh\".<\/p>\n<h2>Z\u00e1v\u011br<\/h2>\n<p>\nNyn\u00ed se Ab Initio pou\u017e\u00edv\u00e1 ve Sberbank pro vytvo\u0159en\u00ed Jednotn\u00e9ho s\u00e9mantick\u00e9ho datov\u00e9ho vrstvy (ESS). Tento projekt zahrnuje vybudov\u00e1n\u00ed jedin\u00e9 verze stavu r\u016fzn\u00fdch bankovn\u00edch obchodn\u00edch entit. Informace p\u0159ich\u00e1zej\u00ed z r\u016fzn\u00fdch zdroj\u016f, jejich\u017e repliky jsou p\u0159ipravov\u00e1ny na Hadoop. Na z\u00e1klad\u011b pot\u0159eb podnik\u00e1n\u00ed se p\u0159iprav\u00ed datov\u00fd model a popisuj\u00ed se transformace dat. Ab Initio nahr\u00e1v\u00e1 informace do ESS a nahran\u00e1 data nejen\u017ee sama o sob\u011b p\u0159edstavuj\u00ed z\u00e1jem pro podnik\u00e1n\u00ed, ale tak\u00e9 slou\u017e\u00ed jako zdroj pro vytv\u00e1\u0159en\u00ed datov\u00fdch vitr\u00edn. P\u0159itom funkce produktu umo\u017e\u0148uje pou\u017e\u00edvat jako p\u0159\u00edjemce r\u016fzn\u00e9 syst\u00e9my (Hive, Greenplum, Teradata, Oracle), co\u017e umo\u017e\u0148uje bez v\u011bt\u0161\u00edch obt\u00ed\u017e\u00ed p\u0159ipravit data pro podnik\u00e1n\u00ed v r\u016fzn\u00fdch form\u00e1tech, kter\u00e9 pot\u0159ebuje.<\/p>\n<p>Mo\u017enosti Ab Initio jsou \u0161irok\u00e9, nap\u0159\u00edklad p\u0159ilo\u017een\u00fd r\u00e1mec MDW umo\u017e\u0148uje vytv\u00e1\u0159et technickou a obchodn\u00ed historii dat \u201ez krabice\u201c. Pro v\u00fdvoj\u00e1\u0159e Ab Initio umo\u017e\u0148uje \u201eneobjevovat kolo\u201c a vyu\u017e\u00edvat mno\u017estv\u00ed existuj\u00edc\u00edch funk\u010dn\u00edch komponent\u016f, kter\u00e9 v podstat\u011b p\u0159edstavuj\u00ed knihovny pot\u0159ebn\u00e9 p\u0159i pr\u00e1ci s daty.<\/p>\n<blockquote><p>Autor \u2014 expert profesion\u00e1ln\u00ed komunity Sberbank SberProfi DWH\/BigData. Profesion\u00e1ln\u00ed komunita SberProfi DWH\/BigData je odpov\u011bdn\u00e1 za rozvoj kompetenc\u00ed v oblastech jako ekosyst\u00e9m Hadoop, Teradata, Oracle DB, GreenPlum, a tak\u00e9 BI n\u00e1stroje Qlik, SAP BO, Tableau a dal\u0161\u00ed.<\/p><\/blockquote>\n<p>Zdroj: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/sberbank\/blog\/509936\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438. \u0415\u0451 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 \u0441\u0432\u0435\u043b\u043e\u0441\u044c \u043a \u0444\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a\u0443 \u043f\u043e \u0437\u0430\u043f\u0443\u0441\u043a\u0443 \u043a\u043e\u043c\u0430\u043d\u0434 spark-submit. \u041d\u0430 \u0440\u044b\u043d\u043a\u0435 \u0438\u043c\u0435\u043b\u043e\u0441\u044c \u043d\u0435 \u0442\u0430\u043a \u043c\u043d\u043e\u0433\u043e \u0430\u043d\u0430\u043b\u043e\u0433\u043e\u0432, \u0432 \u043f\u0440\u0438\u043d\u0446\u0438\u043f\u0435 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u044b\u0445 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0441 \u0442\u0435\u043c \u043e\u0431\u044a\u0451\u043c\u043e\u043c \u0434\u0430\u043d\u043d\u044b\u0445, \u0441 \u043a\u043e\u0442\u043e\u0440\u044b\u043c \u043c\u044b \u0438\u043c\u0435\u0435\u043c \u0434\u0435\u043b\u043e \u043a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":87610,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-87609","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.3 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.3\" \/>\n\t\t<meta property=\"og:locale\" content=\"cs_CZ\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043e\u0433\u0434\u0430 \u0443 \u0432\u0430\u0441 \u0441\u0431\u0435\u0440\u043e\u0432\u0441\u043a\u0438\u0435 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u044b. \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 Ab Initio \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 Hive \u0438 GreenPlum | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-08T23:42:11+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-08T23:42:11+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kdy\u017e m\u00e1te rozm\u011bry Sberbank. Pou\u017eit\u00ed Ab Initio p\u0159i pr\u00e1ci s Hive a GreenPlum | ProHoster","description":"Ned\u00e1vno jsme se dostali p\u0159ed ot\u00e1zku v\u00fdb\u011bru ETL n\u00e1stroje pro pr\u00e1ci s BigData. P\u0159edchoz\u00ed \u0159e\u0161en\u00ed Informatica BDM n\u00e1m nevyhovovalo kv\u016fli omezen\u00e9 funk\u010dnosti.","canonical_url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"cs_CZ","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u0433\u0434\u0430 \u0443 \u0432\u0430\u0441 \u0441\u0431\u0435\u0440\u043e\u0432\u0441\u043a\u0438\u0435 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u044b. \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 Ab Initio \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 Hive \u0438 GreenPlum | ProHoster","og:description":"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.","og:url":"https:\/\/prohoster.info\/cs\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-08T23:42:11+00:00","article:modified_time":"2020-07-08T23:42:11+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"87609","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 13:47:29","updated":"2026-02-22 15:29:30","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/87609","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/comments?post=87609"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/87609\/revisions"}],"predecessor-version":[{"id":162163,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/posts\/87609\/revisions\/162163"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media\/87610"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/media?parent=87609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/categories?post=87609"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/cs\/wp-json\/wp\/v2\/tags?post=87609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}