{"id":87609,"date":"2020-07-09T01:42:11","date_gmt":"2020-07-08T23:42:11","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum"},"modified":"2020-07-09T01:42:11","modified_gmt":"2020-07-08T23:42:11","slug":"kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","title":{"rendered":"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Acum ceva timp, ne-am confruntat cu \u00eentrebarea alegerii unui instrument ETL pentru lucrul cu Big Data. Solu\u021bia anterioar\u0103, Informatica BDM, nu ne-a satisf\u0103cut din cauza func\u021bionalit\u0103\u021bii sale limitate. Folosirea ei s-a redus la un cadru pentru lansarea comenzilor spark-submit. Pe pia\u021b\u0103 nu erau multe alternative capabile s\u0103 lucreze cu volumul de date cu care ne confrunt\u0103m zilnic. \u00cen cele din urm\u0103, am ales Ab Initio. \u00cen timpul demonstra\u021biilor pilot, produsul a ar\u0103tat o vitez\u0103 de procesare a datelor foarte mare. Informa\u021biile despre Ab Initio \u00een limba rom\u00e2n\u0103 sunt aproape inexistente, a\u0219a c\u0103 am decis s\u0103 \u00eemp\u0103rt\u0103\u0219im experien\u021ba noastr\u0103 pe Habr.<\/p>\n<p>Ab Initio dispune de numeroase transform\u0103ri clasice \u0219i neobi\u0219nuite, codul acestora put\u00e2nd fi extins folosind propriul limbaj PDL. Pentru micile afaceri, un instrument at\u00e2t de puternic va fi probabil excesiv, iar majoritatea func\u021bionalit\u0103\u021bilor sale ar putea fi costisitoare \u0219i nefolosite. Dar dac\u0103 dimensiunile afacerii tale se apropie de cele ale Sberbank, atunci Ab Initio \u00ee\u021bi poate fi de interes. <\/p>\n<p>Acesta ajut\u0103 afacerile s\u0103 acumuleze global cuno\u0219tin\u021be \u0219i s\u0103 dezvolte un ecosistem, oferindu-le dezvoltatorilor ocazia de a-\u0219i \u00eembun\u0103t\u0103\u021bi abilit\u0103\u021bile \u00een ETL, de a beneficia de cuno\u0219tin\u021be \u00een shell, oferind posibilitatea de a \u00eenv\u0103\u021ba limbajul PDL, oferind o imagine vizual\u0103 a proceselor de \u00eenc\u0103rcare \u0219i facilit\u00e2nd dezvoltarea datorit\u0103 abunden\u021bei componentelor func\u021bionale.<\/p>\n<p>\u00cen acest post, voi vorbi despre posibilit\u0103\u021bile Ab Initio \u0219i voi prezenta caracteristici comparative ale performan\u021bei sale \u00een lucrul cu Hive \u0219i GreenPlum.<\/p>\n<ul>\n<li>Descrierea cadrului MDW \u0219i lucr\u0103rile de ajustare a acestuia pentru GreenPlum<\/li>\n<li>Caracteristici comparative ale performan\u021bei Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum<\/li>\n<li>Utilizarea Ab Initio cu GreenPlum \u00een modul Near Real Time<\/li>\n<\/ul>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nFunc\u021bionalitatea acestui produs este foarte vast\u0103 \u0219i necesit\u0103 timp considerabil pentru a fi \u00eenv\u0103\u021bat\u0103. Cu toate acestea, cu abilit\u0103\u021bile de lucru adecvate \u0219i configura\u021bii corecte ale performan\u021bei, rezultatele proces\u0103rii datelor sunt destul de impresionante. Folosirea Ab Initio poate oferi dezvoltatorului o experien\u021b\u0103 interesant\u0103. Este o nou\u0103 viziune asupra dezvolt\u0103rii ETL, un hibrid \u00eentre un mediu vizual \u0219i dezvoltarea \u00eenc\u0103rc\u0103rilor \u00eentr-un limbaj similar cu scripturile.<\/p>\n<p>Afacerea \u00ee\u0219i dezvolt\u0103 ecosistemele, iar acest instrument se dovede\u0219te mai util ca niciodat\u0103. Cu ajutorul Ab Initio, se pot acumula cuno\u0219tin\u021be despre afacerea curent\u0103 \u0219i folosi aceste cuno\u0219tin\u021be pentru a extinde afacerile vechi \u0219i a deschide altele noi. Printre alternativele Ab Initio se num\u0103r\u0103 mediile de dezvoltare vizual\u0103 Informatica BDM \u0219i mediile non-vizuale \u2013 Apache Spark.<\/p>\n<h3>Descrierea Ab Initio<\/h3>\n<p>\nAb Initio, ca \u0219i alte instrumente ETL, reprezint\u0103 un set de produse.<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/c467fe1e8455805075856967b577f8b7.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAb Initio GDE (Graphical Development Environment) este mediu pentru dezvoltatori, \u00een care ace\u0219tia configureaz\u0103 transform\u0103rile de date \u0219i le conecteaz\u0103 prin fluxuri de date sub form\u0103 de s\u0103ge\u021bi. Astfel, acest set de transform\u0103ri este denumit grafic:<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/7b0c19a82ae8b461a9bd85117555e106.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nConexiunile de intrare \u0219i ie\u0219ire ale componentelor func\u021bionale sunt porturi \u0219i con\u021bin c\u00e2mpuri calculate \u00een interiorul transform\u0103rilor. Mai multe grafice conectate prin fluxuri de s\u0103ge\u021bi \u00een ordinea execu\u021biei lor se numesc plan.<\/p>\n<p>Exist\u0103 c\u00e2teva sute de componente func\u021bionale, ceea ce este foarte mult. Multe dintre ele sunt specializate. Capacit\u0103\u021bile transform\u0103rilor clasice \u00een Ab Initio sunt mai extinse dec\u00e2t \u00een alte instrumente ETL. De exemplu, Join are mai multe ie\u0219iri. Pe l\u00e2ng\u0103 rezultatul conect\u0103rii seturilor de date, se pot ob\u021bine \u0219i \u00eenregistr\u0103rile seturilor de date de intrare pentru cheile care nu au putut fi conectate. De asemenea, se pot ob\u021bine rejects, errors \u0219i jurnalul de lucru al transform\u0103rii, care poate fi citit \u00een acela\u0219i grafic ca un fi\u0219ier text \u0219i procesat prin alte transform\u0103ri:<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/c7d70879daad364a3d4aa41a0a1b738d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDe asemenea, se poate materializa receptorul de date sub form\u0103 de tabel \u0219i, \u00een acela\u0219i grafic, se pot citi datele din acesta.<\/p>\n<p>Exist\u0103 transform\u0103ri originale. De exemplu, transformarea Scan are func\u021bionalitate similar\u0103 cu func\u021biile analitice. Exist\u0103 transform\u0103ri cu denumiri sugestive: Create Data, Read Excel, Normalize, Sort within Groups, Run Program, Run SQL, Join with DB etc. Graficele pot folosi parametrii de execu\u021bie, inclusiv este posibil\u0103 transmisia parametrilor din sistemul de operare sau c\u0103tre sistemul de operare. Fi\u0219ierele cu seturile de parametrii transmise graficului se numesc parameter sets (psets).<\/p>\n<p>A\u0219a cum este firesc, Ab Initio GDE are propriul s\u0103u depozit, numit EME (Enterprise Meta Environment). Dezvoltatorii au posibilitatea de a lucra cu versiuni locale ale codului \u0219i de a face check in al dezvolt\u0103rilor lor \u00een depozitul central.<\/p>\n<p>Exist\u0103 posibilitatea, \u00een timpul sau dup\u0103 execu\u021bia graficului, s\u0103 faci clic pe orice conexiune care leag\u0103 transform\u0103rile fluxului \u0219i s\u0103 vizualizezi datele care au trecut \u00eentre aceste transform\u0103ri:<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/a674507b2cd25b980cb862bda2e10cbe.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDe asemenea, exist\u0103 posibilitatea de a face clic pe orice flux \u0219i a vizualiza detaliile de urm\u0103rire \u2013 c\u00e2te paralele au func\u021bionat \u00een transformare, c\u00e2te linii \u0219i c\u00e2te bite au fost \u00eenc\u0103rcate \u00een fiecare din paralele respective:<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/6188d17bebe756ee59e6956f9a40ec43.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nEste posibil s\u0103 \u00eemp\u0103r\u021bi execu\u021bia graficului \u00een faze \u0219i s\u0103 marchezi ce transform\u0103ri trebuie s\u0103 fie efectuate mai \u00eent\u00e2i (\u00een faza zero), urm\u0103toarele \u00een faza \u00eent\u00e2i, urm\u0103toarele \u00een faza a doua \u0219i a\u0219a mai departe.<\/p>\n<p>Fiecare transformare permite alegerea unui a\u0219a-numit layout (unde va fi executat\u0103): f\u0103r\u0103 paralele sau \u00een fluxuri paralele, num\u0103rul c\u0103rora poate fi specificat. De asemenea, fi\u0219ierele temporare create de Ab Initio \u00een timpul execu\u021biei transform\u0103rilor pot fi plasate fie \u00een sistemul de fi\u0219iere <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/ro\/server\/dts-los-angeles\/\"   title=\"server\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"3635\">server<\/a>, fie \u00een HDFS.<\/p>\n<p>\u00cen fiecare transformare, pe baza \u0219ablonului implicit, po\u021bi crea propriul script \u00een limbajul PDL, care se aseam\u0103n\u0103 oarecum cu shell-ul. <\/p>\n<p>Folosind limbajul PDL, po\u021bi extinde func\u021bionalitatea transform\u0103rilor \u0219i, \u00een special, po\u021bi genera dinamic (\u00een timpul execu\u021biei) fragmente de cod arbitrare \u00een func\u021bie de parametrii de execu\u021bie.<\/p>\n<p>De asemenea, \u00een Ab Initio integrarea cu sistemul de operare prin shell este bine dezvoltat\u0103. Concret, \u00een Sberbank se folose\u0219te linux ksh. Se pot schimba variabile cu shell-ul \u0219i le po\u021bi folosi ca parametrii pentru grafuri. Este posibil s\u0103 apelezi execu\u021bia graficelor Ab Initio din shell \u0219i s\u0103 administrezi Ab Initio.<\/p>\n<p>Pe l\u00e2ng\u0103 Ab Initio GDE, livrarea include multe alte produse. Exist\u0103 propriul Co&gt;Operation System care se pretinde a fi un sistem de operare. Exist\u0103 Control&gt;Center, unde po\u021bi programa \u0219i monitoriza fluxurile de \u00eenc\u0103rcare. Exist\u0103 produse pentru dezvoltare la un nivel mai primitiv dec\u00e2t permite Ab Initio GDE.<\/p>\n<h3>Descrierea cadrului MDW \u0219i lucr\u0103rile de ajustare a acestuia pentru GreenPlum<\/h3>\n<p>\n\u00cempreun\u0103 cu produsele sale, furnizorul ofer\u0103 produsul MDW (Metadata Driven Warehouse), care este un configurator de grafuri, destinat s\u0103 ajute \u00een sarcinile tipice de umplere a depozitelor de date sau data vaults.<\/p>\n<p>Acesta con\u021bine parseri de metadate personaliza\u021bi (specifice pentru proiect) \u0219i generatoare de cod gata f\u0103cute 'din cutie'.<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/0842ecb9550f369875103ccb23cdf121.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nLa intrare, MDW prime\u0219te un model de date, un fi\u0219ier de configura\u021bie pentru set\u0103rile de conexiune cu baza de date (Oracle, Teradata sau Hive) \u0219i alte set\u0103ri suplimentare. Partea specific\u0103 proiectului, de exemplu, desf\u0103\u0219oar\u0103 modelul \u00een baza de date. Partea standard a produsului genereaz\u0103 grafice \u0219i fi\u0219iere de configurare pentru acestea la \u00eenc\u0103rcarea datelor \u00een tabelele modelului. Astfel, graficile (\u0219i psets) sunt create pentru mai multe moduri de lucru ini\u021bial \u0219i incremental pentru actualizarea entit\u0103\u021bilor.<\/p>\n<p>\u00cen cazurile Hive \u0219i RDBMS, se genereaz\u0103 grafice diferite pentru actualizarea ini\u021bial\u0103 \u0219i incremental\u0103 a datelor.<\/p>\n<p>\u00cen cazul Hive, datele primite ale delta sunt unite prin Ab Initio Join cu datele existente \u00een tabel \u00eenainte de actualizare. \u00cenc\u0103rc\u0103toarele de date \u00een MDW (at\u00e2t \u00een Hive, c\u00e2t \u0219i \u00een RDBMS) nu doar c\u0103 introduc datele noi din delta, dar \u0219i \u00eenchid perioadele de valabilitate a datelor, \u00een func\u021bie de cheile primare asociate cu delta primit\u0103. \u00cen plus, este necesar s\u0103 se rescrie partea de date care nu s-a modificat. Asta trebuie f\u0103cut, deoarece \u00een Hive nu exist\u0103 opera\u021bii de \u0219tergere sau actualizare.<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/36b6dd7c4d45727ad7d7aa1183fdd5fd.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u00cen cazul RDBMS, graficele de actualizare incremental\u0103 a datelor arat\u0103 mai optim, deoarece RDBMS au capacit\u0103\u021bi reale de actualizare.<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/b62ac222bed575a60b632b13e012f603.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDelta primit\u0103 este \u00eenc\u0103rcat\u0103 \u00eentr-un tabel intermediar \u00een baza de date. Ulterior, delta este conectat\u0103 cu datele care existau \u00een tabel \u00eenainte de actualizare. Aceasta se realizeaz\u0103 prin intermediul SQL, utiliz\u00e2nd o interogare SQL generat\u0103. Apoi, cu ajutorul comenzilor SQL delete+insert, datele noi din delta sunt inserate \u00een tabelul \u021bint\u0103 \u0219i sunt \u00eenchise perioadele de valabilitate a datelor, \u00een func\u021bie de cheile primare asociate cu delta primit\u0103. <br \/>\nNu este necesar s\u0103 rescriem datele neschimbate.<\/p>\n<p>Astfel, ajungem la concluzia c\u0103 \u00een cazul Hive, MDW trebuie s\u0103 rescrie \u00eentreaga tabel\u0103, deoarece Hive nu are func\u021bia de actualizare. \u0218i nu exist\u0103 nimic mai bun dec\u00e2t rescrierea complet\u0103 a datelor \u00een timpul actualiz\u0103rii. \u00cen schimb, \u00een cazul RDBMS, creatorii produsului au considerat c\u0103 este necesar s\u0103 \u00eencredin\u021beze conectarea \u0219i actualizarea tabelelor utiliz\u0103rii SQL.<\/p>\n<p>Pentru proiectul din Sberbank, am creat o nou\u0103 implementare reutilizabil\u0103 a loader-ului de baze de date pentru GreenPlum. Aceasta a fost realizat\u0103 pe baza versiunii generate de MDW pentru Teradata. Teradata, \u0219i nu Oracle, s-a dovedit a fi mai potrivit\u0103, deoarece este de asemenea un sistem MPP. Modalit\u0103\u021bile de operare, precum \u0219i sintaxa Teradata \u0219i GreenPlum s-au dovedit a fi asem\u0103n\u0103toare.<\/p>\n<p>Exemplu de diferen\u021be critice pentru MDW \u00eentre diferite RDBMS sunt urm\u0103toarele. \u00cen GreenPlum, spre deosebire de Teradata, la crearea tabelelor trebuie s\u0103 scrii clauza<\/p>\n<pre><code class=\"sql\">distributed by<\/code><\/pre>\n<p>\n\u00cen Teradata scrii<\/p>\n<pre><code class=\"sql\">delete &lt;table&gt; all<\/code><\/pre>\n<p>\n, iar \u00een GreenPlum scrii<\/p>\n<pre><code class=\"sql\">\u015fterge din &lt;table&gt;<\/code><\/pre>\n<p>\n\u00cen Oracle, \u00een scopuri de optimizare se scrie<\/p>\n<pre><code class=\"sql\">delete from t where rowid in ()<\/code><\/pre>\n<p>\n, iar \u00een Teradata \u0219i GreenPlum se scrie<\/p>\n<pre><code class=\"sql\">delete from t where exists (select * from delta where delta.pk=t.pk)<\/code><\/pre>\n<p>\nDe asemenea, s\u0103 men\u021bion\u0103m c\u0103 pentru a func\u021biona Ab Initio cu GreenPlum a fost necesar s\u0103 instal\u0103m clientul GreenPlum pe toate nodurile clusterului Ab Initio. Acest lucru se datoreaz\u0103 faptului c\u0103 ne-am conectat la GreenPlum simultan de pe toate nodurile clusterului nostru. Iar pentru ca citirea din GreenPlum s\u0103 fie paralel\u0103 \u0219i fiecare fir paralel Ab Initio s\u0103 citeasc\u0103 por\u021biunea sa de date din GreenPlum, a fost necesar s\u0103 includem \u00een sec\u021biunea \u201ewhere\u201d a interog\u0103rilor SQL o construc\u021bie \u00een\u021beleas\u0103 de Ab Initio<\/p>\n<pre><code class=\"sql\">where ABLOCAL()<\/code><\/pre>\n<p>\n\u0219i s\u0103 definim valoarea acestei construc\u021bii, specific\u00e2nd \u00een parametrul de transformare care cite\u0219te din baza de date<\/p>\n<pre><code class=\"sql\">ablocal_expr=\u00abstring_concat(&quot;mod(t.&quot;, string_filter_out(&quot;{$TABLE_KEY}&quot;,&quot;{}&quot;), &quot;,&quot;, (decimal(3))(number_of_partitions()),&quot;)=&quot;, (decimal(3))(this_partition()))\u00bb<\/code><\/pre>\n<p>\n, care se compileaz\u0103 \u00eentr-un ceva de genul<\/p>\n<pre><code class=\"sql\">mod(sk,10)=3<\/code><\/pre>\n<p>\n, adic\u0103 este necesar s\u0103 specific\u0103m explicit GreenPlum filtrul pentru fiecare parti\u021bie. Pentru alte baze de date (Teradata, Oracle), Ab Initio poate realiza aceast\u0103 paralelizare automat.<\/p>\n<h3>Caracteristici comparative ale performan\u021bei Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum<\/h3>\n<p>\n\u00cen Sberbank a fost efectuat un experiment comparativ al performan\u021bei graficelor generate de MDW \u00een raport cu Hive \u0219i GreenPlum. \u00cen cadrul experimentului, \u00een cazul Hive au fost 5 noduri \u00een acela\u0219i cluster cu Ab Initio, iar \u00een cazul GreenPlum au fost 4 noduri \u00eentr-un cluster separat. Astfel, Hive a avut un anumit avantaj \u201ehardware\u201d fa\u021b\u0103 de GreenPlum.<\/p>\n<p>Au fost analizate dou\u0103 perechi de grafice care \u00eendeplinesc aceea\u0219i sarcin\u0103 de actualizare a datelor \u00een Hive \u0219i GreenPlum. \u00cen acest context s-au rulat graficele generate de configuratorul MDW:<\/p>\n<ul>\n<li>\u00eenc\u0103rcare ini\u021bial\u0103 + \u00eenc\u0103rcare incremental\u0103 a datelor generate aleatoriu \u00een tabelul Hive<\/li>\n<li>\u00eenc\u0103rcare ini\u021bial\u0103 + \u00eenc\u0103rcare incremental\u0103 a datelor generate aleatoriu \u00een un tabel similar GreenPlum<\/li>\n<\/ul>\n<p>\n\u00cen ambele cazuri (Hive \u0219i GreenPlum), au fost rulate \u00eenc\u0103rc\u0103ri \u00een 10 fluxuri paralele pe acela\u0219i cluster Ab Initio. Datele intermediare pentru calculele Ab Initio au fost stocate \u00een HDFS (\u00een termenii Ab Initio a fost folosit layout MFS folosind HDFS). O linie de date generate aleatoriu ocupa \u00een ambele cazuri c\u00e2te 200 de bi\u021bi.<\/p>\n<p>Rezultatul a fost urm\u0103torul:<\/p>\n<p><b>Hive:<\/b><\/p>\n<p><b>\u00cenc\u0103rcare ini\u021bial\u0103 \u00een Hive<\/b><\/p>\n<p>Linii inserate<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Durata \u00eenc\u0103rc\u0103rii ini\u021biale<br \/>\n\u00een secunde<br \/>\n41<br \/>\n203<br \/>\n1 601<\/p>\n<p><b>\u00cenc\u0103rcare incremental\u0103 \u00een Hive<\/b><\/p>\n<p>Num\u0103rul de linii, existente \u00een<br \/>\ntabelul \u021bint\u0103 la \u00eenceputul experimentului<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Num\u0103rul de linii delta, aplicate \u00een<br \/>\ntabelul \u021bint\u0103 pe parcursul experimentului<br \/>\n6 000 000<br \/>\n6 000 000<br \/>\n6 000 000<\/p>\n<p>Durata \u00eenc\u0103rc\u0103rii incrementale<br \/>\n\u00een secunde<br \/>\n88<br \/>\n299<br \/>\n<b>2 541<\/b><\/p>\n<p>\n<b>GreenPlum:<\/b><\/p>\n<p><b>\u00cenc\u0103rcare ini\u021bial\u0103 \u00een GreenPlum<\/b><\/p>\n<p>Linii inserate<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Durata \u00eenc\u0103rc\u0103rii ini\u021biale<br \/>\n\u00een secunde<br \/>\n72<br \/>\n360<br \/>\n3 631<\/p>\n<p><b>\u00cenc\u0103rcare incremental\u0103 \u00een GreenPlum<\/b><\/p>\n<p>Num\u0103rul de linii, existente \u00een<br \/>\ntabelul \u021bint\u0103 la \u00eenceputul experimentului<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Num\u0103rul de linii delta, aplicate \u00een<br \/>\ntabelul \u021bint\u0103 pe parcursul experimentului<br \/>\n6 000 000<br \/>\n6 000 000<br \/>\n6 000 000<\/p>\n<p>Durata \u00eenc\u0103rc\u0103rii incrementale<br \/>\n\u00een secunde<br \/>\n159<br \/>\n199<br \/>\n<b>321<\/b><\/p>\n<p>\nObserv\u0103m c\u0103 viteza \u00eenc\u0103rc\u0103rii ini\u021biale at\u00e2t \u00een Hive, c\u00e2t \u0219i \u00een GreenPlum depinde liniar de volumul de date \u0219i, din cauze legate de hardware superior, aceasta este pu\u021bin mai rapid\u0103 pentru Hive dec\u00e2t pentru GreenPlum.<\/p>\n<p>\u00cenc\u0103rcarea incremental\u0103 \u00een Hive depinde de asemenea liniar de volumul datelor din tabelul \u021bint\u0103 care au fost \u00eenc\u0103rcate anterior \u0219i se desf\u0103\u0219oar\u0103 destul de lent odat\u0103 cu cre\u0219terea volumului. Acest lucru este cauzat de necesitatea de a rescrie complet tabelul \u021bint\u0103. Asta \u00eenseamn\u0103 c\u0103 aplicarea unor modific\u0103ri mici la tabele imense nu este o variant\u0103 foarte bun\u0103 de utilizare \u00een Hive.<\/p>\n<p>\u00cenc\u0103rcarea incremental\u0103 \u00een GreenPlum depinde slab de volumul datelor deja existente \u00een tabelul \u021bint\u0103 \u0219i se desf\u0103\u0219oar\u0103 destul de rapid. Acest lucru s-a realizat datorit\u0103 SQL Joins \u0219i arhitecturii GreenPlum, care permite opera\u021bia delete.<\/p>\n<p>Astfel, GreenPlum introduce delta prin metoda delete+insert, \u00een timp ce \u00een Hive nu exist\u0103 opera\u021bii delete sau update, a\u0219a c\u0103 \u00eentregul set de date la actualizarea incremental\u0103 a fost obligat s\u0103 fie rescris complet. Cea mai reprezentativ\u0103 este compara\u021bia celulelor marcate cu bold, deoarece aceasta corespunde celei mai frecvente variante de exploatare a \u00eenc\u0103rc\u0103rilor consumatoare de resurse. Vedem c\u0103 GreenPlum a \u00eenvins Hive \u00een acest test cu 8 ori.<\/p>\n<h3>Utilizarea Ab Initio cu GreenPlum \u00een modul Near Real Time<\/h3>\n<p>\n\u00cen acest experiment vom verifica capacitatea Ab Initio de a actualiza tabelul GreenPlum cu por\u021biuni de date generate aleatoriu \u00eentr-un mod apropiat de real-time. Vom lua \u00een considerare tabelul GreenPlum dev42_1_db_usl.TESTING_SUBJ_org_finval, cu care se va lucra.<\/p>\n<p>Vom folosi trei grafuri Ab Initio pentru a lucra cu acesta:<\/p>\n<p>1) Graf Create_test_data.mp \u2013 genereaz\u0103 fi\u0219iere de date \u00een HDFS cu 6.000.000 de r\u00e2nduri folosind 10 fire paralele. Datele sunt generate aleatoriu, structura lor este organizat\u0103 pentru inserarea \u00een tabelul nostru<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/8cac605c1c5931313bd016a48fb6384e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/b86988bf1722329bde60013ba025d600.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n2) Graf mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset \u2013 grafic MDW generat pentru inser\u021bia ini\u021bial\u0103 a datelor \u00een tabelul nostru folosind 10 fire paralele (se folosesc date de test generate de graficul (1))<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/b4228a13c088c6b96abba35e9e58fe8e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n3) Graf mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset \u2013 grafic MDW generat pentru actualizarea incremental\u0103 a tabelului nostru folosind 10 fire paralele cu o por\u021bie de date proaspete (delta) generate de graficul (1)<\/p>\n<p><img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/714a901616cf84e5a1fd7a764373e30f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nVom executa scenariul de mai jos \u00een modul NRT:<\/p>\n<ul>\n<li>genereaz\u0103 6.000.000 de r\u00e2nduri de test<\/li>\n<li>efectueaz\u0103 \u00eenc\u0103rcarea ini\u021bial\u0103, inser\u00e2nd 6.000.000 de r\u00e2nduri de test \u00een tabelul gol<\/li>\n<li>repet\u0103 de 5 ori \u00eenc\u0103rcarea incremental\u0103\n<ul>\n<li>genereaz\u0103 6.000.000 de r\u00e2nduri de test<\/li>\n<li>efectueaz\u0103 inser\u021bia incremental\u0103 a 6.000.000 de r\u00e2nduri de test \u00een tabel (datele vechi primesc un timp de expirare valid_to_ts \u0219i se insereaz\u0103 date mai recente cu aceea\u0219i cheie primar\u0103)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\nAcest scenariu emuleaz\u0103 modul de func\u021bionare real al unui sistem de afaceri \u2013 o por\u021bie considerabil\u0103 de date noi apare \u00een timp real \u0219i este imediat \u00eencorporat\u0103 \u00een GreenPlum.<\/p>\n<p>Acum s\u0103 vedem log-ul lucrului din scenariu:<\/p>\n<p><i>Start Create_test_data.input.pset at 2020-06-04 11:49:11<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:49:37<br \/>\nStart mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:49:37<br \/>\nFinish mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:50:42<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:50:42<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:51:06<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:51:06<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:53:41<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:53:41<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:54:04<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:54:04<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:56:51<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:56:51<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 11:57:14<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:57:14<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 11:59:55<br \/>\nStart Create_test_data.input.pset at 2020-06-04 11:59:55<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 12:00:23<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 12:00:23<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 12:03:23<br \/>\nStart Create_test_data.input.pset at 2020-06-04 12:03:23<br \/>\nFinish Create_test_data.input.pset at 2020-06-04 12:03:49<br \/>\nStart mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 12:03:49<br \/>\nFinish mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset at 2020-06-04 12:06:46<\/i><\/p>\n<p>Rezultatul este urm\u0103torul:<\/p>\n<p>Grafic<br \/>\nStart time<br \/>\nTimp de finalizare<br \/>\nLungime<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:49:11<br \/>\n04.06.2020 11:49:37<br \/>\n00:00:26<\/p>\n<p>mdw_load.day_one.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:49:37<br \/>\n04.06.2020 11:50:42<br \/>\n00:01:05<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:50:42<br \/>\n04.06.2020 11:51:06<br \/>\n00:00:24<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:51:06<br \/>\n04.06.2020 11:53:41<br \/>\n00:02:35<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:53:41<br \/>\n04.06.2020 11:54:04<br \/>\n00:00:23<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:54:04<br \/>\n04.06.2020 11:56:51<br \/>\n00:02:47<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:56:51<br \/>\n04.06.2020 11:57:14<br \/>\n00:00:23<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:57:14<br \/>\n04.06.2020 11:59:55<br \/>\n00:02:41<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:59:55<br \/>\n04.06.2020 12:00:23<br \/>\n00:00:28<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 12:00:23<br \/>\n04.06.2020 12:03:23<br \/>\n00:03:00<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 12:03:23<br \/>\n04.06.2020 12:03:49<br \/>\n00:00:26<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 12:03:49<br \/>\n04.06.2020 12:06:46<br \/>\n00:02:57<\/p>\n<p>\nSe observ\u0103 c\u0103 6 000 000 de \u00eenregistr\u0103ri sunt procesate \u00een 3 minute, ceea ce este destul de rapid.<br \/>\nDatele din tabela \u021bint\u0103 au fost distribuite astfel:<\/p>\n<pre><code class=\"sql\">select valid_from_ts, valid_to_ts, count(1), min(sk), max(sk) from dev42_1_db_usl.TESTING_SUBJ_org_finval group by valid_from_ts, valid_to_ts order by 1,2;<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"C\u00e2nd ai dimensiuni pe scara Sberbank. Utilizarea Ab Initio \u00een lucrul cu Hive \u0219i GreenPlum.\" src=\"\/wp-content\/uploads\/2020\/07\/baefcfd8142f4ad6275333e3dcfccf94.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nSe poate observa coresponden\u021ba datelor introduse cu momentele de lansare ale graficelor.<br \/>\nAsta \u00eenseamn\u0103 c\u0103 putem rula \u00een Ab Initio \u00eenc\u0103rcarea incremental\u0103 a datelor \u00een GreenPlum cu o frecven\u021b\u0103 foarte mare \u0219i putem observa o vitez\u0103 ridicat\u0103 de inserare a acestor date \u00een GreenPlum. Desigur, nu va fi posibil s\u0103 se execute la fiecare secund\u0103, deoarece Ab Initio, asemenea oric\u0103rei solu\u021bii ETL, necesit\u0103 timp \u201epentru a porni\u201d.<\/p>\n<h2>Concluzie<\/h2>\n<p>\n\u00cen prezent, Ab Initio este utilizat de Sberbank pentru construirea unei versiuni unificate a stratului semantic de date (ESS). Acest proiect implic\u0103 crearea unei versiuni unice a st\u0103rii diferitelor entit\u0103\u021bi de afaceri bancare. Informa\u021biile provin din diverse surse, replicile acestora fiind preg\u0103tite pe Hadoop. \u00cen func\u021bie de nevoile afacerii, se preg\u0103te\u0219te un model de date \u0219i se descriu transform\u0103rile datelor. Ab Initio \u00eencarc\u0103 informa\u021biile \u00een ESS, iar datele \u00eenc\u0103rcate nu sunt doar de interes pentru afacere, ci servesc \u0219i ca surs\u0103 pentru construc\u021bia vitrinelor de date. Func\u021bionalitatea produsului permite utilizarea ca receptor a diverselor sisteme (Hive, Greenplum, Teradata, Oracle), ceea ce ofer\u0103 posibilitatea de a preg\u0103ti datele pentru afacere \u00een diferite formate necesare f\u0103r\u0103 eforturi mari.<\/p>\n<p>Capacit\u0103\u021bile Ab Initio sunt largi, de exemplu, cadrul MDW inclus permite construirea istoricului tehnic \u0219i de afaceri al datelor \u201edin cutie\u201d. Pentru dezvoltatori, Ab Initio ofer\u0103 posibilitatea de a \u201enu reinventa roata\u201d, ci de a folosi numeroasele componente func\u021bionale existente, care sunt, practic, biblioteci necesare \u00een munca cu datele.<\/p>\n<blockquote><p>Autorul \u2014 expert \u00een comunitatea profesional\u0103 Sberbank SberProfi DWH\/BiData. Comunitatea profesional\u0103 SberProfi DWH\/BiData este responsabil\u0103 pentru dezvoltarea competen\u021belor \u00een domenii precum ecosistemul Hadoop, Teradata, Oracle DB, GreenPlum \u0219i instrumentele BI Qlik, SAP BO, Tableau, etc.<\/p><\/blockquote>\n<p>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/sberbank\/blog\/509936\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438. \u0415\u0451 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 \u0441\u0432\u0435\u043b\u043e\u0441\u044c \u043a \u0444\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a\u0443 \u043f\u043e \u0437\u0430\u043f\u0443\u0441\u043a\u0443 \u043a\u043e\u043c\u0430\u043d\u0434 spark-submit. \u041d\u0430 \u0440\u044b\u043d\u043a\u0435 \u0438\u043c\u0435\u043b\u043e\u0441\u044c \u043d\u0435 \u0442\u0430\u043a \u043c\u043d\u043e\u0433\u043e \u0430\u043d\u0430\u043b\u043e\u0433\u043e\u0432, \u0432 \u043f\u0440\u0438\u043d\u0446\u0438\u043f\u0435 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u044b\u0445 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0441 \u0442\u0435\u043c \u043e\u0431\u044a\u0451\u043c\u043e\u043c \u0434\u0430\u043d\u043d\u044b\u0445, \u0441 \u043a\u043e\u0442\u043e\u0440\u044b\u043c \u043c\u044b \u0438\u043c\u0435\u0435\u043c \u0434\u0435\u043b\u043e \u043a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":87610,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-87609","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043e\u0433\u0434\u0430 \u0443 \u0432\u0430\u0441 \u0441\u0431\u0435\u0440\u043e\u0432\u0441\u043a\u0438\u0435 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u044b. \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 Ab Initio \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 Hive \u0438 GreenPlum | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-08T23:42:11+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-08T23:42:11+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47C\u00e2nd ave\u021bi dimensiuni mari. Utilizarea Ab Initio pentru lucrul cu Hive \u0219i GreenPlum | ProHoster","description":"Cu ceva timp \u00een urm\u0103, ne-am confruntat cu problema alegerii unui instrument ETL pentru lucrul cu Big Data. Solu\u021bia utilizat\u0103 anterior, Informatica BDM, nu ne-a satisf\u0103cut din cauza func\u021bionalit\u0103\u021bii limitate.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u0433\u0434\u0430 \u0443 \u0432\u0430\u0441 \u0441\u0431\u0435\u0440\u043e\u0432\u0441\u043a\u0438\u0435 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u044b. \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 Ab Initio \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 Hive \u0438 GreenPlum | ProHoster","og:description":"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-08T23:42:11+00:00","article:modified_time":"2020-07-08T23:42:11+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"87609","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 13:47:29","updated":"2026-02-22 15:29:30","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/87609","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=87609"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/87609\/revisions"}],"predecessor-version":[{"id":162163,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/87609\/revisions\/162163"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/87610"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=87609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=87609"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=87609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}