{"id":87609,"date":"2020-07-09T01:42:11","date_gmt":"2020-07-08T23:42:11","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum"},"modified":"2020-07-09T01:42:11","modified_gmt":"2020-07-08T23:42:11","slug":"kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","title":{"rendered":"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Disa disa m\u00eb par\u00eb, na doli nj\u00eb pyetje p\u00ebr t\u00eb zgjedhur nj\u00eb mjet ETL p\u00ebr pun\u00eb me BigData. Zgjidhja e m\u00ebparshme, Informatica BDM, nuk na p\u00ebrmbushi p\u00ebr shkak t\u00eb funksionalitetit t\u00eb kufizuar. P\u00ebrdorimi i saj u reduktua n\u00eb nj\u00eb framework p\u00ebr ekzekutimin e komandave spark-submit. N\u00eb treg kishte jo shum\u00eb alternativa q\u00eb ishin n\u00eb gjendje t\u00eb punonin me volumin e t\u00eb dh\u00ebnave me t\u00eb cilat merremi \u00e7do dit\u00eb. P\u00ebrfundimisht, ne zgjodh\u00ebm Ab Initio. Gjat\u00eb demonstrimeve piloto, produkti tregoi nj\u00eb shpejt\u00ebsi shum\u00eb t\u00eb lart\u00eb t\u00eb p\u00ebrpunimit t\u00eb t\u00eb dh\u00ebnave. Nuk ka shum\u00eb informacione n\u00eb gjuh\u00ebn ruse p\u00ebr Ab Initio, prandaj ne vendos\u00ebm t\u00eb ndajm\u00eb p\u00ebrvoj\u00ebn ton\u00eb n\u00eb Habr.<\/p>\n<p>Ab Initio ka shum\u00eb transformime klasike dhe t\u00eb pazakonta, kodi i t\u00eb cilave mund t\u00eb zgjerohet me nj\u00eb gjuh\u00eb t\u00eb vetme PDL. P\u00ebr bizneset e vogla, ky mjet kaq i fuqish\u00ebm ndoshta do t\u00eb jet\u00eb i tep\u00ebrt, dhe shumica e funksioneve t\u00eb tij mund t\u00eb duken t\u00eb shtrenjta dhe t\u00eb pap\u00ebrdorura. Por n\u00ebse shkall\u00ebt tuaja afrohen ato t\u00eb Sberbank, at\u00ebher\u00eb Ab Initio mund t\u00eb jet\u00eb interesante p\u00ebr ju. <\/p>\n<p>Ai ndihmon biznesin t\u00eb akumuloj\u00eb njohuri dhe t\u00eb zhvilloj\u00eb ekologjin\u00eb, nd\u00ebrsa zhvilluesit p\u00ebrmir\u00ebsojn\u00eb aft\u00ebsit\u00eb e tyre n\u00eb ETL, mbledhjn\u00eb njohuri n\u00eb shell, ofron mund\u00ebsin\u00eb p\u00ebr t\u00eb m\u00ebsuar gjuh\u00ebn PDL, jep nj\u00eb pamje vizuale t\u00eb proceseve t\u00eb ngarkes\u00ebs dhe e leht\u00ebson zhvillimin p\u00ebr shkak t\u00eb shumtis\u00eb s\u00eb komponent\u00ebve funksional\u00eb.<\/p>\n<p>N\u00eb k\u00ebt\u00eb postim do t\u00eb flas p\u00ebr mund\u00ebsit\u00eb e Ab Initio dhe do t\u00eb sjell krahasime t\u00eb karakteristikave t\u00eb tij me Hive dhe GreenPlum.<\/p>\n<ul>\n<li>P\u00ebrshkrimi i framework-ut MDW dhe pun\u00ebve p\u00ebr p\u00ebrshtatjen e tij p\u00ebr GreenPlum<\/li>\n<li>Karakteristikat e krahasimit t\u00eb performanc\u00ebs s\u00eb Ab Initio me Hive dhe GreenPlum<\/li>\n<li>Puna e Ab Initio me GreenPlum n\u00eb m\u00ebnyr\u00ebn Near Real Time<\/li>\n<\/ul>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nFunksionaliteti i k\u00ebtij produkti \u00ebsht\u00eb shum\u00eb i gjer\u00eb dhe k\u00ebrkon koh\u00eb p\u00ebr t'u m\u00ebsuar. Megjithat\u00eb, me aft\u00ebsit\u00eb e duhura dhe konfigurimet e sakta t\u00eb performanc\u00ebs, rezultatet e p\u00ebrpunimit t\u00eb t\u00eb dh\u00ebnave jan\u00eb mjaft mbres\u00ebl\u00ebn\u00ebse. P\u00ebrdorimi i Ab Initio p\u00ebr zhvilluesit mund t'i ofroj\u00eb nj\u00eb p\u00ebrvoj\u00eb interesante. Ky \u00ebsht\u00eb nj\u00eb korniz\u00eb e re p\u00ebr zhvillimin ETL, nj\u00eb hibrid mes nj\u00eb ambienti vizual dhe zhvillimit t\u00eb ngarkesave n\u00eb nj\u00eb gjuh\u00eb t\u00eb ngjashme me skenarin.<\/p>\n<p>Bizneset po zhvillojn\u00eb ekologjit\u00eb e tyre dhe ky mjet \u00ebsht\u00eb si kurr\u00eb m\u00eb pare i p\u00ebrshtatsh\u00ebm p\u00ebr ta. Me Ab Initio, mund t\u00eb grumbullohet njohuri p\u00ebr biznesin aktual dhe t\u00eb p\u00ebrdoren k\u00ebto njohuri p\u00ebr t\u00eb zgjeruar bizneset ekzistuese dhe p\u00ebr t\u00eb hapur t\u00eb reja. Alternativat p\u00ebr Ab Initio p\u00ebrfshijn\u00eb mjete si Informatica BDM dhe Apache Spark.<\/p>\n<h3>P\u00ebrshkrimi i Ab Initio<\/h3>\n<p>\nAb Initio, si \u00e7do mjet tjet\u00ebr ETL, p\u00ebrb\u00ebn nj\u00eb grup produktesh.<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/c467fe1e8455805075856967b577f8b7.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAb Initio GDE (Graphical Development Environment) \u00ebsht\u00eb nj\u00eb mjedis p\u00ebr zhvilluesit, ku ata konfiguroni transformimet e t\u00eb dh\u00ebnave dhe i lidhin ato me rrjedha t\u00eb dh\u00ebnash n\u00eb form\u00ebn e shigjetave. Nj\u00eb kombinim i k\u00ebtij grupi transformimesh quhet graf:<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/7b0c19a82ae8b461a9bd85117555e106.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLidhjet hyr\u00ebse dhe dal\u00ebse t\u00eb komponent\u00ebve funksional\u00eb jan\u00eb porte dhe p\u00ebrmbajn\u00eb fusha t\u00eb llogaritura brenda transformimeve. Disa grafe, t\u00eb lidhura me rrjedha n\u00eb form\u00ebn e shigjetave n\u00eb rendin e ekzekutimit quhen plane.<\/p>\n<p>Ka disa qindra komponent\u00eb funksional\u00eb, q\u00eb jan\u00eb shum\u00eb. Shum\u00eb prej tyre jan\u00eb t\u00eb specializuar. Mund\u00ebsit\u00eb e transformimeve klasike n\u00eb Ab Initio jan\u00eb m\u00eb t\u00eb gjera se n\u00eb mjete t\u00eb tjera ETL. P\u00ebr shembull, Join ka disa dalje. P\u00ebrve\u00e7 rezultatit t\u00eb bashkimit t\u00eb dataset-eve, mund t\u00eb merrni daljet e dataset-eve hyr\u00ebse, p\u00ebr t\u00eb cilat nuk ishte e mundur t\u00eb bashkoheshin. Po ashtu mund t\u00eb merrni rejects, errors dhe logun e pun\u00ebs s\u00eb transformimeve, i cili mund t\u00eb lexohet si nj\u00eb skedar tekstual dhe t\u00eb p\u00ebrpunohet me transformime t\u00eb tjera:<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/c7d70879daad364a3d4aa41a0a1b738d.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nOse, p\u00ebr shembull, mund t\u00eb materializoni marr\u00ebsin e t\u00eb dh\u00ebnave n\u00eb form\u00ebn e nj\u00eb tabele dhe n\u00eb t\u00eb nj\u00ebjtin graf t\u00eb lexoni t\u00eb dh\u00ebnat prej saj.<\/p>\n<p>Ka transformime origjinale. P\u00ebr shembull, transformimi Scan ka funksionalitetin si funksionet analitike. Ka transformime me emra dometh\u00ebn\u00ebs: Create Data, Read Excel, Normalize, Sort within Groups, Run Program, Run SQL, Join with DB dhe t\u00eb tjera. Grafet mund t\u00eb p\u00ebrdorin parametra n\u00eb koh\u00eb t\u00eb ekzekutimit, p\u00ebrfshir\u00eb mund\u00ebsin\u00eb p\u00ebr t\u00eb kaluar parametra nga sistemi operativ ose n\u00eb sistemin operativ. Skedar\u00ebt me nj\u00eb grup t\u00eb gatsh\u00ebm t\u00eb parametrave t\u00eb kaluar n\u00eb graf quhen parameter sets (psets).<\/p>\n<p>Si\u00e7 e pret, Ab Initio GDE ka nj\u00eb depo t\u00eb quajtur EME (Enterprise Meta Environment). Zhvilluesit kan\u00eb mund\u00ebsin\u00eb t\u00eb punojn\u00eb me versione lokale t\u00eb kodit dhe t\u00eb b\u00ebjn\u00eb check in-in e zhvillimeve t\u00eb tyre n\u00eb depozitat qendrore.<\/p>\n<p>Ka mund\u00ebsin\u00eb q\u00eb gjat\u00eb ekzekutimit ose pas ekzekutimit t\u00eb grafit, t\u00eb klikoni mbi \u00e7do lidhje q\u00eb bashkon transformimet dhe t\u00eb shihni t\u00eb dh\u00ebnat q\u00eb kalojn\u00eb mes k\u00ebtyre transformimeve:<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/a674507b2cd25b980cb862bda2e10cbe.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKa \u00ebsht\u00eb gjithashtu mund\u00ebsia p\u00ebr t\u00eb klikoni n\u00eb \u00e7do rrjedh\u00eb dhe t\u00eb shihni detajet e ndjekjes - sa paralellet punoi transformimi, sa rreshta dhe byte u ngarkuan n\u00eb cil\u00ebn nga paralellet:<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/6188d17bebe756ee59e6956f9a40ec43.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKa mund\u00ebsi t\u00eb ndani ekzekutimin e grafikut n\u00eb faza dhe t\u00eb sh\u00ebnoni se disa transformime duhen realizuar s\u00eb pari (n\u00eb faz\u00ebn zero), t\u00eb tjerat n\u00eb faz\u00ebn e par\u00eb, t\u00eb tjerat n\u00eb faz\u00ebn e dyt\u00eb etj.<\/p>\n<p>P\u00ebr \u00e7do transformim, mund t\u00eb zgjidhni at\u00eb q\u00eb quhet layout (ku do t\u00eb ekzekutohet): pa paralellet ose n\u00eb rrjedha paralele, numrin e t\u00eb cilave mund ta caktoni. N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, skedar\u00ebt p\u00ebrkoh\u00ebsor\u00eb q\u00eb krijon Ab Initio gjat\u00eb pun\u00ebs s\u00eb transformimeve mund t\u00eb vendosen si n\u00eb sistemin e skedar\u00ebve <a class=\"wpil_keyword_link\" href=\"https:\/\/prohoster.info\/sq\/server\/dts-los-angeles\/\"   title=\"server\" data-wpil-keyword-link=\"linked\"  data-wpil-monitor-id=\"3635\">server<\/a>, ashtu edhe n\u00eb HDFS.<\/p>\n<p>N\u00eb \u00e7do transformim bazuar n\u00eb nj\u00eb templates t\u00eb paracaktuar, mund t\u00eb krijoni skenarin tuaj n\u00eb gjuh\u00ebn PDL, q\u00eb pak a shum\u00eb i ngjan shell. <\/p>\n<p>Me gjuh\u00ebn PDL, mund t\u00eb zgjerohet funksionaliteti i transformimeve dhe, p\u00ebr m\u00eb tep\u00ebr, mund t\u00eb generoni dinamikisht (durante ekzekutimit) fragmente t\u00eb rastit t\u00eb kodit sipas parametrave t\u00eb koh\u00ebs s\u00eb ekzekutimit.<\/p>\n<p>Po ashtu, Ab Initio ka nj\u00eb integrim t\u00eb mir\u00eb t\u00eb zhvilluar me sistemin operativ p\u00ebrmes shell. Konkretisht n\u00eb Sberbank p\u00ebrdoret linux ksh. Mund t\u00eb shk\u00ebmbehen variabla me shell dhe t'i p\u00ebrdorni ato si parametra grafik\u00ebsh. Mund t\u00eb thirreni ekzekutimin e grafik\u00ebve Ab Initio nga shell dhe t\u00eb menaxhoni Ab Initio.<\/p>\n<p>P\u00ebrve\u00e7 Ab Initio GDE, ofrohen shum\u00eb produkte t\u00eb tjera. Ka sistemin e saj Co&gt;Operation me pretendimin p\u00ebr t'u quajtur sistem operativ. Ka Control&gt;Center, n\u00eb t\u00eb cilin mund t\u00eb planifikoni dhe monitoroni rrjedhat e ngarkes\u00ebs. Ka produkte p\u00ebr zhvillimin n\u00eb nj\u00eb nivel m\u00eb primitiv se sa lejon Ab Initio GDE.<\/p>\n<h3>P\u00ebrshkrimi i framework-ut MDW dhe pun\u00ebve p\u00ebr p\u00ebrshtatjen e tij p\u00ebr GreenPlum<\/h3>\n<p>\nS\u00eb bashku me produktet e saj, shit\u00ebsi ofron produktin MDW (Metadata Driven Warehouse), i cili p\u00ebrb\u00ebn nj\u00eb konfigurues grafik\u00ebsh, i dizajnuar p\u00ebr t\u00eb ndihmuar n\u00eb detyrat tipike t\u00eb mbushjes s\u00eb depozita t\u00eb t\u00eb dh\u00ebnave ose data vaults.<\/p>\n<p>Ai p\u00ebrmban parser\u00eb t\u00eb metadatas p\u00ebrdoruesore (specifike p\u00ebr projektin) dhe gjenerator\u00eb t\u00eb kodit \u201cnga kutia\u201d.<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/0842ecb9550f369875103ccb23cdf121.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nN\u00eb hyrje, MDW merr modelin e t\u00eb dh\u00ebnave, nj\u00eb skedarin konfigurues p\u00ebr caktimin e lidhjes me baz\u00ebn e t\u00eb dh\u00ebnave (Oracle, Teradata ose Hive) dhe disa konfigurime t\u00eb tjera. Pjesa specifike p\u00ebr projektin, p\u00ebr shembull, zhvillon modelin n\u00eb baz\u00ebn e t\u00eb dh\u00ebnave. Pjesa produktit q\u00eb vjen \u201cnga kutia\u201d gjeneron graf\u00ebt dhe skedar\u00ebt e konfigurimit p\u00ebr ta p\u00ebrngarkimin e t\u00eb dh\u00ebnave n\u00eb tabelat e modelit. N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, krijohen graf\u00eb (dhe psets) p\u00ebr disa m\u00ebnyra t\u00eb pun\u00ebs fillestare dhe inkrementale p\u00ebr p\u00ebrdit\u00ebsimin e entiteteve.<\/p>\n<p>N\u00eb rastet e Hive dhe RDBMS, gjenerohen graf\u00eb t\u00eb ndrysh\u00ebm p\u00ebr p\u00ebrdit\u00ebsimin fillestar dhe inkremental t\u00eb t\u00eb dh\u00ebnave.<\/p>\n<p>N\u00eb rastin e Hive, delta e ardhur lidhet p\u00ebrmes Ab Initio Join me t\u00eb dh\u00ebnat q\u00eb ishin n\u00eb tabel\u00eb para p\u00ebrdit\u00ebsimit. Ngarkuesit e t\u00eb dh\u00ebnave n\u00eb MDW (si n\u00eb Hive ashtu edhe n\u00eb RDBMS) jo vet\u00ebm q\u00eb fusin t\u00eb dh\u00ebnat e reja nga delta, por gjithashtu mbyllin periudhat e aktualitetit t\u00eb t\u00eb dh\u00ebnave, sipas \u00e7el\u00ebsave t\u00eb par\u00eb me t\u00eb cil\u00ebt ka ardhur delta. P\u00ebr m\u00eb tep\u00ebr, \u00ebsht\u00eb e nevojshme t\u00eb rikopjoni p\u00ebrs\u00ebri pjes\u00ebn e t\u00eb dh\u00ebnave q\u00eb nuk \u00ebsht\u00eb ndryshuar. Por \u00ebsht\u00eb e nevojshme t\u00eb b\u00ebhet k\u00ebshtu, pasi n\u00eb Hive nuk ka operacione delete ose update.<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/36b6dd7c4d45727ad7d7aa1183fdd5fd.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nN\u00eb rastin e RDBMS, graf\u00ebt p\u00ebr p\u00ebrdit\u00ebsimin inkremental t\u00eb t\u00eb dh\u00ebnave duket se jan\u00eb m\u00eb optimal\u00eb, sepse RDBMS kan\u00eb mund\u00ebsi reale p\u00ebr p\u00ebrdit\u00ebsim.<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/b62ac222bed575a60b632b13e012f603.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDelta e ardhur ngarkohet n\u00eb nj\u00eb tabel\u00eb p\u00ebrkoh\u00ebsisht n\u00eb baz\u00ebn e t\u00eb dh\u00ebnave. Pas k\u00ebsaj, ndodh lidhja e delta-s me t\u00eb dh\u00ebnat q\u00eb ishin n\u00eb tabel\u00eb para p\u00ebrdit\u00ebsimit. Dhe kjo b\u00ebhet me an\u00eb t\u00eb SQL me nj\u00eb pyetje SQL t\u00eb gjeneruar. M\u00eb pas, me ndihm\u00ebn e komandave SQL delete+insert, ndodhte futja e t\u00eb dh\u00ebnave t\u00eb reja nga delta n\u00eb tabel\u00ebn q\u00eb \u00ebsht\u00eb synuar dhe mbyllja e periudhave t\u00eb aktualitetit t\u00eb t\u00eb dh\u00ebnave, sipas \u00e7el\u00ebsave t\u00eb par\u00eb me t\u00eb cil\u00ebt ka ardhur delta. <br \/>\nNuk ka nevoj\u00eb t\u00eb rikopjoni t\u00eb dh\u00ebnat q\u00eb nuk jan\u00eb ndryshuar.<\/p>\n<p>K\u00ebshtu, arrijm\u00eb n\u00eb p\u00ebrfundimin se n\u00eb rastin e Hive, MDW duhet t\u00eb shkoj\u00eb n\u00eb rikopimin e t\u00eb gjith\u00eb tabel\u00ebs, sepse Hive nuk ka funksionin e p\u00ebrdit\u00ebsimit. Dhe nuk \u00ebsht\u00eb menduar di\u00e7ka m\u00eb mir\u00eb se rikopimi i plot\u00eb i t\u00eb dh\u00ebnave gjat\u00eb p\u00ebrdit\u00ebsimit. N\u00eb rastin e RDBMS, p\u00ebrkundrazi, krijuesit e produktit e paskan konsideruar t\u00eb nevojshme q\u00eb lidhja dhe p\u00ebrdit\u00ebsimi i tabelave t'i lihen p\u00ebrdorimit t\u00eb SQL.<\/p>\n<p>P\u00ebr projektin n\u00eb Sberbank, krijuam nj\u00eb zbatim t\u00eb ri q\u00eb mund t\u00eb p\u00ebrdoret shum\u00eb her\u00eb p\u00ebr ngarkuesin e baz\u00ebs t\u00eb dh\u00ebnave p\u00ebr GreenPlum. Kjo u b\u00eb duke u bazuar n\u00eb versionin q\u00eb MDW gjeneron p\u00ebr Teradata. Pik\u00ebrisht Teradata, e jo Oracle, p\u00ebr k\u00ebt\u00eb i p\u00ebrshtatet m\u00eb mir\u00eb dhe \u00ebsht\u00eb m\u00eb af\u00ebr, pasi gjithashtu \u00ebsht\u00eb nj\u00eb sistem MPP. Metodat e pun\u00ebs, si dhe sintaksa e Teradata dhe GreenPlum, u treguan t\u00eb ngjashme.<\/p>\n<p>Shembuj kritike p\u00ebr ndryshimet midis RDBMS-ve jan\u00eb t\u00eb tilla. N\u00eb GreenPlum, ndryshe nga Teradata, kur krijoni tabela duhet t\u00eb shkruani klauzol\u00ebn<\/p>\n<pre><code class=\"sql\">distributed by<\/code><\/pre>\n<p>\nN\u00eb Teradata shkruhet<\/p>\n<pre><code class=\"sql\">fshi &lt;table&gt; all<\/code><\/pre>\n<p>\n, nd\u00ebrsa n\u00eb Gre\u00ebPlum shkruhet<\/p>\n<pre><code class=\"sql\">fshi nga &lt;table&gt;<\/code><\/pre>\n<p>\nN\u00eb Oracle, p\u00ebr q\u00ebllime optimizimi, shkruhet<\/p>\n<pre><code class=\"sql\">delete from t where rowid in ()<\/code><\/pre>\n<p>\n, dhe n\u00eb Teradata dhe GreenPlum shkruajn\u00eb<\/p>\n<pre><code class=\"sql\">fshi nga t ku ekziston (shiko * nga delta ku delta.pk=t.pk)<\/code><\/pre>\n<p>\nGjithashtu, do t\u00eb theksojm\u00eb se p\u00ebr pun\u00ebn e Ab Initio me GreenPlum ishte e nevojshme t\u00eb instaloheshin klienti GreenPlum n\u00eb t\u00eb gjitha nodet e klasterit Ab Initio. Kjo p\u00ebr shkak se ne u lidh\u00ebm me GreenPlum nj\u00ebkoh\u00ebsisht nga t\u00eb gjitha nyjet e klasterit ton\u00eb. Dhe q\u00eb leximi nga GreenPlum t\u00eb ishte paralel dhe \u00e7do rrjedh\u00eb paralel e Ab Initio t\u00eb lexonte pjes\u00ebt e saj t\u00eb t\u00eb dh\u00ebnave nga GreenPlum, duhej t\u00eb vendosnim n\u00eb seksionin \"ku\" t\u00eb k\u00ebrkesave SQL nj\u00eb konstruksion t\u00eb kuptuesh\u00ebm p\u00ebr Ab Initio<\/p>\n<pre><code class=\"sql\">ku ABLOCAL()<\/code><\/pre>\n<p>\ndhe t\u00eb p\u00ebrcaktonim vler\u00ebn e k\u00ebtij konstruksioni, duke treguar parametrin q\u00eb lexon nga Baza e t\u00eb Dh\u00ebnave t\u00eb transformimit<\/p>\n<pre><code class=\"sql\">ablocal_expr=\u00abstring_concat(&quot;mod(t.&quot;, string_filter_out(&quot;{$TABLE_KEY}&quot;,&quot;{}&quot;), &quot;,&quot;, (decimal(3))(number_of_partitions()),&quot;)=&quot;, (decimal(3))(this_partition()))\u00bb<\/code><\/pre>\n<p>\n, i cili kompilohet n\u00eb di\u00e7ka t\u00eb till\u00eb<\/p>\n<pre><code class=\"sql\">mod(sk,10)=3<\/code><\/pre>\n<p>\n, dmth. duhet t'i japim GreenPlum nj\u00eb filtrin t\u00eb duksh\u00ebm p\u00ebr \u00e7do parti. P\u00ebr baza t\u00eb tjera t\u00eb t\u00eb dh\u00ebnave (Teradata, Oracle) Ab Initio mund ta ekzekutoj\u00eb k\u00ebt\u00eb paralelizim automatikisht.<\/p>\n<h3>Karakteristikat e krahasimit t\u00eb performanc\u00ebs s\u00eb Ab Initio me Hive dhe GreenPlum<\/h3>\n<p>\nN\u00eb Sberbank, u zhvillua nj\u00eb eksperiment p\u00ebr t\u00eb krahasuar performanc\u00ebn e graf\u00ebve t\u00eb gjeneruar MDW n\u00eb lidhje me Hive dhe GreenPlum. N\u00eb kuad\u00ebr t\u00eb eksperimenteve, n\u00eb rastin e Hive kishte 5 nyje n\u00eb t\u00eb nj\u00ebjtin klaster si Ab Initio, nd\u00ebrsa n\u00eb rastin e GreenPlum kishte 4 nyje n\u00eb nj\u00eb klaster t\u00eb ve\u00e7ant\u00eb. Pra, Hive kishte nj\u00eb avantazh t\u00eb caktuar mbi GreenPlum \"p\u00ebr shkak t\u00eb harduerit\".<\/p>\n<p>Iu shqyrtuan dy \u00e7ift graf\u00ebsh, t\u00eb cil\u00ebt kryenin t\u00eb nj\u00ebjt\u00ebn detyr\u00eb t\u00eb p\u00ebrdit\u00ebsimit t\u00eb t\u00eb dh\u00ebnave n\u00eb Hive dhe n\u00eb GreenPlum. N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, u nis\u00ebn graf\u00ebt e gjeneruar nga konfiguratori MDW:<\/p>\n<ul>\n<li>ngarkimi fillestar + ngarkimi inkremental t\u00eb t\u00eb dh\u00ebnave t\u00eb gjeneruara rast\u00ebsisht n\u00eb tabel\u00ebn Hive<\/li>\n<li>ngarkimi fillestar + ngarkimi inkremental t\u00eb t\u00eb dh\u00ebnave t\u00eb gjeneruara rast\u00ebsisht n\u00eb t\u00eb nj\u00ebjt\u00ebn tabel\u00eb GreenPlum<\/li>\n<\/ul>\n<p>\nN\u00eb t\u00eb dyja rastet (Hive dhe GreenPlum), ngarkimet ishin n\u00eb 10 rrjedha paralel n\u00eb t\u00eb nj\u00ebjtin klaster Ab Initio. T\u00eb dh\u00ebnat nd\u00ebrmjet\u00ebsore p\u00ebr llogaritjet Ab Initio u ruajt\u00ebn n\u00eb HDFS (n\u00eb termat e Ab Initio u p\u00ebrdor MFS layout using HDFS). Nj\u00eb rresht i t\u00eb dh\u00ebnave t\u00eb gjeneruara rast\u00ebsisht zinte n\u00eb t\u00eb dyja rastet 200 byte.<\/p>\n<p>Rezultati ishte i till\u00eb:<\/p>\n<p><b>Hive:<\/b><\/p>\n<p><b>Ngarkimi fillestar n\u00eb Hive<\/b><\/p>\n<p>Numri i rreshtave t\u00eb futur<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Koh\u00ebzgjatja e ngarkimit fillestar<br \/>\nn\u00eb sekonda<br \/>\n41<br \/>\n203<br \/>\n1 601<\/p>\n<p><b>Ngarkimi inkremental n\u00eb Hive<\/b><\/p>\n<p>Numri i rreshtave, t\u00eb cil\u00ebt ishin n\u00eb<br \/>\ntabel\u00ebn e synuar n\u00eb fillim t\u00eb eksperimenti<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Numri i rreshtave t\u00eb delta-s, t\u00eb aplikuar n\u00eb<br \/>\ntabel\u00ebn e synuar gjat\u00eb eksperimenti<br \/>\n6 000 000<br \/>\n6 000 000<br \/>\n6 000 000<\/p>\n<p>Koh\u00ebzgjatja e ngarkimit inkremental<br \/>\nn\u00eb sekonda<br \/>\n88<br \/>\n299<br \/>\n<b>2 541<\/b><\/p>\n<p>\n<b>GreenPlum:<\/b><\/p>\n<p><b>Ngarkimi fillestar n\u00eb GreenPlum<\/b><\/p>\n<p>Numri i rreshtave t\u00eb futur<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Koh\u00ebzgjatja e ngarkimit fillestar<br \/>\nn\u00eb sekonda<br \/>\n72<br \/>\n360<br \/>\n3 631<\/p>\n<p><b>Ngarkimi inkremental n\u00eb GreenPlum<\/b><\/p>\n<p>Numri i rreshtave, t\u00eb cil\u00ebt ishin n\u00eb<br \/>\ntabel\u00ebn e synuar n\u00eb fillim t\u00eb eksperimenti<br \/>\n6 000 000<br \/>\n60 000 000<br \/>\n600 000 000<\/p>\n<p>Numri i rreshtave t\u00eb delta-s, t\u00eb aplikuar n\u00eb<br \/>\ntabel\u00ebn e synuar gjat\u00eb eksperimenti<br \/>\n6 000 000<br \/>\n6 000 000<br \/>\n6 000 000<\/p>\n<p>Koh\u00ebzgjatja e ngarkimit inkremental<br \/>\nn\u00eb sekonda<br \/>\n159<br \/>\n199<br \/>\n<b>321<\/b><\/p>\n<p>\nV\u00ebrejm\u00eb se shpejt\u00ebsia e ngarkimit fillestar si n\u00eb Hive ashtu edhe n\u00eb GreenPlum varet n\u00eb m\u00ebnyr\u00eb lineare nga sasia e t\u00eb dh\u00ebnave dhe p\u00ebr shkak t\u00eb harduerit m\u00eb t\u00eb mir\u00eb, ajo \u00ebsht\u00eb pak m\u00eb e shpejt\u00eb p\u00ebr Hive, sesa p\u00ebr GreenPlum.<\/p>\n<p>Ngarkimi inkremental n\u00eb Hive gjithashtu varet n\u00eb m\u00ebnyr\u00eb lineare nga sasia e t\u00eb dh\u00ebnave t\u00eb ngarkuara m\u00eb par\u00eb n\u00eb tabel\u00ebn e synuar dhe kalon mjaft ngadalt\u00eb me rritjen e sasis\u00eb. Kjo lidhet me nevoj\u00ebn p\u00ebr t\u00eb rishtuar plot\u00ebsisht tabel\u00ebn e synuar. Kjo do t\u00eb thot\u00eb se aplikimi i ndryshimeve t\u00eb vogla n\u00eb tabela t\u00eb m\u00ebdha - nuk \u00ebsht\u00eb nj\u00eb variant i mir\u00eb p\u00ebr p\u00ebrdorim p\u00ebr Hive.<\/p>\n<p>Ngarkimi inkremental n\u00eb GreenPlum varet pak nga sasia e t\u00eb dh\u00ebnave t\u00eb ngarkuara m\u00eb par\u00eb n\u00eb tabel\u00ebn e synuar dhe kalon mjaft shpejt. Kjo arrihet fal\u00eb SQL Joins dhe arkitektur\u00ebs s\u00eb GreenPlum, e cila lejon operacionin e fshirjes.<\/p>\n<p>Pra, GreenPlum derdh deltat me metod\u00ebn delete+insert, nd\u00ebrsa n\u00eb Hive nuk ka operacione fshir\u00ebse ose p\u00ebrdit\u00ebsuese, k\u00ebshtu q\u00eb e gjith\u00eb sasia e t\u00eb dh\u00ebnave gjat\u00eb p\u00ebrdit\u00ebsimit inkremental duhet t\u00eb rishkruhet plot\u00ebsisht. E ve\u00e7anta \u00ebsht\u00eb krahasimi i qelizave t\u00eb theksuara me shkronja t\u00eb trasha, pasi ai p\u00ebrkon me variantin m\u00eb t\u00eb shpesht\u00eb t\u00eb shfryt\u00ebzimit t\u00eb ngarkimeve intensive me resurse. V\u00ebrejm\u00eb se GreenPlum e fitoi ndaj Hive n\u00eb k\u00ebt\u00eb test 8 her\u00eb.<\/p>\n<h3>Puna e Ab Initio me GreenPlum n\u00eb m\u00ebnyr\u00ebn Near Real Time<\/h3>\n<p>\nN\u00eb k\u00ebt\u00eb eksperiment do t\u00eb verifikohet mund\u00ebsia q\u00eb Ab Initio t\u00eb kryej\u00eb p\u00ebrdit\u00ebsimin e tabel\u00ebs GreenPlum me grupe t\u00eb dh\u00ebnash t\u00eb gjeneruara rast\u00ebsisht n\u00eb nj\u00eb mod t\u00eb af\u00ebrt me koh\u00ebn reale. Do t\u00eb shqyrtojm\u00eb tabel\u00ebn GreenPlum dev42_1_db_usl.TESTING_SUBJ_org_finval, me t\u00eb cil\u00ebn do t\u00eb punojm\u00eb.<\/p>\n<p>Do t\u00eb p\u00ebrdorim tre graf\u00eb Ab Initio p\u00ebr pun\u00ebn me t\u00eb:<\/p>\n<p>1) Grafi Create_test_data.mp \u2013 krijon n\u00eb 10 rrjedha paralel skedar\u00eb me t\u00eb dh\u00ebna n\u00eb HDFS me 6 000 000 rreshta. T\u00eb dh\u00ebnat jan\u00eb rast\u00ebsore, struktura e tyre \u00ebsht\u00eb organizuar p\u00ebr t'u futur n\u00eb tabel\u00ebn ton\u00eb<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/8cac605c1c5931313bd016a48fb6384e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n<img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/b86988bf1722329bde60013ba025d600.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n2) Grafi mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset \u2013 grafi i gjeneruar nga MDW p\u00ebr ngarkimin e fillestar t\u00eb t\u00eb dh\u00ebnave n\u00eb tabel\u00ebn ton\u00eb n\u00eb 10 rrjedha paralel (p\u00ebrdoren t\u00eb dh\u00ebna testimi, t\u00eb gjeneruara nga grafi (1))<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/b4228a13c088c6b96abba35e9e58fe8e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n3) Grafi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset \u2013 grafi i gjeneruar nga MDW p\u00ebr p\u00ebrdit\u00ebsimin inkremental t\u00eb tabel\u00ebs ton\u00eb n\u00eb 10 rrjedha paralel duke p\u00ebrdorur grupin e t\u00eb dh\u00ebnave t\u00eb fresk\u00ebta q\u00eb kan\u00eb mb\u00ebrritur (delta), t\u00eb gjeneruara nga grafi (1)<\/p>\n<p><img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/714a901616cf84e5a1fd7a764373e30f.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDo t\u00eb ekzekutojm\u00eb skenarin e m\u00ebposht\u00ebm n\u00eb modin NRT:<\/p>\n<ul>\n<li>gjeneroni 6 000 000 rreshta testimi<\/li>\n<li>t\u00eb realizoni nj\u00eb ngarkes\u00eb fillestare duke futur 6 000 000 rreshta testimi n\u00eb nj\u00eb tabel\u00eb t\u00eb zbraz\u00ebt<\/li>\n<li>t\u00eb p\u00ebrs\u00ebrisni 5 her\u00eb ngarkes\u00ebn inkrementale\n<ul>\n<li>gjeneroni 6 000 000 rreshta testimi<\/li>\n<li>t\u00eb realizoni nj\u00eb futje inkrementale t\u00eb 6 000 000 rreshtave testimi n\u00eb tabel\u00eb (n\u00eb k\u00ebt\u00eb rast, t\u00eb dh\u00ebnave t\u00eb vjetra u caktohet nj\u00eb koh\u00eb skadence valid_to_ts dhe futen t\u00eb dh\u00ebna m\u00eb t\u00eb reja me t\u00eb nj\u00ebjtin \u00e7el\u00ebs kryesor)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>\nKy skenar imiton nj\u00eb m\u00ebnyr\u00eb reale t\u00eb funksionimit t\u00eb nj\u00eb sistemi biznesi - n\u00eb koh\u00eb reale shfaqet nj\u00eb sasi e madhe t\u00eb dh\u00ebnash t\u00eb reja dhe menj\u00ebher\u00eb futet n\u00eb GreenPlum.<\/p>\n<p>Tani le t\u00eb shikojm\u00eb logun e funksionimit t\u00eb skenarit:<\/p>\n<p><i>Fillimi Create_test_data.input.pset n\u00eb 2020-06-04 11:49:11<br \/>\nP\u00ebrfundimi Create_test_data.input.pset n\u00eb 2020-06-04 11:49:37<br \/>\nFillimi mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:49:37<br \/>\nP\u00ebrfundimi mdw_load.day_one.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:50:42<br \/>\nFillimi Create_test_data.input.pset n\u00eb 2020-06-04 11:50:42<br \/>\nP\u00ebrfundimi Create_test_data.input.pset n\u00eb 2020-06-04 11:51:06<br \/>\nFillimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:51:06<br \/>\nP\u00ebrfundimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:53:41<br \/>\nFillimi Create_test_data.input.pset n\u00eb 2020-06-04 11:53:41<br \/>\nP\u00ebrfundimi Create_test_data.input.pset n\u00eb 2020-06-04 11:54:04<br \/>\nFillimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:54:04<br \/>\nP\u00ebrfundimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:56:51<br \/>\nFillimi Create_test_data.input.pset n\u00eb 2020-06-04 11:56:51<br \/>\nP\u00ebrfundimi Create_test_data.input.pset n\u00eb 2020-06-04 11:57:14<br \/>\nFillimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:57:14<br \/>\nP\u00ebrfundimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 11:59:55<br \/>\nFillimi Create_test_data.input.pset n\u00eb 2020-06-04 11:59:55<br \/>\nP\u00ebrfundimi Create_test_data.input.pset n\u00eb 2020-06-04 12:00:23<br \/>\nFillimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 12:00:23<br \/>\nP\u00ebrfundimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 12:03:23<br \/>\nFillimi Create_test_data.input.pset n\u00eb 2020-06-04 12:03:23<br \/>\nP\u00ebrfundimi Create_test_data.input.pset n\u00eb 2020-06-04 12:03:49<br \/>\nFillimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 12:03:49<br \/>\nP\u00ebrfundimi mdw_load.regular.current.dev42_1_db_usl_testing_subj_org_finval.pset n\u00eb 2020-06-04 12:06:46<\/i><\/p>\n<p>K\u00ebshtu, del nj\u00eb pamje e till\u00eb:<\/p>\n<p>Grafik<br \/>\nKoha e fillimit<br \/>\nKoha e p\u00ebrfundimit<br \/>\nGjat\u00ebsia<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:49:11<br \/>\n04.06.2020 11:49:37<br \/>\n00:00:26<\/p>\n<p>mdw_load.day_one.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:49:37<br \/>\n04.06.2020 11:50:42<br \/>\n00:01:05<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:50:42<br \/>\n04.06.2020 11:51:06<br \/>\n00:00:24<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:51:06<br \/>\n04.06.2020 11:53:41<br \/>\n00:02:35<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:53:41<br \/>\n04.06.2020 11:54:04<br \/>\n00:00:23<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:54:04<br \/>\n04.06.2020 11:56:51<br \/>\n00:02:47<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:56:51<br \/>\n04.06.2020 11:57:14<br \/>\n00:00:23<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 11:57:14<br \/>\n04.06.2020 11:59:55<br \/>\n00:02:41<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 11:59:55<br \/>\n04.06.2020 12:00:23<br \/>\n00:00:28<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 12:00:23<br \/>\n04.06.2020 12:03:23<br \/>\n00:03:00<\/p>\n<p>Create_test_data.input.pset<br \/>\n04.06.2020 12:03:23<br \/>\n04.06.2020 12:03:49<br \/>\n00:00:26<\/p>\n<p>mdw_load.regular.current.<br \/>\ndev42_1_db_usl_testing_subj_org_finval.pset<br \/>\n04.06.2020 12:03:49<br \/>\n04.06.2020 12:06:46<br \/>\n00:02:57<\/p>\n<p>\nShikojm\u00eb se 6 000 000 rreshta inkrementale p\u00ebrpunohen brenda 3 minutash, q\u00eb \u00ebsht\u00eb mjaft shpejt.<br \/>\nT\u00eb dh\u00ebnat n\u00eb tabel\u00ebn e synuar dol\u00ebn t\u00eb shp\u00ebrndara n\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb:<\/p>\n<pre><code class=\"sql\">select valid_from_ts, valid_to_ts, count(1), min(sk), max(sk) from dev42_1_db_usl.TESTING_SUBJ_org_finval group by valid_from_ts, valid_to_ts order by 1,2;<\/code><\/pre>\n<p>\n<img decoding=\"async\" alt=\"Kur keni p\u00ebrmasa t\u00eb m\u00ebdha si ato t\u00eb Sberbank. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum\" src=\"\/wp-content\/uploads\/2020\/07\/baefcfd8142f4ad6275333e3dcfccf94.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nMund t\u00eb shihni p\u00ebrputhshm\u00ebrin\u00eb e t\u00eb dh\u00ebnave t\u00eb futur me momentet e aktivizimit t\u00eb grafik\u00ebve.<br \/>\nPra, mund t\u00eb filloni ngarkes\u00ebn inkrementale t\u00eb t\u00eb dh\u00ebnave n\u00eb GreenPlum me nj\u00eb frekuenc\u00eb shum\u00eb t\u00eb lart\u00eb dhe t\u00eb shihni shpejt\u00ebsin\u00eb e lart\u00eb t\u00eb futjes s\u00eb k\u00ebtyre t\u00eb dh\u00ebnave n\u00eb GreenPlum. Sigurisht, nuk mund t\u00eb aktivizohet \u00e7do sekond\u00eb, pasi Ab Initio, ashtu si \u00e7do mjet ETL, k\u00ebrkon koh\u00eb p\u00ebr \"t\u00eb fjetur\".<\/p>\n<h2>P\u00ebrfundimi<\/h2>\n<p>\nAktualisht, Ab Initio p\u00ebrdoret n\u00eb Sberbank p\u00ebr nd\u00ebrtimin e Shtres\u00ebs s\u00eb Vetme Semantike t\u00eb t\u00eb Dh\u00ebnave (ECCD). Ky projekt supozon nd\u00ebrtimin e nj\u00eb versioni t\u00eb vet\u00ebm t\u00eb gjendjes s\u00eb entiteteve t\u00eb ndryshme t\u00eb biznesit bankar. Informacioni vjen nga burime t\u00eb ndryshme, kopjet e t\u00eb cilave p\u00ebrgatiten n\u00eb Hadoop. Duke u bazuar n\u00eb nevojat e biznesit, p\u00ebrgatitet modeli i t\u00eb dh\u00ebnave dhe p\u00ebrshkruhen transformimet e t\u00eb dh\u00ebnave. Ab Initio ngarkon informacionin n\u00eb ECCD dhe t\u00eb dh\u00ebnat e ngarkuara jo vet\u00ebm q\u00eb paraqesin interes p\u00ebr biznesin p\u00ebr vetvete, por sh\u00ebrbejn\u00eb gjithashtu si burim p\u00ebr nd\u00ebrtimin e vitrinave t\u00eb t\u00eb dh\u00ebnave. Nj\u00ebkoh\u00ebsisht, funksionaliteti i produktit lejon p\u00ebrdorimin e sistemeve t\u00eb ndryshme si marr\u00ebs (Hive, Greenplum, Teradata, Oracle), \u00e7ka ofron mund\u00ebsin\u00eb p\u00ebr t\u00eb p\u00ebrgatitur t\u00eb dh\u00ebnat p\u00ebr biznesin n\u00eb formate t\u00eb ndryshme pa ndonj\u00eb v\u00ebshtir\u00ebsi t\u00eb ve\u00e7ant\u00eb.<\/p>\n<p>Mund\u00ebsit\u00eb e Ab Initio jan\u00eb t\u00eb shumta, p\u00ebr shembull, korniza shoq\u00ebruese MDW ofron mund\u00ebsin\u00eb p\u00ebr t\u00eb nd\u00ebrtuar historin\u00eb teknike dhe biznesore t\u00eb t\u00eb dh\u00ebnave \"nga kutia\". P\u00ebr zhvilluesit, Ab Initio ofron mund\u00ebsin\u00eb p\u00ebr t\u00eb \"mos shpikur rrot\u00ebn p\u00ebrs\u00ebri\" dhe p\u00ebr t\u00eb shfryt\u00ebzuar shum\u00eb komponente funksionale ekzistues, t\u00eb cilat n\u00eb thelb jan\u00eb biblioteka t\u00eb nevojshme p\u00ebr pun\u00ebn me t\u00eb dh\u00ebnat.<\/p>\n<blockquote><p>Autori - ekspert i komunitetit profesional t\u00eb Sberbank SberProfi DWH\/BigData. Komuniteti profesional SberProfi DWH\/BigData \u00ebsht\u00eb p\u00ebrgjegj\u00ebs p\u00ebr zhvillimin e kompetencave n\u00eb fusha si ekosistemi Hadoop, Teradata, Oracle DB, GreenPlum, si dhe veglat BI Qlik, SAP BO, Tableau e t\u00eb tjera.<\/p><\/blockquote>\n<p>Burimi: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/sberbank\/blog\/509936\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438. \u0415\u0451 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 \u0441\u0432\u0435\u043b\u043e\u0441\u044c \u043a \u0444\u0440\u0435\u0439\u043c\u0432\u043e\u0440\u043a\u0443 \u043f\u043e \u0437\u0430\u043f\u0443\u0441\u043a\u0443 \u043a\u043e\u043c\u0430\u043d\u0434 spark-submit. \u041d\u0430 \u0440\u044b\u043d\u043a\u0435 \u0438\u043c\u0435\u043b\u043e\u0441\u044c \u043d\u0435 \u0442\u0430\u043a \u043c\u043d\u043e\u0433\u043e \u0430\u043d\u0430\u043b\u043e\u0433\u043e\u0432, \u0432 \u043f\u0440\u0438\u043d\u0446\u0438\u043f\u0435 \u0441\u043f\u043e\u0441\u043e\u0431\u043d\u044b\u0445 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0441 \u0442\u0435\u043c \u043e\u0431\u044a\u0451\u043c\u043e\u043c \u0434\u0430\u043d\u043d\u044b\u0445, \u0441 \u043a\u043e\u0442\u043e\u0440\u044b\u043c \u043c\u044b \u0438\u043c\u0435\u0435\u043c \u0434\u0435\u043b\u043e \u043a\u0430\u0436\u0434\u044b\u0439 \u0434\u0435\u043d\u044c. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":87610,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-87609","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.0.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.0.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"sq_AL\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043e\u0433\u0434\u0430 \u0443 \u0432\u0430\u0441 \u0441\u0431\u0435\u0440\u043e\u0432\u0441\u043a\u0438\u0435 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u044b. \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 Ab Initio \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 Hive \u0438 GreenPlum | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-07-08T23:42:11+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-07-08T23:42:11+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Kur keni shkall\u00eb Sber. P\u00ebrdorimi i Ab Initio n\u00eb pun\u00ebn me Hive dhe GreenPlum | ProHoster","description":"Disa koh\u00eb m\u00eb par\u00eb na u paraqit nj\u00eb pyetje p\u00ebr t\u00eb zgjedhur nj\u00eb mjet ETL p\u00ebr pun\u00eb me BigData. Zgjidhja e m\u00ebparshme Informatica BDM nuk na p\u00ebrmbushte p\u00ebr shkak t\u00eb funksionalitetit t\u00eb kufizuar.","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u0433\u0434\u0430 \u0443 \u0432\u0430\u0441 \u0441\u0431\u0435\u0440\u043e\u0432\u0441\u043a\u0438\u0435 \u043c\u0430\u0441\u0448\u0442\u0430\u0431\u044b. \u0418\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 Ab Initio \u043f\u0440\u0438 \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 Hive \u0438 GreenPlum | ProHoster","og:description":"\u041d\u0435\u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u0432\u0440\u0435\u043c\u044f \u043d\u0430\u0437\u0430\u0434 \u043f\u0435\u0440\u0435\u0434 \u043d\u0430\u043c\u0438 \u0432\u0441\u0442\u0430\u043b \u0432\u043e\u043f\u0440\u043e\u0441 \u0432\u044b\u0431\u043e\u0440\u0430 ETL-\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u0430 \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441 BigData. \u0420\u0430\u043d\u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0432\u0448\u0435\u0435\u0441\u044f \u0440\u0435\u0448\u0435\u043d\u0438\u0435 Informatica BDM \u043d\u0435 \u0443\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u043b\u043e \u043d\u0430\u0441 \u0438\u0437-\u0437\u0430 \u043e\u0433\u0440\u0430\u043d\u0438\u0447\u0435\u043d\u043d\u043e\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b\u044c\u043d\u043e\u0441\u0442\u0438.","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/kogda-u-vas-sberovskie-masshtaby-ispolzovanie-ab-initio-pri-rabote-s-hive-i-greenplum","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-07-08T23:42:11+00:00","article:modified_time":"2020-07-08T23:42:11+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"87609","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 13:47:29","updated":"2026-02-22 15:29:30","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/87609","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=87609"}],"version-history":[{"count":1,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/87609\/revisions"}],"predecessor-version":[{"id":162163,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/87609\/revisions\/162163"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/87610"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=87609"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=87609"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=87609"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}