{"id":55701,"date":"2020-01-26T00:00:00","date_gmt":"2020-01-25T21:00:00","guid":{"rendered":"https:\/\/prohoster.info\/blog\/blog_prohoster\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh"},"modified":"2020-02-18T14:03:50","modified_gmt":"2020-02-18T11:03:50","slug":"nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","title":{"rendered":"A na nevojitet nj\u00eb liqen t\u00eb dh\u00ebnash? \u00c7far\u00eb duhet b\u00ebr\u00eb me ruajtjen e t\u00eb dh\u00ebnave?","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Ky \u00ebsht\u00eb nj\u00eb artikull i p\u00ebrkthyer nga artikulli im n\u00eb medium \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/medium.com\/rock-your-data\/getting-started-with-data-lake-4bb13643f9\">Hyrja n\u00eb Data Lake<\/a><\/noindex>, q\u00eb rezultoi t\u00eb ishte mjaft popullor, ndoshta p\u00ebr shkak t\u00eb thjesht\u00ebsis\u00eb s\u00eb tij. Prandaj, vendosa ta shkruaj n\u00eb shqip dhe ta pasuroj pak, n\u00eb m\u00ebnyr\u00eb q\u00eb nj\u00eb person i zakonsh\u00ebm, i cili nuk \u00ebsht\u00eb specialist n\u00eb pun\u00ebn me t\u00eb dh\u00ebna, t\u00eb kuptoj\u00eb se \u00e7far\u00eb \u00ebsht\u00eb nj\u00eb depo t\u00eb dh\u00ebnash (DW), \u00e7far\u00eb \u00ebsht\u00eb nj\u00eb liqen t\u00eb dh\u00ebnash (Data Lake), dhe si bashk\u00ebjetojn\u00eb ato s\u00eb bashku. <\/p>\n<p>Pse vendosa t\u00eb shkruaj p\u00ebr liqenin e t\u00eb dh\u00ebnave? Kam punuar me t\u00eb dh\u00ebna dhe analiz\u00eb p\u00ebr m\u00eb shum\u00eb se 10 vjet, dhe tani po punoj me t\u00eb dh\u00ebna t\u00eb m\u00ebdha n\u00eb Amazon Alexa AI n\u00eb Kembrixh, q\u00eb ndodhet n\u00eb Boston, ndon\u00ebse un\u00eb vet\u00eb jetoj n\u00eb Viktori n\u00eb ishullin Vancouver dhe shpesh vizitoj Bostonin, Siattlin dhe Vancouverin, dhe ndonj\u00ebher\u00eb madje edhe n\u00eb Mosk\u00eb p\u00ebr t\u00eb folur n\u00eb konferenca. Gjithashtu, nga koha n\u00eb koh\u00eb shkruaj, por kryesisht n\u00eb anglisht, dhe kam shkruar tashm\u00eb <noindex><a rel=\"nofollow\" href=\"https:\/\/www.amazon.com\/Dmitry-Anoshin\/e\/B01A5PVT2M\">disa libra<\/a><\/noindex>, gjithashtu kam nj\u00eb nevoj\u00eb p\u00ebr t\u00eb ndar\u00eb trendet e analiz\u00ebs nga Amerika e Veriut dhe ndonj\u00ebher\u00eb shkruaj n\u00eb <noindex><a rel=\"nofollow\" href=\"https:\/\/t.me\/rockyourdata\">Telegram<\/a><\/noindex>.<\/p>\n<p>Kam punuar gjithmon\u00eb me depo t\u00eb dh\u00ebnash, dhe q\u00eb nga viti 2015 kam filluar t\u00eb punoj ngusht\u00eb me Amazon Web Services, madje kam kaluar n\u00eb analiz\u00ebn cloud (AWS, Azure, GCP). Kam v\u00ebzhguar evolucioni i zgjidhjeve p\u00ebr analiz\u00eb q\u00eb nga viti 2007 dhe madje kam punuar te nj\u00eb ofrues depo t\u00eb dh\u00ebnash, Teradata, dhe e kam implementuar at\u00eb n\u00eb Sberbank, kur at\u00ebher\u00eb nisi era e Big Data me Hadoop. T\u00eb gjith\u00eb filluan t\u00eb thon\u00eb se era e depozita p\u00ebrfundoi dhe tani gjith\u00e7ka \u00ebsht\u00eb n\u00eb Hadoop, dhe m\u00eb von\u00eb filluan t\u00eb flisnin p\u00ebr Data Lake, s\u00ebrish duke th\u00ebn\u00eb se tani me siguri i ka ardhur fundi depozit\u00ebs s\u00eb t\u00eb dh\u00ebnave. Por p\u00ebr fat t\u00eb mir\u00eb (ndoshta p\u00ebr disa fatkeq\u00ebsi, ata q\u00eb fitonin shum\u00eb para nga konfigurimi i Hadoop), depozit\u00eb e t\u00eb dh\u00ebnave nuk iku. <br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nN\u00eb k\u00ebt\u00eb artikel do t\u00eb shqyrtojm\u00eb se \u00e7far\u00eb \u00ebsht\u00eb nj\u00eb liqen t\u00eb dh\u00ebnash. Artikulli \u00ebsht\u00eb i destinuar p\u00ebr njer\u00ebzit, t\u00eb cil\u00ebt kan\u00eb pak p\u00ebrvoj\u00eb me depo t\u00eb dh\u00ebnash ose nuk kan\u00eb fare.<\/p>\n<p><img decoding=\"async\" alt=\"A na nevojitet nj\u00eb liqen t\u00eb dh\u00ebnash? \u00c7far\u00eb duhet b\u00ebr\u00eb me ruajtjen e t\u00eb dh\u00ebnave?\" src=\"\/wp-content\/uploads\/2020\/01\/85b3eda809ce19fc33efb2fad4e93a41.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nN\u00eb imazh \u00ebsht\u00eb liqeni Bled, nj\u00eb nga liqenet e mia t\u00eb preferuara, ndon\u00ebse kam qen\u00eb aty vet\u00ebm nj\u00ebher\u00eb, por e mbajta mend gjat\u00eb gjith\u00eb jet\u00ebs. Por ne do t\u00eb flasim p\u00ebr nj\u00eb tip tjet\u00ebr liqeni \u2014 liqeni i t\u00eb dh\u00ebnave. Ndoshta shum\u00eb prej jush kan\u00eb d\u00ebgjuar tashm\u00eb k\u00ebt\u00eb term, por nj\u00eb p\u00ebrcaktim tjet\u00ebr nuk do t\u00eb d\u00ebmtonte ask\u00ebnd.<\/p>\n<p>Para s\u00eb gjithash, ja p\u00ebrcaktimet m\u00eb t\u00eb njohura t\u00eb Liqenit t\u00eb T\u00eb Dh\u00ebnave:<\/p>\n<blockquote><p>\u00abdepo e skedar\u00ebve t\u00eb t\u00eb gjitha llojeve t\u00eb t\u00eb dh\u00ebnave t\u00eb pap\u00ebrpunuara, q\u00eb jan\u00eb t\u00eb arritshme p\u00ebr analiz\u00eb nga kushdo n\u00eb organizat\u00eb\u00bb \u2014 Martin Fowler.<\/p><\/blockquote>\n<blockquote><p>\u00abN\u00ebse mendoni se nj\u00eb liqen t\u00eb dh\u00ebnash \u00ebsht\u00eb si nj\u00eb shishe uji \u2014 t\u00eb pastruar, t\u00eb paketuar dhe t\u00eb ndar\u00eb p\u00ebr p\u00ebrdorim t\u00eb p\u00ebrshtatsh\u00ebm, at\u00ebher\u00eb liqeni i t\u00eb dh\u00ebnave \u00ebsht\u00eb nj\u00eb rezervuar i madh me uj\u00eb n\u00eb form\u00ebn e tij natyrore. P\u00ebrdoruesit mund t\u00eb marrin uj\u00eb p\u00ebr veten, t\u00eb zhytin thell\u00eb, t\u00eb eksplorojn\u00eb\u00bb \u2014 James Dickson. <\/p><\/blockquote>\n<p> Tani ne e dim\u00eb sakt\u00ebsisht se liqeni i t\u00eb dh\u00ebnave \u00ebsht\u00eb p\u00ebr analiz\u00ebn, ai na lejon t\u00eb ruajm\u00eb sasi t\u00eb m\u00ebdha t\u00eb dh\u00ebnash n\u00eb form\u00ebn e tyre origjinale dhe kemi akses t\u00eb nevojsh\u00ebm dhe t\u00eb p\u00ebrshtatsh\u00ebm n\u00eb t\u00eb dh\u00ebnat.<\/p>\n<p>Un\u00eb shpesh e dua ta thjeshtoj gj\u00ebrat, n\u00ebse mund t\u00eb p\u00ebrshkruaj nj\u00eb term t\u00eb nd\u00ebrlikuar me fjal\u00eb t\u00eb thjeshta, do t\u00eb thot\u00eb se e kam kuptuar si funksionon dhe p\u00ebr \u00e7far\u00eb \u00ebsht\u00eb e nevojshme. Nj\u00ebher\u00eb, duke p\u00ebrshkruar n\u00eb iPhone n\u00eb galerin\u00eb e fotografive, m\u00eb erdhi ndri\u00e7imi, k\u00ebshtu q\u00eb \u00ebsht\u00eb nj\u00eb liqen t\u00eb dh\u00ebnash i v\u00ebrtet\u00eb, madje b\u00ebra nj\u00eb slajd p\u00ebr konferenca:<\/p>\n<p><img decoding=\"async\" alt=\"A na nevojitet nj\u00eb liqen t\u00eb dh\u00ebnash? \u00c7far\u00eb duhet b\u00ebr\u00eb me ruajtjen e t\u00eb dh\u00ebnave?\" src=\"\/wp-content\/uploads\/2020\/01\/1956ddf4091ca00f6d86c33a6780d495.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nE gjith\u00e7ka \u00ebsht\u00eb shum\u00eb e thjesht\u00eb. Ne b\u00ebjm\u00eb nj\u00eb fotografi me telefon, fotografia ruhet n\u00eb telefon dhe mund t\u00eb ruhet n\u00eb iCloud (ruajtja e skedar\u00ebve n\u00eb re). Gjithashtu, telefoni grumbullon metadata t\u00eb fotografis\u00eb: \u00e7far\u00eb \u00ebsht\u00eb e paraqitur, gjeoetiketa, koha. Si rezultat, ne mund t\u00eb p\u00ebrdorim nd\u00ebrfaqen e p\u00ebrshtatshme t\u00eb iPhone p\u00ebr t\u00eb gjetur fotografin\u00eb ton\u00eb dhe madje shohim treguesit, p\u00ebr shembull, kur k\u00ebrkoj fotografit\u00eb me fjal\u00ebn zjarr (fire), un\u00eb gjej 3 fotografi me imazhin e zjarrit. P\u00ebr mua, \u00ebsht\u00eb thjesht si nj\u00eb mjet Business Intelligence q\u00eb funksionon shum\u00eb shpejt dhe sakt\u00eb. <\/p>\n<p>Dhe sigurisht, nuk duhet t\u00eb harrojm\u00eb p\u00ebr sigurin\u00eb (autorizimin dhe autentikimin), p\u00ebrndryshe t\u00eb dh\u00ebnat tona mund t\u00eb p\u00ebrfundojn\u00eb leht\u00ebsisht n\u00eb akses t\u00eb hapur. Ka shum\u00eb lajme p\u00ebr korporata t\u00eb m\u00ebdha dhe startup-e, t\u00eb cilat t\u00eb dh\u00ebnat e tyre p\u00ebrfunduan n\u00eb akses t\u00eb hapur p\u00ebr shkak t\u00eb neglizhenc\u00ebs s\u00eb zhvilluesve dhe mosrespektimit t\u00eb rregullave t\u00eb thjeshta.<\/p>\n<p>Madje nj\u00eb imazh kaq i thjesht\u00eb na ndihmon t\u00eb kuptojm\u00eb se \u00e7far\u00eb \u00ebsht\u00eb nj\u00eb liqen t\u00eb dh\u00ebnash, dallimet e tij nga magazina tradicionale e t\u00eb dh\u00ebnave dhe elementet e tij kryesore:<\/p>\n<ol>\n<li><b>Ngarkimi i t\u00eb dh\u00ebnave <\/b>(Ingestion) \u2014 komponenti ky\u00e7 i liqenit t\u00eb t\u00eb dh\u00ebnave. T\u00eb dh\u00ebnat mund t\u00eb futen n\u00eb magazin\u00ebn e t\u00eb dh\u00ebnave n\u00eb dy m\u00ebnyra \u2014 batch (ngarkes\u00eb me intervale) dhe streaming (rrjedh\u00eb t\u00eb dh\u00ebnash).<\/li>\n<li><b>Ruajtja e skedar\u00ebve<\/b> (Storage) \u2014 komponenti kryesor i Liqenit t\u00eb T\u00eb Dh\u00ebnave. Na nevojitet q\u00eb magazina t\u00eb jet\u00eb e leht\u00eb p\u00ebr t'u zgjeruar, jasht\u00ebzakonisht e besueshme dhe me kosto t\u00eb ul\u00ebt. P\u00ebr shembull, n\u00eb AWS kjo \u00ebsht\u00eb S3.<\/li>\n<li><b>Katalogu dhe K\u00ebrkimi<\/b> (Katalogu dhe K\u00ebrkimi) \u2014 p\u00ebr t\u00eb shmangur Bolot\u00ebn e T\u00eb Dh\u00ebnave (kur ne grumbullojm\u00eb t\u00eb gjitha t\u00eb dh\u00ebnat n\u00eb nj\u00eb vend dhe pastaj \u00ebsht\u00eb e pamundur t\u00eb punojm\u00eb me to), ne duhet t\u00eb krijojm\u00eb nj\u00eb shtres\u00eb t\u00eb metadatat p\u00ebr klasifikimin e t\u00eb dh\u00ebnave, n\u00eb m\u00ebnyr\u00eb q\u00eb p\u00ebrdoruesit t\u00eb mund ta gjejn\u00eb leht\u00ebsisht informacionin e nevojsh\u00ebm p\u00ebr analiz\u00eb. S\u00eb additional, mund t\u00eb p\u00ebrdorim zgjidhje t\u00eb tjera k\u00ebrkimi, si ElasticSearch. K\u00ebrkimi ndihmon p\u00ebrdoruesin t\u00eb k\u00ebrkoj\u00eb t\u00eb dh\u00ebnat e nevojshme p\u00ebrmes nj\u00eb nd\u00ebrfaqeje t\u00eb p\u00ebrshtatshme.<\/li>\n<li><b>P\u00ebrgjigje<\/b> (Procesi) \u2014 ky hap \u00ebsht\u00eb p\u00ebrgjegj\u00ebs p\u00ebr p\u00ebrpunimin dhe transformimin e t\u00eb dh\u00ebnave. Ne mund t\u00eb transformojm\u00eb t\u00eb dh\u00ebnat, t\u00eb ndryshojm\u00eb strukturat e tyre, t'i pastrojm\u00eb dhe shum\u00eb m\u00eb tep\u00ebr. <\/li>\n<li><b>Siguria<\/b> (Siguria) \u2014 \u00ebsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb shpenzohet koh\u00eb p\u00ebr dizajnimin e siguris\u00eb s\u00eb zgjidhjes. P\u00ebr shembull, kriptimi i t\u00eb dh\u00ebnave gjat\u00eb magazinimit, p\u00ebrpunimit dhe ngarkimit. \u00cbsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb p\u00ebrdoren metoda t\u00eb autentifikimit dhe autorizimit. N\u00eb p\u00ebrfundim, nevojitet nj\u00eb mjet auditimi.<\/li>\n<\/ol>\n<p>\nNga nj\u00eb pik\u00ebpamje praktike, ne mund ta karakterizojm\u00eb liqenin e t\u00eb dh\u00ebnave me tre atribute:<\/p>\n<ol>\n<li><b>Mblidhni dhe ruani gjith\u00e7ka<\/b> \u2014 liqeni i t\u00eb dh\u00ebnave p\u00ebrmban t\u00eb gjitha t\u00eb dh\u00ebnat, si t\u00eb dh\u00ebna t\u00eb pap\u00ebrpunuara p\u00ebr \u00e7do periudh\u00eb kohe, ashtu edhe t\u00eb dh\u00ebna t\u00eb p\u00ebrpunuara\/t\u00eb pastruara.<\/li>\n<li><b>Analiza e thell\u00eb<\/b> \u2014 liqeni i t\u00eb dh\u00ebnave u lejon p\u00ebrdoruesve t\u00eb eksploros dhe t\u00eb analizojn\u00eb t\u00eb dh\u00ebnat.<\/li>\n<li><b>Qasje fleksibile<\/b> \u2014 liqeni i t\u00eb dh\u00ebnave ofron qasje fleksibile p\u00ebr t\u00eb dh\u00ebna t\u00eb ndryshme dhe p\u00ebr skenar\u00eb t\u00eb ndrysh\u00ebm.<\/li>\n<\/ol>\n<p>\nTani mund t\u00eb flasim p\u00ebr ndryshimin mes magazin\u00ebs s\u00eb t\u00eb dh\u00ebnave dhe liqenit t\u00eb t\u00eb dh\u00ebnave. Zakonisht, njer\u00ebzit pyesin:<\/p>\n<ul>\n<li>Po p\u00ebrsa i p\u00ebrket magazin\u00ebs s\u00eb t\u00eb dh\u00ebnave?<\/li>\n<li>A e z\u00ebvend\u00ebsojm\u00eb magazin\u00ebn e t\u00eb dh\u00ebnave me liqenin e t\u00eb dh\u00ebnave apo e zgjerojm\u00eb at\u00eb?<\/li>\n<li>A mund t\u00eb shkojm\u00eb pa liqenin e t\u00eb dh\u00ebnave?<\/li>\n<\/ul>\n<p>\nN\u00ebse do t\u00eb flasim shkurt, nuk ka nj\u00eb p\u00ebrgjigje t\u00eb qart\u00eb. \u00c7do gj\u00eb varet nga situata specifike, aft\u00ebsit\u00eb n\u00eb ekip dhe buxheti. P\u00ebr shembull, migrohet magazina e t\u00eb dh\u00ebnave n\u00eb Oracle n\u00eb AWS dhe krijimi i liqenit t\u00eb t\u00eb dh\u00ebnave nga nj\u00eb kompani n\u00ebnshkruese e Amazon \u2014 Woot \u2014 <noindex><a rel=\"nofollow\" href=\"https:\/\/aws.amazon.com\/blogs\/big-data\/our-data-lake-story-how-woot-com-built-a-serverless-data-lake-on-aws\/\">Historia jon\u00eb e liqenit t\u00eb t\u00eb dh\u00ebnave: Si Woot.com ndihmoni p\u00ebr nd\u00ebrtimin e nj\u00eb liqeni t\u00eb dh\u00ebnash pa server\u00eb n\u00eb AWS<\/a><\/noindex>. <\/p>\n<p>Nga ana tjet\u00ebr, ofruesi Snowflake deklaron se nuk ka nevoj\u00eb t\u00eb mendoni m\u00eb p\u00ebr liqenin e t\u00eb dh\u00ebnave, pasi platforma e tij t\u00eb dh\u00ebnash (deri n\u00eb 2020 ishte magazina e t\u00eb dh\u00ebnave) ju lejon t\u00eb kombinoni si liqenin e t\u00eb dh\u00ebnave ashtu edhe magazin\u00ebn e t\u00eb dh\u00ebnave. Kam punuar pak me Snowflake, dhe \u00ebsht\u00eb v\u00ebrtet nj\u00eb produkt unik q\u00eb mund ta b\u00ebj\u00eb k\u00ebt\u00eb. \u00c7mimi \u00ebsht\u00eb nj\u00eb \u00e7\u00ebshtje tjet\u00ebr.<\/p>\n<p>N\u00eb p\u00ebrfundim, mendimi im personal \u00ebsht\u00eb se na nevojitet akoma nj\u00eb depo e dh\u00ebnash si burim kryesor t\u00eb dh\u00ebnash p\u00ebr raportimin ton\u00eb, dhe gjith\u00e7ka q\u00eb nuk p\u00ebrfshihet, e ruajm\u00eb n\u00eb liqenin e t\u00eb dh\u00ebnave. Roli i analitik\u00ebs \u00ebsht\u00eb t\u00eb ofroj\u00eb qasje t\u00eb leht\u00eb p\u00ebr biznesin p\u00ebr marrjen e vendimeve. Si\u00e7 do t\u00eb donim, p\u00ebrdoruesit e biznesit punojn\u00eb m\u00eb efikasht me depo t\u00eb dh\u00ebnash sesa me liqenin e t\u00eb dh\u00ebnave, p\u00ebr shembull n\u00eb Amazon \u2014 kemi Redshift (depo analitike e t\u00eb dh\u00ebnave) dhe Redshift Spectrum\/Athena (interfaci SQL p\u00ebr liqenin e t\u00eb dh\u00ebnave n\u00eb S3 mbi baz\u00ebn e Hive\/Presto). E nj\u00ebjta gj\u00eb i referohet depozita moderne t\u00eb t\u00eb dh\u00ebnave analitike.<\/p>\n<p>Le t\u00eb shqyrtojm\u00eb arkitektur\u00ebn tipike t\u00eb depozit\u00ebs s\u00eb t\u00eb dh\u00ebnave:<\/p>\n<p><img decoding=\"async\" alt=\"A na nevojitet nj\u00eb liqen t\u00eb dh\u00ebnash? \u00c7far\u00eb duhet b\u00ebr\u00eb me ruajtjen e t\u00eb dh\u00ebnave?\" src=\"\/wp-content\/uploads\/2020\/01\/a2015f7f5e28e8a671699682105e011e.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKjo \u00ebsht\u00eb nj\u00eb zgjidhje klasike. Ne kemi sistemet burimore, me ndihm\u00ebn e ETL\/ELT ne kopjojm\u00eb t\u00eb dh\u00ebnat n\u00eb depo t\u00eb dh\u00ebnash analitike dhe i lidhim me zgjidhjen e Business Intelligence (preferenca ime \u00ebsht\u00eb Tableau, e juaja?). <\/p>\n<p>Kjo zgjidhje ka k\u00ebto disavantazhe:<\/p>\n<ul>\n<li>Operacionet ETL\/ELT k\u00ebrkojn\u00eb koh\u00eb dhe burime.<\/li>\n<li>Si rregull, kuadri p\u00ebr ruajtjen e t\u00eb dh\u00ebnave n\u00eb depo t\u00eb dh\u00ebnash analitike nuk \u00ebsht\u00eb i lir\u00eb (p\u00ebr shembull Redshift, BigQuery, Teradata), pasi ne duhet t\u00eb blejm\u00eb nj\u00eb klaster t\u00eb t\u00ebr\u00eb.<\/li>\n<li>P\u00ebrdoruesit e biznesit kan\u00eb qasje n\u00eb t\u00eb dh\u00ebnat e pastruara dhe shpesh t\u00eb agreguara dhe nuk kan\u00eb mund\u00ebsi t\u00eb marrin t\u00eb dh\u00ebna t\u00eb pap\u00ebrpunuara.<\/li>\n<\/ul>\n<p>\nSigurisht, gjith\u00e7ka varet nga rasti juaj. N\u00ebse nuk keni probleme me depozitat tuaja t\u00eb dh\u00ebnash, at\u00ebher\u00eb nuk keni nevoj\u00eb p\u00ebr nj\u00eb liqen t\u00eb dh\u00ebnash. Por kur paraqiten problemet me munges\u00ebn e hap\u00ebsir\u00ebs, fuqis\u00eb ose kostoja ka nj\u00eb rol ky\u00e7, at\u00ebher\u00eb mund t\u00eb shqyrtoni mund\u00ebsin\u00eb e nj\u00eb liqeni t\u00eb dh\u00ebnash. Kjo \u00ebsht\u00eb arsyeja pse, liqeni i t\u00eb dh\u00ebnave \u00ebsht\u00eb shum\u00eb popullor. Ja nj\u00eb shembull i arkitektur\u00ebs s\u00eb liqenit t\u00eb t\u00eb dh\u00ebnave:<br \/>\n<img decoding=\"async\" alt=\"A na nevojitet nj\u00eb liqen t\u00eb dh\u00ebnash? \u00c7far\u00eb duhet b\u00ebr\u00eb me ruajtjen e t\u00eb dh\u00ebnave?\" src=\"\/wp-content\/uploads\/2020\/01\/c952e2202a6ce2c8d7d91215aa5390cc.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\nDuke p\u00ebrdorur qasjen e liqenit t\u00eb t\u00eb dh\u00ebnave, ne ngarkojm\u00eb t\u00eb dh\u00ebnat e pap\u00ebrpunuara n\u00eb liqenin ton\u00eb t\u00eb t\u00eb dh\u00ebnave (batch ose streaming), dhe m\u00eb pas procesojm\u00eb t\u00eb dh\u00ebnat sipas nevoj\u00ebs. Liqeni i t\u00eb dh\u00ebnave lejon q\u00eb p\u00ebrdoruesit e biznesit t\u00eb krijojn\u00eb transformimet e tyre t\u00eb dh\u00ebnave (ETL\/ELT) ose t\u00eb analizojn\u00eb t\u00eb dh\u00ebnat n\u00eb zgjidhjet e Business Intelligence (n\u00ebse ka drejtuesin e nevojsh\u00ebm).<\/p>\n<blockquote><p>Q\u00ebllimi i \u00e7do zgjidhjeje analitike \u00ebsht\u00eb t\u00eb sh\u00ebrbej\u00eb p\u00ebr p\u00ebrdoruesit e biznesit. Prandaj ne gjithmon\u00eb duhet t\u00eb punojm\u00eb nga k\u00ebrkesat e biznesit. (N\u00eb Amazon, ky \u00ebsht\u00eb nj\u00eb nga parimet \u2014 punimi mbrapsht).<\/p><\/blockquote>\n<p> Duke punuar dhe me depozit\u00ebn e dh\u00ebnave dhe me liqenin e t\u00eb dh\u00ebnave, ne mund t\u00eb krahasojm\u00eb t\u00eb dyja zgjidhjet:<\/p>\n<p><img decoding=\"async\" alt=\"A na nevojitet nj\u00eb liqen t\u00eb dh\u00ebnash? \u00c7far\u00eb duhet b\u00ebr\u00eb me ruajtjen e t\u00eb dh\u00ebnave?\" src=\"\/wp-content\/uploads\/2020\/01\/07b62ac9838438a0b28caf1b22b2dccd.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nKonkluzioni kryesor q\u00eb mund t\u00eb b\u00ebhet \u00ebsht\u00eb se depoja e t\u00eb dh\u00ebnave nuk konkuron me liqenin e t\u00eb dh\u00ebnave, p\u00ebrkundrazi, e plot\u00ebson at\u00eb. Por kjo \u00ebsht\u00eb n\u00eb dor\u00ebn tuaj se \u00e7far\u00eb i p\u00ebrshtatet rastit tuaj. Gjithmon\u00eb \u00ebsht\u00eb interesante ta provosh vet\u00eb dhe t\u00eb b\u00ebsh p\u00ebrfundime t\u00eb sakta.<\/p>\n<p>Dua gjithashtu t\u00eb flas p\u00ebr nj\u00eb nga rastet kur fillova t\u00eb p\u00ebrdor qasjen e liqenit t\u00eb t\u00eb dh\u00ebnave. E gjith\u00eb kjo \u00ebsht\u00eb mjaft banale, p\u00ebrpiqesha t\u00eb p\u00ebrdorja nj\u00eb mjet ELT (ne kishim Matillion ETL) dhe Amazon Redshift, zgjidhja ime funksionoi, por nuk p\u00ebrmbushte k\u00ebrkesat.<\/p>\n<p>M\u00eb nevojitej t\u00eb merrja log-et e uebit, t'i transformoja dhe t'i agregoja, p\u00ebr t\u00eb ofruar t\u00eb dh\u00ebna p\u00ebr dy raste:<\/p>\n<ol>\n<li>Ekipa e marketingut donte t\u00eb analizonin aktivitetin e bot\u00ebve p\u00ebr SEO<\/li>\n<li>IT-i donte t\u00eb shikonte metrikat e funksionimit t\u00eb faqeve t\u00eb internetit<\/li>\n<\/ol>\n<p>\nLog-e shum\u00eb t\u00eb thjeshta, mjaft t\u00eb thjeshta. Ja nj\u00eb shembull:<\/p>\n<pre><code class=\"plaintext\">https 2018-07-02T22:23:00.186641Z app\/my-loadbalancer\/50dc6c495c0c9188 \n192.168.131.39:2817 10.0.0.1:80 0.086 0.048 0.037 200 200 0 57 \n\"GET https:\/\/www.example.com:443\/ HTTP\/1.1\" \"curl\/7.46.0\" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 \narn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup\/my-targets\/73e2d6bc24d8a067\n\"Root=1-58337281-1d84f3d73c47ec4e58577259\" \"www.example.com\" \"arn:aws:acm:us-east-2:123456789012:certificate\/12345678-1234-1234-1234-123456789012\"\n1 2018-07-02T22:22:48.364000Z \"authenticate,forward\" \"-\" \"-\"<\/code><\/pre>\n<p>\nNj\u00eb skedar peshoi 1-4 megabajt.<\/p>\n<p>Por kishte nj\u00eb v\u00ebshtir\u00ebsi. Ne kishim 7 domain n\u00eb mbar\u00eb bot\u00ebn, dhe n\u00eb nj\u00eb dit\u00eb krijoheshin 7000 skedar\u00eb. Kjo nuk ishte nj\u00eb volum shum\u00eb i madh, gjith\u00e7ka 50 gigabajt. Por madh\u00ebsia e klasterit ton\u00eb Redshift ishte gjithashtu e vog\u00ebl (4 nod). Ngarkimi n\u00eb m\u00ebnyr\u00eb tradicionale i nj\u00eb skedari merrte rreth nj\u00eb minut\u00eb. Pra, n\u00eb m\u00ebnyr\u00eb t\u00eb drejtp\u00ebrdrejt\u00eb, problemi nuk zgjidhej. Dhe kjo ishte ajo her\u00eb q\u00eb vendosa t\u00eb p\u00ebrdor qasjen e liqenit t\u00eb t\u00eb dh\u00ebnave. Zgjidhja dukej p\u00ebraf\u00ebrsisht k\u00ebshtu:<\/p>\n<p><img decoding=\"async\" alt=\"A na nevojitet nj\u00eb liqen t\u00eb dh\u00ebnash? \u00c7far\u00eb duhet b\u00ebr\u00eb me ruajtjen e t\u00eb dh\u00ebnave?\" src=\"\/wp-content\/uploads\/2020\/01\/21ca33e82da56f83b3deab4c32eb2345.png\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIshte mjaft e thjesht\u00eb (dua t\u00eb theksoj se p\u00ebrpar\u00ebsia e pun\u00ebs n\u00eb re \u00ebsht\u00eb thjesht\u00ebsia). Un\u00eb p\u00ebrdora:<\/p>\n<ul>\n<li>AWS Elastic Map Reduce (Hadoop) si kapacitet llogarit\u00ebs <\/li>\n<li>AWS S3 si depo e skedave me mund\u00ebsi p\u00ebr kriptimin e t\u00eb dh\u00ebnave dhe ndarjen e aksesit<\/li>\n<li>Spark si kapacitet llogarit\u00ebs InMemory dhe PySpark p\u00ebr logjik\u00ebn dhe transformimin e t\u00eb dh\u00ebnave<\/li>\n<li>Parquet si rezultat i pun\u00ebs s\u00eb Spark<\/li>\n<li>AWS Glue Crawler si mbledh\u00ebs i metadatumit p\u00ebr t\u00eb dh\u00ebna t\u00eb reja dhe partit\u00eb<\/li>\n<li>Redshift Spectrum si nd\u00ebrfaqe SQL p\u00ebr liqenin e t\u00eb dh\u00ebnave p\u00ebr p\u00ebrdoruesit ekzistues t\u00eb Redshift<\/li>\n<\/ul>\n<p>\nKlasteri m\u00eb i vog\u00ebl EMR+Spark p\u00ebrpunoi t\u00eb gjith\u00eb grupin e skedar\u00ebve p\u00ebr 30 minuta. Ka dhe raste t\u00eb tjera p\u00ebr AWS, sidomos shum\u00eb t\u00eb lidhura me Alexa, ku ka shum\u00eb t\u00eb dh\u00ebna. <\/p>\n<p>Koh\u00eb m\u00eb par\u00eb m\u00ebsova nj\u00eb nga disavantazhet e liqenit t\u00eb t\u00eb dh\u00ebnave \u2013 \u00ebsht\u00eb GDPR. Problemi \u00ebsht\u00eb se kur klienti k\u00ebrkon ta fshij\u00eb, nd\u00ebrsa t\u00eb dh\u00ebnat gjenden n\u00eb nj\u00eb nga skedar\u00ebt, nuk mund t\u00eb p\u00ebrdorim Data Manipulation Language dhe operacionin DELETE si n\u00eb nj\u00eb baz\u00eb t\u00eb dh\u00ebnash.<\/p>\n<p>Shpresoj se artikulli sqaroi ndryshimin midis depozit\u00ebs s\u00eb t\u00eb dh\u00ebnave dhe liqenit t\u00eb t\u00eb dh\u00ebnave. N\u00ebse ishte interesante, mund t\u00eb p\u00ebrkthej edhe artikujt e mi ose artikujt e profesionist\u00ebve q\u00eb lexoj. Po ashtu mund t\u00eb flas p\u00ebr zgjidhjet me t\u00eb cilat punoj dhe arkitektur\u00ebn e tyre.<br \/>\n<br \/>Burimi: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/485180\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u043e\u043a\u0430\u0437\u0430\u043b\u0430\u0441\u044c \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u043e\u0439, \u043d\u0430\u0432\u0435\u0440\u043d\u043e\u0435 \u0438\u0437-\u0437\u0430 \u0441\u0432\u043e\u0435\u0439 \u043f\u0440\u043e\u0441\u0442\u043e\u0442\u044b. \u041f\u043e\u044d\u0442\u043e\u043c\u0443 \u044f \u0440\u0435\u0448\u0438\u043b \u043d\u0430\u043f\u0438\u0441\u0430\u0442\u044c \u0435\u0435 \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u043e\u043c \u044f\u0437\u044b\u043a\u0435 \u0438 \u043d\u0435\u043c\u043d\u043e\u0433\u043e \u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u044c, \u0447\u0442\u043e\u0431\u044b \u043f\u0440\u043e\u0441\u0442\u043e\u043c\u0443 \u0447\u0435\u043b\u043e\u0432\u0435\u043a\u0443, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043d\u0435 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u0438\u0441\u0442\u043e\u043c \u043f\u043e \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 \u0434\u0430\u043d\u043d\u044b\u043c\u0438 \u0441\u0442\u0430\u043b\u043e \u043f\u043e\u043d\u044f\u0442\u043d\u043e, \u0447\u0442\u043e \u0442\u0430\u043a\u043e\u0435 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435 \u0434\u0430\u043d\u043d\u044b\u0445 (DW), \u0430 \u0447\u0442\u043e \u0442\u0430\u043a\u043e\u0435 \u043e\u0437\u0435\u0440\u043e \u0434\u0430\u043d\u043d\u044b\u0445 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-55701","post","type-post","status-publish","format-standard","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"sq_AL\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041d\u0443\u0436\u043d\u043e \u043b\u0438 \u043d\u0430\u043c \u043e\u0437\u0435\u0440\u043e \u0434\u0430\u043d\u043d\u044b\u0445? \u0410 \u0447\u0442\u043e \u0434\u0435\u043b\u0430\u0442\u044c \u0441 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435\u043c \u0434\u0430\u043d\u043d\u044b\u0445? | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-01-25T21:00:00+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-02-18T11:03:50+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47A na nevojitet liqeni i t\u00eb dh\u00ebnave? \u00c7far\u00eb t\u00eb b\u00ebjm\u00eb me depozit\u00ebn e t\u00eb dh\u00ebnave? | ProHoster","description":"Ky \u00ebsht\u00eb nj\u00eb artikull p\u00ebrkthim i artikullit tim n\u00eb medium \u2013 Getting Started with Data Lake.","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041d\u0443\u0436\u043d\u043e \u043b\u0438 \u043d\u0430\u043c \u043e\u0437\u0435\u0440\u043e \u0434\u0430\u043d\u043d\u044b\u0445? \u0410 \u0447\u0442\u043e \u0434\u0435\u043b\u0430\u0442\u044c \u0441 \u0445\u0440\u0430\u043d\u0438\u043b\u0438\u0449\u0435\u043c \u0434\u0430\u043d\u043d\u044b\u0445? | ProHoster","og:description":"\u042d\u0442\u043e \u0441\u0442\u0430\u0442\u044c\u044f \u043f\u0435\u0440\u0435\u0432\u043e\u0434 \u043c\u043e\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0438 \u043d\u0430 medium \u2014 Getting Started with Data Lake.","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/nuzhno-li-nam-ozero-dannyh-a-chto-delat-s-hranilishhem-dannyh","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-01-25T21:00:00+00:00","article:modified_time":"2020-02-18T11:03:50+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"55701","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 19:38:32","updated":"2022-09-27 20:27:01","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/55701","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=55701"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/55701\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=55701"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=55701"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=55701"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}