{"id":33630,"date":"2019-10-31T21:53:49","date_gmt":"2019-10-31T18:53:49","guid":{"rendered":"https:\/\/prohoster.info\/blog\/chto-osobennogo-v-cloudera-i-kak-ee-gotovit\/"},"modified":"2019-10-31T21:53:49","modified_gmt":"2019-10-31T18:53:49","slug":"chto-osobennogo-v-cloudera-i-kak-ee-gotovit","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/chto-osobennogo-v-cloudera-i-kak-ee-gotovit","title":{"rendered":"\u00c7far\u00eb \u00ebsht\u00eb e ve\u00e7ant\u00eb te Cloudera dhe si ta p\u00ebrgatisim","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Tregu i llogarive t\u00eb shp\u00ebrndara dhe t\u00eb dh\u00ebnave t\u00eb m\u00ebdha, n\u00ebse besohet, statistikave<\/a><\/noindex>, rritet me 18-19% n\u00eb vit. Kjo do t\u00eb thot\u00eb se pyetja e zgjedhjes s\u00eb softuerit p\u00ebr k\u00ebto q\u00ebllime mbetet aktuale. N\u00eb k\u00ebt\u00eb postim, do t\u00eb fillojm\u00eb me arsyen pse jan\u00eb t\u00eb nevojshme llogarit\u00eb e shp\u00ebrndara, do t\u00eb ndalemi m\u00eb n\u00eb detaje n\u00eb zgjedhjen e softuerit, do t\u00eb flasim p\u00ebr p\u00ebrdorimin e Hadoop me Cloudera dhe n\u00eb fund do t\u00eb diskutojm\u00eb p\u00ebr zgjedhjen e harduerit dhe p\u00ebr m\u00ebnyrat se si ndikon n\u00eb performanc\u00eb.<\/p>\n

\"\u00c7far\u00eb<\/a><\/noindex>
\nPse jan\u00eb t\u00eb nevojshme llogarit\u00eb e shp\u00ebrndara n\u00eb biznesin e zakonsh\u00ebm? K\u00ebtu \u00ebsht\u00eb e thjesht\u00eb dhe e komplikuar nj\u00ebkoh\u00ebsisht. E thjesht\u00eb \u2014 sepse n\u00eb shumic\u00ebn e rasteve, ne b\u00ebjm\u00eb llogaritje relativisht t\u00eb thjeshta p\u00ebr \u00e7do nj\u00ebsi informacioni. E komplikuar \u2014 sepse ka shum\u00eb t\u00eb till\u00eb. Shum\u00eb shum\u00eb. Si pasoj\u00eb, na duhet t\u00eb p\u00ebrpunojm\u00eb tera bayt t\u00eb dh\u00ebnash n\u00eb 1000 kanale<\/a><\/noindex>. K\u00ebshtu, skenar\u00ebt e p\u00ebrdorimit jan\u00eb mjaft universale: llogarit\u00eb mund t\u00eb aplikohen kudo ku k\u00ebrkohet t\u00eb merren parasysh nj\u00eb num\u00ebr i madh metrikash mbi nj\u00eb masiv edhe m\u00eb t\u00eb madh t\u00eb dh\u00ebnash.<\/p>\n

Nj\u00eb nga shembujt e fundit: rrjeti i picerive Dodo Pizza kryesoi<\/a><\/noindex> n\u00eb baz\u00eb t\u00eb analiz\u00ebs s\u00eb baz\u00ebs s\u00eb t\u00eb dh\u00ebnave t\u00eb porosive t\u00eb klient\u00ebve, se kur zgjedhin nj\u00eb pic\u00eb me p\u00ebrb\u00ebrje t\u00eb rast\u00ebsishme, p\u00ebrdoruesit zakonisht operojn\u00eb vet\u00ebm me gjasht\u00eb grupe themelore p\u00ebrb\u00ebr\u00ebsish plus disa t\u00eb rast\u00ebsish\u00ebm. N\u00eb p\u00ebrputhje me k\u00ebt\u00eb, piceria rregulloi blerjet. P\u00ebr m\u00eb tep\u00ebr, ajo arriti t\u00eb rekomandoj\u00eb m\u00eb mir\u00eb p\u00ebrdoruesve produkte shtes\u00eb t\u00eb ofruara n\u00eb faz\u00ebn e porosis\u00eb, \u00e7ka rriti fitimin.<\/p>\n

Nj\u00eb tjet\u00ebr shembull: bashk\u00eb me t\u00eb gjitha referencat dhe<\/a><\/noindex> reduktimi i numrit t\u00eb artikujve ndihmoi dyqanin H&M t\u00eb zvog\u00ebloj\u00eb asortimentin n\u00eb disa dyqane me 40%, duke ruajtur k\u00ebshtu nivelin e shitjeve. Kjo u arrit duke p\u00ebrjashtuar pozitat q\u00eb shiteshin keq, p\u00ebr m\u00eb tep\u00ebr, n\u00eb llogaritje u mor parasysh sezonaliteti.<\/p>\n

Zgjedhja e mjetit<\/h2>\n

\nStandarti i industris\u00eb p\u00ebr llogarit\u00eb e k\u00ebtij lloji \u00ebsht\u00eb Hadoop. Pse? Sepse Hadoop \u00ebsht\u00eb nj\u00eb framework i shk\u00eblqyer, i dokumentuar mir\u00eb (po ashtu, Habr publikojn\u00eb shum\u00eb artikuj t\u00eb detajuar mbi k\u00ebt\u00eb tem\u00eb), q\u00eb shoq\u00ebrohet me nj\u00eb set t\u00eb t\u00ebr\u00eb utilitaresh dhe librarish. Ju mund t\u00eb jepni si input grupe t\u00eb m\u00ebdha t\u00eb t\u00eb dh\u00ebnave si t\u00eb strukturuara ashtu edhe t\u00eb pa-struktuara, dhe sistemi do t\u2019i shp\u00ebrndaj\u00eb ato vet\u00eb mes kapaciteteve llogaritative. M\u00eb tej, k\u00ebto kapacitete mund t\u00eb rriten ose \u00e7aktivizohen n\u00eb \u00e7do moment \u2014 ajo \u00ebsht\u00eb horizontale e shkall\u00ebzueshm\u00ebris\u00eb n\u00eb veprim. <\/p>\n

N\u00eb vitin 2017, kompania e njohur e konsulenc\u00ebs Gartner n\u00ebnshkroi<\/a><\/noindex>, q\u00eb Hadoop do t\u00eb shkoj\u00eb drejt zhdukjes. Arsyeja \u00ebsht\u00eb mjaft e thjesht\u00eb: analist\u00ebt besojn\u00eb se kompanit\u00eb do t\u00eb fillojn\u00eb t\u00eb migrojn\u00eb masivisht n\u00eb cloud, sepse atje ata do t\u00eb mund t\u00eb paguajn\u00eb p\u00ebr p\u00ebrdorimin real t\u00eb kapaciteteve kompjuterike. Faktori i dyt\u00eb i r\u00ebnd\u00ebsish\u00ebm, i cili p\u00ebrflitet se mund t\u00eb dhemb\u00eb Hadoop \u2014 \u00ebsht\u00eb shpejt\u00ebsia e funksionimit. Sepse alternativa si Apache Spark ose Google Cloud DataFlow punojn\u00eb m\u00eb shpejt se MapReduce, q\u00eb \u00ebsht\u00eb baza e Hadoop.<\/p>\n

Hadoop mb\u00ebshtetet n\u00eb disa k\u00ebrkesa, nd\u00ebr t\u00eb cilat m\u00eb t\u00eb dukshme jan\u00eb teknologjit\u00eb MapReduce (sistemi i shp\u00ebrndarjes s\u00eb t\u00eb dh\u00ebnave p\u00ebr kalkulimet nd\u00ebrmjet server\u00ebve) dhe sistemi i skedar\u00ebve HDFS. Ky i fundit \u00ebsht\u00eb krijuar posa\u00e7\u00ebrisht p\u00ebr ruajtjen e informacionit t\u00eb shp\u00ebrndar\u00eb nd\u00ebrmjet nyjave t\u00eb klasterit: \u00e7do bllok me madh\u00ebsi t\u00eb caktuar mund t\u00eb vendoset n\u00eb disa nyje, dhe p\u00ebrmes replikimit sigurohet q\u00ebndrueshm\u00ebria e sistemit ndaj d\u00ebshtimeve t\u00eb nyjave t\u00eb caktuara. N\u00eb vend t\u00eb tabel\u00ebs s\u00eb skedar\u00ebve p\u00ebrdoret nj\u00eb server i ve\u00e7ant\u00eb, i quajtur NameNode.<\/p>\n

N\u00eb ilustrimin m\u00eb posht\u00eb \u00ebsht\u00eb paraqitur skema e pun\u00ebs s\u00eb MapReduce. N\u00eb faz\u00ebn e par\u00eb, t\u00eb dh\u00ebnat ndahen sipas nj\u00eb kritere t\u00eb caktuar, n\u00eb t\u00eb dyt\u00ebn \u2014 shp\u00ebrndahen sipas kapaciteteve kompjuterike, n\u00eb t\u00eb tret\u00ebn \u2014 ndodhin llogaritjet. <\/p>\n

\"\u00c7far\u00eb
\nFillimisht, MapReduce u krijua nga Google p\u00ebr nevojat e k\u00ebrkimit t\u00eb saj. M\u00eb pas, MapReduce kaloi n\u00eb kodin e lir\u00eb dhe projekti u mor nga Apache. Nd\u00ebrsa Google gradualisht kaloi n\u00eb zgjidhje t\u00eb tjera. Nj\u00eb detaj interesant: n\u00eb momentin aktual, Google ka nj\u00eb projekt t\u00eb quajtur Google Cloud Dataflow, e cila pozicionohet si hapi tjet\u00ebr pas Hadoop, si nj\u00eb z\u00ebvend\u00ebsim t\u00eb shpejt\u00eb p\u00ebr t\u00eb.<\/p>\n

P\u00ebr nj\u00eb shqyrtim m\u00eb t\u00eb af\u00ebrt, v\u00ebrehet se Google Cloud Dataflow bazohet n\u00eb nj\u00eb variant t\u00eb Apache Beam, dhe n\u00eb t\u00eb, Apache Beam p\u00ebrfshin nj\u00eb framework t\u00eb dokumentuar mir\u00eb Apache Spark, duke lejuar t\u00eb flitet p\u00ebr nj\u00eb shpejt\u00ebsi ekzekutimi praktikisht t\u00eb nj\u00ebjt\u00eb. Nd\u00ebrsa Apache Spark punon shum\u00eb mir\u00eb n\u00eb sistemin e skedar\u00ebve HDFS, \u00e7ka lejon q\u00eb t\u00eb zhvillohet n\u00eb server\u00eb Hadoop.<\/p>\n

Shtojm\u00eb k\u00ebtu volumet e dokumentacionit dhe zgjidhjeve t\u00eb gatshme p\u00ebr Hadoop dhe Spark n\u00eb krahasim me Google Cloud Dataflow, dhe zgjedhja e mjetit b\u00ebhet e qart\u00eb. P\u00ebr m\u00eb tep\u00ebr, inxhinier\u00ebt mund t\u00eb vendosin vet\u00eb se cilin kod \u2014 p\u00ebr Hadoop ose Spark \u2014 ta ekzekutojn\u00eb, duke u bazuar n\u00eb detyr\u00ebn, p\u00ebrvoj\u00ebn dhe kualifikimin.<\/p>\n

Cloud ose server lokal<\/h2>\n

\nTendenca p\u00ebr kalimin n\u00eb cloud ka sjell\u00eb edhe nj\u00eb termin interesant si Hadoop-as-a-service. N\u00eb k\u00ebt\u00eb skenar, administrimi i server\u00ebve t\u00eb lidhur \u00ebsht\u00eb b\u00ebr\u00eb shum\u00eb i r\u00ebnd\u00ebsish\u00ebm. Sepse, fatkeq\u00ebsisht, p\u00ebrkund\u00ebr popullaritetit t\u00eb tij, Hadoop-i i past\u00ebr \u00ebsht\u00eb nj\u00eb mjet mjaft i komplikuar p\u00ebr t'u konfiguruar, pasi shum\u00eb gj\u00ebra duhen b\u00ebr\u00eb manualisht. P\u00ebr shembull, konfiguroi server\u00ebt ve\u00e7as, monitoroi treguesit e tyre, konfiguroi me kujdes shum\u00eb parametrat. N\u00eb p\u00ebrgjith\u00ebsi, \u00ebsht\u00eb nj\u00eb pun\u00eb p\u00ebr entuziast\u00eb dhe ka nj\u00eb mund\u00ebsi t\u00eb madhe p\u00ebr t\u00eb b\u00ebr\u00eb gabime apo p\u00ebr t\u00eb l\u00ebn\u00eb di\u00e7ka pas dore.<\/p>\n

Prandaj, kan\u00eb fituar popullaritet t\u00eb madh distribucione t\u00eb ndryshme, t\u00eb cilat p\u00ebrfshijn\u00eb mjetet e p\u00ebrshtatshme p\u00ebr zbatimin dhe administrimin. Nj\u00eb nga distribucionet m\u00eb t\u00eb njohura, t\u00eb cilat p\u00ebrkrahin Spark dhe thjeshtojn\u00eb gjith\u00e7ka, \u00ebsht\u00eb Cloudera. Ai ka versione t\u00eb paguara dhe t\u00eb lira \u2014 dhe n\u00eb versionin e fundit \u00ebsht\u00eb n\u00eb dispozicion e gjith\u00eb funksionaliteti kryesor, pa kufizim n\u00eb numrin e nod-\u00ebve. <\/p>\n

\"\u00c7far\u00eb
\n
\nGjat\u00eb konfigurimit, Cloudera Manager do t\u00eb lidh\u00eb p\u00ebr SSH me server\u00ebt tuaj. Nj\u00eb moment interesant: gjat\u00eb instalimit \u00ebsht\u00eb m\u00eb mir\u00eb t\u00eb caktuash q\u00eb t\u00eb realizohet me paket\u00eb speciale<\/i>: paketat speciale, n\u00eb secil\u00ebn nga t\u00eb cilat p\u00ebrfshihen t\u00eb gjitha komponent\u00ebt e nevojsh\u00ebm, t\u00eb konfiguruara p\u00ebr t\u00eb punuar s\u00eb bashku. N\u00eb thelb, kjo \u00ebsht\u00eb nj\u00eb version i p\u00ebrmir\u00ebsuar i menaxherit t\u00eb paketave.<\/p>\n

Pas instalimit, ne marrim nj\u00eb panel menaxhimi t\u00eb klastrit, ku mund t\u00eb shihni telemetrin\u00eb lidhur me klastrat, sh\u00ebrbimet e instaluara, plus do t\u00eb jeni n\u00eb gjendje t\u00eb shtoni\/hiqni burime dhe t\u00eb redaktoni konfigurimin e klastrit. <\/p>\n

\"\u00c7far\u00eb
\n
\nSi rezultat, para jush shfaqet pjesa e raket\u00ebs q\u00eb do t'ju \u00e7oj\u00eb n\u00eb t\u00eb ardhmen e ndritur t\u00eb BigData. Por p\u00ebrpara se t\u00eb themi \"n\u00eb rrug\u00eb\", le t\u00eb hedhim nj\u00eb sy n\u00ebn kapak.<\/p>\n

K\u00ebrkesat p\u00ebr harduer<\/h2>\n

\nN\u00eb faqen e saj, Cloudera p\u00ebrmend konfigurime t\u00eb ndryshme t\u00eb mundshme. Parimet e p\u00ebrgjithshme mbi t\u00eb cilat ato bazohen jan\u00eb paraqitur n\u00eb ilustrovimin:<\/p>\n

\"\u00c7far\u00eb
\nNj\u00eb imazh optimist mund t\u00eb prish\u00eb MapReduce. N\u00ebse e shikojm\u00eb p\u00ebrs\u00ebri diagramin nga seksioni i m\u00ebparsh\u00ebm, b\u00ebhet e qart\u00eb se n\u00eb pothuajse t\u00eb gjitha rastet, nj\u00eb detyr\u00eb MapReduce mund t\u00eb p\u00ebrballen me \"ngushticat\" kur lexon t\u00eb dh\u00ebna nga disku ose nga rrjeti. Kjo gjithashtu \u00ebsht\u00eb e theksuar n\u00eb blogun e Cloudera. Si rezultat, p\u00ebr \u00e7do lloj llogaritjesh t\u00eb shpejta, p\u00ebrfshir\u00eb p\u00ebrmes Spark, i cili shpesh p\u00ebrdoret p\u00ebr llogaritje n\u00eb koh\u00eb reale, \u00ebsht\u00eb shum\u00eb e r\u00ebnd\u00ebsishme shpejt\u00ebsia e hyrjes\/daljes. Prandaj, kur p\u00ebrdoret Hadoop, \u00ebsht\u00eb shum\u00eb e r\u00ebnd\u00ebsishme q\u00eb n\u00eb kluster t\u00eb p\u00ebrfshihen makina t\u00eb balancuara dhe t\u00eb shpejta, \u00e7ka, but\u00ebsisht th\u00ebn\u00eb, nuk sigurohet gjithmon\u00eb n\u00eb infrastruktur\u00ebn cloud.<\/p>\n

Balancimi n\u00eb shp\u00ebrndarjen e ngarkesave arrihet p\u00ebrmes p\u00ebrdorimit t\u00eb virtualizimit Openstack n\u00eb server\u00ebt me CPU t\u00eb fuqishme me shum\u00eb b\u00ebrtham\u00eb. Nodalet e t\u00eb dh\u00ebnave jan\u00eb ndar\u00eb me burime t\u00eb specificuara t\u00eb procesor\u00ebve dhe disqeve. N\u00eb zgjidhjen ton\u00eb Atos Codex Data Lake Engine<\/i> arrihet nj\u00eb virtualizim i gjer\u00eb, prej s\u00eb cil\u00ebs p\u00ebrfitojm\u00eb si n\u00eb performanc\u00eb (minimizimi i ndikimit t\u00eb infrastruktur\u00ebs rrjetit), ashtu edhe n\u00eb TCO (eliminohem server\u00ebt fizik\u00eb t\u00eb panevojsh\u00ebm).<\/p>\n

\"\u00c7far\u00eb
\nN\u00eb rastin e p\u00ebrdorimit t\u00eb server\u00ebve BullSequana S200, ne marrim nj\u00eb ngarkes\u00eb shum\u00eb t\u00eb barabart\u00eb, t\u00eb liruar nga disa ngushtica. Konfigurimi minimal p\u00ebrmban 3 server\u00eb BullSequana S200, secili me dy JBOD, plus mund\u00ebsisht lidhen server\u00eb t\u00eb tjer\u00eb S200, q\u00eb p\u00ebrmbajn\u00eb kat\u00ebr nodale t\u00eb dh\u00ebnash. Ja nj\u00eb shembull ngarkese n\u00eb testin TeraGen:<\/p>\n

\"\u00c7far\u00eb
\n
\nTestet me volumin e ndrysh\u00ebm t\u00eb t\u00eb dh\u00ebnave dhe vlerat e replikimit tregojn\u00eb rezultate t\u00eb nj\u00ebjta n\u00eb p\u00ebrmes shp\u00ebrndarjes s\u00eb ngarkes\u00ebs nd\u00ebrmjet nyjave t\u00eb klustrit. M\u00eb posht\u00eb \u00ebsht\u00eb nj\u00eb grafik i shp\u00ebrndarjes s\u00eb aksesit n\u00eb disk nga testet e performanc\u00ebs.<\/p>\n

\"\u00c7far\u00eb
\n
\nLlogaritjet jan\u00eb kryer mbi baz\u00ebn e konfigurimit minimal t\u00eb 3 server\u00ebve BullSequana S200. Ajo p\u00ebrfshin 9 nyje t\u00eb dh\u00ebnash dhe 3 nyje kryesore, si dhe makinat virtuale t\u00eb rezervuara p\u00ebr rastin e mobilizimit t\u00eb mbrojtjes mbi baz\u00ebn e Virtualizimit OpenStack. Rezultati i testit TeraSort: madh\u00ebsia e bllokut 512 MB me faktor replikimi tre me enkriptim \u00ebsht\u00eb 23.1 minuta.<\/p>\n

Si mund t\u00eb zgjerohet sistemi? P\u00ebr Data Lake Engine jan\u00eb t\u00eb disponueshme lloje t\u00eb ndryshme t\u00eb zgjerimeve:<\/p>\n