{"id":33630,"date":"2019-10-31T21:53:49","date_gmt":"2019-10-31T18:53:49","guid":{"rendered":"https:\/\/prohoster.info\/blog\/chto-osobennogo-v-cloudera-i-kak-ee-gotovit\/"},"modified":"2019-10-31T21:53:49","modified_gmt":"2019-10-31T18:53:49","slug":"chto-osobennogo-v-cloudera-i-kak-ee-gotovit","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/chto-osobennogo-v-cloudera-i-kak-ee-gotovit","title":{"rendered":"\u00c7far\u00eb \u00ebsht\u00eb e ve\u00e7ant\u00eb n\u00eb Cloudera dhe si ta p\u00ebrgatisni","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Tregu i komputimeve t\u00eb shp\u00ebrndara dhe t\u00eb dh\u00ebnave t\u00eb m\u00ebdha, n\u00ebse besoni statistikave<\/a><\/noindex>, po rritet me 18-19% n\u00eb vit. Kjo n\u00ebnkupton se \u00e7\u00ebshtja e zgjedhjes s\u00eb softuerit p\u00ebr k\u00ebto q\u00ebllime mbetet e r\u00ebnd\u00ebsishme. N\u00eb k\u00ebt\u00eb post, do t\u00eb fillojm\u00eb me arsyet e nevoj\u00ebs p\u00ebr komputime t\u00eb shp\u00ebrndara, do t\u00eb ndalemi m\u00eb n\u00eb detay n\u00eb zgjedhjen e software-it, do t\u00eb flasim p\u00ebr p\u00ebrdorimin e Hadoop me Cloudera dhe n\u00eb fund, do t\u00eb diskutojm\u00eb se si zgjedhja e harduerit ndikon n\u00eb performanc\u00eb n\u00eb m\u00ebnyra t\u00eb ndryshme.<\/p>\n

\"\u00c7far\u00eb<\/a><\/noindex>
\nPse jan\u00eb t\u00eb nevojshme komputimet e shp\u00ebrndara n\u00eb biznesin e zakonsh\u00ebm? K\u00ebtu \u00ebsht\u00eb e thjesht\u00eb dhe e komplikuar nj\u00ebkoh\u00ebsisht. E thjesht\u00eb \u2014 sepse n\u00eb shumic\u00ebn e rasteve ne kryejm\u00eb llogaritje relativisht t\u00eb thjeshta p\u00ebr nj\u00ebsi informacioni. E komplikuar \u2014 sepse ka shum\u00eb informacion. Shum\u00eb shum\u00eb. Si pasoj\u00eb, na nevojitet t\u00eb p\u00ebrpunojm\u00eb terabajt t\u00eb dh\u00ebnash n\u00eb 1000 shpejt\u00ebsi<\/a><\/noindex>. K\u00ebshtu, skenar\u00ebt e p\u00ebrdorimit jan\u00eb mjaft universale: llogaritjet mund t\u00eb p\u00ebrdoren kudo ku k\u00ebrkohet t\u00eb merret parasysh nj\u00eb num\u00ebr i madh metrikash mbi nj\u00eb grumbull akoma m\u00eb t\u00eb madh t\u00eb dh\u00ebnash.<\/p>\n

Nj\u00eb nga shembujt e fundit: rrjeti i picerive Dodo Pizza p\u00ebrcaktoi<\/a><\/noindex> bazuar n\u00eb analiz\u00ebn e baz\u00ebs s\u00eb porosive t\u00eb klient\u00ebve, q\u00eb gjat\u00eb zgjedhjes s\u00eb pic\u00ebs me p\u00ebrb\u00ebrje t\u00eb rast\u00ebsishme, p\u00ebrdoruesit zakonisht operojn\u00eb vet\u00ebm me gjasht\u00eb grupe bazike p\u00ebrb\u00ebr\u00ebsish plus disa t\u00eb rast\u00ebsishme. N\u00eb p\u00ebrputhje me k\u00ebt\u00eb, piceria ka rregulluar blerjet. P\u00ebr m\u00eb tep\u00ebr, ajo ka arritur t\u00eb rekomandoj\u00eb m\u00eb mir\u00eb produkte shtes\u00eb p\u00ebr p\u00ebrdoruesit n\u00eb faz\u00ebn e porosis\u00eb, gj\u00eb q\u00eb ka rritur fitimet.<\/p>\n

Nj\u00eb tjet\u00ebr shembull: analiza<\/a><\/noindex> pozitat e produkteve kan\u00eb lejuar dyqanin H&M t\u00eb reduktoj\u00eb asortimentin n\u00eb dyqane t\u00eb caktuara me 40%, duke ruajtur nj\u00ebkoh\u00ebsisht nivelin e shitjeve. K\u00ebto rezultate jan\u00eb arritur duke p\u00ebrjashtuar pozitat me shitje t\u00eb dob\u00ebta, duke marr\u00eb parasysh sezonin.<\/p>\n

Zgjedhja e mjetit<\/h2>\n

\nStandardi industrial p\u00ebr llogaritjet e k\u00ebtij lloji \u00ebsht\u00eb Hadoop. Pse? Sepse Hadoop \u00ebsht\u00eb nj\u00eb struktur\u00eb e shk\u00eblqyer, e dokumentuar mir\u00eb (po aty Habr jep shum\u00eb artikuj t\u00eb detajuar mbi k\u00ebt\u00eb tem\u00eb), e cila shoq\u00ebrohet me nj\u00eb grup t\u00eb t\u00ebr\u00eb utilitar\u00ebsh dhe bibliotekash. Ju mund t\u00eb jepni n\u00eb hyrje grumbuj t\u00eb m\u00ebdha si t\u00eb dh\u00ebnash t\u00eb strukturuara ashtu edhe t\u00eb pa strukturuara, dhe sistemi do t'i shp\u00ebrndaj\u00eb vet\u00eb ato mes kapaciteteve t\u00eb llogaritjes. Plus, k\u00ebto kapacitete mund t\u00eb rriten ose \u00e7aktivizohen n\u00eb \u00e7do moment \u2014 ajo q\u00eb quhet shkall\u00ebzim horizontal n\u00eb veprim. <\/p>\n

N\u00eb vitin 2017, kompania e njohur e konsulenc\u00ebs Gartner n\u00ebnshkroi<\/a><\/noindex>, tha se Hadoop do t\u00eb zgjidhte vetveten. Arsyetimi \u00ebsht\u00eb mjaft banal: analist\u00ebt mendojn\u00eb se kompanit\u00eb do t\u00eb migrojn\u00eb masivisht n\u00eb re, p\u00ebr shkak se aty mund t\u00eb paguajn\u00eb sipas p\u00ebrdorimit t\u00eb kapaciteteve t\u00eb llogaritjes. Faktori i dyt\u00eb i r\u00ebnd\u00ebsish\u00ebm, q\u00eb mund t\u00eb \"varros\u00eb\" Hadoop \u2014 \u00ebsht\u00eb shpejt\u00ebsia e pun\u00ebs. Sepse opsionet si Apache Spark ose Google Cloud DataFlow punojn\u00eb m\u00eb shpejt se MapReduce, q\u00eb \u00ebsht\u00eb baza e Hadoop.<\/p>\n

Hadoop mb\u00ebshtetet n\u00eb disa shtylla, m\u00eb t\u00eb dukshmet prej t\u00eb cilave jan\u00eb teknologjit\u00eb MapReduce (sistemi i shp\u00ebrndarjes s\u00eb t\u00eb dh\u00ebnave p\u00ebr llogaritje mes server\u00ebve) dhe sistemi i skedar\u00ebve HDFS. Ky i fundit \u00ebsht\u00eb posa\u00e7\u00ebrisht i dizajnuar p\u00ebr ruajtjen e informacionit t\u00eb shp\u00ebrndar\u00eb mes nyjave t\u00eb klasterit: \u00e7do bllok me nj\u00eb madh\u00ebsi fikse mund t\u00eb vendoset n\u00eb disa nyje, dhe p\u00ebrmes replikimit sigurohet q\u00eb sistemi \u00ebsht\u00eb i q\u00ebndruesh\u00ebm ndaj d\u00ebshtimeve t\u00eb nyjeve t\u00eb ve\u00e7anta. N\u00eb vend t\u00eb tabel\u00ebs s\u00eb skedave, p\u00ebrdoret nj\u00eb server i ve\u00e7ant\u00eb, i quajtur NameNode.<\/p>\n

N\u00eb ilustrimin m\u00eb posht\u00eb \u00ebsht\u00eb paraqitur skema e pun\u00ebs s\u00eb MapReduce. N\u00eb faz\u00ebn e par\u00eb t\u00eb dh\u00ebnat ndahen sipas nj\u00eb kriteri t\u00eb caktuar, n\u00eb t\u00eb dyt\u00ebn \u2014 ndahen mes kapaciteteve t\u00eb llogaritjes, dhe n\u00eb t\u00eb tret\u00ebn \u2014 kryhet llogaritja. <\/p>\n

\"\u00c7far\u00eb
\nFillimisht MapReduce u krijua nga Google p\u00ebr nevojat e k\u00ebrkimit t\u00eb tij. M\u00eb pas MapReduce kaloi n\u00eb kodin e lir\u00eb, dhe projekti u mor nga Apache. Nd\u00ebrsa Google gradualisht migroi n\u00eb zgjidhje t\u00eb tjera. Nj\u00eb detaj interesant: aktualisht Google ka nj\u00eb projekt, t\u00eb quajtur Google Cloud Dataflow, i pozicionuar si hapi pasues pas Hadoop, si nj\u00eb z\u00ebvend\u00ebsim t\u00eb shpejt\u00eb.<\/p>\n

Me nj\u00eb v\u00ebshtrim m\u00eb t\u00eb af\u00ebrt, v\u00ebrehet se Google Cloud Dataflow bazohet n\u00eb nj\u00eb lloj t\u00eb Apache Beam, nd\u00ebrsa n\u00eb Apache Beam p\u00ebrfshihet nj\u00eb struktur\u00eb e dokumentuar mir\u00eb, Apache Spark, q\u00eb lejon t\u00eb flitet p\u00ebr shpejt\u00ebsi t\u00eb ngjashme n\u00eb zgjidhje. Nd\u00ebrsa Apache Spark punon shk\u00eblqyesh\u00ebm n\u00eb sistemin e skedar\u00ebve HDFS, \u00e7ka lejon ta zhvillosh at\u00eb n\u00eb serverat Hadoop.<\/p>\n

Shtojm\u00eb k\u00ebtu sasin\u00eb e dokumentacionit dhe zgjidhjeve t\u00eb gatshme p\u00ebr Hadoop dhe Spark krahasuar me Google Cloud Dataflow, dhe zgjedhja e mjetit b\u00ebhet e qart\u00eb. M\u00eb shum\u00eb, inxhinier\u00ebt mund t\u00eb vendosin vet\u00eb se cilin kod \u2014 n\u00ebn Hadoop ose Spark \u2014 duhet t\u00eb kryejn\u00eb, duke u bazuar n\u00eb detyr\u00ebn, p\u00ebrvoj\u00ebn dhe kualifikimin.<\/p>\n

Re ose server lokal<\/h2>\n

\nTendenca p\u00ebr kalimin n\u00eb cloud ka sjell\u00eb nj\u00eb term interesant si Hadoop-as-a-service. N\u00eb k\u00ebt\u00eb skenar, administrimi i server\u00ebve t\u00eb lidhur \u00ebsht\u00eb b\u00ebr\u00eb shum\u00eb i r\u00ebnd\u00ebsish\u00ebm. Sepse, fatkeq\u00ebsisht, megjith\u00ebse \u00ebsht\u00eb i njohur, Hadoop i past\u00ebr \u00ebsht\u00eb nj\u00eb mjet mjaft i komplikuar p\u00ebr t'u konfiguruar, duke pasur parasysh se shum\u00eb gj\u00ebra duhet t\u00eb b\u00ebhen manualisht. P\u00ebr shembull, konfigurimi i ve\u00e7ant\u00eb i server\u00ebve, monitorimi i treguesve t\u00eb tyre, dhe p\u00ebrshtatja me kujdes e shum\u00eb parametrave. N\u00eb p\u00ebrgjith\u00ebsi, \u00ebsht\u00eb pun\u00eb p\u00ebr entuziast\u00eb dhe ka nj\u00eb shans t\u00eb madh q\u00eb ndonj\u00ebher\u00eb t\u00eb gabosh ose t\u00eb humbas\u00ebsh di\u00e7ka.<\/p>\n

Prandaj, jan\u00eb b\u00ebr\u00eb shum\u00eb t\u00eb njohura distribucione t\u00eb ndryshme, t\u00eb cilat fillimisht vijn\u00eb me mjete t\u00eb p\u00ebrshtatshme p\u00ebr shp\u00ebrndarje dhe administrim. Nj\u00eb nga distribucionet m\u00eb t\u00eb njohura q\u00eb mb\u00ebshtet Spark dhe e thjeshton gj\u00ebrat \u00ebsht\u00eb Cloudera. Ai ka versione si t\u00eb paguara, ashtu edhe falas - dhe versioni falas ofron gjith\u00eb funksionalitetin kryesor, pa kufizim n\u00eb numrin e nodave. <\/p>\n

\"\u00c7far\u00eb
\n
\nGjat\u00eb konfigurimit, Cloudera Manager do t\u00eb lidh\u00eb p\u00ebrmes SSH me server\u00ebt tuaj. Nj\u00eb pik\u00eb interesante: gjat\u00eb instalimit, \u00ebsht\u00eb m\u00eb mir\u00eb t\u00eb specifikohet q\u00eb ky proces t\u00eb zhvillohet me at\u00eb q\u00eb quhet paket\u00eb<\/i>: paketa speciale, n\u00eb secil\u00ebn prej t\u00eb cilave p\u00ebrfshihen t\u00eb gjith\u00eb komponent\u00ebt e nevojsh\u00ebm, t\u00eb konfiguruar p\u00ebr t\u00eb punuar p\u00ebrkrah nj\u00ebri-tjetrit. N\u00eb thelb, kjo \u00ebsht\u00eb nj\u00eb version i p\u00ebrmir\u00ebsuar i menaxherit t\u00eb paketave.<\/p>\n

Pas instalimit, ne marrim nj\u00eb konsol menaxhimi p\u00ebr klasterin, ku mund t\u00eb shihni telemetrin\u00eb p\u00ebr klasteret, sh\u00ebrbimet e instaluara, plus mund t\u00eb shtoni\/hiqni burime dhe t\u00eb redaktoni konfigurimin e klasterit. <\/p>\n

\"\u00c7far\u00eb
\n
\nSi rezultat, para jush shfaqet nj\u00eb model i raket\u00ebs q\u00eb do t'ju \u00e7oj\u00eb n\u00eb t\u00eb ardhmen e ndritur t\u00eb Big Data. Por para se t\u00eb themi \"lem\u00eb\", le t\u00eb hedhim nj\u00eb v\u00ebshtrim n\u00ebn kapak.<\/p>\n

K\u00ebrkesat p\u00ebr harduer<\/h2>\n

\nN\u00eb faqen e saj, Cloudera p\u00ebrmend konfigurime t\u00eb ndryshme t\u00eb mundshme. Parimet e p\u00ebrgjithshme mbi t\u00eb cilat ato nd\u00ebrtohen jan\u00eb paraqitur n\u00eb ilustrim:<\/p>\n

\"\u00c7far\u00eb
\nNj\u00eb tablo optimiste mund t\u00eb prish\u00eb MapReduce. N\u00ebse e shqyrtojm\u00eb s\u00ebrish skem\u00ebn nga seksioni i m\u00ebparsh\u00ebm, b\u00ebhet e qart\u00eb se p\u00ebr pothuajse \u00e7do rast, detyra MapReduce mund t\u00eb has\u00eb nj\u00eb \"gryk\u00eb\" kur lexon t\u00eb dh\u00ebna nga disku ose nga rrjeti. Kjo gjithashtu \u00ebsht\u00eb p\u00ebrmendur n\u00eb blogun e Cloudera. Si rezultat, p\u00ebr \u00e7do llogaritje t\u00eb shpejt\u00eb, p\u00ebrfshir\u00eb edhe p\u00ebrmes Spark, i cili p\u00ebrdoret shpesh p\u00ebr llogaritje n\u00eb koh\u00eb reale, \u00ebsht\u00eb shum\u00eb e r\u00ebnd\u00ebsishme shpejt\u00ebsia e hyrjes\/daljes. Prandaj, kur p\u00ebrdoren Hadoop, \u00ebsht\u00eb shum\u00eb e r\u00ebnd\u00ebsishme q\u00eb klasteri t\u00eb p\u00ebrfshij\u00eb makina t\u00eb balancuara dhe t\u00eb shpejta, gj\u00eb q\u00eb, nd\u00ebr t\u00eb tjera, nuk \u00ebsht\u00eb gjithmon\u00eb e garantuar n\u00eb infrastruktur\u00ebn cloud.<\/p>\n

Balancimi n\u00eb shp\u00ebrndarjen e ngarkesave arrihet duke p\u00ebrdorur virtualizimin Openstack n\u00eb server\u00eb me CPU t\u00eb fuqishme me shum\u00eb b\u00ebrtham\u00eb. Nody t\u00eb dh\u00ebna kan\u00eb burimet e tyre procesor dhe disqe t\u00eb caktuara. N\u00eb zgjidhjen ton\u00eb Atos Codex Data Lake Engine<\/i> arrihet nj\u00eb virtualizim i gjer\u00eb, nga i cili p\u00ebrfitojm\u00eb si n\u00eb performanc\u00eb (minimizohet ndikimi i infrastruktur\u00ebs rrjet\u00eb) ashtu edhe n\u00eb TCO (eleminohet nevoja p\u00ebr server\u00eb fizik\u00eb t\u00eb tep\u00ebrt).<\/p>\n

\"\u00c7far\u00eb
\nN\u00ebse p\u00ebrdoren server\u00ebt BullSequana S200, ne marrim nj\u00eb ngarkes\u00eb shum\u00eb t\u00eb balancuar, t\u00eb liruar nga disa ngushtica. Konfigurimi minimal p\u00ebrfshin 3 server\u00eb BullSequana S200, secili me dy JBOD, plus munden t\u00eb lidhen opsionalisht S200 shtes\u00eb me kat\u00ebr nod\u00eb t\u00eb dh\u00ebnash secili. Ky \u00ebsht\u00eb nj\u00eb shembull i ngarkes\u00ebs n\u00eb testin TeraGen:<\/p>\n

\"\u00c7far\u00eb
\n
\nTestet me sasi t\u00eb ndryshme t\u00eb t\u00eb dh\u00ebnave dhe vlera replikimi tregojn\u00eb rezultate t\u00eb nj\u00ebjta n\u00eb lidhje me shp\u00ebrndarjen e ngarkes\u00ebs midis nodave t\u00eb klasterit. M\u00eb posht\u00eb \u00ebsht\u00eb grafiku i shp\u00ebrndarjes s\u00eb qasjes n\u00eb disk p\u00ebr testet e performanc\u00ebs.<\/p>\n

\"\u00c7far\u00eb
\n
\nLlogaritjet jan\u00eb kryer n\u00eb baz\u00ebn e konfigurimit minimal prej 3 server\u00ebsh BullSequana S200. Ajo p\u00ebrfshin 9 nod\u00eb t\u00eb dh\u00ebnash dhe 3 nod\u00eb kryesore, si dhe makina virtuale t\u00eb rezervuara n\u00eb rast se d\u00ebshirohet implementimi i mbrojtjes p\u00ebrmes Virtualizimit OpenStack. Rezultati i testit TeraSort: madh\u00ebsia e bllokut 512 MB me faktor replikimi tre me enkriptim \u00ebsht\u00eb 23.1 minuta.<\/p>\n

Si mund t\u00eb zgjerohet sistemi? P\u00ebr Data Lake Engine jan\u00eb t\u00eb disponueshme lloje t\u00eb ndryshme zgjerimesh:<\/p>\n