{"id":30548,"date":"2019-10-31T21:36:08","date_gmt":"2019-10-31T18:36:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/konfigurirovanie-spark-na-yarn\/"},"modified":"2019-10-31T21:36:08","modified_gmt":"2019-10-31T18:36:08","slug":"konfigurirovanie-spark-na-yarn","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn","title":{"rendered":"Konfigurimi i Spark n\u00eb YARN","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Habr, p\u00ebrsh\u00ebndetje! Dje n\u00eb mitap dedikuar Apache Spark<\/a><\/noindex>, nga djemt e Rambler&Co, pati mjaft pyetje nga pjes\u00ebmarr\u00ebsit q\u00eb lidhen me konfigurimin e k\u00ebtij instrumenti. Vendos\u00ebm t\u00eb ndajm\u00eb p\u00ebrvoj\u00ebn ton\u00eb n\u00eb lidhje me t\u00eb. Tema \u00ebsht\u00eb e nd\u00ebrlikuar \u2014 k\u00ebshtu q\u00eb ju inkurajojm\u00eb t\u00eb ndaheni me p\u00ebrvoj\u00ebn tuaj gjithashtu n\u00eb komente, ndoshta ne gjithashtu nuk po e kuptojm\u00eb di\u00e7ka si\u00e7 duhet dhe po e p\u00ebrdorim gabim.
\n<\/a><\/noindex>
\nNj\u00eb hyrje e vog\u00ebl \u2014 si ne p\u00ebrdorim Spark. Ne kemi nj\u00eb program tre-mujor \u201cSpecialist p\u00ebr t\u00eb dh\u00ebna t\u00eb m\u00ebdha\u201d<\/a><\/noindex>, dhe gjith\u00eb moduli i dyt\u00eb, pjes\u00ebmarr\u00ebsit tan\u00eb punojn\u00eb me k\u00ebt\u00eb instrument. Ndaj, detyra jon\u00eb si organizator\u00eb \u00ebsht\u00eb t\u00eb p\u00ebrgatisim nj\u00eb klaster p\u00ebr p\u00ebrdorim n\u00eb kuad\u00ebr t\u00eb nj\u00eb rasti t\u00eb till\u00eb.<\/p>\n

Karakteristika e p\u00ebrdorimit ton\u00eb \u00ebsht\u00eb se numri i njer\u00ebzve q\u00eb punojn\u00eb nj\u00ebkoh\u00ebsisht n\u00eb Spark mund t\u00eb jet\u00eb i barabart\u00eb me gjith\u00eb grupin. P\u00ebr shembull, n\u00eb nj\u00eb seminar, kur t\u00eb gjith\u00eb provon nj\u00eb gj\u00eb nj\u00ebkoh\u00ebsisht dhe p\u00ebrs\u00ebrisin pas m\u00ebsuesit ton\u00eb. Dhe kjo \u00ebsht\u00eb pak shum\u00eb \u2014 deri n\u00eb 40 njer\u00ebz ndonj\u00ebher\u00eb. Ndoshta, nuk ka shum\u00eb kompani n\u00eb bot\u00eb q\u00eb ballafaqohen me nj\u00eb skenar t\u00eb till\u00eb p\u00ebrdorimi.<\/p>\n

M\u00eb tej do t\u00eb flas p\u00ebr m\u00ebnyrat dhe arsyet se si i p\u00ebrzgjodh\u00ebm k\u00ebto parametro t\u00eb konfigurimit.<\/p>\n

T\u00eb fillojm\u00eb nga fillimi. Spark ka 3 m\u00ebnyra p\u00ebr t\u00eb punuar n\u00eb nj\u00eb klaster: standalone, duke p\u00ebrdorur Mesos dhe duke p\u00ebrdorur YARN. Ne vendos\u00ebm t\u00eb zgjidhim opsionin e tret\u00eb, sepse ishte logjike p\u00ebr ne. Ne tashm\u00eb kemi nj\u00eb klaster Hadoop. Pjes\u00ebtar\u00ebt tan\u00eb jan\u00eb t\u00eb njohur me arkitektur\u00ebn e tij. Le ta p\u00ebrdorim YARN.<\/p>\n

spark.master=yarn<\/code><\/pre>\n
\nM\u00eb pas b\u00ebhet m\u00eb interesante. \u00c7do nj\u00eb nga k\u00ebto 3 opsione p\u00ebr shp\u00ebrndarje ka 2 m\u00ebnyra p\u00ebr deploy: client dhe cluster. Nga dokumentacion<\/a><\/noindex> dhe lidhjet e ndryshme n\u00eb internet, mund t\u00eb arrihet n\u00eb p\u00ebrfundimin se client \u00ebsht\u00eb i p\u00ebrshtatsh\u00ebm p\u00ebr pun\u00eb interaktive \u2014 p\u00ebr shembull, p\u00ebrmes jupyter notebook, nd\u00ebrsa cluster \u00ebsht\u00eb m\u00eb i p\u00ebrshtatsh\u00ebm p\u00ebr zgjidhjet n\u00eb prodhim. N\u00eb rastin ton\u00eb, na interesonte puna interaktive, k\u00ebshtu q\u00eb:<\/p>\n
spark.deploy-mode=client<\/code><\/pre>\n
\nN\u00eb parim, q\u00eb nga ky moment, Spark do t\u00eb funksionoj\u00eb n\u00eb YARN, por kjo nuk ishte e mjaftueshme p\u00ebr ne. Duke qen\u00eb se kemi nj\u00eb program p\u00ebr t\u00eb dh\u00ebna t\u00eb m\u00ebdha, ndonj\u00ebher\u00eb pjes\u00ebmarr\u00ebsve u mungonte gjith\u00e7ka q\u00eb mund t\u00eb arrihej p\u00ebrmes ndarjes s\u00eb barabart\u00eb t\u00eb burimeve. Dhe k\u00ebtu gjet\u00ebm nj\u00eb gj\u00eb interesante \u2014 alokimin dinamik t\u00eb burimeve. N\u00eb p\u00ebrmbledhje, thelbi \u00ebsht\u00eb ky: n\u00ebse keni nj\u00eb detyr\u00eb t\u00eb r\u00ebnd\u00eb dhe klasteri \u00ebsht\u00eb i lir\u00eb (p.sh., n\u00eb m\u00ebngjes), at\u00ebher\u00eb me k\u00ebt\u00eb opsion Spark mund t'ju jap\u00eb burime shtes\u00eb. Nevoja llogaritet atje sipas nj\u00eb formule interesante. Nuk do t\u00eb hyjm\u00eb n\u00eb detaje \u2014 ajo funksionon mjaft mir\u00eb.<\/p>\n
spark.dynamicAllocation.enabled=true<\/code><\/pre>\n
\nNe e vendos\u00ebm k\u00ebt\u00eb parameter, dhe kur e ekzekutuam Spark, u ankuar dhe nuk nisi. E drejt\u00eb, sepse duhej t\u00eb lexohej dokumentacioni<\/a><\/noindex> m\u00eb me kujdes. Aty tregohet se p\u00ebr t\u00eb pasur gjith\u00e7ka n\u00eb rregull, duhet t\u00eb aktivizohet edhe nj\u00eb parameter shtes\u00eb.<\/p>\n
spark.shuffle.service.enabled=true<\/code><\/pre>\n
\nP\u00ebr \u00e7far\u00eb \u00ebsht\u00eb e nevojshme? Kur puna jon\u00eb nuk k\u00ebrkon m\u00eb aq shum\u00eb burime, Spark duhet t'i kthej\u00eb ato n\u00eb rezerv\u00ebn e p\u00ebrgjithshme. Faza m\u00eb e pun intensive n\u00eb pothuajse \u00e7do detyr\u00eb MapReduce \u00ebsht\u00eb faza Shuffle. Ky parametr lejon ruajtjen e t\u00eb dh\u00ebnave q\u00eb krijohen n\u00eb k\u00ebt\u00eb faz\u00eb dhe, p\u00ebr pasoj\u00eb, lirimin e executors. Nj\u00eb executor \u00ebsht\u00eb procesi q\u00eb llogarit gjith\u00e7ka n\u00eb pun\u00eb. Ai ka nj\u00eb num\u00ebr t\u00eb caktuar b\u00ebrthamash procesori dhe nj\u00eb sasi t\u00eb caktuar memories.<\/p>\n
E kemi shtuar k\u00ebt\u00eb parametr. T\u00eb gjitha duket se funksionojn\u00eb. Ka filluar t\u00eb v\u00ebrehet se pjes\u00ebmarr\u00ebsit realisht po merrnin m\u00eb shum\u00eb burime kur ishin t\u00eb nevojshme. Por u shfaq nj\u00eb problem tjet\u00ebr \u2014 n\u00eb nj\u00eb moment t\u00eb caktuar pjes\u00ebmarr\u00ebsit e tjer\u00eb u zgjuan dhe gjithashtu donin t\u00eb p\u00ebrdorin Spark, por ajo ishte e mbushur, dhe ata ishin t\u00eb pak\u00ebnaqur. Mund t\u2019i kuptojm\u00eb. Filluam t\u00eb shohim dokumentacionin. Aty dol\u00ebn se kishte edhe disa parametra t\u00eb tjer\u00eb q\u00eb mund t\u00eb ndikojn\u00eb n\u00eb proces. P\u00ebr shembull, n\u00ebse executor \u00ebsht\u00eb n\u00eb modalitetin e pritjes \u2014 pas sa koh\u00ebsh mund t\u2019i merret burimet?<\/p>\n
spark.dynamicAllocation.executorIdleTimeout=120s<\/code><\/pre>\n
\nN\u00eb rastin ton\u00eb \u2014 n\u00ebse executor\u00ebt tuaj nuk b\u00ebjn\u00eb asgj\u00eb p\u00ebr dy minuta, ju lutemi, ktheni ata n\u00eb rezerv\u00ebn e p\u00ebrbashk\u00ebt. Por as ky paramet\u00ebr nuk ishte gjithmon\u00eb i mjaftuesh\u00ebm. Ishte e qart\u00eb se nj\u00eb person nuk po b\u00ebnte asgj\u00eb prej koh\u00ebsh, por burimet nuk po liroheshin. Doli se kishte edhe nj\u00eb paramet\u00ebr t\u00eb ve\u00e7ant\u00eb \u2014 pas sa koh\u00ebsh t\u00eb merreshin executor\u00ebt q\u00eb kishin t\u00eb dh\u00ebna t\u00eb ruajtura n\u00eb cache. Nga parazgjedhja, ky paramet\u00ebr ishte vendosur n\u00eb \u2014 pafund\u00ebsi! Ne e rregulluam at\u00eb.<\/p>\n
spark.dynamicAllocation.cachedExecutorIdleTimeout=600s<\/code><\/pre>\n
\nK\u00ebshtu q\u00eb, n\u00ebse p\u00ebr 5 minuta executor\u00ebt tuaj nuk b\u00ebjn\u00eb asgj\u00eb, ktheni ata n\u00eb rezerv\u00ebn e p\u00ebrbashk\u00ebt. N\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb, shpejt\u00ebsia e lirimit dhe ndarjes s\u00eb burimeve p\u00ebr nj\u00eb num\u00ebr t\u00eb madh p\u00ebrdoruesish ka marr\u00eb nj\u00eb nivel t\u00eb k\u00ebnaqsh\u00ebm. Numri i pak\u00ebnaq\u00ebsive u ul. Por ne vendos\u00ebm t\u00eb shkojm\u00eb m\u00eb tej dhe ta kufizojm\u00eb numrin maksimal t\u00eb executor\u00ebve p\u00ebr nj\u00eb aplikacion \u2014 n\u00eb thelb p\u00ebr nj\u00eb pjes\u00ebmarr\u00ebs t\u00eb programit.<\/p>\n
spark.dynamicAllocation.maxExecutors=19<\/code><\/pre>\n
\nTani, natyrisht, u shfaq\u00ebn pak\u00ebnaq\u00ebsi nga ana tjet\u00ebr \u2014 \u201cklistri po pushon, nd\u00ebrsa un\u00eb kam vet\u00ebm 19 executor\u00eb\u201d, por \u00e7far\u00eb mund t\u00eb b\u00ebjm\u00eb \u2014 duhet nj\u00eb balancim i duhur. Nuk do t\u00eb arrijm\u00eb t\u00eb b\u00ebjm\u00eb t\u00eb gjith\u00eb t\u00eb lumtur.<\/p>\n
Dhe nj\u00eb histori e vog\u00ebl tjet\u00ebr e lidhur me specifikat e rastit ton\u00eb. Nj\u00ebher\u00eb disa njer\u00ebz vonuan n\u00eb nj\u00eb nga ushtrimet praktike, dhe p\u00ebr nj\u00eb arsye, Spark nuk u nis. Ne e kontrolluam numrin e burimeve t\u00eb lira \u2014 duke dukur, kishte. Spark duhet t\u00eb niste. Fatmir\u00ebsisht, n\u00eb at\u00eb koh\u00eb dokumentacioni ishte tashm\u00eb ndar\u00eb n\u00eb n\u00ebnnd\u00ebrgjegje, dhe ne p\u00ebrmend\u00ebm se kur niset Spark, ai k\u00ebrkon nj\u00eb port p\u00ebr t'u nisur. N\u00ebse porta e par\u00eb nga diapazoni \u00ebsht\u00eb e z\u00ebn\u00eb, ai kalon n\u00eb portin tjet\u00ebr n\u00eb rend. N\u00ebse ai \u00ebsht\u00eb i lir\u00eb, ai e kap. Dhe ka nj\u00eb paramet\u00ebr q\u00eb tregon numrin maksimal t\u00eb p\u00ebrpjekjeve p\u00ebr k\u00ebt\u00eb. N\u00eb m\u00ebnyr\u00eb t\u00eb paracaktuar, kjo \u00ebsht\u00eb 16. Numri \u00ebsht\u00eb m\u00eb i vog\u00ebl se njer\u00ebzit n\u00eb grupin ton\u00eb gjat\u00eb ushtrimit. Prandaj, pas 16 p\u00ebrpjekjeve Spark e ndaloi at\u00eb dhe tha se nuk mund t\u00eb nisa. Ne e rregulluam k\u00ebt\u00eb paramet\u00ebr.<\/p>\n
spark.port.maxRetries=50<\/code><\/pre>\n
\nM\u00eb tutje do t\u00eb tregoj disa konfigurime, q\u00eb tashm\u00eb nuk jan\u00eb shum\u00eb t\u00eb lidhura me specifikat e rastit ton\u00eb.<\/p>\n
P\u00ebr nj\u00eb nisje m\u00eb t\u00eb shpejt\u00eb t\u00eb Spark, ekziston nj\u00eb rekomandim p\u00ebr t\u00eb kompresuar folderin jars, q\u00eb ndodhet n\u00eb direktorin\u00eb sht\u00ebpiake SPARK_HOME, dhe ta vendosni n\u00eb HDFS. K\u00ebshtu ai nuk do ta humbas\u00eb koh\u00ebn n\u00eb ngarkimin e k\u00ebtyre jar-ve p\u00ebr pun\u00ebtor\u00ebt.<\/p>\n
spark.yarn.archive=hdfs:\/\/\/tmp\/spark-archive.zip<\/code><\/pre>\n
\nPo ashtu, p\u00ebr nj\u00eb funksionim m\u00eb t\u00eb shpejt\u00eb, rekomandohet t\u00eb p\u00ebrdorni kryo si serializues. Ai \u00ebsht\u00eb m\u00eb i optimizuar se ai q\u00eb \u00ebsht\u00eb parazgjedhur.<\/p>\n
spark.serializer=org.apache.spark.serializer.KryoSerializer<\/code><\/pre>\n
\nDhe ka nj\u00eb problem t\u00eb njohur me Spark, q\u00eb shpesh p\u00ebrben probleme me memorie. Kjo ndodh shpesh kur pun\u00ebtor\u00ebt kan\u00eb p\u00ebrfunduar t\u00eb gjitha llogaritjet dhe d\u00ebrgojn\u00eb rezultatet te drejtuesi. Ne e vendos\u00ebm k\u00ebt\u00eb parametr m\u00eb t\u00eb lart\u00eb. K\u00ebshtu, parazgjedhja \u00ebsht\u00eb 1Gb, ne e b\u00ebm\u00eb 3.<\/p>\n
spark.driver.maxResultSize=3072<\/code><\/pre>\n
\nDhe e fundit, si nj\u00eb \u00ebmb\u00eblsir\u00eb. Si ta p\u00ebrdit\u00ebsojm\u00eb Spark n\u00eb versionin 2.1 n\u00eb distribucionin HortonWorks \u2014 HDP 2.5.3.0. Ky version HDP p\u00ebrmban nj\u00eb version t\u00eb parazgjedhur 2.0, por ne her\u00ebn e par\u00eb vendos\u00ebm se Spark po zhvillohej va\u017ehdimisht, dhe \u00e7do version i ri rregullon disa defekte dhe jep mund\u00ebsi t\u00eb reja, p\u00ebrfshir\u00eb edhe p\u00ebr API-n\u00eb python, k\u00ebshtu q\u00eb vendos\u00ebm ta b\u00ebjm\u00eb p\u00ebrdit\u00ebsimin.<\/p>\n
E kemi shkarkuar versionin nga faqja zyrtare p\u00ebr Hadoop 2.7. E kemi shp\u00ebrngulur, e kemi vendosur n\u00eb dosjen me HDP. Kemi vendosur simoling si duhet. E nisim \u2014 nuk fillon. Shfaq nj\u00eb gabim shum\u00eb t\u00eb paqart\u00eb.<\/p>\n
java.lang.NoClassDefFoundError: com\/sun\/jersey\/api\/client\/config\/ClientConfig<\/code><\/pre>\n
\nPas duke k\u00ebrkuar n\u00eb Google, q\u00eb e kuptuam se Spark vendosi t\u00eb mos priste q\u00eb Hadoop t\u00eb zgjidhte problemin dhe vendosi t\u00eb p\u00ebrdorte versionin e ri t\u00eb jersey. Ata vet\u00eb po grindeshin p\u00ebr k\u00ebt\u00eb \u00e7\u00ebshtje n\u00eb JIRA. Zgjidhja ishte - shkarko jersey version 1.17.1<\/a><\/noindex>. V\u00ebzhgoje k\u00ebt\u00eb n\u00eb dosjen jars n\u00eb SPARK_HOME, p\u00ebrs\u00ebri b\u00ebj zip dhe d\u00ebrgoje n\u00eb HDFS.<\/p>\n
Ne e anashaluam k\u00ebt\u00eb gabim, por lind nj\u00eb e re dhe mjaft e paqart\u00eb.<\/p>\n
org.apache.spark.SparkException: Aplikimi Yarn ka p\u00ebrfunduar tashm\u00eb! Mund t\u00eb ket\u00eb qen\u00eb i vrar\u00eb ose i paaft\u00eb p\u00ebr t\u00eb nisur masterin e aplikacionit.<\/code><\/pre>\n
\nN\u00eb k\u00ebt\u00eb rast, po provojm\u00eb t\u00eb lan\u00e7ojm\u00eb versionin 2.0 - gjith\u00e7ka \u00ebsht\u00eb n\u00eb rregull. Provo t\u00eb gjesh se \u00e7far\u00eb ndodh. Ne shkuam n\u00eb log-et e k\u00ebtij aplikacioni dhe pam\u00eb di\u00e7ka t\u00eb till\u00eb:<\/p>\n
\/usr\/hdp\/${hdp.version}\/hadoop\/lib\/hadoop-lzo-0.6.0.${hdp.version}.jar<\/code><\/pre>\n
\nN\u00eb p\u00ebrgjith\u00ebsi, p\u00ebr ndonj\u00eb arsye, hdp.version nuk u zgjodh. Pas k\u00ebrkimeve, gjet\u00ebm zgjidhjen. Duhet t\u00eb hysh n\u00eb Ambari, n\u00eb cil\u00ebsimet e YARN dhe t\u00eb shtosh atje nj\u00eb paramet\u00ebr n\u00eb yarn-site-n e personalizuar:<\/p>\n
hdp.version=2.5.3.0-37<\/code><\/pre>\n
\nKjo magji ndihmoi, dhe Spark fluturoi. Testuam disa nga jupyter-notebook-et tona. \u00c7do gj\u00eb funksionon. Jemi gati p\u00ebr sesionin e par\u00eb p\u00ebr Spark t\u00eb shtun\u00ebn (n\u00eb dit\u00ebn e nes\u00ebrme)!<\/p>\n
UPD<\/b>. Gjat\u00eb sesionit doli nj\u00eb problem tjet\u00ebr. N\u00eb nj\u00eb moment, YARN ndaloi s\u00eb dh\u00ebni konteiner\u00eb p\u00ebr Spark. Duhej t\u00eb rregullohej nj\u00eb paramet\u00ebr n\u00eb YARN, i cili nga default ishte 0.2:<\/p>\n
yarn.scheduler.capacity.maximum-am-resource-percent=0.8<\/code><\/pre>\n
\nDo t\u00eb thot\u00eb se vet\u00ebm 20% e burimeve mor\u00ebn pjes\u00eb n\u00eb shp\u00ebrndarjen e burimeve. Duke ndryshuar parametrat, e ngarkuam p\u00ebrs\u00ebri YARN. Problemi u zgjidh dhe pjes\u00ebmarr\u00ebsit e tjer\u00eb arrit\u00ebn gjithashtu t\u00eb aktivizojn\u00eb kontekstin e spark.
\n
Burimi: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-30548","post","type-post","status-publish","format-standard","hentry","category-novosti-interneta"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Konfigurimi i Spark n\u00eb YARN | ProHoster","description":"Habr, p\u00ebrsh\u00ebndetje! Dje n\u00eb takimin e dedikuar Apache Spark, nga djemt\u00eb e Rambler&Co, kishte shum\u00eb pyetje nga pjes\u00ebmarr\u00ebsit n\u00eb lidhje me konfigurimin e k\u00ebtij \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442i. Vendos\u00ebm t\u00eb ndajm\u00eb p\u00ebrvoj\u00ebn ton\u00eb. Tema \u00ebsht\u00eb e komplikuar \u2014 k\u00ebshtu q\u00eb e inkurajojm\u00eb t\u00eb ndajm\u00eb eksperiencat po ashtu n\u00eb komentet, ndoshta ne gjithashtu nuk e kuptojm\u00eb di\u00e7ka si\u00e7 duhet dhe e p\u00ebrdorim gabimisht. Nj\u00eb hyrje e vog\u00ebl \u2014 si ne","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 Spark \u043d\u0430 YARN | ProHoster","og:description":"\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b","og:url":"https:\/\/prohoster.info\/sq\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:36:08+00:00","article:modified_time":"2019-10-31T18:36:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"30548","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 01:45:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:33:26","updated":"2026-01-21 01:45:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/30548","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=30548"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/30548\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=30548"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=30548"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=30548"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}