{"id":30548,"date":"2019-10-31T21:36:08","date_gmt":"2019-10-31T18:36:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/konfigurirovanie-spark-na-yarn\/"},"modified":"2019-10-31T21:36:08","modified_gmt":"2019-10-31T18:36:08","slug":"konfigurirovanie-spark-na-yarn","status":"publish","type":"post","link":"https:\/\/prohoster.info\/et\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn","title":{"rendered":"Sparki seadistamine YARN-il","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Tere, \u0425\u0430\u0431\u0440! Eelmine p\u00e4ev <noindex><a rel=\"nofollow\" href=\"https:\/\/www.facebook.com\/events\/1825957590998380\/\">toimus mitmik, mis k\u00e4sitles Apache Sparki<\/a><\/noindex>, Rambler&amp;Co poolt tuli palju k\u00fcsimusi osalejatelt, mis olid seotud selle t\u00f6\u00f6riista seadistamisega. Otsustasime jagada oma kogemusi. Teema on keeruline \u2013 seet\u00f5ttu kutsume \u00fcles jagama oma arvamusi ka kommentaarides, v\u00f5ib-olla on meil samuti midagi valesti arusaadav v\u00f5i kasutatav.<br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\nPisike sissejuhatus \u2014 kuidas me Spark'i kasutame. Meil on kolm kuud kestev programm <noindex><a rel=\"nofollow\" href=\"http:\/\/newprolab.com\/ru\/bigdata?utm_source=habr&amp;utm_campaign=spark\">\u201eSuure andmete spetsialist\u201c<\/a><\/noindex>, ja kogu teises moodulis t\u00f6\u00f6tavad meie osalejad selle t\u00f6\u00f6riistaga. Seega on meie \u00fclesanne, kui korraldajatena, ette valmistada klaster selle kasutamiseks sellistes tingimustes.<\/p>\n<p>Meie kasutamise erip\u00e4ra seisneb selles, et spark'iga samaaegselt t\u00f6\u00f6tavate inimeste arv v\u00f5ib olla v\u00f5rdne kogu grupiga. N\u00e4iteks seminaril, kui k\u00f5ik \u00fchiselt midagi proovivad ja kordavad meie \u00f5petaja j\u00e4rgi. Ja see pole sugugi v\u00e4ike \u2014 vahel kuni 40 inimest. Ilmselt ei ole maailmas paljusid ettev\u00f5tteid, kes sellise kasutusstsenaariumiga silmitsi seisavad.<\/p>\n<p>J\u00e4rgmine r\u00e4\u00e4gin, kuidas ja miks me valisime erinevad konfiguratsiooniparametrid.<\/p>\n<p>Alustame algusest. Sparkil on 3 v\u00f5imalust t\u00f6\u00f6tada klastris: standalone, Mesose abil ja YARNi abil. Otsustasime valida viimase variandi, kuna see oli meie jaoks loogiline. Meil on juba olemas Hadoopi klaster. Meie liikmed on selle arhitektuuriga juba h\u00e4sti tuttavad. Kasutame YARNi.<\/p>\n<pre><code class=\"apache\">spark.master=yarn<\/code><\/pre>\n<p>\nEdasi on huvitavam. Igal kolmel disainiv\u00f5imalusel on kaks j\u00f5udlust: client ja cluster. L\u00e4htuvalt <noindex><a rel=\"nofollow\" href=\"http:\/\/spark.apache.org\/docs\/latest\/running-on-yarn.html\">dokumentatsioonis<\/a><\/noindex> ja erinevatest linkidest internetis v\u00f5ib j\u00e4reldada, et client sobib interaktiivseks t\u00f6\u00f6ks \u2014 n\u00e4iteks Jupyteri m\u00e4rkmikus, samas kui cluster sobib paremini tootmislahendustele. Meie puhul huvitas meid interaktiivne t\u00f6\u00f6tamine, seega:<\/p>\n<pre><code class=\"apache\">spark.deploy-mode=client<\/code><\/pre>\n<p>\nP\u00f5him\u00f5tteliselt hakkab Spark alates sellest hetkest YARN-is mingil moel t\u00f6\u00f6le, kuid see ei olnud meile piisav. Kuna meil on programm suurte andmete kohta, siis m\u00f5nikord ei piisanud osalistest ressursside jaotustest. Siit leidsime huvitava asja \u2014 d\u00fcnaamiline ressursside jaotamine. Lihtsalt \u00f6eldes: kui teil on keeruline \u00fclesanne ja klaster on vaba (n\u00e4iteks hommikul), v\u00f5ib Spark selle v\u00f5imaluse abil teile lisarekursse anda. Vajaduse arvutamine toimub saladuslikul valemil. S\u00fcgavale detailidesse ei lasku \u2014 see t\u00f6\u00f6tab h\u00e4sti.<\/p>\n<pre><code class=\"apache\">spark.dynamicAllocation.enabled=true<\/code><\/pre>\n<p>\nMe seadsime selle parameetri ja kui k\u00e4ivitasime Sparki, nurises ta ja ei k\u00e4ivitanud. \u00d5igesti, sest oli vaja lugeda <noindex><a rel=\"nofollow\" href=\"http:\/\/spark.apache.org\/docs\/latest\/configuration.html\">dokumentatsiooni<\/a><\/noindex> t\u00e4helepanelikumalt. Seal on m\u00e4rgitud, et k\u00f5ik peab olema korras, peame sisse l\u00fclitama ka t\u00e4iendava parameetri.<\/p>\n<pre><code class=\"apache\">spark.shuffle.service.enabled=true<\/code><\/pre>\n<p>\nMiks see vajalik on? Kui meie t\u00f6\u00f6 lihtsustub ja ei vaja enam nii palju ressursse, peab Spark need tagasi \u00fcldisse basseinisse andma. K\u00f5ige t\u00f6\u00f6mahukam etapp peaaegu igas MapReduce \u00fclesandes on Shuffle etapp. See parameeter v\u00f5imaldab salvestada andmeid, mis tekivad sellel etapil, ja seega vabastada executoreid. Executor on protsess, mis k\u00f5ike t\u00f6\u00f6tleb t\u00f6\u00f6tajas. Tal on teatav arv protsessorituumasid ja teatav kogus m\u00e4lu.<\/p>\n<p>Lisatud see parameeter. K\u00f5ik tundub t\u00f6\u00f6tavat. On m\u00e4rgatav, et osalejatele antakse t\u00f5eliselt rohkem ressursse, kui neid vajavad. Kuid tekkis teine probleem \u2014 mingil hetkel \u00e4rkasid teised osalejad ja soovisid samuti Spark'i kasutada, aga k\u00f5ik oli h\u00f5ivatud ja nad olid rahulolematud. Nende tundeid on lihtne m\u00f5ista. Hakkasime dokumentatsiooni uurima. Selgus, et seal on veel mitmeid parameetreid, millega saab protsessile m\u00f5ju avaldada. N\u00e4iteks, kui executor on oote-re\u017eiimis \u2014 kui kaua v\u00f5ib oodata, kuni ressursid temalt \u00e4ra v\u00f5tta?<\/p>\n<pre><code class=\"apache\">spark.dynamicAllocation.executorIdleTimeout=120s<\/code><\/pre>\n<p>\nMeie puhul, kui teie executors ei tee midagi kahe minuti jooksul, siis palun tagastage need \u00fcldsesse puuli. Kuid isegi see parameeter ei olnud alati piisav. Oli n\u00e4ha, et inimene ei ole pikka aega midagi teinud, kuid ressursse ei vabastatud. Selgus, et on olemas veel eriline parameeter \u2014 p\u00e4eva jooksul, kui kaua tohib hoida executors, mis sisaldavad vahem\u00e4lus andmeid. Vaikimisi oli selle parameetri v\u00e4\u00e4rtus \u2014 infinity! Me parandasime selle.<\/p>\n<pre><code class=\"apache\">spark.dynamicAllocation.cachedExecutorIdleTimeout=600s<\/code><\/pre>\n<p>\nSee t\u00e4hendab, et kui 5 minuti jooksul teie executors ei tee midagi, tagastage need \u00fcldsesse puuli. Sel viisil on suurte kasutajate arvu jaoks ressursi vabastamise ja jagamise kiirus saanud korralikuks. Rahulolematute arv on v\u00e4henenud. Kuid otsustasime minna edasi ja piirata maksimaalset executorside arvu \u00fche rakenduse kohta \u2014 p\u00f5him\u00f5tteliselt \u00fche programmi osaleja kohta.<\/p>\n<pre><code class=\"apache\">spark.dynamicAllocation.maxExecutors=19<\/code><\/pre>\n<p>\nN\u00fc\u00fcd on muidugi tekkinud rahulolematud ka teiselt poolt \u2014 \"kluster seisab, aga mul on ainult 19 executors\", aga mis teha \u2014 vajalik on leida mingi \u00f5ige tasakaal. K\u00f5iki \u00f5nnelikuks teha ei \u00f5nnestu.<\/p>\n<p>Ja veel \u00fcks v\u00e4ike lugu, mis on seotud meie juhtumi erip\u00e4radega. \u00dchel korral hilinesid m\u00f5ned inimesed praktilisele tunnile ning nende Spark ei k\u00e4ivitunud. Vaatasime vabaressursse \u2013 k\u00f5ik n\u00e4is olevat korras. Spark peaks k\u00e4ivituma. \u00d5nneks oli selleks ajaks dokumentatsioon juba kuskil alateadvuses, ja meenutasime, et Spark otsib k\u00e4ivitamise ajal endale porti. Kui esimene port vahemikus on h\u00f5ivatud, siis liigub ta j\u00e4rgmise juurde. Kui see on vaba, siis haarab ta selle. Ja seal on seade, mis m\u00e4\u00e4rab maksimaalse katsete arvu selle jaoks. Vaikes\u00e4tetena on see 16. See number on v\u00e4iksem kui meie grupis praktilisel tunnil osalejate arv. Seega, p\u00e4rast 16 katset viskas Spark selle asja k\u00e4est ja \u00fctles, et ei saa k\u00e4ivituda. Muutsime seda seadet.<\/p>\n<pre><code class=\"apache\">spark.port.maxRetries=50<\/code><\/pre>\n<p>\nR\u00e4\u00e4gin n\u00fc\u00fcd m\u00f5ningatest seadistustest, mis ei ole enam tugevalt seotud meie juhtumi erip\u00e4radega.<\/p>\n<p>Selleks, et Spark k\u00e4ivituks kiiremini, on soovitus tihendada jars kaust, mis asub SPARK_HOME kodu kataloogis, ja paigutada see HDFS-i. Nii ei pea ta kulutama aega nende jar-failide laadimisele t\u00f6\u00f6tajatele.<\/p>\n<pre><code class=\"apache\">spark.yarn.archive=hdfs:\/\/\/tmp\/spark-archive.zip<\/code><\/pre>\n<p>\nSelleks, et saavutada kiiremat t\u00f6\u00f6d, soovitatakse kasutada serialiseerijana kryo. See on paremini optimeeritud kui vaikimisi variant.<\/p>\n<pre><code class=\"apache\">spark.serializer=org.apache.spark.serializer.KryoSerializer<\/code><\/pre>\n<p>\nJa on veel \u00fcks vana probleem Sparkis, et see kipub sageli m\u00e4lu \u00fcletama. See juhtub sageli hetkel, kui t\u00f6\u00f6lised on k\u00f5ik arvutused teinud ja saadavad tulemuse draiverile. Me muutsime selle parameetri suuremaks. Vaikimisi on see 1 GB, meil on 3.<\/p>\n<pre><code class=\"apache\">spark.driver.maxResultSize=3072<\/code><\/pre>\n<p>\nJa viimane asi, magustoiduks. Kuidas uuendada Spark 2.1 versiooniks HortonWorks'i jaotises HDP 2.5.3.0. See HDP versioon sisaldab eelnevalt installitud 2.0 versiooni, kuid me otsustasime, et Spark areneb \u00fcsna aktiivselt ning iga uus versioon lahendab m\u00f5ningaid vigu ja annab t\u00e4iendavaid v\u00f5imalusi, sealhulgas python API jaoks, seega otsustasime, et uuendamine on vajalik.<\/p>\n<p>Laaditud versioon ametlikult veebilehelt Hadoop 2.7 jaoks. Lahti pakkida, paigutada HDP kausta. Panime s\u00fcmboolikud \u00f5igesti. K\u00e4ivitame - ei k\u00e4ivitu. N\u00e4itab v\u00e4ga arusaamatut viga.<\/p>\n<pre><code class=\"apache\">java.lang.NoClassDefFoundError: com\/sun\/jersey\/api\/client\/config\/ClientConfig<\/code><\/pre>\n<p>\nOtsisime Google'ist, et Spark otsustas mitte oodata, kuni Hadoop v\u00e4lja tuleb, ja kasutama hakata uut jersey versiooni. Nad t\u00fclitsevad selle \u00fcle JIRA-s. Lahenduseks oli - alla laadida <noindex><a rel=\"nofollow\" href=\"https:\/\/mvnrepository.com\/artifact\/com.sun.jersey\/jersey-bundle\/1.17.1\">jersey versioon 1.17.1<\/a><\/noindex>. Panna see SPARK_HOME'i jars kausta, j\u00e4lle zipida ja HDFS-i \u00fcles laadida.<\/p>\n<p>Selle vea me \u00fcletasime, kuid ilmnes uus ja \u00fcsna keeruline.<\/p>\n<pre><code class=\"apache\">org.apache.spark.SparkException: Yarn rakendus on juba l\u00f5petatud! See v\u00f5is olla l\u00f5petatud v\u00f5i ei suutnud rakenduse pead k\u00e4ivitada<\/code><\/pre>\n<p>\nSamas proovime k\u00e4ivitada versiooni 2.0 - k\u00f5ik on korras. Proovi arvata, mis siin toimub. Me vaatlesime selle rakenduse logisid ja n\u00e4gime midagi sellist:<\/p>\n<pre><code class=\"apache\">\/usr\/hdp\/${hdp.version}\/hadoop\/lib\/hadoop-lzo-0.6.0.${hdp.version}.jar<\/code><\/pre>\n<p>\n\u00dches\u00f5naga, mingil p\u00f5hjusel hdp.version ei lahendunud. Otsides leidsime lahenduse. Ambari seadetes tuleb minna YARN-i seadetesse ja lisada custom yarn-site'i parameeter:<\/p>\n<pre><code class=\"apache\">hdp.version=2.5.3.0-37<\/code><\/pre>\n<p>\nSee maagia aitas ning Spark t\u00f5usis. Testisime mitmeid meie jupyter-notebook'e. K\u00f5ik t\u00f6\u00f6tab. Oleme valmis laup\u00e4evale (juba homme) toimuvale Spark'i esimeses tunnis!<\/p>\n<p><b>UPD<\/b>. Tunnis selgus veel \u00fcks probleem. M\u00f5nel hetkel l\u00f5petas YARN Spark'i konteinerite andmise. YARN-is pidi muutma parameetrit, mis vaikimisi oli 0.2:<\/p>\n<pre><code class=\"apache\">yarn.scheduler.capacity.maximum-am-resource-percent=0.8<\/code><\/pre>\n<p>\nSee, only 20% of the resources were involved in resource distribution. After changing the parameters, we restarted YARN. The problem was solved, and the other participants were also able to launch the spark context.<br \/>\n<br \/>Allikas: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/newprolab\/blog\/327556\/\">habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&#038;Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-30548","post","type-post","status-publish","format-standard","hentry","category-novosti-interneta"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.0.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&amp;Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/et\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.0.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"et_EE\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 Spark \u043d\u0430 YARN | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&amp;Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/et\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2019-10-31T18:36:08+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2019-10-31T18:36:08+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Spark Configuration on YARN | ProHoster","description":"Habr, hello! Yesterday at the meetup dedicated to Apache Spark, hosted by the guys from Rambler&amp;Co, there were quite a few questions from participants related to configuring this tool. We decided to share our experience in light of this. The topic is complex \u2014 therefore, we encourage sharing experiences in the comments as well; maybe there\u2019s something we don\u2019t understand or use properly. A brief introduction \u2014 how we","canonical_url":"https:\/\/prohoster.info\/et\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"et_EE","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 Spark \u043d\u0430 YARN | ProHoster","og:description":"\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&amp;Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b","og:url":"https:\/\/prohoster.info\/et\/blog\/novosti-interneta\/konfigurirovanie-spark-na-yarn","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:36:08+00:00","article:modified_time":"2019-10-31T18:36:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"30548","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 01:45:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:33:26","updated":"2026-01-21 01:45:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/30548","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/comments?post=30548"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/posts\/30548\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/media?parent=30548"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/categories?post=30548"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/et\/wp-json\/wp\/v2\/tags?post=30548"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}