{"id":30548,"date":"2019-10-31T21:36:08","date_gmt":"2019-10-31T18:36:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/konfigurirovanie-spark-na-yarn\/"},"modified":"2019-10-31T21:36:08","modified_gmt":"2019-10-31T18:36:08","slug":"konfigurirovanie-spark-na-yarn","status":"publish","type":"post","link":"https:\/\/prohoster.info\/no\/blog\/news\/konfigurirovanie-spark-na-yarn","title":{"rendered":"Konfigurering av Spark p\u00e5 YARN","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Habr, hei! I g\u00e5r p\u00e5 meetupen om Apache Spark<\/a><\/noindex>, fra folkene hos Rambler&Co, var det en god del sp\u00f8rsm\u00e5l fra deltakerne relatert til konfigurasjonen av dette verkt\u00f8yet. Vi bestemte oss for \u00e5 dele erfaringene v\u00e5re. Temaet er komplisert \u2014 derfor oppfordrer vi ogs\u00e5 til \u00e5 dele erfaringer i kommentarene, kanskje vi ogs\u00e5 misforst\u00e5r og bruker noe feil.
\n<\/a><\/noindex>
\nEn liten introduksjon \u2014 hvordan vi bruker Spark. Vi har et tre m\u00e5neders program \u201cStore dataspesialist\u201d<\/a><\/noindex>, og i den andre modulen jobber deltakerne v\u00e5re med dette verkt\u00f8yet. Dermed er det v\u00e5r oppgave, som arrang\u00f8rer, \u00e5 forberede en klynge til bruk i en slik sammenheng.<\/p>\n

En spesiell egenskap ved bruken v\u00e5r er at antallet personer som jobber med Spark samtidig kan v\u00e6re lik hele gruppen. For eksempel, p\u00e5 et seminar, n\u00e5r alle pr\u00f8ver \u00e5 f\u00f8lge med p\u00e5 hva l\u00e6reren v\u00e5r gj\u00f8r. Og det er ikke lite \u2014 noen ganger opp mot 40 personer. Det er sikkert ikke s\u00e5 mange selskaper i verden som m\u00f8ter et slikt bruksopplegg.<\/p>\n

Videre vil jeg fortelle hvordan og hvorfor vi valgte de forskjellige konfigurasjonsparameterne.<\/p>\n

La oss begynne fra starten. Spark har 3 m\u00e5ter \u00e5 operere p\u00e5 en klynge: standalone, ved \u00e5 bruke Mesos eller ved \u00e5 bruke YARN. Vi valgte det tredje alternativet fordi det virket logisk for oss. Vi har allerede en hadoop-klynge. Deltakerne v\u00e5re er godt kjent med arkitekturen. La oss bruke YARN.<\/p>\n

spark.master=yarn<\/code><\/pre>\n
\nDet blir enda mer interessant. Hver av disse 3 distribusjonsalternativene har 2 distribusjonsmoduser: client og cluster. Basert p\u00e5 dokumentasjonen<\/a><\/noindex> og forskjellige lenker p\u00e5 nettet, kan man konkludere med at client-modus passer for interaktivt arbeid \u2014 for eksempel gjennom jupyter notebook, mens cluster-modus er mer passende for produksjonsl\u00f8sninger. I v\u00e5rt tilfelle var vi interessert i interaktivt arbeid, s\u00e5:<\/p>\n
spark.deploy-mode=client<\/code><\/pre>\n
\nFra dette \u00f8yeblikket begynner Spark \u00e5 fungere p\u00e5 YARN, men det var ikke nok for oss. Siden programmet v\u00e5rt handler om store data, falt det noen ganger deltakerne kort n\u00e5r det kom til den jevne fordelingen av ressurser. Og her fant vi noe interessant \u2013 dynamisk ressursallokering. Kort fortalt er essensen som f\u00f8lger: hvis du har en tung oppgave og klyngen er ledig (for eksempel om morgenen), kan Spark med denne opsjonen gi deg ekstra ressurser. Behovet beregnes etter en smart formel. Vi skal ikke g\u00e5 n\u00e6rmere inn p\u00e5 detaljene \u2013 den fungerer ganske godt.<\/p>\n
spark.dynamicAllocation.enabled=true<\/code><\/pre>\n
\nVi satte denne parameteren, og da vi startet Spark, klaget den og startet ikke. Riktig, fordi vi burde ha lest dokumentasjonen<\/a><\/noindex> n\u00f8ye. Det st\u00e5r at for at alt skal v\u00e6re greit, m\u00e5 vi ogs\u00e5 aktivere en ekstra parameter.<\/p>\n
spark.shuffle.service.enabled=true<\/code><\/pre>\n
\nHvorfor er den n\u00f8dvendig? N\u00e5r jobben v\u00e5r ikke lenger krever s\u00e5 mange ressurser, m\u00e5 Spark returnere dem til det felles pulje. Den mest ressurskrevende fasen i nesten hver MapReduce-oppgave er Shuffle-fasen. Denne parameteren lar oss bevare dataene som dannes i denne fasen og dermed frigj\u00f8re executors. En executor er en prosess som beregner alt p\u00e5 arbeideren. Den har et visst antall prosesseringskjerner og en viss mengde minne.<\/p>\n
Vi la til denne parameteren. Alt s\u00e5 ut til \u00e5 fungere. Det ble merkbart at deltakerne faktisk fikk tildelt flere ressurser n\u00e5r de trengte det. Men et nytt problem oppsto \u2013 p\u00e5 et tidspunkt v\u00e5knet andre deltakere og \u00f8nsket ogs\u00e5 \u00e5 bruke Spark, men alt var opptatt, og de var misforn\u00f8yde. Det er forst\u00e5elig. Vi begynte \u00e5 se i dokumentasjonen. Der viste det seg at det er flere parametere ved hjelp av hvilke vi kan p\u00e5virke prosessen. For eksempel, hvis executor er i ventemodus \u2013 hvor lenge etter kan vi hente ressursene?<\/p>\n
spark.dynamicAllocation.executorIdleTimeout=120s<\/code><\/pre>\n
\nI v\u00e5rt tilfelle \u2014 hvis executorsene dine ikke gj\u00f8r noe i l\u00f8pet av to minutter, vennligst returner dem til den felles puljen. Men denne parameteren var ikke alltid tilstrekkelig. Det var tydelig at en person ikke gjorde noe p\u00e5 lenge, men ressursene ble ikke frigjort. Det viste seg at det fantes en spesialparameter \u2014 hvor lenge det skulle g\u00e5 f\u00f8r man tok tilbake executors som inneholdt bufrede data. Som standard sto denne parameteren p\u00e5 \u2014 uendelig! Vi har justert den.<\/p>\n
spark.dynamicAllocation.cachedExecutorIdleTimeout=600s<\/code><\/pre>\n
\nDet vil si, hvis executorsene dine ikke gj\u00f8r noe i l\u00f8pet av 5 minutter, returner dem til den felles puljen. I dette moduset ble hastigheten p\u00e5 frigj\u00f8ring og utlevering av ressurser til mange brukere tilfredsstillende. Antallet misforn\u00f8yde personer ble redusert. Men vi bestemte oss for \u00e5 g\u00e5 videre og begrense det maksimale antallet executors for \u00e9n applikasjon \u2014 i prinsippet for \u00e9n deltaker i programmet.<\/p>\n
spark.dynamicAllocation.maxExecutors=19<\/code><\/pre>\n
\nN\u00e5, selvsagt, har det dukket opp misforn\u00f8yde fra en annen side \u2014 \u201cklusteret st\u00e5r stille, og jeg har bare 19 executors\u201d, men hva skal man gj\u00f8re \u2014 vi trenger en riktig balanse. \u00c5 gj\u00f8re alle glade vil ikke v\u00e6re mulig.<\/p>\n
Og en liten historie knyttet til spesifisiteten i v\u00e5rt tilfelle. Noen personer kom for sent til praktisk undervisning, og Spark startet ikke av en eller annen grunn. Vi s\u00e5 p\u00e5 antallet frie ressurser \u2014 det s\u00e5 ut til at det var nok. Spark burde starte. Heldigvis, p\u00e5 det tidspunktet, hadde dokumentasjonen allerede festet seg i hukommelsen v\u00e5r, og vi husket at n\u00e5r Spark starter, leter det etter en port \u00e5 starte p\u00e5. Hvis den f\u00f8rste porten i omr\u00e5det er opptatt, g\u00e5r det videre til neste i rekken. Hvis den er ledig, tar det tak i den. Og det finnes en parameter som angir det maksimale antallet fors\u00f8k for dette. Som standard \u2014 er det 16. Tallet er mindre enn antallet personer i gruppen v\u00e5r p\u00e5 undervisningen. Dermed, etter 16 fors\u00f8k, ga Spark opp og sa at det ikke kunne starte. Vi har justert denne parameteren.<\/p>\n
spark.port.maxRetries=50<\/code><\/pre>\n
\nVidere vil jeg fortelle om noen innstillinger som ikke er s\u00e5 sterkt knyttet til spesifisiteten i v\u00e5rt tilfelle.<\/p>\n
For raskere oppstart av Spark er det anbefalt \u00e5 komprimere mappen jars, som ligger i hjemmemappen SPARK_HOME, og legge den p\u00e5 HDFS. Da vil den ikke bruke tid p\u00e5 \u00e5 laste ned disse jar-filene til arbeiderne.<\/p>\n
spark.yarn.archive=hdfs:\/\/\/tmp\/spark-archive.zip<\/code><\/pre>\n
\nFor faster performance, it is also recommended to use Kryo as the serializer. It is more optimized than the default one.<\/p>\n
spark.serializer=org.apache.spark.serializer.KryoSerializer<\/code><\/pre>\n
\nThere is also a long-standing problem with Spark that it often crashes due to memory issues. This frequently happens when workers have computed everything and send the result to the driver. We increased this parameter for ourselves. By default, it is 1GB, we made it 3.<\/p>\n
spark.driver.maxResultSize=3072<\/code><\/pre>\n
\nLastly, as a bonus. How to upgrade Spark to version 2.1 on the HortonWorks distribution \u2014 HDP 2.5.3.0. This HDP version contains a pre-installed version 2.0, but we once decided that Spark is actively developing, and each new version fixes certain bugs and adds additional capabilities, including for the Python API, so we decided it was time for an update.<\/p>\n
We downloaded the version from the official website for Hadoop 2.7. Unzipped it, placed it in the HDP folder. We created symlinks as needed. We try to run it \u2014 it doesn't start. It gives a very unclear error.<\/p>\n
java.lang.NoClassDefFoundError: com\/sun\/jersey\/api\/client\/config\/ClientConfig<\/code><\/pre>\n
\nAfter googling, we found out that Spark decided not to wait for Hadoop and opted to use a new version of Jersey. They are arguing about this matter in JIRA. The solution was to download Jersey version 1.17.1<\/a><\/noindex>. Place it in the jars folder in SPARK_HOME, zip it again, and upload it to HDFS.<\/p>\n
We bypassed this error, but a new one appeared that was quite vague.<\/p>\n
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master<\/code><\/pre>\n
\nAt the same time, we tried running version 2.0 \u2014 everything is fine. Try to guess what the issue is. We looked into the logs of this application and saw something like:<\/p>\n
\/usr\/hdp\/${hdp.version}\/hadoop\/lib\/hadoop-lzo-0.6.0.${hdp.version}.jar<\/code><\/pre>\n
\nIn general, for some reason hdp.version was not resolved. After googling, we found a solution. We need to go into Ambari settings for YARN and add this parameter in custom yarn-site:<\/p>\n
hdp.version=2.5.3.0-37<\/code><\/pre>\n
\nThis magic worked, and Spark took off. We tested several of our Jupyter notebooks. Everything works. We are ready for the first Spark class on Saturday (already tomorrow)!<\/p>\n
UPD<\/b>. During the class, another problem emerged. At some point, YARN stopped issuing containers for Spark. We needed to adjust the parameter in YARN, which by default was set to 0.2:<\/p>\n
yarn.scheduler.capacity.maximum-am-resource-percent=0.8<\/code><\/pre>\n
\nDet vil si at bare 20 % av ressursene deltok i distribusjonen av ressursene. Ved \u00e5 endre parameterne ble YARN omstartet. Problemet ble l\u00f8st, og de andre deltakerne kunne ogs\u00e5 starte spark-konteksten.
\n
Kilde: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-30548","post","type-post","status-publish","format-standard","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Konfigurering av Spark p\u00e5 YARN | ProHoster","description":"Habr, hei!","canonical_url":"https:\/\/prohoster.info\/no\/blog\/news\/konfigurirovanie-spark-na-yarn","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"nn_NO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 Spark \u043d\u0430 YARN | ProHoster","og:description":"\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442!","og:url":"https:\/\/prohoster.info\/no\/blog\/news\/konfigurirovanie-spark-na-yarn","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:36:08+00:00","article:modified_time":"2019-10-31T18:36:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"30548","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 01:45:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:33:26","updated":"2026-01-21 01:45:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts\/30548","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/comments?post=30548"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/posts\/30548\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/media?parent=30548"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/categories?post=30548"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/no\/wp-json\/wp\/v2\/tags?post=30548"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}