{"id":30548,"date":"2019-10-31T21:36:08","date_gmt":"2019-10-31T18:36:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/konfigurirovanie-spark-na-yarn\/"},"modified":"2019-10-31T21:36:08","modified_gmt":"2019-10-31T18:36:08","slug":"konfigurirovanie-spark-na-yarn","status":"publish","type":"post","link":"https:\/\/prohoster.info\/da\/blog\/news\/konfigurirovanie-spark-na-yarn","title":{"rendered":"Konfiguration af Spark p\u00e5 YARN","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Hej Habr! I g\u00e5r p\u00e5 meetup'en om Apache Spark<\/a><\/noindex>, fra folkene hos Rambler&Co, var der ganske mange sp\u00f8rgsm\u00e5l fra deltagerne relateret til konfigurationen af dette v\u00e6rkt\u00f8j. Vi besluttede at dele vores erfaringer efterf\u00f8lgende. Emnet er ikke simpelt \u2014 derfor opfordrer vi til at dele erfaringer i kommentarerne ogs\u00e5, m\u00e5ske forst\u00e5r og anvender vi ogs\u00e5 noget forkert.
\n<\/a><\/noindex>
\nEn lille introduktion \u2014 hvordan vi bruger Spark. Vi har et tre-m\u00e5neders program \u201cSpecialist i store data\u201d<\/a><\/noindex>, og i hele den anden modul arbejder vores deltagere p\u00e5 dette v\u00e6rkt\u00f8j. Derfor er vores opgave som arrang\u00f8rer at forberede et cluster til brug i forbindelse med et s\u00e5dant case.<\/p>\n

Det s\u00e6rlige ved vores brug er, at antallet af personer, der arbejder p\u00e5 Spark samtidigt, kan v\u00e6re lig med hele gruppen. For eksempel, p\u00e5 seminaret, n\u00e5r alle samtidigt pr\u00f8ver noget og f\u00f8lger efter vores instrukt\u00f8r. Det er en del mennesker \u2014 nogle gange omkring 40. M\u00e5ske er der ikke s\u00e5 mange virksomheder i verden, der st\u00e5r over for et s\u00e5dant brugsscenario.<\/p>\n

Dern\u00e6st vil jeg fort\u00e6lle, hvordan og hvorfor vi valgte bestemte konfigurationsparametre.<\/p>\n

Lad os begynde fra starten. Spark har 3 m\u00e5der at arbejde p\u00e5 et cluster: standalone, ved at bruge Mesos og ved at bruge YARN. Vi valgte den tredje mulighed, fordi den gav mening for os. Vi har allerede et Hadoop-cluster. Vores deltagere er allerede godt bekendt med dets arkitektur. Lad os bruge YARN.<\/p>\n

spark.master=yarn<\/code><\/pre>\n
\nHerefter bliver det interessant. Hver af disse 3 distributionsmuligheder har 2 implementeringsmetoder: client og cluster. Baseret p\u00e5 dokumentationen<\/a><\/noindex> og forskellige links p\u00e5 internettet, kan vi konkludere, at client er velegnet til interaktivt arbejde \u2014 f.eks. gennem jupyter notebook, mens cluster er mere passende til produktionsl\u00f8sninger. I vores tilf\u00e6lde var vi interesseret i interaktivt arbejde, derfor:<\/p>\n
spark.deploy-mode=client<\/code><\/pre>\n
\nFra dette \u00f8jeblik begynder Spark egentlig allerede at fungere p\u00e5 YARN, men det var ikke nok for os. Da vi arbejder med store data, mangler deltagerne nogle gange det, der opn\u00e5s ved en j\u00e6vn fordeling af ressourcer. Her fandt vi en interessant l\u00f8sning \u2014 dynamisk ressourceallokering. Kort sagt g\u00e5r det ud p\u00e5, at hvis du har en kr\u00e6vende opgave og klyngen er ledig (f.eks. om morgenen), kan Spark ved hj\u00e6lp af denne indstilling give dig ekstra ressourcer. Behovet beregnes efter en smart formel. Vi g\u00e5r ikke i detaljer \u2014 den fungerer ganske godt.<\/p>\n
spark.dynamicAllocation.enabled=true<\/code><\/pre>\n
\nVi satte denne parameter, og da vi startede Spark, klagede det og kunne ikke starte. Det var korrekt, fordi vi skulle have l\u00e6st dokumentationen<\/a><\/noindex> mere omhyggeligt. Der st\u00e5r, at for at alt skal v\u00e6re i orden, er det n\u00f8dvendigt at aktivere en ekstra parameter.<\/p>\n
spark.shuffle.service.enabled=true<\/code><\/pre>\n
\nHvad er den n\u00f8dvendig for? N\u00e5r vores job ikke l\u00e6ngere kr\u00e6ver s\u00e5 mange ressourcer, skal Spark returnere dem til den samlede pulje. Den mest ressourcekr\u00e6vende fase i n\u00e6sten enhver MapReduce-opgave er Shuffle-fasen. Denne parameter g\u00f8r det muligt at gemme data, der genereres i denne fase, og dermed frig\u00f8re eksekutorerne. En eksekutor er en proces, der beregner alt p\u00e5 en worker. Den har et bestemt antal CPU-kerner og en bestemt m\u00e6ngde hukommelse.<\/p>\n
Vi tilf\u00f8jede denne parameter. Alt syntes at fungere. Det blev tydeligt, at deltagerne faktisk fik tildelt flere ressourcer, n\u00e5r de havde brug for det. Men der opstod et andet problem \u2014 p\u00e5 et tidspunkt v\u00e5gnede de andre deltagere og ville ogs\u00e5 bruge Spark, men der var allerede optaget, og de var utilfredse. Man kan forst\u00e5 dem. Vi begyndte at kigge i dokumentationen. Der viste sig at v\u00e6re et par andre parametre, som kan p\u00e5virke processen. For eksempel, hvis eksekutoren er i ventetilstand \u2014 efter hvor lang tid kan man tage ressourcerne tilbage?<\/p>\n
spark.dynamicAllocation.executorIdleTimeout=120s<\/code><\/pre>\n
\nI vores tilf\u00e6lde \u2014 hvis dine executors ikke laver noget i l\u00f8bet af to minutter, skal du venligst returnere dem til den f\u00e6lles pulje. Men selv denne parameter var ikke altid tilstr\u00e6kkelig. Det kunne ses, at personen ikke havde gjort noget i lang tid, men ressourcerne blev ikke frigivet. Det viste sig, at der ogs\u00e5 er en speciel parameter \u2014 efter hvor lang tid der skal v\u00e6lges executors, der indeholder cachede data. Som standard stod denne parameter p\u00e5 \u2014 uendelig! Vi har rettet det.<\/p>\n
spark.dynamicAllocation.cachedExecutorIdleTimeout=600s<\/code><\/pre>\n
\nS\u00e5 hvis dine executors ikke laver noget i l\u00f8bet af 5 minutter, s\u00e5 returner dem til den f\u00e6lles pulje. P\u00e5 denne m\u00e5de er hastigheden for frigivelse og tildeling af ressourcer til et stort antal brugere blevet anst\u00e6ndig. Antallet af utilfredse er faldet. Men vi besluttede at g\u00e5 videre og begr\u00e6nse det maksimale antal executors pr. application \u2014 i princippet pr. deltager i programmet.<\/p>\n
spark.dynamicAllocation.maxExecutors=19<\/code><\/pre>\n
\nNu er der selvf\u00f8lgelig utilfredse p\u00e5 den anden side \u2014 \"klassen st\u00e5r stille, og jeg har kun 19 executors\", men hvad kan man g\u00f8re \u2014 der skal v\u00e6re en korrekt balance. Det er ikke muligt at g\u00f8re alle glade.<\/p>\n
Og en lille historie, der er relateret til vores specifikke sag. En dag kom flere personer for sent til praktisk undervisning, og deres Spark startede af en eller anden grund ikke. Vi s\u00e5 p\u00e5 antallet af ledige ressourcer \u2014 der syntes at v\u00e6re nok. Spark skulle starte. Heldigvis var dokumentationen allerede et sted gemt i hukommelsen, og vi huskede, at n\u00e5r Spark starter, leder den efter en port, den kan starte p\u00e5. Hvis den f\u00f8rste port i omr\u00e5det er optaget, g\u00e5r den videre til den n\u00e6ste. Hvis den er ledig, tager den den. Og der er en parameter, der angiver det maksimale antal fors\u00f8g for dette. Som standard er det 16. Antallet er mindre end folkene i vores gruppe til undervisningen. Derfor efter 16 fors\u00f8g opgav Spark og sagde, at den ikke kunne starte. Vi har rettet denne parameter.<\/p>\n
spark.port.maxRetries=50<\/code><\/pre>\n
\nLad mig videre fort\u00e6lle om nogle indstillinger, der ikke er s\u00e5 meget relateret til vores specifikke sag.<\/p>\n
For hurtigere opstart af Spark anbefales det at zippe mappen jars, der ligger i hjemmemappen SPARK_HOME, og l\u00e6gge den p\u00e5 HDFS. S\u00e5 vil den ikke bruge tid p\u00e5 at indl\u00e6se disse jar-filer p\u00e5 arbejderne.<\/p>\n
spark.yarn.archive=hdfs:\/\/\/tmp\/spark-archive.zip<\/code><\/pre>\n
\nFor faster operation, it is also recommended to use Kryo as the serializer. It is more optimized than the default one.<\/p>\n
spark.serializer=org.apache.spark.serializer.KryoSerializer<\/code><\/pre>\n
\nThere is also a long-standing problem with Spark, which often crashes due to memory issues. This often happens when the workers have calculated everything and send the result to the driver. We increased this parameter. By default, it is 1GB, we set it to 3.<\/p>\n
spark.driver.maxResultSize=3072<\/code><\/pre>\n
\nAnd lastly, as a dessert. How to upgrade Spark to version 2.1 on the HortonWorks distribution - HDP 2.5.3.0. This HDP version comes with a pre-installed version 2.0, but we once decided that Spark is developing quite actively, and each new version fixes some bugs and adds features, including for the Python API, so we decided to do the update.<\/p>\n
We downloaded the version from the official website for Hadoop 2.7. We unpacked it and placed it in the HDP folder. We created the symlinks as needed. When we start it, it doesn\u2019t launch. It shows a very unclear error.<\/p>\n
java.lang.NoClassDefFoundError: com\/sun\/jersey\/api\/client\/config\/ClientConfig<\/code><\/pre>\n
\nAfter googling, we found out that Spark decided not to wait for Hadoop and opted to use a new version of Jersey. They are arguing about this in JIRA. The solution was to download Jersey version 1.17.1<\/a><\/noindex>. Place it in the jars folder in SPARK_HOME, zip it again, and upload it to HDFS.<\/p>\n
We bypassed this error, but a new and quite vague one arose.<\/p>\n
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch the application master.<\/code><\/pre>\n
\nAt the same time, we tried to run version 2.0 \u2014 everything was fine. Can you guess what the issue is? We looked into the logs of this application and saw something like this:<\/p>\n
\/usr\/hdp\/${hdp.version}\/hadoop\/lib\/hadoop-lzo-0.6.0.${hdp.version}.jar<\/code><\/pre>\n
\nIn general, for some reason, hdp.version did not resolve. After googling, we found the solution. We need to go to Ambari, access the YARN settings, and add the parameter in custom yarn-site:<\/p>\n
hdp.version=2.5.3.0-37<\/code><\/pre>\n
\nThis magic helped, and Spark took off. We tested several of our Jupyter notebooks. Everything works. We are ready for the first Spark class on Saturday (tomorrow)!<\/p>\n
UPD<\/b>. During the class, another problem arose. At some point, YARN stopped issuing containers for Spark. We had to adjust the parameter in YARN, which by default was set to 0.2:<\/p>\n
yarn.scheduler.capacity.maximum-am-resource-percent=0.8<\/code><\/pre>\n
\nDet vil sige, at kun 20 % af ressourcerne deltog i fordelingen af ressourcer. Vi \u00e6ndrede indstillingerne og genstartede YARN. Problemet blev l\u00f8st, og de \u00f8vrige deltagere kunne ogs\u00e5 starte spark-kontexten.
\n
Kilde: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-30548","post","type-post","status-publish","format-standard","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Konfiguration af Spark p\u00e5 YARN | ProHoster","description":"Habr, hej!","canonical_url":"https:\/\/prohoster.info\/da\/blog\/news\/konfigurirovanie-spark-na-yarn","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"da_DK","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 Spark \u043d\u0430 YARN | ProHoster","og:description":"\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442!","og:url":"https:\/\/prohoster.info\/da\/blog\/news\/konfigurirovanie-spark-na-yarn","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:36:08+00:00","article:modified_time":"2019-10-31T18:36:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"30548","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 01:45:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:33:26","updated":"2026-01-21 01:45:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/30548","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/comments?post=30548"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/posts\/30548\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/media?parent=30548"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/categories?post=30548"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/da\/wp-json\/wp\/v2\/tags?post=30548"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}