{"id":30548,"date":"2019-10-31T21:36:08","date_gmt":"2019-10-31T18:36:08","guid":{"rendered":"https:\/\/prohoster.info\/blog\/konfigurirovanie-spark-na-yarn\/"},"modified":"2019-10-31T21:36:08","modified_gmt":"2019-10-31T18:36:08","slug":"konfigurirovanie-spark-na-yarn","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/news\/konfigurirovanie-spark-na-yarn","title":{"rendered":"Configuring Spark on YARN","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Hello, Habr! Yesterday at the meetup dedicated to Apache Spark<\/a><\/noindex>, from the team at Rambler&Co, there were quite a few questions from participants regarding the configuration of this tool. We decided to share our experience based on those queries. It's a complex topic \u2014 we encourage sharing experiences in the comments as well, as there might be things we're also misunderstanding or misusing.
\n<\/a><\/noindex>
\nA brief introduction \u2014 how we use Spark. We have a three-month program \u201cBig Data Specialist\u201d<\/a><\/noindex>, and in the second module, our participants work with this tool. Accordingly, our task as organizers is to prepare a cluster for use within such a case.<\/p>\n

The peculiarity of our use is that the number of people working on Spark simultaneously can be equal to the entire group. For instance, during a seminar, when everyone is trying out something together and following our instructor. That can be quite a lot \u2014 sometimes nearly 40 people. There probably aren't many companies in the world that face such a usage scenario.<\/p>\n

Next, I'll explain how and why we selected certain configuration parameters.<\/p>\n

Let\u2019s start from the very beginning. Spark has three options for running on the cluster: standalone, using Mesos, and using YARN. We decided to choose the third option, as it made sense for us. We already have a Hadoop cluster, and our participants are well acquainted with its architecture. Let\u2019s use YARN.<\/p>\n

spark.master=yarn<\/code><\/pre>\n
\nNext, it gets more interesting. Each of these three deployment options has two deployment modes: client and cluster. Based on the documentation<\/a><\/noindex> various online references, one can conclude that the client mode is suitable for interactive work \u2014 for example, through Jupyter Notebook, while the cluster mode is more suitable for production solutions. In our case, we were interested in interactive work, so:<\/p>\n
spark.deploy-mode=client<\/code><\/pre>\n
\nFrom this moment, Spark will begin functioning on YARN, but that wasn't enough for us. Since our program is focused on big data, sometimes participants lacked the resources provided by the uniform allocation. Here, we discovered an interesting feature \u2014 dynamic resource allocation. In short, the essence is as follows: if you have a heavy task and the cluster is free (for instance, in the morning), Spark can allocate additional resources for you with this option. The necessity is determined by a clever formula. We won't delve into the details \u2014 it works quite well.<\/p>\n
spark.dynamicAllocation.enabled=true<\/code><\/pre>\n
\nWe set this parameter, and upon starting Spark, it complained and didn't launch. Rightly so, because we needed to read documentation<\/a><\/noindex> more carefully. It states that to ensure everything is okay, an additional parameter must also be enabled.<\/p>\n
spark.shuffle.service.enabled=true<\/code><\/pre>\n
\nWhat is it for? When our job no longer requires such a large amount of resources, Spark should return them to the common pool. The most resource-intensive stage in almost any MapReduce job is the Shuffle stage. This parameter allows saving the data generated at this stage and, accordingly, freeing up executors. An executor is the process that performs all the calculations on the worker. It has a certain number of CPU cores and a certain amount of memory.<\/p>\n
We added this parameter. Everything seemed to work. It became apparent that participants were indeed receiving more resources when they needed them. But another problem arose \u2014 at some point, other participants woke up and also wanted to use Spark, but everything was occupied, and they were unhappy. It's understandable. We started looking into the documentation. It turned out there are additional parameters that can influence the process. For example, if an executor is in idle mode \u2014 after how long can resources be reclaimed?<\/p>\n
spark.dynamicAllocation.executorIdleTimeout=120s<\/code><\/pre>\n
\nIn our case, if your executors do nothing for two minutes, please return them to the general pool. However, this parameter was not always sufficient. It was clear that a person had been inactive for a long time, yet resources were not being released. It turned out that there was a special parameter \u2014 how long to wait before reclaiming executors that contain cached data. By default, this parameter was set to infinity! We corrected it.<\/p>\n
spark.dynamicAllocation.cachedExecutorIdleTimeout=600s<\/code><\/pre>\n
\nSo if your executors do nothing for 5 minutes, return them to the general pool. In this mode, the speed of resource release and allocation for a large number of users became satisfactory. The level of dissatisfaction decreased. However, we decided to go further and limit the maximum number of executors per application \u2014 essentially per participant in the program.<\/p>\n
spark.dynamicAllocation.maxExecutors=19<\/code><\/pre>\n
\nOf course, now some were unhappy from the other side \u2014 \"the cluster is idle, and I only have 19 executors,\" but what can you do \u2014 a proper balance is needed. It is impossible to make everyone happy.<\/p>\n
And one more small story related to the specifics of our case. A few people were late for a practical lesson, and for some reason, their Spark did not start. We looked at the number of free resources \u2014 there seemed to be enough. Spark should start. Fortunately, by that time the documentation was already ingrained in our minds, and we remembered that when starting Spark looks for a port to start on. If the first port in the range is occupied, it moves to the next one. If it\u2019s free, it takes it. There is a parameter that specifies the maximum number of attempts for this. By default, it is 16. This is fewer than the number of people in our class. Accordingly, after 16 attempts, Spark would give up and say that it could not start. We adjusted this parameter.<\/p>\n
spark.port.maxRetries=50<\/code><\/pre>\n
\nNext, I will tell you about some settings that are not closely related to the specifics of our case.<\/p>\n
For a faster Spark startup, it is recommended to archive the jars folder located in the home directory SPARK_HOME and place it on HDFS. This way, it will not waste time loading these jars on workers.<\/p>\n
spark.yarn.archive=hdfs:\/\/\/tmp\/spark-archive.zip<\/code><\/pre>\n
\nTo achieve better performance, it's recommended to use Kryo as the serializer. It's more optimized than the default one.<\/p>\n
spark.serializer=org.apache.spark.serializer.KryoSerializer<\/code><\/pre>\n
\nThere's also a long-standing issue with Spark where it often runs out of memory. This usually happens when workers have calculated everything and are sending results to the driver. We increased this parameter. By default, it's 1GB; we set it to 3GB.<\/p>\n
spark.driver.maxResultSize=3072<\/code><\/pre>\n
\nLastly, as a bonus. How to update Spark to version 2.1 on the HortonWorks distribution \u2014 HDP 2.5.3.0. This version of HDP comes with a pre-installed version 2.0, but we decided some time ago that Spark is rapidly evolving, and each new version fixes bugs and provides additional capabilities, including for the Python API, so we decided to do the update.<\/p>\n
We downloaded the version from the official website for Hadoop 2.7. Unzipped it and placed it in the HDP folder. We created symlinks as needed. When we tried to run it \u2014 it doesn't start. It shows a very unclear error.<\/p>\n
java.lang.NoClassDefFoundError: com\/sun\/jersey\/api\/client\/config\/ClientConfig<\/code><\/pre>\n
\nAfter some research, we found out that Spark decided not to wait for Hadoop to release its updates, opting to use a new version of Jersey. They argue about this in JIRA. The solution was to download Jersey version 1.17.1<\/a><\/noindex>. Place this in the jars folder in SPARK_HOME, zip it again, and place it on HDFS.<\/p>\n
We bypassed this error, but a new and somewhat vague one appeared.<\/p>\n
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.<\/code><\/pre>\n
\nMeanwhile, we tried running version 2.0 \u2014 it's all fine. Try to guess what the issue is. We checked the logs of this application and saw something like this:<\/p>\n
\/usr\/hdp\/${hdp.version}\/hadoop\/lib\/hadoop-lzo-0.6.0.${hdp.version}.jar<\/code><\/pre>\n
\nIn general, for some reason, hdp.version did not resolve. After some research, we found the solution. We need to go into Ambari's settings for YARN and add a parameter to the custom yarn-site:<\/p>\n
hdp.version=2.5.3.0-37<\/code><\/pre>\n
\nThis magic helped, and Spark took off. We tested several of our Jupyter notebooks. Everything works. We are ready for the first Spark class on Saturday (tomorrow)!<\/p>\n
UPD<\/b>. During the class, another problem arose. At some point, YARN stopped providing containers for Spark. We needed to adjust a parameter in YARN that was set to 0.2 by default:<\/p>\n
yarn.scheduler.capacity.maximum-am-resource-percent=0.8<\/code><\/pre>\n
\nThat is, only 20% of the resources were involved in resource allocation. After changing the parameters, we restarted YARN. The issue was resolved, and the other participants were also able to start the Spark context.
\n
Source: habr.com<\/a><\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"
\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442! \u0412\u0447\u0435\u0440\u0430 \u043d\u0430 \u043c\u0438\u0442\u0430\u043f\u0435, \u043f\u043e\u0441\u0432\u044f\u0449\u0435\u043d\u043d\u043e\u043c Apache Spark, \u043e\u0442 \u0440\u0435\u0431\u044f\u0442 \u0438\u0437 Rambler&Co, \u0431\u044b\u043b\u043e \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043c\u043d\u043e\u0433\u043e \u0432\u043e\u043f\u0440\u043e\u0441\u043e\u0432 \u043e\u0442 \u0443\u0447\u0430\u0441\u0442\u043d\u0438\u043a\u043e\u0432, \u0441\u0432\u044f\u0437\u0430\u043d\u043d\u044b\u0445 \u0441 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435\u043c \u044d\u0442\u043e\u0433\u043e \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442\u0430. \u0420\u0435\u0448\u0438\u043b\u0438 \u043f\u043e \u0435\u0433\u043e \u0441\u043b\u0435\u0434\u0430\u043c \u043f\u043e\u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u0441\u0432\u043e\u0438\u043c \u043e\u043f\u044b\u0442\u043e\u043c. \u0422\u0435\u043c\u0430 \u043d\u0435\u043f\u0440\u043e\u0441\u0442\u0430\u044f \u2014 \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u043f\u0440\u0435\u0434\u043b\u0430\u0433\u0430\u0435\u043c \u0434\u0435\u043b\u0438\u0442\u044c\u0441\u044f \u043e\u043f\u044b\u0442\u043e\u043c \u0442\u043e\u0436\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u044f\u0445, \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c, \u043c\u044b \u0442\u043e\u0436\u0435 \u0447\u0442\u043e-\u0442\u043e \u043d\u0435 \u0442\u0430\u043a \u043f\u043e\u043d\u0438\u043c\u0430\u0435\u043c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u043c. \u041d\u0435\u0431\u043e\u043b\u044c\u0448\u0430\u044f \u0432\u0432\u043e\u0434\u043d\u0430\u044f \u2014 \u043a\u0430\u043a \u043c\u044b […]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[702],"tags":[],"class_list":["post-30548","post","type-post","status-publish","format-standard","hentry","category-news"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\n","aioseo_head_json":{"title":"\ud83e\udd47Configuring Spark on YARN | ProHoster","description":"Habr, hello!","canonical_url":"https:\/\/prohoster.info\/en\/blog\/news\/konfigurirovanie-spark-na-yarn","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"en_US","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 Spark \u043d\u0430 YARN | ProHoster","og:description":"\u0425\u0430\u0431\u0440, \u043f\u0440\u0438\u0432\u0435\u0442!","og:url":"https:\/\/prohoster.info\/en\/blog\/news\/konfigurirovanie-spark-na-yarn","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2019-10-31T18:36:08+00:00","article:modified_time":"2019-10-31T18:36:08+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"30548","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":"2026-01-21 01:45:22","breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-03-01 03:33:26","updated":"2026-01-21 01:45:22","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/30548","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/comments?post=30548"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/posts\/30548\/revisions"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/media?parent=30548"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/categories?post=30548"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/en\/wp-json\/wp\/v2\/tags?post=30548"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}