Recommended allocation of the number of execution units, memory and CPU of spark on yarn


This article is translated from: managing relations inside elasticsearch | elastic

Have you ever had a question:--num-executors, --executor-memory and --execuor-coresHow to configure these parameters?

  1. Let’s look at some suggestions for a better understanding
  2. Do it yourself and use some sample clusters to push the appropriate parameters

Theory first

Here are some suggestions for configuration:

  • Hadoop / yarn / OS daemons: when we run spark applications in cluster managers such as yarn, the background will be accompanied by daemons, such as namenode, secondary namenode, datanode, jobtracker, and tasktracker. Therefore, when thenum-executorsWe need to ensure that there is enough CPU core tree (1 core per node) to ensure that these daemons can run smoothly.
  • Yan applicationmaster (AM): the applicationmaster requests resources from the ResourceManager, executes and monitors the container and the resource consumption in it through nodemanager. If we start the application in yarn, we need to flow out the required resources for AM (about 1024MB of memory and an executor)
  • HDFS throughput: HDFS clients have poor performance in dealing with a large number of concurrent threads. It has been observed that when five tasks per execution unit are reached, HDFS will reach the upper throughput. Therefore, it is best to ensure that the number of cores per executor is lower than this value.
  • Memory overhead: the following figure describes the memory usage of spark on yarn:

Recommended allocation of the number of execution units, memory and CPU of spark on yarn

From the above picture, we need to pay attention to two things:

  1. Memory requested by each executor=spark-executor-memory + spark.yarn.executor.memoryOverhead
  2. spark.yarn.executor.memoryOverhead = Max( 384MB, 7% * spark.executor-memory )

In other words, if we apply for 20GB of memory for each executor, am will actually apply for 20GB + memoryoverhead = 20 + 20 * 7% ~ = 23gb.

  • Too much memory in executor usually leads to excessive GC latency;
  • Thiy executor (with only a single core and enough memory for a single task) loses the advantage of running multiple tasks in a single JVM.

Hands on practice

Now, suppose we have 10 server nodes, and their configurations are as follows:

**Server configuration**

Number of nodes: 10
Number of cores per node: 16
Single node memory: 64GB

Let’s consider different parameter configurations:

Method 1: tiny executor (single core for each executor)

Tiny executor means that each executor has only one core. The following table shows the parameter configuration in this case:

--- num executors = total number of nodes
                    =Core number of single node * number of nodes in cluster 
                    = 16 x 10 = 160
--- executor cores = 1 (single core per executor)
--- executor memory = memory per executor
                     =Memory per node / executors per node
                     = 64GB / 16 = 4GB

analysis:As mentioned above, when each executor has only one core, we cannot take advantage of running multiple tasks with the same JVM. Again, usebroadcastandaccumulatorFor variable sharing / caching, replication is required in each core of each node. In addition, we don’t have enough memory resources for Hadoop / yarn daemons. This method is not good.

Method 2: fat executor (one executor for each node)

Fat executor means one executor for each node. The following table shows the corresponding spark configuration:

--- num executors = number of cluster nodes
                  = 10
--- executor cores = single node cores
                    = 16
--- executor memory = single node memory / executors per node
                     = 64GB / 1 = 64GB

analysis:When each executor allocates 16 cores, the throughput of HDFS will be constrained and the GC of the country will be caused, except that am and daemons are not taken into account. Because, this method is not good.

Method 3: the balance scheme between fat and tiny

According to our previous proposal:

  • Based on the above suggestions, let’s assign 5 cores to each executor = > 0--executor-cores=5 (for better HDFS throughput)
  • In each node, one core is reserved for Hadoop / yarn daemons = > the number of cores available for a single node = 16 – 1 = 15
  • Therefore, the total number of available cores = 15 * 10 = 150
  • Number of executors available = (total cores / cores per executor) = 150 / 530
  • Set aside an executor for applicationmaster = >--num-executors = 29
  • Number of executors per node = 30 / 10 = 3
  • Memory per node = 64GB / 3 ~ = 21 GB
  • Calculate the extra cost = 7% * 21 GB = 3 GB. Therefore, the actual--executor-memory = 21 – 3 = 18 GB

Therefore, the recommended configuration is: 29 executors, each executor: 18 GB of memory and 5 cores.

Analysis: it is not difficult to see how method 3 balances fat and tiny. It guarantees the parallelism of fat executor and the throughput of tiny executor.


As follows:

Remember the following suggestions when configuring spark applications:

  • Reserve the required resources for yarn am;
  • Reserve resources for Hadoop / yarn / OS daemons;
  • Understanding spark memory utilization model

In addition, three parameter configuration methods are analyzed

  1. Tiny executors – Single core executor
  2. Fat executor – one executor for each node
  3. Recommended Practice – keep a balance between fat and tiny as suggested above

--num-executors--executor-cores and --executor-memoryThese three parameters control the number of CPU and memory that spark application can use, which plays an important role in spark performance. It is essential that users understand the correct way to configure. I hope this blog can help you understand this.