This article is translated from: managing relations inside elasticsearch | elastic
Have you ever had a question:
--execuor-coresHow to configure these parameters?
- Let’s look at some suggestions for a better understanding
- Do it yourself and use some sample clusters to push the appropriate parameters
Here are some suggestions for configuration:
- Hadoop / yarn / OS daemons: when we run spark applications in cluster managers such as yarn, the background will be accompanied by daemons, such as namenode, secondary namenode, datanode, jobtracker, and tasktracker. Therefore, when the
num-executorsWe need to ensure that there is enough CPU core tree (1 core per node) to ensure that these daemons can run smoothly.
- Yan applicationmaster (AM): the applicationmaster requests resources from the ResourceManager, executes and monitors the container and the resource consumption in it through nodemanager. If we start the application in yarn, we need to flow out the required resources for AM (about 1024MB of memory and an executor)
- HDFS throughput: HDFS clients have poor performance in dealing with a large number of concurrent threads. It has been observed that when five tasks per execution unit are reached, HDFS will reach the upper throughput. Therefore, it is best to ensure that the number of cores per executor is lower than this value.
- Memory overhead: the following figure describes the memory usage of spark on yarn:
From the above picture, we need to pay attention to two things:
- Memory requested by each executor=
spark.yarn.executor.memoryOverhead= Max( 384MB, 7% *
In other words, if we apply for 20GB of memory for each executor, am will actually apply for 20GB + memoryoverhead = 20 + 20 * 7% ~ = 23gb.
- Too much memory in executor usually leads to excessive GC latency;
- Thiy executor (with only a single core and enough memory for a single task) loses the advantage of running multiple tasks in a single JVM.
Hands on practice
Now, suppose we have 10 server nodes, and their configurations are as follows:
**Server configuration** Number of nodes: 10 Number of cores per node: 16 Single node memory: 64GB
Let’s consider different parameter configurations:
Method 1: tiny executor (single core for each executor)
Tiny executor means that each executor has only one core. The following table shows the parameter configuration in this case:
--- num executors = total number of nodes =Core number of single node * number of nodes in cluster = 16 x 10 = 160 --- executor cores = 1 (single core per executor) --- executor memory = memory per executor =Memory per node / executors per node = 64GB / 16 = 4GB
analysis:As mentioned above, when each executor has only one core, we cannot take advantage of running multiple tasks with the same JVM. Again, use
accumulatorFor variable sharing / caching, replication is required in each core of each node. In addition, we don’t have enough memory resources for Hadoop / yarn daemons. This method is not good.
Method 2: fat executor (one executor for each node)
Fat executor means one executor for each node. The following table shows the corresponding spark configuration:
--- num executors = number of cluster nodes = 10 --- executor cores = single node cores = 16 --- executor memory = single node memory / executors per node = 64GB / 1 = 64GB
analysis:When each executor allocates 16 cores, the throughput of HDFS will be constrained and the GC of the country will be caused, except that am and daemons are not taken into account. Because, this method is not good.
Method 3: the balance scheme between fat and tiny
According to our previous proposal:
- Based on the above suggestions, let’s assign 5 cores to each executor = > 0
--executor-cores=5 (for better HDFS throughput)
- In each node, one core is reserved for Hadoop / yarn daemons = > the number of cores available for a single node = 16 – 1 = 15
- Therefore, the total number of available cores = 15 * 10 = 150
- Number of executors available = (total cores / cores per executor) = 150 / 530
- Set aside an executor for applicationmaster = >
- Number of executors per node = 30 / 10 = 3
- Memory per node = 64GB / 3 ~ = 21 GB
- Calculate the extra cost = 7% * 21 GB = 3 GB. Therefore, the actual
--executor-memory= 21 – 3 = 18 GB
Therefore, the recommended configuration is: 29 executors, each executor: 18 GB of memory and 5 cores.
Analysis: it is not difficult to see how method 3 balances fat and tiny. It guarantees the parallelism of fat executor and the throughput of tiny executor.
Remember the following suggestions when configuring spark applications:
- Reserve the required resources for yarn am;
- Reserve resources for Hadoop / yarn / OS daemons;
- Understanding spark memory utilization model
In addition, three parameter configuration methods are analyzed
- Tiny executors – Single core executor
- Fat executor – one executor for each node
- Recommended Practice – keep a balance between fat and tiny as suggested above
--executor-memoryThese three parameters control the number of CPU and memory that spark application can use, which plays an important role in spark performance. It is essential that users understand the correct way to configure. I hope this blog can help you understand this.