Detailed explanation of Flink taskmanager memory model

Time:2021-12-4

Apache Flink provides an efficient workload on top of the JVM by strictly controlling the memory usage of its components. Although the community strives to provide reasonable default values for all configurations, it cannot be applied to all applications deployed by users on Flink. In order to provide maximum production value to our users, Flink allows advanced and fine-grained memory allocation tuning within the cluster.

We all know that no matter what cluster Flink runs on, the real work is taskmanager (hereinafter referred to as TM), and jobmanager (hereinafter referred to as JM) is only responsible for task scheduling, so it is very necessary to understand TM’s memory model. Today’s article will talk about TM’s memory model. JM’s memory model is relatively simple, so I won’t talk about it here

In Flink version 1.10.0, the community has further improved and upgraded TM’s memory model. Although the memory division has been very clear, it is still dazzling and easy to be confused. Let’s take a look at the following figure first

Flink TM memory model

Detailed explanation of Flink taskmanager memory model

image-20210403163557657.png

In version 1.12.0 of Flink, the UI is improved. A memory model diagram is added on the TM page to clearly display the memory configuration and usage of each area

Total process memory contains all the memory resources used by the Flink application: total Flink memory + memory used to run the Flink JVM. Total process memory corresponds to Yan / mesos and other container deployment modes (to be specified by the user), which is equivalent to the size of the application container, and total Flink memory corresponds to standalone deployment mode (to be specified by the user).

Total Flink memory is internally divided into: in heap memory + out of heap memory:

The heap memory consists of two parts: freamework heap memory + task heap memory

The off heap memory consists of three parts: managed memory + framework off heap memory + network memory

The following describes the functions of these memories and how to configure them according to the numbering sequence in the figure above

Framework Heap

Meaning description

The memory occupied by the Flink framework itself, which generally does not need to be modified, and may need to be adjusted under special circumstances

Parameter setting
  • Taskmanager.memory.framework.heap.size: in heap part (framework heap), the default value is 128M;

  • Taskmanager.memory.framework.off-heap.size: Framework off heap, which is allocated in the form of direct memory. The default value is 128M.

Task Heap

Meaning description

The memory occupied by the operator and user code used for Flink application.

Parameter setting
  • Taskmanager.memory.task.heap.size: there is no default value for task heap, which is generally not recommended. It will be calculated automatically by subtracting the memory of framework, hosting and network from the total memory of Flink.
  • Taskmanager.memory.task.off-heap.size: the off heap part (task off heap) is allocated in the form of direct memory. The default value is 0, that is, it is not used. If the code needs to call native method and allocate out of heap memory, you can specify this parameter. Generally not used, so it can be kept at 0 most of the time.

Managed Memory

Meaning description

Pure out of heap memory, managed by memorymanager, is used for intermediate result caching, sorting, hash table, etc., as well as the backend of rocksdb status. It can be seen that the memory consumed by rocksdb can be explicitly controlled by the user, which is no longer as difficult to predict and adjust as the old version.

Parameter setting
  • Taskmanager.memory.managed.fraction: the proportion of managed memory in the total memory taskmanager.memory.flink.size of Flink. The default value is 0.4;
  • Taskmanager.memory.managed.size: the size of managed memory. It has no default value and is generally not specified. It is estimated according to the above proportion, which is more flexible.

Network

Meaning description

Network memory uses directory memory. When data is exchanged between tasks (shuffle), the data needs to be cached. The memory that can be used by the cache is this network memory. It is determined by three parameters:

Parameter setting
  • Taskmanager.memory.network.min: the minimum value of network cache, 64MB by default;
  • Taskmanager.memory.network.max: the maximum value of network cache, 1GB by default;
  • Taskmanager.memory.network.composition: the proportion of network cache in the total memory taskmanager.memory.flink.size of Flink. The default value is 0.1. If the amount of memory calculated according to this ratio is less than the minimum or greater than the maximum, it will be limited to the minimum or maximum.

JVM Metaspace

Meaning description

Starting with JDK 8, the JVM removed the permanent generation. Some metadata of the class is placed in the native memory called Metaspace. The JVM Metaspace memory in Flink is the same. It configures the Metaspace memory size of the task manager JVM.

Parameter setting
  • Taskmanager.memory.jvm-metaspace.size: the default value is 256MB.

JVM Overhead

Meaning description

Additional memory overhead reserved for the JVM. For example, thread stack, code cache, GC reclaim space, and so on. The configuration method is similar to that of network memory. It is also determined by three configurations

Parameter setting
  • Taskmanager.memory.jvm-overhead.min: the minimum value of JVM overhead, 192mb by default;
  • Taskmanager.memory.jvm-overhead.max: the maximum value of JVM overhead, 1GB by default;
  • Taskmanager.memory.jvm-overhead.fraction: the additional overhead of JVM accounts for the total memory of TM process
  • The proportion of taskmanager.memory.process.size (note that it is not the total memory of Flink). The default value is 0.1. If the amount of memory calculated according to this ratio is less than the minimum or greater than the maximum, it will be limited to the minimum or maximum.

Let’s take another look at the memory related configuration information in the TM startup log as follows:

< pre data tool = “mdnice editor” style = “margin top: 10px; margin bottom: 10px; border radius: 5px; box shadow: RGBA (0, 0, 0, 0.55) 0px 2px 10px;” >INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - Program Arguments: INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.memory.framework.off-heap.size=134217728b(128M) INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.memory.network.max=214748368b(204.8M) INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.memory.network.min=214748368b(204.8M) INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.memory.framework.heap.size=134217728b(128M) INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.memory.managed.size=858993472b(819.2M) INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.cpu.cores=4.0 INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.memory.task.heap.size=805306352b(767.9M) INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - -D INFO org.apache.flink.yarn.YarnTaskExecutorRunner [] - taskmanager.memory.task.off-heap.size=0b </pre>

This memory is the value shown in the configured values in the figure above, We add these values to taskmanager.memory.framework.off-heap.size + taskmanager.memory.network.max + taskmanager.memory.framework.heap.size + taskmanager.memory.managed.size + taskmanager.memory.task.heap.size = 128 + 205 + 128 + 819 + 768 = taskmanager.memory.flink.size = 2048m. This result is consistent with our configuration in flink-conf.yaml On the

summary

How to configure so much memory above? First of all, the official website does not recommend setting the total memory of the process and the total memory of Flink at the same time. This may cause memory configuration conflicts, resulting in deployment failure. When configuring other memory parts, you should also pay attention to possible configuration conflicts.

In fact, the parameters taskmanager.memory.framework.heap.size, taskmanager.memory.framework.off-heap.size, JVM Metaspace and JVM overhead generally do not need to be configured. We can take the default value. We mainly focus on the memory of task heap, managed memory and network. Of course, Flink also calculates the memory of these three parts, We also need to adjust according to the characteristics of the task, such as traffic and status

Recommended reading

How does Flink SQL implement column to row conversion?

Flink SQL is used with hivecatalog

Flink SQL parses nested JSON data

Dynamically modify the properties of DDL in Flink SQL

Source code analysis of Flink windowassigner

Flink 1.11. X watermarkstrategy incompatibility

[image upload failed… (image-6a4c29-1617465251937)]

If you think the article is helpful to you, please click itfabulousandLookingWell, your support is the biggest driving force for my creation