Solution of ES memory overflow in docker startup

Time:2022-5-13

Add JVM. In the config of elasticsearch Options file. Modify the stack size. The default is 2GB. Just start es directly to ensure that the configuration file has been mapped before.


-Xms5g
-Xmx5g

Full JVM The options file is as follows:


## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms5g
-Xmx5g
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# force the server VM (remove on 32-bit client JVMs)
-server
# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# use old-style file permissions on JDK9
-Djdk.io.permissionsUseCanonicalPath=true
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}
## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime
# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}
# By default, the GC log file will not rotate.
# By uncommenting the lines below, the GC log file
# will be rotated every 128MB at most 32 times.
#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32
#-XX:GCLogFileSize=128M
# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true

Supplement: docker container memory limit

Docker memory limit


docker run -d -i -t -m 256M --memory-swap 512M --name centos2.12 centos /bin/bash

View container instance memory limit:

Limit container memory size;

docker run -d -i -t -m 256M –memory-swap 512M –name centos centos /bin/bash

-m, --memory 
#Memory limit size, unit: B, K, m, G; 4m minimum
--memory-swap
#Total limit of memory + swap partition size
--Memory reservation # reserved memory size; Minimum memory occupied by the container in the host;
--oom-kill-disable
#Out of memory overflow; Restrict the kill container process. It is not set by default
--oom-score-adj
#The priority of the container killed by the oom killer. The range is [- 1000, 1000]. The default is 0
--memory-swappiness
#Used to set the virtual memory control behavior of the container. The value is an integer between 0 and 100
--kernel-memory
Core memory limit, minimum 4m.

1. Memory sets the memory size of the container;

--Memory swap is not a swap partition, but the size of memory + swap;
Swap partition of container = memory swap - memory

2. Docker’s default container swap partition has the same size and memory

Memory swap is not set or set to 0;
The swap size of the container's swap partition is the size of memory;
Maximum memory used by the process of the container = memory + swap

3. Memory swap settings

When memory swap is set to - 1;
The container memory size is the size set by memory;
The swap partition size is the host swap size;
The maximum memory that can be used by the container process = memory + host swap size;

4. Memory overflow

--oom-kill-disable
Restrict the kill container process; (it must be set after memory to be limited.)
docker run -d -i -t -m 256M --oom-kill-disable --name Centos-1 centos /bin/bash

5. Core memory & user memory

The difference between core memory and user memory is that core memory cannot be swapped out.

The non swappable feature allows the container to block some system services by consuming too much memory.

Core memory includes:
Stack pages
slab pages
socket memory pressure
tcp memory pressure

You can constrain this memory by setting core memory limits.

Each process consumes some stack pages. By limiting the core memory, you can prevent new processes from being created when the core memory is used too much.

docker run -d -i -t -m 500M --kernel-memory 128M --name Centos-2 centos /bin/bash
Limit the container memory to 256M; Limit core memory to 128M.
docker run -d -i -t --kernel-memory 128M --name Centos-3 centos /bin/bash
The memory is the size of the host memory, and the core memory is limited to 128M

6. Swappiness memory reclamation page

The kernel of the container can exchange a certain proportion of anonymous pages.

--Memory swing is used to set this ratio.
--Memory swappiness can be set from 0 to 100.
#0 means that anonymous page exchange is turned off.
#100 means that all anonymous pages can be exchanged. By default, if -- memory swappiness is not applicable, the value is inherited from the parent process.
docker run -d -i -t --memory-swappiness=0 --name Centos-4 centos /bin/bash
Setting -- memory swappiness to 0 can maintain the working set of the container and avoid the performance loss of the exchange agent.

The larger the value of swap, the more active the use of swap partitions, and the smaller the value, the more active the use of physical memory. The default value is swappiness = 60


sysctl vm.swappiness = 100 
# cat /proc/sys/vm/swappiness

The above is my personal experience. I hope I can give you a reference, and I hope you can support developpaer. If there are mistakes or not fully considered, please don’t hesitate to comment.