Solve the server process exit problem (Metaspace overflow)



The planning response server can’t get in. After a remote look, the process crashes. Sometimes it can log in but can’t execute the operation (the process is still there). It can’t be shut down normally. It appears in the process root directoryjava_pid16298.hprofWhen I saw the file, it was a memory overflow. I was surprised that it should not be a heap memory overflow because there were not many people. It was initially suspected that it was a Java 8#metaspace overflow. The following was verified because the startup parameters were added-XX:+HeapDumpOnOutOfMemoryError, there are alsohs_err_pid.log, the JVM fatal error log.

Log query (VIM / grep / less / more)

  1. vim std.log
    esc /OutOfMemoryError ?OutOfMemoryError
    N / N next

  2. less std.log | grep OutOfMemoryError
    Caused by: java.lang.OutOfMemoryError: Metaspace

  3. grep OutOfMemory std.log -A 50 -B 50 | less
    /Outofmemoryerror n next Q exit

  4. less std.log
    If colon / outofmemoryerror appears, search Q to exit? OutOfMemoryError

  5. more std.log
    /Outofmemoryerror search Q exit only/

    Caused by: java.lang.OutOfMemoryError: Metaspace
    at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_40]
    at java.lang.ClassLoader.defineClass( ~[na:1.8.0_40]
    at ~[na:1.8.0_40]
    at ~[na:1.8.0_40]
    at$100( ~[na:1.8.0_40]
    at$ ~[na:1.8.0_40]
    at$ ~[na:1.8.0_40]
    at Method) ~[na:1.8.0_40]
    at ~[na:1.8.0_40]
    at java.lang.ClassLoader.loadClass( ~[na:1.8.0_40]
    at java.lang.ClassLoader.loadClass( ~[na:1.8.0_40]

From the log output, it is: Metaspace memory overflows. The size of my startup parameter setting is 48m

From the output of the fatal log, it can be seen that the JVM also has a fatal error during Metaspace:: allocateMetaspace used 47519K, capacity 48950K, committed 49152K, reserved 1093632K… you can also see similar logs and find that the Metaspace is almost full.


Understanding of Metaspace concept:Metaspace decryption of JVM source code analysis
In java7 and java8, part of the original data in permgen has been transferred to the heap. The removal of the permanent generation has started from JDK7. Part of the data stored in the permanent generation has been transferred to Java heap or native heap. However, the permanent generation still exists in JDK7 and has not been completely removed: symbol references have been transferred to native heap; The literals (interned strings) are transferred to Java heap; Class statics is transferred to Java heap.

In JDK 8, classes metadata is now stored in the native heap and this space is called Metaspace.

Which occupy space (personal analysis is mainly generated classes)

Fastjson#asm (debug)

ASMDeserializerFactory#createJavaBeanDeserializer, when called asJSON#parseObject(String text, Class clazz)For example, a corresponding to clazz will be generatedFastjsonASMDeserializer_53_xxClass, which is used for deserialization. At present, the places used include configuration files, data tables, player related data, etc.

  • serializer

Asmserializerfactory #createjavabeanserializer. When calling json.tojsonstring (object object), it will also generate an asmserializer corresponding to the object #clazz_ 1_ XX class, used to write / serialize.

The two add up to about 200.

The internal class of lambda expression. Where lambda expression is used, an example XX will be generatedThere are about 150 classes in lambda $1. For example, there are about 200 classes generated by protobuf. For other internal classes, no special classes are found. Search for classes containing numbers, because similar classes generated dynamically usually have numbers, and a large number of classes are foundsun.reflect.GeneratedMethodAccessor344… there are about 350, and the same number of sun.reflect.delegatingclassloaders are found at the same time (there is only one class, but there are a corresponding number of instances)sun.reflect.GeneratedConstructorAccessor… analysis – this is the optimization of reflection,It can use a JNI accessor, or a Java bytecode accessor
At the beginning, the JVM uses JNI by default. When the number of calls of the same class reaches a certain value, it changes to Java bytecode call (there will be a new classloader and a clazz)

There are a lot of content on the Internet about the memory overflow problem because of this problem. You can search and consult it yourself
At present, reflection is frequently called in business logic:

  • Reflection execution of handler logic method
  • Deserialization of protobuf
  • Reflection of other third-party libraries, etc

Tool use

Jvisualvm# loads hprof# installable plug-ins

Date generated: mon Sep 25 14:30:30 CST 2017
    File: D: \ XX \ London \ task \ September 2017 \ server_ err\java_ pid16298.hprof
    File size: 56.1 MB

    Total bytes: 47508830
    Total number of classes: 7743
    Total number of instances: 568577
    Class loader: 380
    Garbage collection root node: 2703
    Number of pending objects waiting to end: 0

    Heap dump occurred when outofmemoryerror exception error occurred
    Thread causing outofmemoryerror exception: queue-executor-handler-8
Copy code

From the two hprofs, the memory overflow error is thrown when more than 7700 classes are loaded. The query # permgen saved in the # lower right # of the OQL console # analyzes the # class loader types and finds an interesting one: a large number of classes such as XX $lambda $143 are found, and the lambda expression is to generate internal classes. From the output, The internal class number generated by lambda expression starts from 1 and then + +. At present, there are 151 lambda internal classes. You can directly search for matching under the class information.

Using mat

open heap dump
Size: 22.6 MB Classes: 7.5k Objects: 578k Class Loader: 357

JavaBasics#class loader explorer

Class Name                                           | Defined Classes | No. of Instances
sun.misc.Launcher$ExtClassLoader @ 0x800230b0        |           4,312 |           93,289
                                |           2,617 |          484,122 @ 0x805fd848|             129 |              129 @ 0x805e2858|              73 |               73

Class Name                                               | Shallow Heap | Retained Heap
class sun.reflect.GeneratedMethodAccessor344 @ 0x80593e18|            0 |           568
class sun.reflect.GeneratedMethodAccessor343 @ 0x80593ee0|            0 |           568
class sun.reflect.GeneratedMethodAccessor342 @ 0x80593fa8|            0 |           568
class sun.reflect.GeneratedMethodAccessor341 @ 0x80594070|            0 |           568
class sun.reflect.GeneratedMethodAccessor340 @ 0x80594138|            0 |           568
class sun.reflect.GeneratedMethodAccessor339 @ 0x80594200|            0 |           568
class sun.reflect.GeneratedMethodAccessor338 @ 0x805942c8|            0 |           568
class sun.reflect.GeneratedMethodAccessor337 @ 0x80594390|            0 |           568
class sun.reflect.GeneratedMethodAccessor336 @ 0x80594458|            0 |           568

Class Name                                                     | Defined Classes | No. of Instances
sun.misc.Launcher$ExtClassLoader @ 0x800230b0                  |           4,312 |           93,289
                                          |           2,617 |          484,122 @ 0x805fd848          |             129 |              129 @ 0x805e2858          |              73 |               73 @ 0x806fc4f8|               1 |                0 @ 0x806fc5d0|               1 |                0
sun.reflect.DelegatingClassLoader @ 0x80593db8                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80593e80                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80593f48                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594010                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x805940d8                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x805941a0                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594268                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594330                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x805943f8                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x805944c0                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594588                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594650                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x805947c8                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594890                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594958                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594a20                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594ae8                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594bb0                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594c78                 |               1 |                1
sun.reflect.DelegatingClassLoader @ 0x80594d40                 |               1 |                1
Copy code

Mainclassloader#sun.misc.Launch$ExtClassLoader#defined class 4312 53_ Xxconfig, why are all config objects generated with deserialized inner classes? There are others such as XX_ Redisconfig, etc. (just understand fastjson#asm principle)
After looking at the code here, it is found that there is an xxmonsterconfig. When serializing to redis, serializerfeature.ignorenonfieldgetter is not added. The players serializing Mongo have uniformly added this feature

Here is an explanation of why extclassloader loads most of the categories in the business, because I started it with – DJava. Ext.dirs = lib, that is, extclassloader instead of appclassloader. OQL, OQL syntax, select distinct objects classof (s) from “com. XX. *” s can be used for both
There are about 600 classes of query objects under the com.xx package.

Summary and Solutions

From the above analysis, it is true that the space allocated by Metaspace is too small. 48m is ready to be adjusted to 128M for actual running test

  • Lambda expressions generate inner classes
  • Reflection calls are frequent, and the JVM will also generate corresponding classes


Because the process crash costs a lot, although a large Metaspace can be set, if it is leaked, the process will crash directly, which will have a great impact
Therefore, it is recommended not to set this parameter and the JVM should adjust it by itself. If there is a leak, the memory will always grow wildly
At this time, our operation and maintenance monitoring system can monitor, give an alarm immediately, then go to normal shutdown (jmap hprof before shutdown), and then troubleshoot the problem
PS: during OUTOFMEMORY crash, the shutdown hook will be executed. However, the sudden crash of the process will affect the player’s experience and may cause loss

After reading three things ❤ ️

If you think this article is very helpful to you, I’d like to invite you to help me with three small things:

  1. Praise, forwarding, and your “praise and comments” are the driving force for my creation.

  2. Pay attention to the official account.Rotten pig skin“And share original knowledge from time to time.

  3. At the same time, we can look forward to the follow-up articles

  4. Welcome to the authorgitee, hope to learn together

Author: Landon 30…