A magic bug: oom? Graceful thread termination? High system memory usage?

Time:2021-4-7

Abstract:This project is the data development (DLF) of Dayu platform. An important function of data development is ETL (data cleaning). ETL is from the source end to the destination end, and the middle business logic is generally implemented by SQL templates written by users themselves. Velocity is a template language involved in ETL.

Velocity’s OOM

Basic use of velocity

The basic code of velocity template language is as follows:

1. Initialize the template engine

2. Get the template file

3. Set variables

4. Output

In ETL business, the output of velocity template is the ETL SQL statement set of users, which is equivalent to. SQL file. Here, the official API needs to pass in a java.io.Writer Class is used to store the set of SQL statements generated by the template. Then, these statements are split according to our business and executed one by one.

java.io.Writer Class is an abstract class. There are many implementations in JDK1.8, including but not limited to the following:

A magic bug: oom? Graceful thread termination? High system memory usage?

Because the security requirements of the user’s file read-write creation and other permissions in the cloud environment are more stringent, we use the java.io.StringWriter The underlying layer is a StringBuffer object, and the underlying layer of StringBuffer is a char array.

Simple template Hellovelocity.vm :

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

set($iAMVariable = ‘good!’)

set($person.password = ‘123’)

Welcome ${name} to velocity.com
today is ${date}

foreach($one in $list)

$one

end

Name: ${person.name}
Password: ${person.password}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

HelloVelocity.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf;

import org.apache.velocity.Template;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.app.VelocityEngine;
import org.apache.velocity.runtime.RuntimeConstants;
import org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader;

import java.io.StringWriter;
import java.util.ArrayList;
import java.util.Date;
import java.util.List ; public class hellowvelocity {public static void main (string [] args) {/ / initialize template engine

VelocityEngine ve = new VelocityEngine();
    ve.setProperty(RuntimeConstants.RESOURCE_LOADER, "classpath");
    ve.setProperty("classpath.resource.loader.class", ClasspathResourceLoader.class.getName());
    ve.init (); // get template file
    Template template = ve.getTemplate("Hellovelocity.vm");
    Velocitycontext CTX = new velocitycontext(); // set variables
    ctx.put("name", "velocity");
    ctx.put("date", (new Date()));

    List temp = new ArrayList();
    temp.add("Hey");
    temp.add("Volecity!");
    ctx.put("list", temp);

    Person person = new Person();
    ctx.put ("person", person); // output
    StringWriter sw = new StringWriter();
    template.merge(ctx, sw);
    System.out.println(sw.toString());
}

}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

console output

A magic bug: oom? Graceful thread termination? High system memory usage?

Oom reappearance

Large template file BigVelocity.template.vm

(the number of words in the file exceeds the blog limit, which will be given in the attachment later ~ ~)

The template file itself is 379kb, which is not big. The key is that it defines a string array containing more than 90000 elements, and each element of the array is “1”. Then it writes 79 layers of nested loops, and each layer of the loop traverses the string array. The innermost loop is called once

show table;

This means that the template will generate 79 SQL statements containing 96372, each of which is:

show table;

To fill such a huge amount of characters into the stringwriter object requires at least 10 gigabytes of memory, which is almost unrealistic. Therefore, oom overflow is inevitable.

BigVelocity.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf;

import org.apache.velocity.Template;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.app.VelocityEngine;
import org.apache.velocity.runtime.RuntimeConstants;
import org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader;

import java.io.StringWriter ; public class bigvelocity {public static void main (string [] args) {/ / initialize template engine

VelocityEngine ve = new VelocityEngine();
    ve.setProperty(RuntimeConstants.RESOURCE_LOADER, "classpath");
    ve.setProperty("classpath.resource.loader.class", ClasspathResourceLoader.class.getName());
    ve.init (); // get template file
    Template template = ve.getTemplate("BigVelocity.template.vm");
    VelocityContext ctx = new VelocityContext();
    StringWriter sw = new StringWriter();
    template.merge(ctx, sw);
}

}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

console output

A magic bug: oom? Graceful thread termination? High system memory usage?

Analysis of oom causes

The results generated by the velocity template are written into the stringwriter object. As analyzed above, the bottom layer is a char array. The code for generating oom directly lies in java.util.Array Copyof() function:

A magic bug: oom? Graceful thread termination? High system memory usage?

Capacity limit test of stringwriter underlying char array

StringWriterOOMTest.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf;

import java.io.StringWriter; public class StringWriterOOMTest { public static void main(String[] args) {

    System.out.println("The maximum value of Integer is: " + Integer.MAX_VALUE);
    StringWriter sw = new StringWriter(); int count = 0; for (int i = 0; i < 100000; i++) { for (int j = 0; j < 100000; j++) {
            sw.write("This will cause OOMn");
            System.out.println("sw.getBuffer().length(): " + sw.getBuffer().length() + ", count: " + (++count));
        }
    }
}

}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

JVM parameter setting (refer to hardware configuration)

Environment: jdk8 + windows10 desktop + 32GB memory + 1TB SSD + i7-8700

If your hardware configuration is not enough, please do not try!

A magic bug: oom? Graceful thread termination? High system memory usage?

test result

When the memory size of the whole process reaches 10300 MB in Windows Task Manager, the program stops.

console output

A magic bug: oom? Graceful thread termination? High system memory usage?

Analysis of test results

The maximum value of char array elements will not exceed Integer.MAX_ Value, a very close value, I have a difference of more than 20. Online search for a time, the more reliable statement is: really better than Integer.MAX_ If the value is smaller, it will not be equal to Integer.MAX_ Value, because the char [] object has some other space, such as the object header. It should be said that the total space can’t exceed Integer.MAX_ VALUE。 If you are interested, you can explore the number of elements of other types of arrays. I have a humble opinion here.

Oom solution

Summary of reasons

Through the above series of reproduction and analysis, we know that the root cause of oom is that the stringwriter object rendered by template file is too large. The specific performance is as follows

  1. If the system does not have enough memory space to allocate to the JVM, it will lead to oom, because this part of memory is not useless memory, and the JVM cannot recycle it
  2. If the system has enough memory space to allocate to the JVM, the number of elements in the char array is close to max_ Value throws an oom error.

Solution

As previously analyzed, for security reasons, we can only use the stringwriter object to receive the output of template rendering results. Can’t use file. Therefore, we can only make improvements in the stringwriter itself

Inherit the stringwriter class and rewrite its write method as follows:

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

StringWriter sw = new StringWriter() { public void write(String str) { int length = this.getBuffer ().length() + str.length (); / / limit size to 10MB

    if (length > 10 * 1024 * 1024) { this.getBuffer().delete(0, this.getBuffer().length()); throw new RuntimeException("Velocity template size exceeds limit!");
    } this.getBuffer().append(str);
}

};

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

The rest of the code remains the same

BigVelocitySolution.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf;

import org.apache.velocity.Template;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.app.VelocityEngine;
import org.apache.velocity.runtime.RuntimeConstants;
import org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader;

import java.io.StringWriter ; public class bigvelocitysolution {public static void main (string [] args) {/ / initialize template engine

VelocityEngine ve = new VelocityEngine();
    ve.setProperty(RuntimeConstants.RESOURCE_LOADER, "classpath");
    ve.setProperty("classpath.resource.loader.class", ClasspathResourceLoader.class.getName());
    ve.init (); // get template file
    Template template = ve.getTemplate("BigVelocity.template.vm");
    VelocityContext ctx = new VelocityContext();
    StringWriter sw = new StringWriter() { public void write(String str) { int length =  this.getBuffer ().length() +  str.length (); // limit size to 10MB
            if (length > 10 * 1024 * 1024) { this.getBuffer().delete(0, this.getBuffer().length()); throw new RuntimeException("Velocity template size exceeds limit!");
            } this.getBuffer().append(str);
        }
    };
    template.merge(ctx, sw);
}

}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

console output

If the size of the SQL statement set rendered by the velocity template is within the allowable range, these statement sets will be split according to our business and executed sentence by sentence.

How to terminate thread gracefully

In the subsequent process of executing SQL sentence by sentence, each SQL sentence is executed by calling the peripheral services (DLI, OBS, mysql, etc.), and the result will be returned to our job development scheduling service (DLF) background every time. Our DLF platform supports the function of stopping a job in time. That is to say, if the job needs to execute 10000 SQL in the scheduling process, I will stop in the middle and not execute the following SQL. This function is supported.

After modifying the oom bug mentioned above and passing the test, the test students found that our homework could not be stopped. In other words, the java thread where our homework was located could not be stopped.

Thread stop failure recurrence

After a thorough study of debugging and code, we found that our project did call the interrupt method of the corresponding thread object thread.interrupt (); to terminate a thread.

So why can’t the thread be terminated after calling the interrupt method?

TestForInterruptedException.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf; public class TestForInterruptedException { public static void main(String[] args) {

    StringBuilder sb = new StringBuilder(); for (int i = 0; i < 10; i++) {
        sb.append("show tables;n");
    } int i = 0; for (String str : sb.toString().split("n")) { if (i > 4) {
            Thread.currentThread().interrupt();
            System.out.println(i + " after interrupt");
        }
        System.out.println(str);
        System.out.println(i++);
    }

}

}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

console output

A magic bug: oom? Graceful thread termination? High system memory usage?

Analysis of test results

TestFor InterruptedException.main The function is simple enough to generate a larger string, split it into 10 small strings, print the small strings one by one in the for loop, and try to terminate the thread from the fifth segment (the initial segment is 0). It turns out that the thread did not terminate!

What’s going on? Why does calling the thread’s interrupt method not terminate the thread? Or is it because the JVM needs some time to respond to this method? In fact, this is not the case. Interested students can increase the number of cycles and interrupt several times at the beginning of the cycle. You will find that the result is still the same.

After some exploration, there are no more than two methods of thread termination

  • Using the stop () method of the thread object can stop the thread immediately, but this method is too violent to be used. For details, please refer to the comments of JDK1.8
  • Deprecated. This method is inherently unsafe. Stopping a thread with Thread.stop causes it to unlock all of the monitors that it has locked (as a natural consequence of the unchecked ThreadDeath exception propagating up the stack). If any of the objects previously protected by these monitors were in an inconsistent state, the damaged objects become visible to other threads, potentially resulting in arbitrary behavior. Many uses of stop should be replaced by code that simply modifies some variable to indicate that the target thread should stop running. The target thread should check this variable regularly, and return from its run method in an orderly fashion if the variable indicates that it is to stop running. If the target thread waits for long periods (on a condition variable, for example), the interrupt method should be used to interrupt the wait…
  • The second method is to set the flag bit mentioned in the above JDK comment. There are two ways to do this. Either way, the terminated thread itself needs to “actively” judge the status of the flag bit
  1. Set a general flag bit, such as true / false of boolean type variable, to determine whether the thread will continue to run according to the state of the variable — to actively judge the state of the variable in the code. This method is generally used in loops. When the corresponding state is detected, it will break, return or throw exception.
  2. Use the instance method interrupt of thead class to terminate the thread represented by the thread object. But the interrupt method essentially sets an interrupt flag bit, and once the flag bit is captured (read), “most of the time” will be reset (invalid). Therefore, it does not guarantee that the thread can be stopped, and it does not guarantee that it can be stopped immediately
  3. After the interrupt flag bit set by the interrupt method, if the thread executes the wait / join / sleep of the object class in the subsequent program execution logic, these three methods will capture the interrupt flag bit in time, reset and throw interruptedexception.
  4. Similar to the previous point, java.nio.channels The interruptechannel class under the package will also actively capture the interrupt flag bit, that is, when the thread is in the I / O block of the interruptechannel, it will be interrupted, and then the flag bit will also be reset, and then the channel will be closed and thrown java.nio.channels . ClosedByInterruptException; the same example java.nio.channels . selector, see ja vaDoc
  5. The instance method isinterrupted() of thread class can also capture and reset the interrupt identification bit. This method is used where it is necessary to determine the termination of the program, which can be understood as actively and explicitly capturing the interrupt identification bit.
  6. It is worth noting that throwing and catching interruptedexception does not involve catching and resetting thread ID bits
  7. How to understand that once the interrupt flag bit is captured, it will be reset “most of the time”? There is private native Boolean isinterrupted (Boolean clearinterrupted) in thread class. When the parameter is false, it can not be reset after the interrupt identification bit is captured. In general, however, it will only be used in two places
  8. Static method of thread class: the interrupt identification bit will be reset here, and a thread object cannot be specified. It can only be judged by the current thread

A magic bug: oom? Graceful thread termination? High system memory usage?

  1. Instance method of thread class: this method is also a common method to determine the thread interrupt identification bit, and will not reset the identification bit.

A magic bug: oom? Graceful thread termination? High system memory usage?

Summary

To terminate a thread, the following methods are available in JDK:

  1. Set variables to identify whether a thread has been interrupted
  2. Use the thread interrupt bit of JDK to judge whether the thread is interrupted or not

Both of these methods need to be followed up, such as breaking the loop, returning the method or throwing an exception.

When does the thread terminate?

Generally speaking, there are two reasons for thread termination:

  1. After that, his logic ends naturally.
  2. If a throwable object is thrown during thread execution and is not explicitly captured, the JVM will terminate the thread. As we all know: throwable class is the parent of exception and error!

Explicit catchex with abnormal thread termination ceptionAndDoNotThrow.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf; public class ExplicitlyCatchExceptionAndDoNotThrow { public static void main(String[] args) throws Exception {

    boolean flag = true;
    System.out.println("Main started!"); try { throw new InterruptedException();
    } catch (InterruptedException exception) {
        System.out.println("InterruptedException is caught!");
    }
    System.out.println("Main doesn't stop!"); try { throw new Throwable();
    } catch (Throwable throwable) {
        System.out.println("Throwable is caught!");
    }
    System.out.println("Main is still here!"); if (flag) { throw new Exception("Main is dead!");
    }
    System.out.println("You'll never see this!");
}

}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

console output

A magic bug: oom? Graceful thread termination? High system memory usage?

Analysis of test results

This test verifies the previous conclusion about thread abnormal termination

If a throwable object is thrown during thread execution and is not explicitly captured, the JVM will terminate the thread.

Graceful manual thread termination

The thread execution needs to be terminated manually. The best way is to set the identification bit (which can be interrupt or self-defined), then catch the identification bit in time and throw an exception. At the end of the business logic, catch the exception and do some finishing cleaning actions: for example, calculate the proportion of failed tasks, or close some flows, etc. In this way, the execution of the program takes into account both normal and abnormal conditions, and has been handled gracefully.

TerminateThreadGracefully.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf; public class TerminateThreadGracefully { public static void main(String[] args) {

StringBuilder sb = new StringBuilder(); for (int i = 0; i < 10; i++) {
        sb.append("show tables;n");
    } int i = 0; try { for (String str : sb.toString().split("n")) { if (i > 4) {
                Thread.currentThread().interrupt(); if (Thread.currentThread().isInterrupted()) { throw new InterruptedException();
                }
                System.out.println(i + " after interrupt");
            }
            System.out.println(str);
            System.out.println(i++);
        }
    }Catch (interruptedexception exception) {// todo: some cleanup may be done here
        System.out.println(Thread.currentThread().isInterrupted());
    }
    System.out.println("Thread main stops normally!");
}

}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

console output

A magic bug: oom? Graceful thread termination? High system memory usage?

Why does thread termination fail in a project?

Our project does call the interrupt method of the corresponding thread object thread.interrupt (); to terminate a thread.

So why can’t the thread interrupt the bit and terminate?

Back to the business logic of our project:

The whole job is divided into three stages: template reading, rendering and SQL execution. Generally speaking, the first two stages are faster. In the subsequent process of executing SQL sentence by sentence, each SQL sentence is executed by calling the peripheral services (DLI, OBS, mysql, etc.), and the result will be returned to our job development scheduling service (DLF) background every time. Our DLF platform supports the function of stopping a job in time. That is to say, if the job needs to execute 10000 SQL in the scheduling process, I will stop in the middle and not execute the following SQL. This function is supported.

So the problem lies in the process of SQL execution. After several debugs, it is found that in the process of SQL execution, you need to write log to OBS (Huawei self-developed, third-party package) every time, and this process cannot be omitted. Call the interrupt method of the thread object thread.interrupt (), interrupt bit was first used by OBS bottom layer java.util.concurrent . The await() method of the countdownlatch class catches, resets the flag bit and throws an exception, which is then converted into other exception types when throwing up one layer at a time. Moreover, we can’t judge whether it is caused by our manual termination of the job according to the type of the last thrown exception.

For the third-party package OBS, there is nothing wrong with handling the exception thrown by countdownlatch according to its own underlying logic. But we can’t terminate the program! In order to terminate the thread, I add a custom flag variable to it. When calling thread.interrupt () to set the state of the variable, and at several key points, such as OBS log, to judge the state of my custom ID bit, if the state changes, throw runtimeException (can not be caught, minimize code changes). In order to reuse the thread objects in the thread pool, reset the custom identification bit at the beginning of each job. Finally, the goal of graceful manual termination of job is achieved.

This part of the source code related to the details of the project will not be posted, but the relevant logic has been shown in the previous code.

The system memory consumption is high and inaccurate

The common local variables defined in the running process of a thread, which are not ThreadLocal, are generally recycled with the end of the thread. The phenomenon I encountered was that the thread above could not be stopped. After the bug was solved, the thread stopped, but the memory consumption of the corresponding process running the top command on Linux was still very high.

  1. First, I use jmap- histo:alive pid The command forced GC on the JVM, and found that the heap memory was not used much at this time (about 1% regardless of the old or young band) But the top command takes about 18% * 7g (total Linux memory).
  2. Secondly, I used the jcmd command to analyze the external memory, excluding the problem of memory leakage outside the heap
  3. The next step is to use the jstack command to check the state of each thread in the JVM process. Several threads related to jobs are all in waiting on condition state.
  4. Now, a preliminary conclusion is that the heap memory and out of heap memory used by the JVM process are very small (compared with the explicit 18% * 8g occupation of the top command). So can you guess: the JVM only applied for so much memory from the operating system, but did not return it for the time being, leaving it to be reused the next time there are new tasks in the thread pool? The last part of this paper focuses on one point.

The phenomenon reappears

In the following tests

Set the JVM parameters to:

-Xms100m -Xmx200m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Its significance lies in:

The initial memory of the JVM is limited to 100m and the maximum heap memory is 200m. The detailed GC information and time stamp are printed in time when garbage collection occurs in the JVM. What I have to do in my code is to reproduce that the memory of the JVM is not enough and garbage collection has to happen. At the same time, observe the memory occupation of the java process at the operating system level.

A magic bug: oom? Graceful thread termination? High system memory usage?

SystemMemoryOccupiedAndReleaseTest.java

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

package com.xlf;

import java.util.concurrent.SynchronousQueue;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit; public class SystemMemoryOccupiedAndReleaseTest { public static void main(String[] args) { try {

System.out.println("start");
        Thread.sleep(5000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    ThreadPoolExecutor executor = new ThreadPoolExecutor(3, 3, 30, TimeUnit.SECONDS, new SynchronousQueue<Runnable>(), new ThreadFactory() { public Thread newThread(Runnable r) { return new Thread(r);
            }
        }, new ThreadPoolExecutor.AbortPolicy()); try {
        System.out.println ("(executor initialized):");
        Thread.sleep(1000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    Thread t1 = new Thread(new Runnable() {
        {
            System.out.println ("T1 has been initialized");
        }
        @Override public void run() { byte[] b = new byte[100 * 1024 * 1024];
            System.out.println (T1 allocates 100m space to array); try{
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                e.printStackTrace(); throw new RuntimeException("t1 stop");
            }
            System.out.println("t1 stop");
        }
    }, "t1"); try {
        Thread.sleep(1000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    Thread t2 = new Thread(new Runnable() {
        {
            System.out.println ("T2 has been initialized");
        }
        @Override public void run() { byte[] b = new byte[100 * 1024 * 1024];
            System.out.println (T2 allocates 100m space to array); try{
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                e.printStackTrace(); throw new RuntimeException("t2 stop");
            }
            System.out.println("t2 stop");
        }
    }, "t2"); try {
        Thread.sleep(1000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    Thread t3 = new Thread(new Runnable() {
        {
            System.out.println ("T3 has been initialized");
        }
        @Override public void run() { byte[] b = new byte[100 * 1024 * 1024];
            System.out.println (T3 allocates 100m space to array); try{
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                e.printStackTrace(); throw new RuntimeException("t3 stop");
            }
            System.out.println("t3 stop");
        }
    }, "t3"); try {
        Thread.sleep(1000);
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

    executor.execute(t1);
    System.out.println("t1 executed!"); try {
        Thread.sleep(10000);
    } catch (InterruptedException e) {
        e.printStackTrace();

    }
    executor.execute(t2);
    System.out.println("t2 executed!"); try {
        Thread.sleep(10000);
    } catch (InterruptedException e) {
        e.printStackTrace();

    }
    executor.execute(t3);
    System.out.println("t3 executed!"); try {
        Thread.sleep(10000);
    } catch (InterruptedException e) {
        e.printStackTrace();

    }

    System.out.println("jmap -histo:live pid by cmd:"); try {
        Thread.sleep(20000);
    } catch (InterruptedException e) {
        e.printStackTrace();

    }
    System.out.println("After jmap!"); // You may run jmap -heap pid using cmd here // executor.shutdown();

}
}

A magic bug: oom? Graceful thread termination? High system memory usage?; “copy code”)

In the above code, I first defined three thread objects, all of which are allocated 100m char in the run() method, and then the thread sleeps for 5 seconds. Then, create a new thread pool and give the three thread objects to the thread pool to execute in turn. There is an interval of 10 seconds between the execution of thread pool every two times. This is to give enough time for the last thread to run and let the JVM reclaim this part of memory (200m of maximum heap memory, a thread object takes up more than 100m, and GC will occur if it wants to run to the next thread). In this way, the GC information can be printed for observation. Finally, wait until all three threads are executed, and sleep for a period of time (about 20 seconds). Let me have time to manually execute jmap – histo live PID in CMD. This command will forcibly trigger fullgc. After jmap command, you can also try to execute jmap – heap PID. This command will not trigger GC, but you can see the details of the whole JVM heap

console output

A magic bug: oom? Graceful thread termination? High system memory usage?

In JMP- histo:live Before execution, the process occupies memory in the operating system:

A magic bug: oom? Graceful thread termination? High system memory usage?

Execute JMP- histo:live after

A magic bug: oom? Graceful thread termination? High system memory usage?

Result of jmap – heap PID execution:

A magic bug: oom? Graceful thread termination? High system memory usage?

Test result analysis / win10 task manager inaccurate

After T1 allocates 100m space to the array, T2 ends

A magic bug: oom? Graceful thread termination? High system memory usage?

Memory: 107042k, total heap space: 166400k

Unable to allocate 100m to T2, trigger fullgc:

103650K->1036K(98304K)

After T2 allocates 100m space to the array, T2 ends:

A magic bug: oom? Graceful thread termination? High system memory usage?

Memory usage: 104461k, total available heap space: 166400k

Unable to allocate 100m to T3, trigger fullgc:

103532K->1037K(123904K)

After T3 allocates 100m space to the array, T3 ends

jmap -histo:live pid by cmd:

103565K->997K(123904K)

Finally, the heap size in jmap – heap PID result is also 123m.

In this process, the memory consumption of the JVM process at the operating system level will not exceed 122m, jmap- histo:live pid After triggering the fullgc, it was maintained at about 87m (this was the result of repeated tests)

So why is the stack information size of the JVM inconsistent with that of the resource manager?

This question has been searched on the Internet and the conclusion is as follows:

Committed memory refers to the minimum size that the program requires the system to run for the program. If it is not met, the prompt of insufficient memory will appear.

Working set memory is the real memory occupied by the program, and working set memory = shared memory + dedicated memory

The purpose of shared memory is that when you open more and larger software or organize memory, this part will be distributed to other software. Therefore, this piece of memory is reserved for program operation. The special memory refers to the exclusive memory of program operation. This piece is different from the shared memory. No matter how tight the system memory is, this piece is not the same Dedicated memory will not take the initiative to make room for other programs

So to sum up, the memory displayed by the task manager is actually the dedicated memory of the displayed program, and the memory really occupied by the program is the working set memory

A magic bug: oom? Graceful thread termination? High system memory usage?A magic bug: oom? Graceful thread termination? High system memory usage?

The two pictures above can be matched

The following two figures can be “barely” matched:

A magic bug: oom? Graceful thread termination? High system memory usage?A magic bug: oom? Graceful thread termination? High system memory usage?

However, there is still a little gap between jmap and the 123904k heap memory after GC triggered by jmap. For the time being, there is no reliable answer on the Internet. I guess this part may use the shared memory of other processes. I went to Linux for a try, but there was also the problem of inaccurate memory. Let’s leave it to the next pit filling~~

conclusion

  1. Thread end can be normal end, or it can be abnormal end by throwing throwable object which is not caught
  2. After the thread ends, the memory space occupied by the thread will be recycled when the JVM needs space, which mainly includes: the object allocated on the heap, whose unique reference only exists in the thread
  3. Although the heap space occupied by JVM is very small after full GC, it does not only apply for XMS of memory from the operating system. This seemingly large amount of available memory is actually utilized when there are new thread tasks allocated
  4. The memory consumption of the JVM process heap is slightly higher than that of the process according to the statistics at the operating system level, which may be the reason for the shared memory. This will be filled next time!

Write at the end

All the codes and corresponding resource files described in this article are attached for your reference! Also welcome to comment and ask questions!

 VelocityExperiment.zip 19.40KB

This article is from Huawei cloud community “a magic bug: oom? Graceful thread termination? High system memory usage? 》Author: unstoppable rock.

Click follow to learn about Huawei’s new cloud technology for the first time~

Recommended Today

Third party calls wechat payment interface

Step one: preparation 1. Wechat payment interface can only be called if the developer qualification has been authenticated on wechat open platform, so the first thing is to authenticate. It’s very simple, but wechat will charge 300 yuan for audit 2. Set payment directory Login wechat payment merchant platform( pay.weixin.qq . com) — > Product […]