20210322-20210326 technical weekly report

Time:2021-10-19

preface

The main technical content this week is to share the basic knowledge of JVM and some production accident cases

JVM

First ask a question: in the company, what are the parameters of the JVM of the project you are responsible for? For example, the size of the heap, maxheapsize, the size of the new generation, the promotion age, and what is the garbage collector?

If you don’t know, haven’t seen anything, and don’t even know how to view the parameters of the JVM, you’ve read the article correctly.

First of all, we know that the parameters of the JVM are nothing more than read and write. Next, let’s take a look at the commands of the JVM?

How to view JVM parameters

Command 1: Java – xshowsettings: VM – version

[email protected] ~ % java -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 3.56G
    Ergonomics Machine Class: server
    Using VM: Java HotSpot(TM) 64-Bit Server VM

java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)
[email protected] ~ % 

This command displays relatively little JVM configuration information: only the maximum heap size, JDK version and JDK operation mode. This command is rarely used in production. Just understand it.

Command 2: jcmd
You can query the real-time JVM parameters of the running program with the jcmd – L command

[work(tanghailin)@tjtx167-48-41 ~]$ jcmd -l
1041671 sun.tools.jcmd.JCmd -l
1467 com.bj58.spat.scf.server.bootstrap.main - DSCF. Service. Name = testdemo // project name
[work(tanghailin)@tjtx167-48-41 ~]$ jcmd 1467 VM.flags
1467:
-20: Cicompilercount = 4 // maximum number of parallel compilations
-20: Cmsinitiatingoccupancyfraction = 80 // garbage collection is required when 80% of the old space is occupied
-XX:InitialHeapSize=2147483648 // -Xms=2G
-XX:MaxHeapSize=2147483648// -Xmx=2G
-20: Maxnewsize = 1073741824 // Cenozoic = 1g
-20: Maxtenuringthreshold = 6 // promotion age = 6
-XX:MinHeapDeltaBytes=196608 //
-20: Newsize = 1073741824 // Cenozoic = 1g
-XX:OldPLABSize=16 
-20: Oldsize = 1073741824 // old age = 1g
-20: Parallelgcthreads = 20 // number of concurrent threads = 20
-20: Threadsacksize = 1024 // equivalent to - XSS, stack size
-20: + usecmsampactfullcollection // compress when full GC occurs https://blog.csdn.net/weixin_ 33978044/article/details/94321406
-20: + usecompressedclasspointers // type pointer klassponiter in compressed object
-20: + usecompressedoops // OOP: Order object pointer the of the compressed object. Generally, usecompressedcoops and usecompressedclasspointers are used together. Both are designed to improve memory utilization
-20: + useconcmarksweepgc // use CMS garbage collector to collect older generations
-XX:+UseFastUnorderedTimeStamps 
-20: + useparnewgc // collect the new generation using the concurrent newgc garbage collector

Jcmd command usage:

  • Step 1: find out the corresponding process number first.
  • Step 2: jcmd PID vm.flags to check the JVM parameter configuration of the corresponding process

You see, there are a lot more parameters above, but not all JVM parameters.

If you want to view all JVM parameters, you can use the command

java -XX:+PrintFlagsFinal -version
Note:
=Represents the initial parameter value;
: = represents the modified parameter value

[email protected] ~ % java -XX:+PrintFlagsFinal -version
[Global flags]
     intx ActiveProcessorCount                      = -1                                  {product}
    uintx AdaptiveSizeDecrementScaleFactor          = 4                                   {product}
    uintx AdaptiveSizeMajorGCDecayTimeScale         = 10                                                       {product}
                                    {product}
   
    uintx InitialHeapSize                          := 268435456                           {product}
    uintx InitialRAMFraction                        = 64                                  {product}
   double InitialRAMPercentage                      = 1.562500                            {product}
    uintx InitialSurvivorRatio                      = 8                                   {product}
    uintx InitialTenuringThreshold                  = 7                                   {product}
    uintx InitiatingHeapOccupancyPercent            = 45                                  {product}
     bool Inline                                    = true                                {product}
    ccstr InlineDataFile                            =                                     {product}
     intx InlineSmallCode                           = 2000                                {pd product}
     bool InlineSynchronizedMethods                 = true                                {C1 product}
     bool InsertMemBarAfterArraycopy                = true                                {C2 product}
     intx InteriorEntryAlignment                    = 16                                  {C2 pd product}
     intx InterpreterProfilePercentage              = 33                                  {product}
     bool JNIDetachReleasesMonitors                 = true                                {product}
    
    uintx MaxHeapSize                              := 4294967296                          {product}
        
                                  {product}
     intx SelfDestructTimer                         = 0                                   {product}
    uintx SharedBaseAddress                         = 34359738368                         {product}
    ccstr SharedClassListFile                       =                                     {product}
    uintx SharedMiscCodeSize                        = 122880                              {product}
    uintx SharedMiscDataSize                        = 4194304                             {product}
    uintx SharedReadOnlySize                        = 16777216                            {product}
    uintx SharedReadWriteSize                       = 16777216                            {product}
     bool ShowMessageBoxOnError                     = false                               {product}
     intx SoftRefLRUPolicyMSPerMB                   = 1000                                {product}
     bool SpecialEncodeISOArray                     = true                                {C2 product}
     bool SplitIfBlocks                             = true                                {C2 product}
     intx StackRedPages                             = 1                                   {pd product}
     intx StackShadowPages                          = 20                                  {pd product}
     bool StackTraceInThrowable                     = true                                {product}
     intx StackYellowPages                          = 2                                   {pd product}
     bool StartAttachListener                       = false                               {product}
     intx StarvationMonitorInterval                 = 200                                 {product}
     bool StressLdcRewrite                          = false                               {product}
    uintx StringDeduplicationAgeThreshold           = 3                                   {product}
    uintx StringTableSize                           = 60013                               {product}
     bool SuppressFatalErrorMessage                 = false                               {product}
    uintx SurvivorPadding                           = 3                                   {product}
    uintx SurvivorRatio                             = 8                                   {product}
     intx SuspendRetryCount                         = 50                                  {product}
     intx SuspendRetryDelay                         = 5                                   {product}
     
     bool UseCompressedClassPointers               := true                                {lp64_product}
     bool UseCompressedOops                        := true                                {lp64_product}
                                    {C1 product}
     intx ValueMapMaxLoopSize                       = 8      
java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)
[email protected] ~ % 

Among the above three commands, I often use the second command jcmd, because the parameters provided by this command are enough for me to troubleshoot JVM problems. The third command has too much information. It is used when the parameter information provided by the second command is not enough.

How to modify JVM parameters

Once you know how to read it, you should start to modify JVM parameters and then tune them.

  • Method 1: start parameters (not commonly used in production)
  • Method 2: configure catalina.sh file

Add in catalina.sh file:
JAVA_OPTS=-Xms64m -Xmx256m -XX:PermSize=128M -XX:MaxNewSize=256m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./

You can configure different parameters according to different needs. The specific meaning and function of parameters will not be expanded here.

Here we need to answer two questions:

  • 1. If the maximum heap memory limit of the JVM is set to 2G here, do all wars in the Tomcat share 2G, or can each war have 2G independently?
  • 2. Tomcat itself is also a program running on the JVM. In that case, how should its own JVM parameters be adjusted?

Question 1: all war packages share 2G. Problem 2: share 2G with the war package because they are all in the same container.

After knowing the read and write of JVM parameters, how to tune them

When it comes to tuning, there must be a reference, right. Otherwise, how do you know whether it is good or bad after adjusting JVM parameters.

In fact, there are only three references.Latency, throughput, memory usage.

Memory usage: the amount of memory required for the normal operation of the program.

Latency: the amount of time a program stalls due to garbage collection.

Throughput: the ratio of user program running time to the total time occupied by user program and garbage collection.

Of course, like the cap principle, it is impossible to meet the requirements of small memory occupation, low latency and high throughput of a program at the same time. Three references,If it is impossible to meet at the same time, one must be sacrificed.

What do you use to look at these three references? There are a lot of tools on the Internet, such as mat and visualvm.

I am lazy, so I use online tools directly. For example:http://gceasy.io/

20210322-20210326 technical weekly report

Because I set that when there is oom, a dump file will be generated, and then I download the file locally and use a tool to view it.

However, if your project does not set the dump file, an oom exception occurs. Remember to pick the traffic first and then dump to minimize the impact on the business.

Because of different businesses, the garbage collector used for each project is also different.

Here are two GC guidelines.

Guidelines for parallel GC tuning

1. Do not set the maximum heap memory unless you are sure
2. Set throughput target first
3. If the throughput target cannot be reached, increase the maximum memory and prevent the OS from using swap. If it still cannot be reached, reduce the target

4. If the throughput is reached and the GC time is too long, set the pause time target

Guidelines for CMS GC tuning

Direct reference:CMS GC tuning guidelines

Well, this week’s JVM technical weekly report is probably like this. Just read and write the JVM, and then tune it according to different guiding principles and reference objects. When tuning, just use the control variable method. Don’t change multiple values at one time, otherwise you don’t know which value is optimized.

production accident

JVM production accident

20210322-20210326 technical weekly report

This alarm is a little powerful. Tell me the reason directly.

The memory of the new generation is full and replaced to the old generation.

Seeing this, I immediately thought of itSpace allocation guarantee rules.

After reading the data view and heap file analysis, I located the local message queue in a project according to the reference tree and suspicious reports. (relevant pictures are not easy to disclose because they involve Project Secrets)

The reasons why the memory of the new generation of JVM is full and replaced to the old generation are as follows:

As a producer, the upstream system produces an average of 700 messages / s, up to 1300 messages / s during peak periods. Our system is a downstream system. As consumers, we take the initiative to pull messages for consumption. The pulled message object has many fields, which makes the object itself large. Moreover, the consumer system establishes a local message queue locally. Under normal circumstances, 700 messages per second can be consumed normally. At this time, the consumption speed of consumers is greater than that of producers. This will not cause message accumulation. When the traffic is large, there are up to 1300 messages per second. At this time, the consumption speed is less than the production speed. This causes the message accumulation of the local message queue, and because the object itself is relatively large. Thus, the memory of the new generation will soon be full. When it is full, more new messages are actively pulled to the downstream system, and the messages are not consumed in time, so the objects can not be recycled in time. At this time, the new generation is full and the object cannot be recycled, so that the object directly enters the old generation. Thus, the space of the old age is soon full. How to solve it in the end? Simple and rough horizontal expansion, saying human words is adding machines.

Let me briefly summarize: in fact, I encountered a JVM. The general troubleshooting ideas are as follows: first, you will certainly receive the alarm of oom, and then you need to analyze the dump file to locate whether oom is a memory leak or a memory overflow. After determining the cause, you need to use some visual tools, such as mat, to locate the specific business code by analyzing the reference tree and suspicious reports. Then locate what the business code is and design according to different problems.

Non payment of orders caused by long and long types

Go to Tuha first
20210322-20210326 technical weekly report

Bug phenomenon: the order cannot be paid
Severity: This is a very serious bug. When the money should be collected, it can’t be collected.

In fact, when you look at the code, you can find out what the problem is. Long type, can I use = = and= Compare. Yes, yes. But only between – 128-127. Any number outside this range is a new long object.

Expand: in Java, only integer wrapper classes have cache, and floating-point types have no cache. To put it simply:Only short, integer and long have cache, and the range is – 128-127.

Integer class
public static Integer valueOf(int i) {
    if (i >= IntegerCache.low && i <= IntegerCache.high)
        return IntegerCache.cache[i + (-IntegerCache.low)];
    return new Integer(i);
}
Long class
public static Long valueOf(long l) {
    final int offset = 128;
    if (l >= -128 && l <= 127) { // will cache
        return LongCache.cache[(int)l + offset];
    }
    return new Long(l);
}
Short
public static Short valueOf(short s) {
    final int offset = 128;
    int sAsInt = s;
    if (sAsInt >= -128 && sAsInt <= 127) { // must cache
        return ShortCache.cache[sAsInt + offset];
    }
    return new Short(s);
}

Now that we know the problem. Then you must know how to solve it. Compare directly with equals.

Let’s focus on one more thing. Why do developers make this mistake?

In fact, developers did not make mistakes= Comparison error for and = =. At first, the buserid of updateorderstate is actually a long basic type, so it can be used completely= And = = for comparison. But later, the buserid was changed to the long packing type. After modifying the long type, I didn’t check it. The development time is urgent, which is understandable.

To sum up: first, after the interface is defined, it should not be modified directly in the future. You can open a new interface and be compatible with the old interface. However, you must not modify the old interface directly, even if the data type is modified. Because you don’t know the interface, who is using it, and how the internal logic of the interface is. 2、 We should also master the difference and bottom layer between packaging type and basic type. This is why we should know what is and why.

This week’s technology weekly is like this. Bye. See you next week.

Ramble

Thank you very much for reading here. If you think the article is well written, please pay attention, praise and share (very, very useful to me).
If you think the article needs to be improved, I look forward to your suggestions to me. Please leave a message.
If you want to see something, I’m looking forward to your message.
Your support and support are the biggest driving force for my creation!

Recommended Today

SQL exercise 20 – Modeling & Reporting

This blog is used to review and sort out the common topic modeling architecture, analysis oriented architecture and integration topic reports in data warehouse. I have uploaded these reports to GitHub. If you are interested, you can have a lookAddress:https://github.com/nino-laiqiu/TiTanI recorded a relatively complete development process in my hexo blog deployed on GitHub. You can […]