Technical weekly report of 20210322-20210326

Time:2021-5-9

preface

The main technical content of this week is to share the basic knowledge of JVM and some production accident cases

JVM

First ask a question: in the company, what is the number of parameters of the project JVM you are responsible for? For example, the size of the heap, maxheapsize, the size of the new generation, the age of promotion, what is the garbage collector?

If you don’t know, you haven’t seen anything, and you don’t even know how to view the parameters of the JVM, you’re reading the right article.

First of all, we know that the parameters of the JVM are nothing more than read and write. Next, let’s look at the commands of the JVM?

How to view JVM parameters

Command 1: Java – xshows ettings:vm -version

[email protected] ~ % java -XshowSettings:vm -version
VM settings:
    Max. Heap Size (Estimated): 3.56G
    Ergonomics Machine Class: server
    Using VM: Java HotSpot(TM) 64-Bit Server VM

java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)
[email protected] ~ % 

This command, relatively speaking, shows less JVM configuration information: only the maximum heap size, JDK version, and JDK running mode. This order is seldom used in production. You just need to know about it.

Command 2: jcmd
You can use the jcmd – L command to query the real-time JVM parameters of the running program

[work(tanghailin)@tjtx167-48-41 ~]$ jcmd -l
1041671 sun.tools.jcmd.JCmd -l
1467 com. Bj58. Spat. SCF. Server. Bootstrap. Main - DSCF. Service. Name = testdemo // project name
[work(tanghailin)@tjtx167-48-41 ~]$ jcmd 1467 VM.flags
1467:
-20: Cicompilercount = 4 // maximum number of parallel compilations
-20: Cmsinitiating occupancy fraction = 80 // when 80% of the old space is occupied, garbage collection is needed
-XX:InitialHeapSize=2147483648 // -Xms=2G
-XX:MaxHeapSize=2147483648// -Xmx=2G
-20: Maxnewsize = 1073741824 // Cenozoic = 1g
-20: Maxtenuringthreshold = 6 // promotion age = 6
-XX:MinHeapDeltaBytes=196608 //
-20: Newsize = 1073741824 // Cenozoic = 1g
-XX:OldPLABSize=16 
-20: Oldsize = 1073741824 // old age = 1g
-20: Parallelgcthreads = 20 // number of concurrent threads = 20
-20: Threadstacksize = 1024 // equivalent to - XSS, stack size
-20: + usecmpackage atfullcollection // compress when full GC occurs https://blog.csdn.net/weixin_ 33978044/article/details/94321406
-20: + usecompressedclasspointers // type pointer klassponiter in compressed object
-XX:+UseCompressedOops // oop:ordinary object  Pointer compresses objects. In general, usecompressed oops and usecompressed classpointers are used together. Both are to improve the utilization of memory
-20: + useconcmarksweepgc // use CMS garbage collector to collect old generation
-XX:+UseFastUnorderedTimeStamps 
-20: + useparnewgc // use concurrent newgc garbage collector to collect new generation

Jcmd command usage:

  • Step 1: find out the corresponding process number.
  • Step 2: jcmd PID vm.flags to check the JVM parameter configuration of the corresponding process

You see, the above parameters are obviously a lot more, but not all of them are JVM parameters.

If you want to see all the JVM parameters, you can use the command

java -XX:+PrintFlagsFinal -version
Note:
=Represents the initial parameter value;
: = represents the modified parameter value

[email protected] ~ % java -XX:+PrintFlagsFinal -version
[Global flags]
     intx ActiveProcessorCount                      = -1                                  {product}
    uintx AdaptiveSizeDecrementScaleFactor          = 4                                   {product}
    uintx AdaptiveSizeMajorGCDecayTimeScale         = 10                                                       {product}
                                    {product}
   
    uintx InitialHeapSize                          := 268435456                           {product}
    uintx InitialRAMFraction                        = 64                                  {product}
   double InitialRAMPercentage                      = 1.562500                            {product}
    uintx InitialSurvivorRatio                      = 8                                   {product}
    uintx InitialTenuringThreshold                  = 7                                   {product}
    uintx InitiatingHeapOccupancyPercent            = 45                                  {product}
     bool Inline                                    = true                                {product}
    ccstr InlineDataFile                            =                                     {product}
     intx InlineSmallCode                           = 2000                                {pd product}
     bool InlineSynchronizedMethods                 = true                                {C1 product}
     bool InsertMemBarAfterArraycopy                = true                                {C2 product}
     intx InteriorEntryAlignment                    = 16                                  {C2 pd product}
     intx InterpreterProfilePercentage              = 33                                  {product}
     bool JNIDetachReleasesMonitors                 = true                                {product}
    
    uintx MaxHeapSize                              := 4294967296                          {product}
        
                                  {product}
     intx SelfDestructTimer                         = 0                                   {product}
    uintx SharedBaseAddress                         = 34359738368                         {product}
    ccstr SharedClassListFile                       =                                     {product}
    uintx SharedMiscCodeSize                        = 122880                              {product}
    uintx SharedMiscDataSize                        = 4194304                             {product}
    uintx SharedReadOnlySize                        = 16777216                            {product}
    uintx SharedReadWriteSize                       = 16777216                            {product}
     bool ShowMessageBoxOnError                     = false                               {product}
     intx SoftRefLRUPolicyMSPerMB                   = 1000                                {product}
     bool SpecialEncodeISOArray                     = true                                {C2 product}
     bool SplitIfBlocks                             = true                                {C2 product}
     intx StackRedPages                             = 1                                   {pd product}
     intx StackShadowPages                          = 20                                  {pd product}
     bool StackTraceInThrowable                     = true                                {product}
     intx StackYellowPages                          = 2                                   {pd product}
     bool StartAttachListener                       = false                               {product}
     intx StarvationMonitorInterval                 = 200                                 {product}
     bool StressLdcRewrite                          = false                               {product}
    uintx StringDeduplicationAgeThreshold           = 3                                   {product}
    uintx StringTableSize                           = 60013                               {product}
     bool SuppressFatalErrorMessage                 = false                               {product}
    uintx SurvivorPadding                           = 3                                   {product}
    uintx SurvivorRatio                             = 8                                   {product}
     intx SuspendRetryCount                         = 50                                  {product}
     intx SuspendRetryDelay                         = 5                                   {product}
     
     bool UseCompressedClassPointers               := true                                {lp64_product}
     bool UseCompressedOops                        := true                                {lp64_product}
                                    {C1 product}
     intx ValueMapMaxLoopSize                       = 8      
java version "1.8.0_281"
Java(TM) SE Runtime Environment (build 1.8.0_281-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.281-b09, mixed mode)
[email protected] ~ % 

Among the above three commands, the second command jcmd is more commonly used, because the parameters provided by this command are enough for me to check the JVM problems. The third command has too much information. It is used when the parameter information provided by the second command is not enough.

How to modify JVM parameters

When you know how to read it, you need to start modifying the JVM parameters and tuning.

  • Mode 1: start parameters (not commonly used in production)
  • Method 2: configure the catalina.sh file

In the catalina.sh file, add:
JAVA_OPTS=-Xms64m -Xmx256m -XX:PermSize=128M -XX:MaxNewSize=256m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=./

Different parameters can be configured according to different needs. The specific meaning and function of parameters will not be expanded here.

Here we need to answer two questions:

  • 1. If the maximum heap memory limit of the JVM is set to 2G, is 2G shared by all wars in the tomcat, or can each war have 2G independently?
  • 2. Tomcat itself is also a program running on the JVM. In this case, how to adjust its own JVM parameters?

Problem 1: all war packages share 2G. Problem 2: share 2G with war package because they are all in the same container.

After knowing the read and write of JVM parameters, how to tune them

When it comes to tuning, there must be references, right. Otherwise, how do you know whether it is good or bad after adjusting the JVM parameters.

There are actually three references.Latency, throughput, memory footprint.

Memory occupation: the amount of memory required for the normal operation of the program.

Delay: program pause time due to garbage collection.

Throughput: the ratio of the running time of user program to the total time of user program and garbage collection.

Of course, like cap principle, it is impossible to meet the requirements of small memory occupation, low latency and high throughput of a program at the same time. It’s three references,If it is impossible to be satisfied at the same time, one of them must be sacrificed.

What can we use to look at these three references? There are a lot of tools on the Internet, such as mat, visualvm.

I am lazy, so I use online tools directly. For example:http://gceasy.io/

Technical weekly report of 20210322-20210326

Because what I set is that when there is oom, a dump file will be generated, and then I download the file to the local, and use the tool to view it.

However, if the dump file is not set in your project, an oom exception will occur. Remember to remove the traffic first and then dump it to minimize the impact on the business.

Because of different businesses, the garbage collector used for each project is also different.

Here are two guidelines for GC.

Guidelines for parallel GC tuning

1. Do not set the maximum heap memory unless you are sure
2. Set throughput target first
3. If the throughput target cannot be reached, increase the maximum memory and do not allow the OS to use swap. If it still cannot be reached, reduce the target

4. If the throughput is reached and the GC time is too long, set the pause time target

Guidelines for CMS GC tuning

Direct reference:CMS GC tuning guidelines

Well, this week’s JVM technology weekly report is about this. Just read and write the JVM, and then tune it according to different guiding principles and reference objects. When tuning, just use the control variable method. Don’t change more than one value at a time, otherwise you don’t know which value is optimized.

production accident

JVM production accident

Technical weekly report of 20210322-20210326

This alarm is a bit powerful. Tell me the reason directly.

The memory of the new generation is occupied and replaced by the old generation.

See this, immediately thought ofSpace allocation guarantee rules.

At that time, after analyzing the data view and heap file, I located the local message queue in a project according to the reference tree and suspicious report( The related pictures are not easy to disclose due to the confidentiality of the project.)

The reasons for the new generation of JVM memory being occupied and replaced by the old generation are as follows:

The upstream system, as a producer, produces 700 messages / s on average and 1300 messages / s at peak time. Our system is a downstream system. As consumers, we take the initiative to pull information and consume. The pulled message object has many fields, which makes the object itself very large. Moreover, the consumer system establishes a local message queue locally. Under normal circumstances, 700 messages per second can be consumed normally. At this time, the consumer’s consumption speed is greater than the producer’s production speed. This does not result in message accumulation. When the traffic is large, the number of messages is as high as 1300 per second, and the consumption speed is lower than the production speed. This causes the message accumulation of the local message queue, because the object itself is relatively large. This makes the memory of the new generation full soon. When it is full, more new messages are actively pulled to the downstream system, and the messages are not consumed in time, so the objects cannot be recycled in time. At this time, the new generation is full, and the objects cannot be recycled, so that the objects directly enter the old generation. So that the space of the old age will soon be full. How to solve it in the end? Simple and crude horizontal expansion, saying people’s words is adding machines.

Let me just sum it up a little bit: in fact, I met the JVM. The general troubleshooting idea is as follows: first of all, you will definitely receive an alarm from oom, and then you need to analyze the dump file to determine whether the oom is a memory leak or a memory overflow. After determining this reason, we need to use some visual tools, such as mat, to locate the specific business code by analyzing the reference tree and suspicious report. Then locate the problem of the business code and design according to different problems.

Long and long types lead to the problem that orders cannot be paid

Go to Tuha first
Technical weekly report of 20210322-20210326

Bug phenomenon: the order cannot be paid
Severity: This is a very serious bug. When it’s time to collect money, it can’t be collected.

In fact, when you look at the code, you can find out what the problem is. Long type, can use = = and= Comparison. Yes, yes. But only between – 128-127. Any number out of this range is a new long object.

Expand: in Java, only plastic wrapper classes have caching, floating-point types have no caching. In short, it is:Short, integer and long have caches, and the range is – 128-127.

Integer class
public static Integer valueOf(int i) {
    if (i >= IntegerCache.low && i <= IntegerCache.high)
        return IntegerCache.cache[i + (-IntegerCache.low)];
    return new Integer(i);
}
Long class
public static Long valueOf(long l) {
    final int offset = 128;
    if (l >= -128 && l <= 127) { // will cache
        return LongCache.cache[(int)l + offset];
    }
    return new Long(l);
}
Short
public static Short valueOf(short s) {
    final int offset = 128;
    int sAsInt = s;
    if (sAsInt >= -128 && sAsInt <= 127) { // must cache
        return ShortCache.cache[sAsInt + offset];
    }
    return new Short(s);
}

Now that we know the problem. That must know how to solve it. Just compare with equals.

Let’s focus on one more thing. Why do developers make this mistake?

In fact, developers did not make mistakes= The comparison between = = and = = is wrong. At the beginning, the buserid of updateorderstate is actually a long basic type, so it can be used completely= And = = to compare. But later, buserid was changed to long packing type. After modifying the long type, I didn’t check it. The development time is urgent, which is understandable.

To sum up: first, the interface has been defined before, and it should not be modified directly in the future. You can create a new interface and then be compatible with the old interface. However, you must never modify the old interface directly, even if you modify the data type. Because you don’t know the interface, who is using it, and what the internal logic of the interface is. 2、 We should also have a good grasp of the difference and bottom layer between packaging type and basic type. That’s why we need to know what it is and why it is.

That’s what this week’s tech weekly looks like. Bye. See you next week.

Ramble on

Thank you very much for reading here. If you think the article is well written, please pay attention to it, like it and share it (it’s very, very useful to me).
If you think the article needs to be improved, I’m looking forward to your suggestions. Please leave a message.
If you want to see something, I’m looking forward to your message.
Your support and support is the biggest driving force for my creation!