The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

Time:2020-11-25

The most difficult thing is to know yourself!

Personal websiteWelcome to visit!

preface:

Recently, my colleagues in the testing department came to me and said that they found that the service interface request had not been responded in a short time when they were testing. Tomcat was dead, but did not return any error response. They said that I should quickly investigate it. After listening, I was excited. Sister, there are always problems recently, and the leaders will hold a criticism meeting for me. Ha ha, I’m joking. How can the leader have the heart to criticize me for being so handsome as me? Hum, I’ll solve this problem right away, and I won’t let the leader know!

In brief, program deployment: Tomcat + Oracle

Investigation process:

During troubleshooting, you can use the command for troubleshooting, or you can use visual monitoring tools; for example, use the jvisualvm.exe Monitoring tools.

Command troubleshooting process:

1. The request service does not respond. First, check whether Tomcat is really down

Command: PS – EF | grep Tomcat

Check that Tomcat is running through the above command. The execution results are as follows:

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

be careful:If more than one Tomcat is running in this server, you need to check whether the Tomcat address running in the frame in the figure below is correct;

Through the command view, it is found that Tomcat is running normally, so this is in the suspended state. Next, check.

2. To see if the HTTP request has reached Tomcat:

Check the localhost in Tomcat logs directory_ access_ Log file to request record;

Command: Tail – 100F localhost_ access_ Log

View the real-time log through the above command. After executing the above command to view the log, request the next program again. No request record is found in the log, indicating that Tomcat is in a suspended state. Next, check.

3. To view the GC of Tomcat’s JVM:

Check the GC situation to see if the program has been stuck for a long time due to frequent GC and long GC, resulting in the request not being processed in time and waiting in the queue, and the caller can’t get a response for a long time, resulting in Tomcat being suspended;

Command: jstat – GC PID time count

For example:jstat -gc 71129 1000 5 Monitor the GC condition of the JVM of 71129, and output it every 1000ms for 5 times;

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

Command execution result parameter analysis:

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

Check the GC situation through the above command. It is found that garbage collection is not frequent, and the time for GC is not long. It should not be the cause of GC.

4. To view the heap of Tomcat’s JVM:

Check the heap memory to see if there is a heap memory overflow that causes Tomcat to feign death and unable to allocate heap memory resources for new requests;

Command: jmap – heap PID

Example: jmap – heap 7112971129 is the process number where Tomcat is running

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

The command execution results show that the available memory in the heap memory is still very large, and there will be no memory overflow problem. Therefore, it is not because the heap memory is too small to cause Tomcat to die.

5. To view the JVM threads of Tomcat:

① Use the jstack command to export the thread dump snapshot of the current JVM, and then see what the threads are doing in the dump?

Command: jstack PID > > jvmThreadDump.log

Example: jstack 71129 > > jvmThreadDump.log

Generates a thread snapshot of the JVM of the 71129 process and redirects the contents of the snapshot to jvmThreadDump.log In the document;

Note: generated jvmThreadDump.log In the directory where you are currently executing the command.

② , and then use the command more to view jvmThreadDump.log Content;

Command: more jvmThreadDump.log

If the dump file is too large, you need to use the more command to look at it. After executing the more command, press enter to display the file content a little bit;

③ By viewing the thread snapshot file, it is found that the status of many threads isWAITINGWaiting state;

And use the command to check that the thread status isWAITINGPercentage of threads in total threads:

be careful: tomcatDump.log Remember to change the name of the generated thread snapshot file to your own

count=`cat tomcatDump.log | grep java.lang.Thread.State | wc -l`; wait=`cat tomcatDump.log | grep WAITING | wc -l`;  a=`echo | awk "{print $wait/$count*100}"`; echo "$a%"

Execute the command and get the result:91.9786% More than 90% of the threads are waiting;

So far, the cause of Tomcat’s feign death has been found, but it needs to be further determinedWhat causes a large number of threads to wait?

By looking at the service interface code called, we can see that there is no lock set in the business logic of this interface, so it should not be the code written by ourselves. However, many JDBC database operations are involved in this interface. Is the connection in the database connection pool insufficient? Because the database connection is a competitive resource, if the connection in the connection pool is exhausted, the next JDBC thread needs to wait for the connection.

6. To view the TCP connection to the database:

It is found that a large number of threads are in the waiting state. Through analysis, it is possible that the connection in the database connection pool is exhausted. Therefore, you can check whether the number of TCP connections established between the server where the service code is deployed and the server where the database is located has reached the maximum number of connections in the configured database connection pool;

Command: netstat – Pan | grep 1521 | WC – L

Because the database used in this paper is Oracle, the port number matched by grep search is 1521;

For MySQL database, change the port number to 3306, netstat – Pan | grep 3306 | WC – L;

If a user-defined database port number is set, it can be changed to a user-defined port number;

The number of connections to the database that has been used through the command is6Let’s take a look at the maximum number of connections in the database connection pool;

The data source configuration is as follows:

<Resource name="jdbc/testdemo"
      type="javax.sql.DataSource"
      factory="com.alibaba.druid.pool.DruidDataSourceFactory"
      url="jdbc:oracle:thin:@192.168.3.125:1521:ora11g"
      driverClassName="oracle.jdbc.driver.OracleDriver"
      username="root"
      password="root"
      auth="Container"
      initialSize="2"
      maxActive="6"
      minIdle="3"
      maxWait="30000"
      timeBetweenEvictionRunsMillis="30000"
      minEvictableIdleTimeMillis="600000"
      maxEvictableIdleTimeMillis="900000"
      poolPreparedStatements="true"
      maxOpenPreparedStatements="20"
      validationQuery="select 1 from dual"
      testOnBorrow="false"
      testOnReturn="false"
      testWhileIdle="true"
      filters="wall,stat,log4j2"
      />

By looking at the data source, we found that the maximum number of connections configured for the connection pool ismaxActive=”6″; it is found that the current number of connections used in the program has reached the maximum value, and then the thread for JDBC operation will enterWaiting state, waiting for the connection;

At this point, the troubleshooting process of Tomcat feigning death has ended, and the reason has been found, that is, the connection in the database connection pool is exhausted. Therefore, in the later test, it is necessary to set the maximum number of connections in the data source larger, and check the code further to see if there is a problem that the database connection is not closed after use.

In addition to the database connection pool connection exhaustion will lead to Tomcat feign death, there are also some other situations, such as: the redis connection pool connection is exhausted, or the redis connection is not released after use, which eventually leads to the depletion of redis connection.

In addition to using the above command for troubleshooting, you can also directly use visual monitoring tools for troubleshooting, which is more simple and intuitive.

Visual monitoring tools troubleshooting

Use JDK’s own jvisualvm.exe ToolsJMX remoteVisual monitoring Tomcat;

jvisualvm.exe Located in $Java_ Home / bin directory;

1. Using JMX to realize remote monitoring steps:

The following is the content of remote monitoring using JMXRemote monitoring Tomcat with jvisualvm

① In the bin directory of Tomcat startup.sh In the fileOn the last lineAdd the following:

export CATALINA_OPTS="$CATALINA_OPTS
-Dcom.sun.management.jmxremote
-Djava.rmi.server.hostname=192.168.1.130
-Dcom.sun.management.jmxremote.port=7003
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false"

Note: if you directly copy the above content, and then put startup.sh If there are many spaces in the file, Tomcat startup may fail. Please copy the following contents and modify it:

export CATALINA_OPTS="$CATALINA_OPTS -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=192.168.1.130 -Dcom.sun.management.jmxremote.port=7003 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

Parameter analysis of the above content:

- Dcom.sun.management . jmxremote enables JMX remote monitoring
- Djava.rmi.server . hostname = 192.168.1.130 this is the server address where Tomcat connects to you
- Dcom.sun.management . jmxremote.port=7003   JMX connection port
- Dcom.sun.management . jmxremote.ssl=false   Is SSL encrypted
- Dcom.sun.management . jmxremote.authenticate=false   Password authentication is required for remote connection

In startup.sh After adding the above content to the file, Tomcat needs to be restarted before it takes effect;

② , will jvisualvm.exe Open the interface as follows:

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

③ Right click on the remote to add a host and enter the IP address of the server startup.sh The host name in the content added to the file

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

④ Right click on the remote host and addJMX connectionManually add the JMX connection port 7003 after the IP address, and then click OK

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

⑤ Through the above steps, you have completed the remote monitoring connection, and then you can double-click to start the monitoring interface

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

2. To view the monitoring content:

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

After viewing the monitoring screen, we can see that the CPU, GC, and heap heap heap are all OK. Then, check the thread status:

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

Click on the image aboveThread dumpButton to generate a snapshot of the thread. The contents of the snapshot file are as follows:

The database connection pool connection is exhausted, resulting in no response to Tomcat request, showing a feign dead state

By viewing the contents of the snapshot file, it is found that the status of many threads isWAITINGWaiting state;

The next analysis and investigation process is as aboveCommand troubleshooting processIt’s the same.

Summary:

The above two screening methods, I recommend or use the first oneOrder investigationBecause in many cases, you will not be allowed to modify the configuration file for remote monitoring, even if the monitoring tool looks more intuitive and convenient; therefore, you need to remember some common troubleshooting commands in case of emergency.

Due to my limited level, if you have any questions, please ask me;

Mei Don’t forget to leave the footprints of your study

If you don’t like the article, you are “playing rogue”. Haha (◍◍◍◍◍◍) ゙! A joke, move your little hand, praise is finished, you each give a strength (like + comment) will let more learners join in! Thank you very much.  ̄ω ̄=