Recently, an application on line failed to load Tomcat and got stuck. The reason for this problem is that some new functions have been added to an old project of a development colleague, which needs to be launched online. First, after publishing to the pre release environment, restart tomcat, and the following phenomena are found:
Oct 27, 2014 10:31:14 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["ajp-bio-9009"] Oct 27, 2014 10:31:14 AM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 586 ms Oct 27, 2014 10:31:14 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Oct 27, 2014 10:31:14 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.40 10:31:20,011[INFO]MLog:80 MLog clients using log4j logging.
That is, Tomcat log shows that
MLog clients using log4j loggingThat’s not going down.
Although my pre release environment has not changed, I still publish the code to one of the servers in the production cluster for the sake of safety. The result is the same, or the loading is not successful.
However, the development colleagues said that they can load normally in the test environment. In order to prove that it is not an environmental problem, I asked the development colleagues to roll back the code first, and then release it to load normally. So I asked the development colleagues to check the code to see if there was a problem with the code. Because the development colleagues launched two new functions online, I asked them to separate the functions and release them online One of the online functions can be loaded normally, and the other cannot be loaded normally. According to this, it can be judged that it is the code problem of another function. However, after a day’s investigation, it was still uncertain what the problem was. At the same time, open colleagues insist that there are no problems in testing and development, why there are problems in production. So we have to start with the environment.
Finally, I asked the developer to send the Tomcat configuration of the test environment to me to see if it was inconsistent (at last, I checked the Tomcat configuration because our environment Tomcat was similar, and it was always consistent with the test, pre release and production, so I didn’t think about this at that time). In the last comparison, it is found that the
-XssThis value is different. Pre release and production are set at 256K, while the test environment is set
512kFinally, reduce the value of the test environment to be consistent with the pre release environment. This phenomenon is reexamined. Therefore, the problem is that the thread stack used to develop the newly launched code is too large, which leads to the inadequacy of the original setting and needs to be increased. Finally, the pre release and production environment of
-XssAfter the adjustment, the problem is solved.
Note: the operation and maintenance problems can not be taken for granted. We should compare all the differences step by step, and then correct the test one by one.
- One of the JVM optimization series (- XSS adjusts the size of stack space)