Linux timing clear log and Catalina log segmentation


Step on the pit

In addition to the development work, I also have a trivial work, which is to check the Linux machine operation related to the website regularly every day.
At the beginning, the Linux machines were running normally, so I had a lot of loose, from checking every day to checking once or twice a week later.
Later, the website was frequently attacked by unknown sources, resulting in the collapse of some background components (this problem will be described in another blog later), but the website server is still running well.
But the bad thing still came, and then just a month ago, the boss suddenly received a report about the Linux server disk utilization rate reaching more than 90%! Now I’m a little flustered. I’ll check the operation of the machine. It is found that the installation directory of Tomcat on the disk of the formal environment machine accounts for more than 30 g (a total of 44g).

Du - SH / home / Tomcat # - s is the sum of statistics, and the unit is g- H means to count the size of each file
Du - BS / home / Tomcat # - B means the statistical unit is bit
DF - H # statistics to view the system disk utilization, divided into physical utilization and logical utilization

The boss reminded me that it was related to log files. When I went to the logs folder, I found that there were hundreds of log files with different sizes. Well, it’s you!

Pit filling

Measure 1:rm -rf filepath

At the beginning, in order to quickly solve this problem, we simply use RM command to delete it, and only keep the log within one month.

rm -rf logs/*2017*.log && rm -rf logs/*201801*.log && rm -rf logs/*201802*.log 

Reusedf -hCheck the disk, and it’s reduced to about 70%. Mm-hmm, it’s OK. Then we do the same thing and delete the old logs from other machines.

Ha ha, just a few times, I also underestimate my reputation

After the above operation, I think I can have a quiet rest for a week.
But I didn’t expect to receive the warning message the next day. I went and killed myself. The attack some time ago was enough. I can’t have an accident here!
If you look carefully at the few log files left, you can still receive them. By comparison, I found that it was the Catalina. Out file that caused the trouble. Check the size. NIMA has more than 20 g’s. Well, you’re tough!

Measure 2: split Catalina log

After searching for information online, some netizens recommend log rotate log polling for log segmentation. All right, just do it.
Now?/etc/logrotate.d/Now create your own polling task file Tomcat

cat >/etc/logrotate.d/tomcat << EOF
    rotate 7
    size 16M

Here, we set up daily catalina.out file rotation to save at most 7 copies (rotate 7) and compress the split log file (compress). When the log file is larger than 16MB, it will rotate (size 16m).
Here are some points to understand:

  • Every night, the crond daemon will run in the task list in the / etc / cron. Daily directory;
  • The scripts related to logrotate are also in the / etc / cron. Daily directory. The running mode is / usr / bin / logrotate / etc / logrotate.conf;
  • /The / etc / logrotate.conf file includes all the files in the / etc / logrotate.d/directory. It also includes the Tomcat file we just created above;
  • /The / etc / logrotate.d/tomcat file triggers the rotation of the / home / Tomcat HSIP / logs / catalina.out file.

In order to solve the problem as soon as possible, after setting the polling configuration, we immediately run the above configuration file:

Logrotate -- force / etc / logrotate. D / Tomcat # -- force or - F to force polling

After manual execution, after a while, Catalina was split by more than half, and its file format wascatalina.out.1.gzWhere 1 is the order of execution. If we execute the above order again,catalina.out.1.gzWill be renamedcatalina.out.2.gzThe newly generated split file will becomecatalina.out.1.gz
Then execute the command in measure one, delete the separated log, and the disk is quiet^ v^

Measure 3: clear the log regularly

I don’t think I can manually clear this log and the separated Catalina log file every time. Get a regular task and run by yourself!
Since logrotate needs to use crond, I will write the above command into the scriptdeletelogs.shIt’s OK to let crond execute automatically!

#Multiple paths can be filled in
for wdir in ${workdir[@]}; do
  echo -e "filepath is ${wdir} .\n"
  #. log file and. TXT file containing log tag, as well as split Catalina. Out. 1. Gz and other compressed files
  find $wdir -regex "^.*\(log.*\.txt\|\.log\|catalina.*\.gz\)$" -and -mtime +5 -type f -exec  rm  -rf  {} \;
  if [ $? -eq 0 ]; then
    echo -e `date`" delete logs successfully! \n"
    echo -e `date`" delete logs failed! \n"

Here we use the find command to find the corresponding file

  • -regexThis parameter indicates that the following input is written using a regular expression. If it is – name, it will be written with a general string. At this time, wildcards can be used, but regular symbols will be retained.
  • Shell regular: ^ denotes the beginning of the regular matching string, $denotes the end of the regular matching string, and some other non alphabetic symbols used in regular need to be escaped It means to match any character;So the. In the file path needs to be escaped
  • -andIndicates that the relevant parameters of the command are used equally again, such as – Mtime;
  • -mtimeIt means to use the modification time attribute, and the following + 7 means the file or folder that has been modified for more than 7 days, that is, the modification time is more than 7 days; While – 7 means less than 7 days, 7 means just 7 days;
  • -typeIt represents the attributes of the file to be searched, followed by F for the file to be searched, and D for the folder to be searched;
  • -execIndicates that the following command will be executed for the file or folder that matches the previous one.Note that the following command needs a pair of {}, a space and a semicolon to end

Next, we configure the crontab timing task. Follow these steps:

  1. First, judge the crontab running stateservice crond statusTo ensure that crontab is in “is running” state.
  2. Then edit the crontab filevi /etc/crontab, add the following instruction at the end of the file:

    0 5 * * * root sh /home/hsip-monitor/ >> /home/hsip-monitor/deletelogs.log

The configuration is described as follows:

-Crontab timing Configuration Description: * (minutes) * (hours) * (days) * (months) * (weeks)
-Root specifies the user, followed by the. Sh file to be executed. Here we also output the execution log to the log file for future reference.

OK, after setting up, let’s observe the operation for a few days.

Problems and optimization

Crontab mission not executed

  1. After setting the crontab timing task to clear logs, the next day we observed the logs and files and found that the timing task was not executed. I searched a lot of information on the Internet, but I didn’t know the gist. The following points can be made clear:

    • usecrontab -lThe command discovery system does not display any scheduled task list;
    • usecrontab -eAfter creating the task file, after writing the timing task shown in Article 2 in the file, check the timing task list again, and there are the tasks added above;
  2. After the above-mentioned solution has not solved the problem, a closer look at the crontab timing task shows that the original timing task command is missingshThis execution command, although read a lot of blog, execute sh script before no commandsh. I don’t know why other people can implement it, but if I don’t add it here, I willshAfter that, the timing task is easy to use;

Catalina auto rotation not implemented

The partition log configuration is not implemented on the machine in the formal environment, but a test machine that is not often used can partition the Catalina log very well. Many solutions have not been found.
After solving the problem that crontab timing task can’t be solved, I suddenly have an idea. Why can’t Ilogrotate --force /etc/logrotate.d/tomcatWhat about the timing task of crontab?
Do as you say, create filescutoffCatalina.shJust wait for tomorrow’s results.

/usr/sbin/logrotate --force /etc/logrotate.d/tomcat
if [ $? -eq 0 ]; then
echo -e `date`" cut catalina.out successfully! \n"
echo -e `date`" cut catalina.out failed! \n"

At the same time, add the following configuration in the / etc / crontab file

50 23 * * * root sh /home/hsip-monitor/ >> /home/hsip-monitor/cutoffCatalina.log

However, it is a pity to say that writing to the script for execution also fails. It is suspected that the log file is being operated, so that the segmentation cannot be completed. Later, the boss also reminded me whether it had something to do with Catalina’s reading and writing?
So I found a time with less user visits (around 5 a.m.) and reconfigured it

30 4 * * * root sh /home/hsip-monitor/ >> /home/hsip-monitor/cutoffCatalina.log

And then there’s an interesting situation:

  • In the formal environment, logs are divided intocatalina.out.1.gzAnd the test machine is divided intocatalina.out-20180314.gzThat is to say:

    • When Catalina is reading and writing, our script is executed successfully, that is, the — force parameter is valid.
    • The logrotate polling mechanism is triggered. When Catalina reads and writes, it fails to execute the polling. Otherwise, it can be executed successfully. After the success, the split file is given the timestamp command.

Other findings

In the process of searching and solving problems, some netizens didn’t know whether the method was feasible or not, and they didn’t say whyLinux – logrotate management partition nginx log invalid – segmentation fault)。 OK, let’s set up the formal machine first, and then reinsurance.