A flexible log cleaning strategy of trimlog based on bash shell script

Time:2019-12-12

Preface

Through the ZABBIX file system monitoring alarm, it was found that the log was not cleaned regularly. After further investigation, it was confirmed that the trimlog script written by the former colleague had an exception when executing the timing task on some servers. After review, it was found that the code logic itself had no problem, mainly because the cron job did not use escape characters when executing the date parameter. However, trimlog.sh, a log cleaning script code, was found The structure is clear and easy to read. It’s convenient for everyone to reference and revise.

A flexible log cleaning strategy of trimlog based on bash shell script

Update history

October 8, 2019 – first draft

Read the original – https://wsgzao.github.io/post

Extended reading

A simple practice of log backup and cleaning based on bash find command


Problem description

#Check crontab
cat /etc/crontab

#Ansible: gdp_trim_log
0 1 * * * root /data/release/trimlog.sh '/data/release/log/gop_gdp' 30 1 '*' '%Y%m%d' >> /data/release/trimlog.txt
0 1 * * * root /data/release/trimlog.sh '/data/release/log/cacheupdate' 30 1 '*' '%Y%m%d' >> /data/release/trimlog.txt
#Ansible: gdp_trim_backup
0 1 * * * root /data/release/trimbackup.sh '/data/release/backup/gop_gdp' 30 >> /data/release/trimbackup.txt

#Check crontab日志
less /var/log/cron

Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183782]: (root) CMD (/data/release/trimlog.sh '/data/release/log/gop_gdp' 30 1 '*' ')
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183783]: (root) CMD (/data/release/trimbackup.sh '/data/release/backup/gop_gdp' 30 >> /data/release/trimbackup.txt)
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183784]: (root) CMD (/data/release/trimlog.sh '/data/release/log/cacheupdate' 30 1 '*' ')
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183786]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183788]: (root) CMD (cd /tmp &&  ss -tna | awk '{print $1}' | sort | uniq -c > snmpss.tmp && chcon system_u:object_r
:snmpd_var_run_t:s0 snmpss.tmp && mv -f snmpss.tmp snmpss.cache)
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183779]: (root) CMDOUT (/bin/bash: -c: line 0: unexpected EOF while looking for matching `'')
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183779]: (root) CMDOUT (/bin/bash: -c: line 1: syntax error: unexpected end of file)
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183778]: (root) CMDOUT (/bin/bash: -c: line 0: unexpected EOF while looking for matching `'')
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183778]: (root) CMDOUT (/bin/bash: -c: line 1: syntax error: unexpected end of file)
Oct  8 01:00:01 sg-gop-10-71-12-146 CROND[183777]: (root) CMDOUT (find: ‘/data/release/backup/gop_gdp’: No such file or directory)

#Add transfer characters to crontab's date
0 1 * * * root /data/release/trimlog.sh '/data/release/log/gop_gdp' 30 1 '*' '\%Y\%m\%d' >> /data/release/trimlog.txt
0 1 * * * root /data/release/trimlog.sh '/data/release/log/cacheupdate' 30 1 '*' '\%Y\%m\%d' >> /data/release/trimlog.txt

Bash shell code

vim /data/release/trimlog.sh

#!/bin/bash
dir=`dirname $0`
if [ $# -lt 1 ]
then
    echo "usage: $0 DIRECTORY [DELETE_DAYS] [COMPRESS_DAYS] [COMPRESS_FILE_PATTERN] [DATE_PATTERN]"
    echo "multiple matches example for COMPRESS_FILE_PATTERN: data|error"
    exit
fi

current_date=$(date +"%Y-%m-%d %T")
echo "Start trim log at $current_date "

path=$1
if [ $# -gt 1 ]
then
    log_delete_days=$2
else
    log_delete_days=7
fi
if [ $# -gt 2 ]
then
    log_compress_days=$3
else
    log_compress_days=0
fi
if [ $# -gt 3 ]
then
    log_compress_match=$4
else
    log_compress_match='*'
fi
if [ $# -gt 4 ]
then
    date_pattern=$5
else
    date_pattern="%Y-%m-%d"
fi
if [ $log_delete_days -gt 0 ]
then
    files=`ls $path/*.log.*`
    i=0
    while [ $i -le $log_delete_days ]
    do
        d=`date "+$date_pattern" --date "$i day ago"`
        files=`echo "$files"|grep -v $d`
        i=`expr $i + 1`
    done
    echo "$files"|xargs rm -f
fi
if [ $log_compress_days -gt 0 ] && [ -n "$log_compress_match" ]
then
    files=`ls $path/*.log.*|grep -v "\.gz$"|grep -E "$log_compress_match"`
    i=0
    while [ $i -lt $log_compress_days ]
    do
        d=`date "+$date_pattern" --date "$i day ago"`
        files=`echo "$files"|grep -v $d`
        i=`expr $i + 1`
    done
    echo "$files"|xargs gzip
fi

current_date=$(date +"%Y-%m-%d %T")
echo "Finish trim log at $current_date "

vim /data/release/trimbackup.sh

#!/bin/bash
dir=`dirname $0`
if [ $# -lt 1 ]
then
    echo "usage: $0 DIRECTORY [KEEP_BACKUP_NUM]"
    exit
fi

current_date=$(date +"%Y-%m-%d %T")
echo "Start trim backup at $current_date "

path=$1
if [ $# -gt 1 ]
then
    keep_backup_num=$2
else
    keep_backup_num=10
fi

i=0
find $path -maxdepth 1 ! -path $path -type d -printf '%p\n' | sort -r -n | while read dir; do
    i=`expr $i + 1`
    if [ $i -gt $keep_backup_num ]
    then
        echo "del $dir"
        rm -rf $dir
    else
        echo "keep $dir"
    fi
done

current_date=$(date +"%Y-%m-%d %T")
echo "Finish trim backup at $current_date "