Application of process pool in project without regret

Time:2021-4-12

Application of process pool in project without regret

preface:

Recently, after looking at the content of thread pool and learning its internal principles in combination with the source code, I thought, have you ever actually used thread pool in a project? If you think about it, thread pools are used in many parts of the project. Let’s talk about the recent application of multithreading in logging

  • Service interface log asynchronous thread pool storage processing
  • Multithreading for log cleaning in timed tasks

Application of process pool in project without regret

The main line of this paper is as follows

① The basic principle of thread pool is explained;

② Practical application examples of thread pool

  • Structure description of thread pool Application Demo Project
  • Service interface log asynchronous thread pool storage processing
  • Multithreading for log cleaning in timed tasks

Interpretation of the basic principle of thread pool:

Don’t say anything, first paste a brain map, through the brain map to quickly understand the thread pool;

In addition to looking at the picture, you can also use this articleImplementation principle of AVA thread pool and its practice in meituan businessDetailed understanding of thread pool;

Application of process pool in project without regret

Example of thread pool application:

Let’s talk about the recent application of thread pool in the project log;

Thread pool Application Demo Project Description:

Demo address:https://github.com/leishen6/s…

The thread pool application of the two log aspects mentioned above has written demo, which is aSpringBootThe project structure is as follows:

Application of process pool in project without regret

Service interface log asynchronous thread pooling warehousing processing:

In the background service interface project, it is often necessary to store the request message and response message logs of the interface;

The following will be comparedCommon way and thread pool way of storage operationWhy is thread pool storage more elegant;

Normal warehousing operation:

Common warehousing is to directly process the business logic and construct the response, then input the log into the database at the same time, and then return the response after successful warehousing;

The flow chart is as follows:

Application of process pool in project without regret

But there is a big drawback, that is, because of one more database operation (log in), which may lead to slow response speed;

Let’s talk about how to optimize the log storage through thread pool to improve the response speed of the interface;

Thread pool entry operation:

The thread pool method can put the log directly into the queue, and then directly return the response. Finally, the thread in the thread pool is used to take out the log data in the queue for asynchronous warehousing operation;

The flow chart is as follows:

Application of process pool in project without regret

Using the thread pool mode, the main thread processing the request can put the log into the queue, directly return the response, and then use the thread in the thread pool to take out the log data in the queue and store it asynchronously; because one database operation is reduced, the response speed of the interface will be greatly improved.

Let’s take a look at the code implementation

1. The queue for storing request message and response message logs mentioned above:LinkedBlockingDeque

//The bidirectional blocking queue based on linked list can insert and remove elements at both ends of the queue, which is thread safe and more efficient under multithreading concurrency
BlockingQueue<TestLogBean> queue = new LinkedBlockingDeque<TestLogBean>(MAX_QUEUE_SIZE);

In addition to the linkedblockingdeque blocking queue, there are other blocking queues that are often used, as shown in the following figure:

Application of process pool in project without regret

2. Thread pool for log in operation in the project:Single thread pool + fixed number of threads pool

  • Single thread thread pool: it is used to monitor the number of logs in the queue and decide when to take out the logs in the queue, and then send them to the thread pool with a fixed number of threads for storage operation;
  • Thread pool with fixed number of threads: it is mainly used to store logs;

Part of the code is as follows:

/**
 *Initialization
 */
public void init(){
    //The bidirectional blocking queue based on linked list can insert and remove elements at both ends of the queue, which is thread safe and more efficient under multithreading concurrency
    queue = new LinkedBlockingDeque<TestLogBean>(MAX_QUEUE_SIZE);
    lastExecuteTime = System.currentTimeMillis();

    logger.info("LogPoolManager init successfully......");

    logManagerThreadPool.execute(new Runnable() {
        @Override
        public void run() {
            while (run.get()){
                try {
                    //Thread sleep, specific time according to the actual situation of the project configuration
                    Thread.sleep(SLEEP_TIME);
                } catch (InterruptedException e) {
                    logger.error("log Manager Thread sleep fail ", e);
                }
                //Log insertion is performed when 10 logs are stored or the time interval is greater than the set maximum time interval
                if (logCount.get() >= BATCH_SIZE || (System.currentTimeMillis() - lastExecuteTime) > MAX_EXE_TiME) {
                    if (logCount.get() > 0) {
                        logger.info("begin drain log queue to database...");
                        List<TestLogBean> list = new ArrayList<TestLogBean>();
                        /**
                             *Rainto(): get all available data objects from BlockingQueue at one time (you can also specify the number of data to get),
                             *Through this method, the efficiency of data acquisition can be improved, and it does not need to lock or release locks in batches.
                             *Put the extracted data into the specified list set
                             */
                        queue.drainTo(list);
                        //The number of tasks in the task queue is set to 0
                        logCount.set(0);
                        //Remove the thread execution log from the thread pool and insert it
                        logWorkerThreadPool.execute(new InsertThread(testLogService, list));
                        logger.info("end drain log queue to database...");
                    }
                    //Gets the current execution time
                    lastExecuteTime = System.currentTimeMillis();
                }
            }
            logger.info("LogPoolManager shutdown successfully");
        }
    });
}

In this project, test the asynchronous thread pool storage processing of service interface log. After the project starts, enter the following URL in the browser and refresh the page

http://127.0.0.1:8081/v1/api/log/test

Multithreading is used for log cleaning in timed tasks

When the amount of data in the log table is too much, it takes up too much disk space, resulting in constant disk alarms, so it is necessary to slim down the log table;

At this time, you can use multithreading to clean up part of the data in the log table to release disk space;

What data in the log table needs to be cleaned up? The following scenario may appear in the requirements:

The leader said that it is necessary to keep the data of the latest year in the log table, that is, to push 365 days forward from the current date; for example, today is 2020-12-30, and the date of pushing 365 days forward is 2019-12-30, so the logs generated before 2019-12-30 need to be cleaned up;

Let’s take a look at the code below. Since the flexibility of the program needs to be guaranteed as much as possible, the deleted table names need to be flexibly configured according to the deleted fields. The configuration parameters are as follows:application-cleanLog.propertiesconfiguration file

##Thread pool size
threads.pool.num=8

##Tables that need to be cleaned up
log.clean.table=t_test_log

##Fields to clean up by
log.clean.filed=createts

##The amount of data to be cleaned each time
log.clean.batchCount=1000

##The number of cleaning cycles per timed cleaning
log.clean.batchNum=6

##According to the current number of days of the latest data, the rest of the data can be cleaned up
log.clean.dateNum=1

be careful:

  • The thread pool size needs to be reasonably configured according to the server hardware configuration and the actual business log size; if it is set too large, it may occupy too much memory and switch context frequently, which may lead to low efficiency;
  • In this project, the data is cleaned up according to the date field. If the table has been partitioned according to the date, it can be cleaned up directly according to the partition. The speed of partition cleaning is faster, but deleting by partition has not been realized in this project;
  • Size of data to be cleaned each time: refers to the amount of data to be deleted during a delete. It is recommended not to set it too large, because if the amount of data to be deleted is too large, the table may be locked, thus affecting the normal query and addition of the table;
  • The number of times of cycle cleaning during the execution of a scheduled task: it refers to the number of times of delete operation during the execution of a scheduled task. It is said that if the data amount of each delete is not set too large, the number of times of cycle cleaning should be set larger when the total amount of data cleaned remains unchanged;

    Total amount of data cleaned = amount of data cleaned each time * times of cleaning

The flow chart of multi thread log cleaning is as follows:

Application of process pool in project without regret

Let’s take a look at the code implementation

Part of the code implementation is as follows:

/**
 *Multi thread cleaning log start
 */
public void cleanLogStart(){

    //The number of times the cycle cleans up logs
    int whileNum = props.getInt("log.clean.batchNum");
    LogCleanBean logClean = null;

    while (whileNum > 0){

        //Query the time period that matches the amount of data deleted each time
        List<String> list = logCleanService.selectTime(logCleanBean);
        if (list != null && list.size() > 0){
            logClean = new LogCleanBean();
            logClean.setTableName(logCleanBean.getTableName());
            logClean.setFieldName(logCleanBean.getFieldName());
            //Get the minimum generation time for deleting logs
            logClean.setMinTime(list.get(list.size()-1));
            //Get the maximum generation time of the log that can be deleted
            logClean.setMaxTime(list.get(0));
            logCleanBean.setMinTime(logClean.getMinTime());

            //This query does not meet the data size set for each clean-up, indicating that it has been cleaned up
            if (list.size() < logCleanBean.getBatchCleanCount()){
                whileNum = 0;
            }else {
                //Decrease the number of cleanups
                --whileNum;
            }
        }else {
            break;
        }
        //Multithreading
        cleanManagerThreadPool.execute(new CleanThread(this.logCleanService, logClean));
    }
}

Extension:

In this project, delete is used and the data is deleted according to the time fieldMysql databaseIf you want to delete the data and check the disk space, you will find that the available disk space has not increased, and the available disk space may have decreased. Why?

  • Because in InnoDB storage engine, delete does not really delete the data, and MySQL only marks the deleted data as deleted. Therefore, when deleting the data in the table, the space occupied by the table file on the disk will not be reduced, and the storage space will not be released, but the deleted data rows will be set invisible. However, the disk space is not released, but it can still be reused (reuse → overlay) the next time the data is inserted.
  • In addition, when data is deleted in delete, it will be recordedBinlog logIf there are large fields such as text and blob in the deleted data, the log file may become extra large and occupy part of the disk space, which will lead to the further reduction of free disk space;

Solution:

  • You don’t need to wait for MySQL and text to automatically delete big data, but you can use MySQL and text to directly delete big data;
  • It can be used after the delete operationoptimize table table_nameDisk space will be released immediately. However, since optimize will lock the table during execution, it should not be used in peak period or frequently, because it will block normal query, update and other operations.

It can also be usedtruncate 、dropDelete the data,Free up disk space quicklyThe selection is based on the actual situation of the current project.

— END —

This paper introduces the usage scenarios of thread pool in log processing scenarios. In addition, there are many scenarios that use thread pool, and thread pool is also used in many frameworks. You can also learn how to use thread pool by reading the framework source code.

❤ Like + comment + forward

If this article is helpful to you, please wave your little hand to praise. Your support is the driving force of my continuous creation. Thank you!

You can search WeChat Muzi, the official account, and learn a lot of dry cargo articles from Java. You can take a look at it!

Application of process pool in project without regret