Spring boot + redis implements delay queue, which is well written!

Time:2021-10-14

Source: blog.csdn.net/qq330983778/article/details/99341671

operation flow

First, let’s analyze the process

  1. The user submits the task. First, push the task to the delay queue.
  2. After receiving a task, the delay queue first pushes the task to the job pool, and then calculates its execution time.
  3. Then, the delayed task (including only the task ID) is generated and put into a bucket
  4. The time component polls each bucket at any time, and obtains the task meta information from the job pool when the time arrives.
  5. Monitor the legitimacy of the task. If it has been deleted, pass. Continue polling. If the task fits the law, calculate the time again
  6. If the time is calculated according to the law, if the time is legal: put the task into the corresponding ready queue according to the topic, and then remove it from the bucket. If the time is illegal, recalculate the time, put it into the bucket again, and remove the contents of the previous bucket
  7. The consumer polls the ready queue of the corresponding topic. After obtaining the job, make your own business logic. At the same time, the server recalculates the execution time of the job obtained by the consumer according to its set TTR, and puts it into the bucket.
  8. After consumption, send a finish message, and the server deletes the corresponding information according to the job ID.

object

We can now see several components in the middle

  • The delay queue is redis delay queue. Implement messaging
  • The job pool task pool holds job meta information. According to the article, the K / V data structure is used. Key is ID and value is job
  • The delay bucket is used to save the delayed tasks of the business. This article describes the use of polling to place a bucket. You can know that it does not use topic to distinguish it. Individuals use sequential insertion by default
  • The timer time component is responsible for scanning each bucket. According to the article, there are multiple timers, but the same timer can only scan one bucket at a time
  • The ready queue is responsible for storing the tasks to be completed, but there are multiple ready queues according to the description and different topics.

Timer is responsible for polling, and job pool, delay bucket and ready queue are collections of different responsibilities.

Task status

  • Ready: executable status,
  • Delay: non executable status, waiting for the clock cycle.
  • Reserved: it has been read by the consumer, but the consumption has not been completed.
  • Deleted: consumed or deleted.

External interfaces

Additional content

  1. First, according to the state description, both finish and delete operations set the task to the deleted state.
  2. According to the operations described in this article, the task has been removed from the metadata when the finish or delete operation is performed. At this time, the deleted state may only exist for a very short time, so it will be deleted directly in the actual implementation.
  3. The article does not explain how to deal with the response after it times out, so the individual has now put it back into the waiting queue.
  4. In this article, because clusters are used, the setnx lock of redis is used to ensure that multiple time cycles will not occur when processing multiple buckets. Because this is a simple implementation, it is very simple to set a time queue for each bucket. It is also to facilitate simple processing. Distributed locks can be described in my previous articles.

realization

Now we complete the design according to the design content. We finished this design in four steps

Tasks and related objects

At present, two objects are required, one is the task object (job) and the other is the object responsible for saving the task reference (delay job). The spring boot foundation will not be introduced. This practical tutorial is recommended:
https://github.com/javastacks/spring-boot-best-practice

Task object

@Data
@AllArgsConstructor
@NoArgsConstructor
public class Job implements Serializable {

    /**
     *Unique identification of the deferred task, used to retrieve the task
     */
    @JsonSerialize(using = ToStringSerializer.class)
    private Long id;

    /**
     *Task type (specific business type)
     */
    private String topic;

    /**
     *Delay time of task
     */
    private long delayTime;

    /**
     *Task execution timeout
     */
    private long ttrTime;

    /**
     *Task specific message content, which is used to process specific business logic
     */
    private String message;

    /**
     *Number of retries
     */
    private int retryCount;
    /**
     *Task status
     */
    private JobStatus status;
}

Task reference object

@Data
@AllArgsConstructor
public class DelayJob implements Serializable {


    /**
     *Unique identification of the deferred task
     */
    private long jodId;
    
    /**
     *Task execution time
     */
    private long delayDate;

    /**
     *Task type (specific business type)
     */
    private String topic;


    public DelayJob(Job job) {
        this.jodId = job.getId();
        this.delayDate = System.currentTimeMillis() + job.getDelayTime();
        this.topic = job.getTopic();
    }

    public DelayJob(Object value, Double score) {
        this.jodId = Long.parseLong(String.valueOf(value));
        this.delayDate = System.currentTimeMillis() + score.longValue();
    }
}

container

At present, we need to complete the creation of three containers: job task pool, delayed task container and pending task container

The job task pool provides basic operations for common K / V structures

@Component
@Slf4j
public class JobPool {
    
    @Autowired
    private RedisTemplate redisTemplate;

    private String NAME = "job.pool";
    
    private BoundHashOperations getPool () {
        BoundHashOperations ops = redisTemplate.boundHashOps(NAME);
        return ops;
    }

    /**
     *Add task
     * @param job
     */
    public void addJob (Job job) {
        Log.info ("add task to task pool: {}", json.tojsonstring (job));
        getPool().put(job.getId(),job);
        return ;
    }

    /**
     *Get task
     * @param jobId
     * @return
     */
    public Job getJob(Long jobId) {
        Object o = getPool().get(jobId);
        if (o instanceof Job) {
            return (Job) o;
        }
        return null;
    }

    /**
     *Remove task
     * @param jobId
     */
    public void removeDelayJob (Long jobId) {
        Log.info ("remove task from task pool: {}", jobid);
        //Remove task
        getPool().delete(jobId);
    }
}

Delay tasks, save data using sortable Zset, and provide operations such as fetching the minimum value

@Slf4j
@Component
public class DelayBucket {

    @Autowired
    private RedisTemplate redisTemplate;

    private static AtomicInteger index = new AtomicInteger(0);

    @Value("${thread.size}")
    private int bucketsSize;

    private List  bucketNames = new ArrayList <>();

    @Bean
    public List  createBuckets() {
        for (int i = 0; i < bucketsSize; i++) {
            bucketNames.add("bucket" + i);
        }
        return bucketNames;
    }

    /**
     *Get the name of the bucket
     * @return
     */
    private String getThisBucketName() {
        int thisIndex = index.addAndGet(1);
        int i1 = thisIndex % bucketsSize;
        return bucketNames.get(i1);
    }

    /**
     *Get Bucket Set
     * @param bucketName
     * @return
     */
    private BoundZSetOperations getBucket(String bucketName) {
        return redisTemplate.boundZSetOps(bucketName);
    }

    /**
     *Put delay task
     * @param job
     */
    public void addDelayJob(DelayJob job) {
        Log.info ("add deferred task: {}", json.tojsonstring (job));
        String thisBucketName = getThisBucketName();
        BoundZSetOperations bucket = getBucket(thisBucketName);
        bucket.add(job,job.getDelayDate());
    }

    /**
     *Get the latest deferred tasks
     * @return
     */
    public DelayJob getFirstDelayTime(Integer index) {
        String name = bucketNames.get(index);
        BoundZSetOperations bucket = getBucket(name);
        Set set = bucket.rangeWithScores(0, 1);
        if (CollectionUtils.isEmpty(set)) {
            return null;
        }
        ZSetOperations.TypedTuple typedTuple = (ZSetOperations.TypedTuple) set.toArray()[0];
        Object value = typedTuple.getValue();
        if (value instanceof DelayJob) {
            return (DelayJob) value;
        }
        return null;
    }

    /**
     *Remove deferred tasks
     * @param index
     * @param delayJob
     */
    public void removeDelayTime(Integer index,DelayJob delayJob) {
        String name = bucketNames.get(index);
        BoundZSetOperations bucket = getBucket(name);
        bucket.remove(delayJob);
    }

}

For tasks to be completed, topics are used internally for subdivision, and each topic corresponds to a list set

@Component
@Slf4j
public class ReadyQueue {

    @Autowired
    private RedisTemplate redisTemplate;

    private String NAME = "process.queue";

    private String getKey(String topic) {
        return NAME + topic;
    }

    /**
     *Get queue
     * @param topic
     * @return
     */
    private BoundListOperations getQueue (String topic) {
        BoundListOperations ops = redisTemplate.boundListOps(getKey(topic));
        return ops;
    }

    /**
     *Set task
     * @param delayJob
     */
    public void pushJob(DelayJob delayJob) {
        Log.info ("execute queue add task: {}", delayjob);
        BoundListOperations listOperations = getQueue(delayJob.getTopic());
        listOperations.leftPush(delayJob);
    }

    /**
     *Remove and get task
     * @param topic
     * @return
     */
    public DelayJob popJob(String topic) {
        BoundListOperations listOperations = getQueue(topic);
        Object o = listOperations.leftPop();
        if (o instanceof DelayJob) {
            Log.info ("execute queue fetch task: {}", json.tojsonstring ((delayjob) o));
            return (DelayJob) o;
        }
        return null;
    }
    
}

Polling processing

Thread pool is set, and a polling operation is set for each bucket

@Component
public class DelayTimer implements ApplicationListener  {

    @Autowired
    private DelayBucket delayBucket;
    @Autowired
    private JobPool     jobPool;
    @Autowired
    private ReadyQueue  readyQueue;
    
    @Value("${thread.size}")
    private int length;
    
    @Override 
    public void onApplicationEvent(ContextRefreshedEvent contextRefreshedEvent) {
        ExecutorService executorService = new ThreadPoolExecutor(
                length, 
                length,
                0L, TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue ());

        for (int i = 0; i < length; i++) {
            executorService.execute(
                    new DelayJobHandler(
                            delayBucket,
                            jobPool,
                            readyQueue,
                            i));
        }
        
    }
}

Test request

/**
 *Test request
 * @author daify
 **/
@RestController
@RequestMapping("delay")
public class DelayController {
    
    @Autowired
    private JobService jobService;
    /**
     *Add
     * @param request
     * @return
     */
    @RequestMapping(value = "add",method = RequestMethod.POST)
    public String addDefJob(Job request) {
        DelayJob delayJob = jobService.addDefJob(request);
        return JSON.toJSONString(delayJob);
    }

    /**
     *Get
     * @return
     */
    @RequestMapping(value = "pop",method = RequestMethod.GET)
    public String getProcessJob(String topic) {
        Job process = jobService.getProcessJob(topic);
        return JSON.toJSONString(process);
    }

    /**
     *Complete an executed task
     * @param jobId
     * @return
     */
    @RequestMapping(value = "finish",method = RequestMethod.DELETE)
    public String finishJob(Long jobId) {
        jobService.finishJob(jobId);
        return "success";
    }

    @RequestMapping(value = "delete",method = RequestMethod.DELETE)
    public String deleteJob(Long jobId) {
        jobService.deleteJob(jobId);
        return "success";
    }
    
}

test

Add deferred task

Request via postman: localhost: 8000 / delay / add

At this point, the delayed task is added to the thread pool

2019-08-12 21:21:36.589 info 21444 -- [nio-8000-exec-6] d.samples.redis.delay.container.jobpool: add task to task pool: {delaytime ": 10000," Id ": 3," message ":" tag: testid: 3 "," retrycount ": 0," status ":" delay "," topic ":" test "," ttrtime ": 10000}
2019-08-12 21:21:36.609 info 21444 -- [nio-8000-exec-6] d.s.redis.delay.container.delaybucket: add delayed task: {"delaydate": 1565616106609, "jodid": 3, "topic": "test"}

According to the settings, the task will be added to readyqueue after 10 seconds

2019-08-12 21:21:46.744 info 21444 -- [pool-1-thread-4] d.s.redis.delay.container.readyqueue: execute queue addition task: delayjob (jobid = 3, delaydate = 1565616106609, topic = Test)

Get task

At this time, we request localhost: 8000 / delay / pop

At this time, the task is responded, and the timeout is set while modifying the status, and then placed in the delaybucket

2019-08-09 19:36:02.342 info 58456 -- [nio-8000-exec-3] d.s.redis.delay.container.readyqueue: execute queue retrieval task: {"delaydate": 1565321728704, "jodid": 1, "topic": "test"}
2019-08-09 19:36:02.364 info 58456 -- [nio-8000-exec-3] d.samples.redis.delay.container.jobpool: add task to task pool: {delaytime ": 10000," Id ": 1," message ":" delay 10 seconds, timeout 30 seconds "," retrycount ": 0," status ":" reserved "," topic ":" test "," ttrtime ": 30000}
2019-08-09 19:36:02.384 info 58456 -- [nio-8000-exec-3] d.s.redis.delay.container.delaybucket: add delayed task: {"delaydate": 1565321792364, "jodid": 1, "topic": "test"}

According to the design, after 30 seconds, if the task is not consumed, it will be re placed in readyqueue

2019-08-12 21:21:48.239 info 21444 -- [nio-8000-exec-7] d.s.redis.delay.container.readyqueue: execute queue retrieval task: {"delaydate": 15656106609, "jodid": 3, "topic": "test"}
2019-08-12 21:21:48.261 info 21444 -- [nio-8000-exec-7] d.samples.redis.delay.container.jobpool: add task to task pool: {delaytime ": 10000," Id ": 3," message ":" tag: testid: 3 "," retrycount ": 0," status ":" reserved "," topic ":" test "," ttrtime ": 10000}

Task deletion / consumption

Now let’s request: localhost: 8000 / delay / delete

At this time, the task will be removed in the job pool. At this time, the metadata does not exist, but the task is still circulating in the delaybucket. However, in the cycle, when it is detected that the metadata does not exist, the delayed task will be removed.

2019-08-12 21:21:54.880 info 21444 -- [nio-8000-exec-8] d.samples.redis.delay.container.jobpool: task pool removal task: 3
2019-08-12 21:21:59.104 info 21444 -- [pool-1-thread-5] d.s.redis.delay.handler.delayjobhandler: remove nonexistent tasks: {"delaydate": 15656118261, "jodid": 3, "topic": "test"}

Recent hot article recommendations:

1.1000 + java interview questions and answers (2021 latest version)

2.Stop playing if / else on the full screen. Try the strategy mode. It’s really fragrant!!

3.what the fuck! What is the new syntax of XX ≠ null in Java?

4.Spring boot 2.5 heavy release, dark mode is too explosive!

5.Java development manual (Songshan version) is the latest release. Download it quickly!

Feel good, don’t forget to like + forward!

Recommended Today

The selector returned by ngrx store createselector performs one-step debugging of fetching logic

Test source code: import { Component } from ‘@angular/core’; import { createSelector } from ‘@ngrx/store’; export interface State { counter1: number; counter2: number; } export const selectCounter1 = (state: State) => state.counter1; export const selectCounter2 = (state: State) => state.counter2; export const selectTotal = createSelector( selectCounter1, selectCounter2, (counter1, counter2) => counter1 + counter2 ); // […]