Thief easy to use, glacier open source this precise timing task and delay queue framework!!

Time:2021-3-4

Write on the front

In practical work, many small partners in the development of timing tasks, will take the way of timing scan data table. However, this method has major defects: if the amount of data is large, frequent scanning of data tables will cause great pressure on the database; it is difficult to support large-scale distributed timing tasks; it is difficult to support accurate timing tasks; a lot of CPU resources are wasted; most of the scanned data are unnecessary tasks. So, since there are so many disadvantages in the regular scanning data table, is there a way to solve these problems? Today, glacier is coming with his open source project mykit delay!! Open source address:https://github.com/sunshinely…

If you have any questions in the process of using the framework, you can add glacier wechat_ shine_ Lyz.

The article has been included inhttps://github.com/sunshinely…

Project brief

Mykit system provides a simple, stable and scalable delay message queue framework, which provides precise timing tasks and delay queue processing functions.

Project module description

  • Mykit delay common: general tool module of mykit delay message queue framework, which provides global general tool class
  • Mykit delay config: general configuration module of mykit delay message queue framework, providing global configuration
  • Mykit delay queue: the core implementation module of mykit delay message queue framework. At present, all the main functions are implemented in this module
  • Mykit delay controller: Restful interface implementation module of mykit delay message queue framework, which provides restful interface access and is compatible with various language calls
  • Mykit delay core: the entrance of mykit delay message queue framework, and the startup program of the whole framework is implemented in this module
  • Mykit delay test: General test module of mykit delay message queue framework, mainly providing JUnit unit test cases

Demand background

  • The user did not pay after placing the order, and the payment time exceeded 30 minutes later
  • Inform users to participate in system activities at a certain point in time
  • After business execution fails, try again every 10 minutes

There are many similar scenarios, and the simple way to deal with them is to use timing tasks. If there is a lot of data, some data may be delayed seriously, and more and more timing services lead to tedious task scheduling and poor management.

Queue design

The overall architecture design is shown in the figure below.

Thief easy to use, glacier open source this precise timing task and delay queue framework!!

Issues to be considered before development

  • Timely consumer can receive on time
  • Consumption weight of messages at the same time
  • Reliability messages cannot be consumed
  • Recoverable if there are other circumstances that make the message system unavailable, at least the data can be recovered
  • It can be withdrawn because it is a delay message, and the message support can cancel the consumption
  • High availability multiple instances, which means HA / active / standby mode, does not mean that multiple instances work together at the same time
  • How to consume on the consumer side

Of course, the main reason for the initial selection of redis as data cache is that redis supports Zset data structure (score delay time in milliseconds), which reduces the trouble of sorting and has high performance. Our demand is to determine the execution order according to the time dimension, and also supports the map list data structure.

Simply define a message data structure

private String topic;/***topic**/
Private string ID; / ***/
private String bizKey;
Private long delay; / * * delay milliseconds**/
Private int priority; // priority
Private long TTL; / * * TTL for consumer consumption**/
Private string body; / * * message body**/
private long createTime=System.currentTimeMillis();
private int status= Status.WaitPut.ordinal();

Operation principle:

  • Use map to store metadata. ID as the key, the whole message structure is serialized (JSON / )Then, as value, it is put into the metamessage pool.
  • The ID is put into a Zset sequential table (n), and the score is created time + delay + priority. The modification status is in delay
  • The timer is used to monitor the data of top 10 in Zset in real time. If the data score is less than or equal to the current time, it will be taken out in milliseconds. According to the topic, it will be put into a new consumable list again. The data that has been taken out will be deleted in Zset, and the status will be modified to be consumed
  • The client only needs to get the data from the consumable queue. And the status must be to be consumed, and the running time needs to be < = current time. If it does not meet the requirement, it will be put into the Zset list again, and the modification status will be delayed. If the modification status is satisfied, it is consumed. Or delete the metadata directly.

client

Because it involves different programming languages, it supports HTTP access by default.

  • After adding delay message successfully, return consumption unique ID post / push { … message body}
  • Message ID get / delete? ID is required to delete delay message=
  • Recovery delay message get / restore? Expire = true | false? Whether to recover expired and unexecuted messages.
  • To recover a single delay message, the message ID get / restore / ID needs to be passed
  • Getting messages requires a long connection get / get / topic

Using nginx to expose the service, configured as polling, traffic can be evenly distributed when adding delay messages.

At present, the client in the system does not use HTTP long connection to consume messages, but uses MQ to consume data, so that the client can not care about delayed message queue. You only need to intercept MQ when sending it. If it’s a delay message, use the delay message system to process it.

Message recoverable

The principle of recovery is to record logs normally, such as MySQL binlog.

Here we directly use the MySQL database as the log.

At present, the following two tables are created:

  • The message table field includes the whole message body
  • Message flow table fields include message ID, change status, change time, Zset scanning thread name, host / IP

Defining the name of the Zset scanning thread is to see more clearly which Zset messages are distributed to. The premise is that the key of Zset has something to do with the name of the thread monitoring Zset. Here, it can also be the Zset key.

Support message recovery

If the redis server goes down, there will be no data after restart. Therefore, this recovery is very necessary. You only need to redistribute all the data whose message status is not equal to the consumed data to the delay queue from table 1, that is, the message table, and then synchronize the status.

Of course, restoring a single task can do the same.

Data table design
Here, I will directly give the SQL statement to create the data table.

DROP TABLE IF EXISTS `mykit_delay_queue_job`;
CREATE TABLE `mykit_delay_queue_job` (
  `id` varchar(128) NOT NULL,
  `bizkey` varchar(128) DEFAULT NULL,
  `topic` varchar(128) DEFAULT NULL,
  `subtopic` varchar(250) DEFAULT NULL,
  `delay` bigint(20) DEFAULT NULL,
  `create_time` bigint(20) DEFAULT NULL,
  `body` text,
  `status` int(11) DEFAULT NULL,
  `ttl` int(11) DEFAULT NULL,
  `update_time` datetime(3) DEFAULT NULL,
  PRIMARY KEY (`id`),
  KEY `mykit_delay_queue_job_ID_STATUS` (`id`,`status`),
  KEY `mykit_delay_queue_job_STATUS` (`status`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

-- ----------------------------
-- Table structure for mykit_delay_queue_job_log
-- ----------------------------
DROP TABLE IF EXISTS `mykit_delay_queue_job_log`;
CREATE TABLE `mykit_delay_queue_job_log` (
  `id` varchar(128) NOT NULL,
  `status` int(11) DEFAULT NULL,
  `thread` varchar(60) DEFAULT NULL,
  `update_time` datetime(3) DEFAULT NULL,
  `host` varchar(128) DEFAULT NULL,
  KEY `mykit_delay_queue_job_LOG_ID_STATUS` (`id`,`status`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

About high availability

Zookeeper is the best choice for distributed coordination.

If there are multiple instances, at most one can work at the same time, so as to avoid the disadvantages brought by distributed competitive lock. Of course, if the business needs multiple instances to work at the same time, it is also supported, that is, a message can only be processed by one instance at most. You can use zookeeper or redis to implement distributed lock.

In the end, we did a test to run multiple instances at the same time. Maybe because of the lock problem, the performance has declined, but the single machine effect is very good. Therefore, the active standby deployment mode based on docker is recommended.

Operation mode

  • Support master, slave (HA) needs to be configuredmykit.delay.registry.serverListZK cluster address list
  • Supporting cluster will involve distributed lock competition, and the effect of distributed lock competition is not obviousredisOfsetNxrealization
  • StandAlone

At present, after testing,The mode of master slave is recommendedThe cluster mode will be optimized later

How to access

In order to provide a unified precise timing task and delay queue framework, mykit delay provides an HTTP rest interface for other business systems to call. The interface is simple and convenient to use. It only needs a simple calling interface to pass the corresponding parameters.

Message body

In JSON data format, parameters are only providedhttpagreement

  • Body business message body
  • Delay millisecond distancecreateTimeInterval milliseconds of
  • ID task ID the system automatically generates the task and returns if it is created successfully
  • Status is not filled in by default
  • Topic title
  • Subtopic reserved field
  • TTL reserved field
  • Create time is not required by default

Add task

/push  
    POST application/json
{"body":"{hello world}","delay":10000,"id":"20","status":0,"topic":"ces","subtopic":"",ttl":12}

Delete task

To delete a task, you need to record a jobid

/delete?jobId=xxx
   GET

Restore single task

It is used to restore the task according to the log

/reStoreJob?JobId=xxx
   GET

Recover all outstanding tasks

Restore tasks based on logs

/reStore?expire=true
   GET

parameterexpireIndicates whether it is necessary to recover data that has expired but has not yet been executed

Clear queue data

Clear all the data in the queue according to the unfinished data in the log. After clearing, all tasks in the cache are deleted

/clearAll
 GET

Client access queue mode

Now it is implemented by defaultRocketMQAndActiveMQThe method of push. It relies on MQ to realize the coupling between delay framework and specific business system.

Message and message in message bodyRocketMQandActiveMQMessage field correspondence

mykit-delay RocketMQ ActiveMQ remarks
topic topic topic Peer to peer sending queue name or topic name
subtopic subtopic subtopic Send queue sub name or topic sub name point to point
body Message content Message content Message content

About system configuration

The interaction between the delay framework and the specific execution business system is realized through the delay framework configuration, and the specific configuration file location is under the mykit delay config projectresources/properties/starter.propertiesFile.

test

You need to configure the database address and redis address. If it is not in stand-alone mode, you also need to configure zookeeper

Run the test class under mykit delay test moduleio.mykit.delay.test.PushTestAdd task to queue

Start mykit delay test moduleio.mykit.delay.TestDelayQueueAdd data before consumption. For the convenience of query effect, the default consumption mode isconsoleCQconsole output

extend

The number of Zset queues can be configured to avoid the problem of high latency caused by big data.

Short term planning

  • Partition (Buck) supports dynamic setting
  • Data consistency between redis and database(important
  • Realize your own push-pull mechanism
  • At present, it only relies on redis to support switchable implementation mode, which needs to be optimized in the future
  • Support web console to manage queue
  • Realize message consumptionTTLmechanism

If this open source framework is helpful to you, please open the GitHub link:https://github.com/sunshinelyz/mykit-delayTo a star, let more small partners see, reduce the work of tedious scan data table timing task development. Also hope to have more and more small partners to participate in this open source project, we fatten it together!!

Thief easy to use, glacier open source this precise timing task and delay queue framework!!

Well, it’s late. Let’s call it a day. I’m glacier. I’ll see you next time!!

Heavyweight benefits

WeChat search official account official, WeChat’s official account, pay attention to this deep programmer, read hard core technology dry cargo every day, reply to PDF, and I have prepared a big factory interview information and my original PDF technology document, and I have prepared many resume templates for you (constantly updated). I hope you can find your favorite job. Learning is a way of sometimes depressed and sometimes laughing. Come on. If you succeed in entering the company you like, don’t slack off. Career growth is the same as new technology learning. If you are lucky, I’ll see you in the world!

In addition, I open source PDF, I will continue to update and maintain, thank you for your long-term support to Glacier!!

Write at the end

If you think glacier’s writing is good, please search on wechat and follow “Glacier TechnologyThe official account of WeChat is learning high concurrent, distributed, micro services, big data, Internet and cloud native technology with glaciers.Glacier TechnologyThe official account of WeChat has updated a lot of technical topics, and every technical article is full of dry cargo. Many readers have read theGlacier TechnologyThe official account of WeChat, who has been interlocking with the interviewer, has successfully switched to the big factory. Many readers have achieved technological leaps and become the backbone of the company. If you want to improve your ability, realize the leap of technical ability, enter a large factory, get a promotion and raise salary, then pay attention toGlacier TechnologyWeChat official account, updating the hard core technology dry cargo every day, so that you can’t lose your knowledge of how to improve the technical capability.