4-rocketmq Foundation

Time:2021-7-24

architecture design


1 Technical Architecture

4-rocketmq Foundation

Rocketmq architecture is mainly divided into four parts, as shown in the figure above:

  • Producer: the role of message publishing. It supports distributed cluster deployment. Producer selects the corresponding broker cluster queue for message delivery through the load balancing module of MQ. The delivery process supports fast failure and low delay.
  • Consumer: the role of message consumption. It supports distributed cluster deployment. It supports the consumption of messages in push and pull modes. At the same time, it also supports cluster and broadcast consumption. It provides real-time message subscription mechanism, which can meet the needs of most users.
  • Nameserver: nameserver is a very simple topic routing registry. Its role is similar to zookeeper in Dubbo. It supports dynamic registration and discovery of brokers. It mainly includes two functions: broker management, and nameserver accepts the registration information of the broker cluster and saves it as the basic data of routing information. Then provide a heartbeat detection mechanism to check whether the broker is still alive; For routing information management, each nameserver will store the entire routing information about the broker cluster and the queue information for client queries. Then producer and conductor can know the routing information of the whole broker cluster through nameserver, so as to deliver and consume messages. Nameserver is usually deployed in a cluster mode, and there is no information communication between instances. The broker registers its own routing information with each nameserver, so a complete routing information is saved on each nameserver instance. When a nameserver goes offline for some reason, the broker can still synchronize its routing information with other nameservers, and the producer and consumer can still dynamically perceive the routing information of the broker.
  • Brokerserver: broker is mainly responsible for message storage, delivery, query and service high availability guarantee. In order to realize these functions, broker includes the following important sub modules.
  1. Remoting module: the entity of the whole broker, which is responsible for processing requests from clients.
  2. Client manager: responsible for managing the client (Producer / consumer) and maintaining the topic subscription information of the consumer
  3. Store service: provides a convenient and simple API interface to process messages, store them to physical hard disk and query.
  4. Ha service: high availability service, which provides data synchronization between master broker and slave broker.
  5. Index service: index the messages delivered to the broker according to the specific message key to provide quick query of messages.

4-rocketmq Foundation

2 deployment architecture

4-rocketmq Foundation

Rocketmq network deployment features

  • Nameserver is an almost stateless node that can be deployed in a cluster without any information synchronization between nodes.
  • Broker deployment is relatively complex. Brokers are divided into master and slave. A master can correspond to multiple slave, but a slave can only correspond to one master. The corresponding relationship between master and slave is defined by specifying the same broker name and different broker IDs. A broker ID of 0 indicates a master and a non-0 indicates a slave. The master can also deploy multiple. Each broker establishes long-term connections with all nodes in the nameserver cluster and regularly registers topic information with all nameservers. Note: the current version of rocketmq supports one master and multiple slave on the deployment architecture, but only the slave server with brokerid = 1 will participate in the message reading load.
  • Producer establishes a long connection with one of the nodes (randomly selected) in the nameserver cluster, periodically obtains topic routing information from the nameserver, establishes a long connection to the master providing topic services, and regularly sends heartbeat to the master. Producer is completely stateless and can be deployed in a cluster.
  • The consumer establishes a long connection with one of the nodes (randomly selected) in the nameserver cluster, regularly obtains topic routing information from the nameserver, establishes a long connection to the master and slave providing topic services, and regularly sends heartbeat to the master and slave. Consumers can subscribe to messages from the master or slave. When consumers pull messages from the master, the master server will suggest whether to pull messages from the master or slave next time according to the distance between the pull offset and the maximum offset (judge whether to read old messages and generate read I / O), whether the slave server can read and other factors.

Describe the cluster workflow in combination with the deployment architecture diagram:

  • Start the nameserver. After the nameserver is up, listen to the port and wait for the broker, producer and consumer to connect. It is equivalent to a routing control center.
  • The broker starts, maintains a long connection with all nameservers, and sends heartbeat packets regularly. The heartbeat packet contains the current broker information (IP + port, etc.) and all topic information stored. After successful registration, there is a mapping relationship between topic and broker in the nameserver cluster.
  • Before sending and receiving messages, create a topic. When creating a topic, you need to specify which brokers the topic will be stored in. You can also create a topic automatically when sending messages.
  • Producer sends a message. When starting, it first establishes a long connection with one of the nameserver clusters, obtains the brokers of the currently sent topics from the nameserver, polls, selects a queue from the queue list, and then establishes a long connection with the broker where the queue is located, so as to send a message to the broker.
  • Similar to producer, consumer establishes a long connection with one of the nameservers, obtains which brokers the current subscription topic exists on, and then directly establishes a connection channel with the broker to start consuming messages.

Stand alone installation

windows setup

  1. download
https://mirror.bit.edu.cn/apache/rocketmq/4.7.0/rocketmq-all-4.7.0-bin-release.zip
  1. decompression

  2. Configure rocketmq environment variable, rocketmq_ Home = rocketmq decompression directory

  3. Start the name server. In rocketmq_ Double click mqnamesrv.cmd in the home / bin directory to run it. The following message appears to indicate that the startup is successful and keep the command window open (if the window flashes, it means that the environment variable is not configured, please configure the environment variable first)

Java HotSpot(TM) 64-Bit Server VM warning: Using the DefNew young collector with the CMS collector is deprecated and will likely be removed in a future release
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
The Name Server boot success. serializeType=JSON
  1. Start the broker. Open another Windows terminal CMD and enter rocketmq_ Home / bin directory,

    inputmqbroker -n 127.0.0.1:9876Start the broker and keep the MQ broker running. Do not close the terminal.

D:\software\rocketmq-all-4.7.0-bin-release\bin>mqbroker -n 127.0.0.1:9876
The broker[PQSZ-L0039, 10.178.42.122:10911] boot success. serializeType=JSON and name server is 127.0.0.1:9876

Linux Installation

download

wget https://mirror.bit.edu.cn/apache/rocketmq/4.7.0/rocketmq-all-4.7.0-bin-release.zip

decompression

Here, unzip it to the soft folder of the current directory.

unzip rocketmq-all-4.7.0-bin-release.zip -d soft/

Start / shut down name server

start-up

[[email protected] rocketmq-all-4.7.0-bin-release]# nohup sh bin/mqnamesrv &

Verify that OK is started:

tail -f ~/logs/rocketmqlogs/namesrv.log

Run failed

Rocketmq’s default virtual machine memory is large. If starting broker fails because of insufficient memory, you need to edit the following two configuration files to modify the JVM memory size

Runserver.sh (Windows corresponds to runserver.cmd)

JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m"

Runbroker.sh (Windows corresponds to runbroker. CMD)

JAVA_OPT="${JAVA_OPT} -server -Xms1g -Xmx1g -Xmn512m"

close

bin/mqshutdown namesrv

Start / close broker

start-up

nohup sh bin/mqbroker -n localhost:9876 &

Verify that OK is started:

tail -f ~/logs/rocketmqlogs/broker.log

close

sh bin/mqshutdown broker

message console

Download console code

git clone https://github.com/apache/rocketmq-externals.git

After downloading the code, open rocketmq console, a springboot project, and enter the configuration file to configure the address of the name server

rocketmq.config.namesrvAddr=192.168.192.140:9876

Start the springboot project and enter localhost: 8080 in the browser

General message

rely on

<dependencies>
    <dependency>
        <groupId>org.apache.rocketmq</groupId>
        <artifactId>rocketmq-client</artifactId>
        <version>4.7.0</version>
    </dependency>
</dependencies>

Production synchronization message

This reliable synchronous sending method is widely used, such as important message notification and short message notification.

public class SyncProducer {

Send asynchronous message

Asynchronous messages are usually used in business scenarios that are sensitive to response time, that is, the sender cannot tolerate waiting for a broker’s response for a long time.

public class AsyncProducer {

One way send message

This method is mainly used in scenarios that do not particularly care about sending results, such as log sending.

public class OnewayProducer {

Consumption news

public class Consumer {

Sequential message

By default, each topic has four queues. The messages in each queue can be consumed in sequence, so we need to ensure that the messages that need to be consumed in sequence can be saved in the same queue. Therefore, when sending messages, we need to determine which queue this message should be saved in.

Message ordering means that messages can be consumed (FIFO) according to the sending order of messages. Rocketmq can strictly guarantee message order, which can be divided into partition order or global order.

The principle of sequential consumption is analyzed. By default, round robin polling will be used to send messages to different queues (partition queues); When consuming messages, pull messages from multiple queues. In this case, the order of sending and consumption cannot be guaranteed. However, if the control sending sequence messages are only sent to the same queue in turn, and the consumption is only pulled from this queue in turn, the sequence is guaranteed. When there is only one queue for sending and consuming, it is globally ordered; If multiple queues participate, the partition is ordered, that is, the messages are ordered relative to each queue.

The following is an example of order partitioning. The sequential process of an order is: create, pay, push and complete. Messages with the same order number will be sent to the same queue successively. When consuming, the same OrderID must get the same queue.

Production message

public class OrderedProducer {

Consumption news

/**

Delay message

For example, in e-commerce, you can send a delay message after submitting an order, check the status of the order after 1h, and cancel the order and release the inventory if it is still unpaid.

Start consumer

public class ScheduledMessageConsumer {

##Production message

public class ScheduledMessageProducer {

You will see that the consumption of the message is 10 seconds later than the storage time.

Usage restrictions for delayed messages

// org/apache/rocketmq/store/config/MessageStoreConfig.java

private String messageDelayLevel = "1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h";

At present, rocketmq does not support any time delay. It is necessary to set several fixed delay levels from 1s to 2h, corresponding to levels 1 to 18 respectively. If the message consumption fails, it will enter the delay message queue. The message sending time is related to the set delay level and retry times. See the code for detailsSendMessageProcessor.java

Filter messages

In most cases, tag is a simple and useful design to select the messages you want. For example:

DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID_EXAMPLE");
consumer.subscribe("TOPIC", "TAGA || TAGB || TAGC");

The consumer will receive a message containing taga or TagB or TAGC. However, the limitation is that a message can only have one label, which may not work in complex scenarios. In this case, you can filter messages using SQL expressions. The SQL feature can be calculated by the properties when the message is sent. Under the syntax defined by rocketmq, you can implement some simple logic. Here is an example:

------------
| message  |
|----------|  a > 5 AND b = 'abc'
| a = 10   |  --------------------> Gotten
| b = 'abc'|
| c = true |
------------
------------
| message  |
|----------|   a > 5 AND b = 'abc'
| a = 1    |  --------------------> Missed
| b = 'abc'|
| c = true |
------------

Basic grammar

Rocketmq only defines some basic syntax to support this feature. You can also easily expand it.

  • Numerical comparison, such as:>,>=,<,<=,BETWEEN,=;
  • Character comparison, such as:=,<>,IN;
  • IS NULLperhapsIS NOT NULL;
  • Logical symbolsAND,OR,NOT;

Constant support types are:

  • Values, such as:123,3.1415;
  • Characters, such as:‘ABC’ must be enclosed in single quotation marks;
  • NULL, special constants
  • Boolean,TRUEorFALSE

Only consumers using push mode can use SQL92 standard SQL statements. The interfaces are as follows:

public void subscribe(finalString topic, final MessageSelector messageSelector)

Enable configuration (important)

To use the filter function, you need to configure the following options in the startup configuration file

./conf/broker.conf

enablePropertyFilter=true

start-up

start mqbroker.cmd -n 127.0.0.1:9876 -c ../conf/broker.conf

Production message

When sending a message, you canputUserPropertyTo set the properties of the message

public class SyncFilterProducer {

Consumption news

Use messageselector.bysql to filter messages using SQL

public class FilterConsumer {

Things news

Transaction messages have three statuses: commit status, rollback status and intermediate status:

  • Transactionstatus. Committransaction: commit a transaction that allows the consumer to consume this message.
  • Transactionstatus. Rollbacktransaction: rollback transaction, which means that the message will be deleted and cannot be consumed.
  • Transactionstatus. Unknown: intermediate status, which means that the message queue needs to be checked to determine the status.

Production news

useTransactionMQProducerClass to create a producer and specify a uniqueProducerGroup, you can set up a custom thread pool to handle these check requests. After executing a local transaction, you need to reply to the message queue according to the execution result. Please refer to the previous section for the returned transaction status.

public class TransactionProducer {

Implement the transaction listening interface

When sending a semi successful message, we useexecuteLocalTransactionMethod to perform a local transaction. It returns one of the three transaction states mentioned in the previous section.checkLocalTransactionMethod is used to check the local transaction status and respond to the check request of the message queue. It is also one of the three transaction states mentioned in the previous section.

public class TransactionListenerImpl implements TransactionListener {

Consumer News

public class Consumer {

Restrictions on transaction message usage

  1. Delayed messages and bulk messages are not supported for transaction messages.
  2. In order to avoid the accumulation of semi queued messages caused by too many times of single message checking, we limit the number of times of single message checking to 15 by default, but users can check through the broker configuration filetransactionCheckMaxParameter to modify this limit. If a message has been checked more than n times (n)=transactionCheckMax)The broker discards this message and prints the error log at the same time by default. Users can overrideAbstractTransactionalMessageCheckListenerClass to modify this behavior.
  3. The transaction message will be checked after a specific length of time such as the parameter transactiontimeout in the broker configuration file. When sending a transaction message, the user can also set the user attribute check_ IMMUNITY_ TIME_ IN_ Seconds to change this limit. This parameter takes precedence overtransactionTimeoutParameters.
  4. Transactional messages may be checked or consumed more than once.
  5. The target topic message submitted to the user may fail. At present, this depends on the log record. Its high availability is guaranteed by rocketmq’s own high availability mechanism. If you want to ensure that transaction messages are not lost and transaction integrity is guaranteed, it is recommended to use synchronous dual write mechanism.
  6. The producer ID of a transaction message cannot be shared with the producer ID of other types of messages. Unlike other types of messages, transaction messages allow reverse queries, and MQ servers can query consumers through their producer ID.

This work adoptsCC agreement, reprint must indicate the author and the link to this article

Recommended Today

Implementation example of go operation etcd

etcdIt is an open-source, distributed key value pair data storage system, which provides shared configuration, service registration and discovery. This paper mainly introduces the installation and use of etcd. Etcdetcd introduction etcdIt is an open source and highly available distributed key value storage system developed with go language, which can be used to configure sharing […]