Cognition of rocketmq

Time:2021-1-12

Introduction to MQ

Message queue is a kind of middleware which can realize the asynchronous communication of the system, and it is often used to solve the problemAsynchronous decoupling and peak load shedding of request (TPS) in systemIt’s a matter of time. That is to say, it is for developers, not products that end users can use directly;

Three message protocols

JMS (Java Message Service)

  • JMS is essentially a Java API. Three roles of producer, consumer and provider are defined in JMS. Producer is the sender of message, consumer is the receiver of message, provider is the provider of service. Producer and consumer are collectively referred to as client.
  • JMS definesTwo message models: peer to peer and publish subscribeIn the publish subscribe model, the producer can send the message to the specified topic by routing the message through the topic, and the consumer can receive the message sent by the producer by subscribing to the topic.
  • A producer can send messages to one or more topics, a consumer can consume messages from one or more topics, and a topic can have multiple producers or consumers. Producers and consumers only need to associate topics, and they don’t care who sends or consumes the messages. The provider maintains one or more queues for each topic to store messages. Messages are ordered in the queue and follow the principle of first in first out. Messages between different queues are out of order.
  • There is no concept of topic in peer-to-peer mode. Producers directly send messages to the specified queue, and consumers also specify the queue for consumption. Messages can only be consumed by one consumer, not by multiple consumers.Both Kafka and rocketmq implement or partially implement the JMS protocol.

AMQP (Advanced message queuing protocol)

Unlike JMS,AMQP is an application layer network transmission protocol, which defines the message format and has nothing to do with the development language。 In AMQP, there are two roles: producer and consumer, and messages are stored in the queue. However, unlike JMS, which uses topic to route messages, the routing mode of AMQP is determined by exchange and binding.

The client can create a queue and inform exchange when creating a queue. The condition is binding key. When a producer sends a message to exchange, it will specify a router key. After exchange receives the message, it will compare it with the binding key it maintains and send it to the qualified queue. When consumers consume, they specify the queue to consume.

Rabbitmq implements the AMQP protocol.

MQTT(Message Queuing Telemetry Transport)

Mqtt protocol is a lightweight protocol based on publish and subscribe, which supports TCP and UDP connections. It is mainly used in instant messaging, small devices, mobile applications and other fields. There are three roles in mqtt: publish, subscribe and broker. Broker is the service provider. The publisher is the same as the producer in the former two protocols. It sends the message to the broker, and the subscriber gets the message from the broker and does business processing.

In mqtt message, the fixed header is only 2 bytes, and the cost is very small. In addition, it is divided into two parts: variable header and message body. The fixed header contains information such as message type, message level, size of variable length header and total length of message body. The variable length header contains different identification information according to the message category. Mqtt allows the client to create a topic dynamically. After the publisher establishes a session with the server, it can send data to the corresponding topic of the server through the publish method. After the subscriber subscribes to the topic through subscribe, the server will push the message in the topic to the corresponding subscriber.

Rocketmq architecture component

Cognition of rocketmq

NameServer

Nameserver is an almost stateless node, which can be deployed in clusters. There is no information synchronization between nodes. They are independent and parallel, and each keeps a copy of broker, topic and other cluster information.

  • Very simple topic routing registry;
  • Support dynamic registration and discovery of broker;
  • Broker heartbeat detection;
  • Provide topic routing function for producer and consumer clusters (maintain the mapping relationship between topic and broker);

BrokerServer

Broker is mainly responsible for the storage, delivery and query of messages, as well as the guarantee of high availability of services. It consists of four main modules.

  • Remoting Module: the whole broker entity, which is responsible for processing requests from clients;
  • Client Manager: manage the producer / consumer and maintain the consumer’s topic subscription information;
  • Store Service: provide convenient and simple API interface to process messages, store them in physical hard disk and query them;
  • HA Service: high availability service, providing data synchronization between master broker and slave broker;
  • Index Service: index the messages delivered to the broker according to the specific message key to provide fast query of messages;

Producer

Message publishing role, support distributed cluster deployment, no state information.

Consumer

The role of message consumption supports distributed cluster deployment. It supports two modes of push and pull to consume messages. At the same time, it also supports the consumption of cluster mode and broadcast mode. It provides real-time message subscription mechanism.

Characteristics of network deployment

**-

Cognition of rocketmq

  • Nameserver is an almost stateless node, which can be deployed in clusters without any information synchronization between nodes.
  • Broker is divided into master and slave. A master can correspond to multiple slaves, but a slave can only correspond to one master.A broker ID of 0 means master, and a non-0 means slave。 Each broker establishes a long connection with all nodes in the nameserver cluster, and registers topic information to all nameservers regularly.
  • Producer establishes a long connection with one of the nodes (randomly selected) in the nameserver cluster, obtains topic routing information from the nameserver regularly, and establishes a long connection with the master in the broker node that provides topic serviceAnd send heartbeat to master regularly。 Producer is completely stateless and can be deployed in clusters.
  • The consumer establishes a long connection with one of the nodes (randomly selected) in the nameserver cluster, and regularly obtains the topic routing information from the nameserver,And establish long-term connection to the master and slave in the broker node providing topic service, and send heartbeat to the master and slave regularly

Connection relationship

Cognition of rocketmq

  • Nameserver is a cluster;
  • Broker is a cluster, which is divided into master node and slave node. The master node can read and write, while the slave node can only read messages. It sends heartbeat information to all nameservers regularly and registers topic information to nameserver regularly
  • Producer is a cluster
  • Consumer is a cluster
  • Producer randomly establishes a long connection nameserver from which to obtain topic information and connect it with the topicMasterEstablish a long connection, send messages and send heartbeat information regularly.
  • The consumer connects to a nameserver randomly, gets the topic information from it, and compares it with the location of the topicMaster or slaveEstablish a long connection, send messages and send heartbeat information regularly.

Workflow

  • Start nameserver. After nameserver gets up, it listens to the port and waits for broker, producer and consumer to connect. It is equivalent to a routing control center;
  • The broker starts, keeps a long connection with all nameservers, and sends heartbeat packets regularly. The heartbeat packet contains the current broker information (IP + port, etc.) and stores all topic information。 After successful registration, there is a mapping relationship between topic and broker in nameserver cluster;
  • Before sending and receiving messages, create a topic. When creating a topic, you need to specify where the topic will be storedBrokerOn the other hand, topics can be created automatically when sending messages;
  • When producer sends a message, it first establishes a long-term connection with one of the nameserver clusters, obtains the brokers [Master] of the currently sent topic from the nameserver, and then selects a queue from the queue list by polling, and then establishes a long-term connection with the broker where the queue is located, so as to Send a message to the broker;
  • Similar to producer, a consumer establishes a long-term connection with one of the nameservers to obtain the brokers on which the current subscription topic exists, and then directly establishes a connection channel with the broker to start consuming messages;

Overall architecture of message storage

Cognition of rocketmq

  • Commitlog: message body and metadata storage body. It stores the content of message body written by producer. The message content is not fixed length.

The message is written to the log file in sequence. When the file is full, it is written to the next file;

The default size of a single file is 1g, the length of the file name is 20 bits, the left is filled with zeros, and the rest is the starting offset. For example, 00000000000000 represents the first file, the starting offset is 0, and the file size is 1g = 1073741824; when the first file is full, the second file is 0000000000 1073741824, and the starting offset is 1073741824, and so on.

Cognition of rocketmq

  • Consumequeue: Message consumption queue. It is mainly introduced to improve the performance of message consumption. Because rocketmq is aSubscription mode based on topicIf you want to traverse the commitlog file, it is very inefficient to retrieve messages according to topic.
  • The consumer can find the message to be consumed according to the consumequeue. Among them, consumequeue (logical consumption queue) is used as the index of consumption message, which stores the initial physical offset offset offset, message size and hashcode value of message tag in commitlog of the queue message under the specified topic.

The consumequeue file can be regarded as a topic based commitlog index file, so the consumequeue folder is organized as follows: topic / queue / file three-tier structure, and the specific storage path is $home / store / consumequeue / {topic} / {queueid} / {filename}. Similarly, the consumequeue file adopts fixed length design. Each entry has 20 bytes, which are 8-byte commitlog physical offset, 4-byte message length and 8-byte tag hashcode. A single file consists of 30W entries, which can be accessed randomly like an array. The size of each consumequeue file is about 5.72m;

Cognition of rocketmq

  • Index file: index file provides a way to query messages by key or time interval. The storage location of the index file is $home / store / index / {file name}. The file name is named after the time stamp when it was created. The fixed size of a single index file is about 400m. An index file can store 2000W indexes,The underlying storage of indexfile is designed to implement HashMap structure in the file system, so the underlying implementation of index file of rocketmq is hash index;
  • As can be seen from the above overall message storage architecture of rocketmq, rocketmq adopts a hybrid storage structure, that is, all queues in a single broker instance share a log data file (commitlog) for storage;
  • Rocketmq’s hybrid storage structure (the message entity contents of multiple topics are stored in a commitlog) adopts the storage structure of separating the data and index parts for producer and consumer respectively. Producer sends the message to the broker, and then the broker uses synchronous or asynchronous methods to swipe and persist the message to the commitlog;
  • As long as the message is persistent to the disk file commitlog, the message sent by producer will not be lost. Because of this, consumers will certainly have the opportunity to consume this news.When a message cannot be pulled, it can wait for the next message to be pulled. At the same time, the server also supports the long polling mode. If a message is not pulled by a message pull request, the broker is allowed to wait for 30s. As long as a new message arrives during this period, it will be directly returned to the consumer
  • Here, the specific method of rocketmq is to use the broker’s back-end service thread — reputmessageservice to continuously distribute requests and asynchronously build the data of consumequeue (logical consumption queue) and indexfile (index file).
  • Consumer consumption takes “topic” as the granularity, but commitlog is the summary storage of all topic messages. At this time, an index of the offset of commitlog file with topic as the dimension is needed to facilitate consumption of data under this topic. Therefore, consumequeue is generated.It is equivalent to that there are multiple message queues in a topic, and then the message queue is mapped to a consumequeue message consumption queue for consumers to quickly consume the data in this topic

Rocketmq terminology

Producer

The message producer is located in the user’s process,Producer obtains the routing information of all brokers through nameserverAccording to the load balancing policy, select which Broker to send the message to, and then call the Broker interface to submit the message.

Producer Group

A producer group is simply a group of producers that send the same message.

Consumer

Message consumer, located in the user process. After obtaining the routing information of all brokers through nameserver, the consumer sends a pull request to the broker to obtain the message data. The consumer can be started in two modes,Broadcast and clusterIn broadcast mode, a message will be sent to all consumers, while in cluster mode, a message will only be sent to one consumer

Consumer Group

A consumer group, similar to a producer, consists of multiple consumer instances that consume the same type of message.

Topic

Topic is used to divide messages by topic,Producer sends the message to the specified topic, and the consumer subscribes to the topic to receive the message。 Topics have no strong relationship with the sender and the consumer. The sender can send messages to multiple topics at the same time, and the consumer can subscribe to the messages of multiple topics. In rocketmq,Topic is a logical concept. Message stores are not separated by topic

Message

Represents a message, usingMessageIdUnique identification, the user can set the message key when sending, which is convenient for later query and tracking. A message must specify topic, which is equivalent to the mailing address. Message also has an optional tag setting, so that the consumer can filter messages based on tag. You can also add additional key value pairs. For example, you need a business key to find messages on the broker, which is convenient for diagnosing problems in the development process.

Tag

Tags can be considered as further refinements of topics. Generally, tags are introduced to mark messages for different purposes in the same business module.

Broker

Broker is the core module of rocketmq,Responsible for receiving and storing messagesAt the same time, the push / pull interface is provided to send messages to the consumer. The consumer can choose to read data from master or slave. The broker cluster is composed of multiple master / slave nodes, and there is no data interaction between the master nodes in the cluster. Broker also provides message query function, which can query messages through message ID and message key. Borker will synchronize its topic configuration information to nameserver in real time.

Queue

Topic and queue are one to manyA topic can contain multiple queues, mainly used for load balancing. When sending a message, the user only specifies topic, and producer will select which queue to send to according to the routing information of topic. When consumers subscribe to messages, they will decide which queues to subscribe to according to the load balancing policy.

Offset

Rocketmq will generate a message index file for each queue under each topic when storing messages, and each queue corresponds to an offsetRecord the number of messages in the current queue

NameServer

Nameserver can be regarded as the registry of rocketmq, which manages two parts of data: the routing configuration of topic queue of cluster and the real-time configuration information of broker. Other modules get the latest topic configuration and routing information through the interface provided by nameserv.

  • Producer/Consumer: get the address information of the broker corresponding to the topic through the query interface
  • Broker: register configuration information to nameserver and update topic information to nameserver in real time

Recommended Today

Summary of recent use of gin

Recently, a new project is developed by using gin. Some problems are encountered in the process. To sum up, as a note, I hope it can help you. Cross domain problems Middleware: func Cors() gin.HandlerFunc { return func(c *gin.Context) { //Here you can use * or the domain name you specify c.Header(“Access-Control-Allow-Origin”, “*”) //Allow header […]