1. Message producer, message sender, queue
- Message producer: sends a message to the message queue.
- Message consumer consumer: receives messages from the message queue.
- Broker: the concept comes from Apache ActiveMQ, which refers to the server of MQ. It helps you transfer messages from the sender to the receiver.
- Message queue: a first in, first out message storage area. The messages are sent and received in order. Once the message is consumed, it will be deleted from the queue.
2. Main considerations for broker design
1) Message dump: deliver at a more appropriate time point, or assist the message to reach the consumer through a series of means.
2) Standardize a normal form and a general pattern to meet the needs of decoupling, final consistency, peak staggering, etc.
3) In fact, the simple understanding is that a message forwarder can make one RPC into two RPCs. The sender delivers the message to the broker, and the broker forwards the message to the receiver.
In summary, two RPCs plus one dump, and three RPCs for consumption confirmation.
3. Peer to peer message queuing model
The point-to-point model is used for point-to-point communication between message producers and message consumers.
Point to point mode consists of three roles:
- Message queue
Each message is sent to a specific queue, from which the receiver gets the message. Queues hold messages that can be kept in memory or persisted until they are consumed or timed out.
There is only one consumer per message (that is, once consumed, the message is no longer in the message queue)
There is no time dependency between sender and receiver
The receiver needs to respond to the queue successfully after receiving the message successfully
4. Publish / subscribe message model topic
The publish and subscribe model has three roles:
- Publisher (publisher)
Multiple publishers send messages to topics, and the system delivers them to multiple subscribers.
- Each message can have multiple consumers: unlike point-to-point, publishing messages can be consumed by all subscribers
- There is a temporal dependency between publishers and subscribers.
- For a topic subscriber, it must create a subscriber before consuming the publisher’s message.
- In order to consume messages, subscribers must keep running.
5. Differences between point-to-point and publish subscription
The producer sends a message to the queue, and only one consumer can receive it.
The message sent to topic by the publisher will only be received by subscribers who have subscribed to topic.
6. Sequence guarantee of messages
Based on the queue message model, using the FIFO first in first out feature, the order of messages can be guaranteed.
7. Ack mechanism of message
That is, the ackownledge confirmation mechanism of the message,
To ensure that messages are not lost, the message queue provides the message knowledge mechanism, that is, the ACK mechanism. When the consumer confirms that the message has been consumed and sent an ACK to the message queue, the message queue can delete the cancellation
I’m out of breath. If the consumer is down / shut down and no ack is sent, the message queue will think that the message has not been processed and will send the message to other consumers for re consumption.
8. Design idea of final consistency
Mainly in the way of “record” and “compensation”.
The local transaction maintains business changes and notification messages, landing together, and then the RPC arrives at the broker. After the broker successfully lands, the RPC returns success, and the local message can be deleted. Otherwise, the local message is constantly re sent by scheduled task polling, which ensures that the message can land on the broker reliably.
The process of a broker sending a message to a consumer is similar, sending a message until the consumer sends a successful consumer confirmation.
First, we ignore the problem of duplicate messages. Through two message landing and compensation, the downstream will surely receive messages. Then relying on the state machine version number and other methods to make weight judgment and update their own business, the final consistency is achieved.
If the consumer can’t handle the slow consumption, the consumer should be allowed to take the initiative to ack the error, and the next delivery time can be agreed with the broker.
For the message delivered by broker to consumer, it is necessary to record the IP address of the delivery, because it is uncertain whether the loss is in the process of business processing or in the case of message transmission loss. Ask this IP before you decide to resend. Is the message processing successful? If the inquiry fails, resend it.
Transaction: local transaction, local landing, compensation sending. Local transactions do business landing and message landing transactions, not business landing and RPC successful transactions. As long as the message is successfully landed, there is no risk of loss to a large extent.
9. Transaction support for messages
For example, in the task center scenario, a single processing may involve the receiving and processing of multiple messages, which should be within the same transaction range. If a message fails to be processed, the transaction rolls back and the message returns to the queue.
10. Message persistence
Message persistence is very important for some key core businesses. After message persistence is enabled and message queue is down and restarted, message can be recovered from persistent storage without loss and consumption processing can continue.
11. High availability of message queuing
In the actual production environment, when using the Message Queuing service of a single instance, if there are system problems such as downtime and restart, message queuing cannot provide services. Therefore, in many scenarios, we hope that message queuing has high availability support, such as
Rabbitmq’s high availability scheme of image cluster mode, ActiveMQ’s high availability scheme based on leveldb + zookeeper, and Kafka’s replication mechanism, etc.
12. Selection and application scenario of message queue
For details, please refer to: high concurrency Architecture Series: features, selection, and application scenarios of distributed message queuing