It can be seen that the wind direction of the technology circle has been changing, the heat of big data and clouds has been slowly fading, and AI and IOT are now popular. These hot concepts will eventually be transformed from thesis and PPT to a real problem-solving system, otherwise it will be a castle in the air. What’s the constant? (Feelings of some digressions)
What’s the difference between a topic and a queue?
The original message queue is a queue in a strict sense
- In fact, there is a competitive relationship between consumers, and each consumer can only receive a part of messages in the queue
If a message data needs to be distributed to multiple consumers, each consumer is required to receive a full amount of messages. For example, for an order data, risk control system, analysis system and payment system need to receive messages. At this time, a single queue can not meet the demand, and a feasible solution is to create a separate queue for each consumer, so that the customer can be satisfiedProducers send multiple copies (bad practice).
In order to solve this problem, another message model is evolved“Publish subscribe model（Publish-Subscribe Pattern）”。
In the publish subscribe model, the sender of the message is called publisher, the receiver of the message is called subscriber, and the container of the server is called topic. Publishers send messages to topics, and subscribers need to “subscribe to topics” before receiving messages.“Subscription” is not only an action here, but also a logical copy of the topic when it is consumed. In each subscription, the subscriber can receive all messages of the topic.
Most of the modern message queuing products use this publish subscribe model
Message model of rabbitmq
It is one of the few products that still adhere to the queue model .
In rabbitmq, exchange is located between the producer and the queue. The producer does not care which queue the message is sent to, but sends the message to exchange. The policy configured on exchange determines which queues the message is delivered to.
If the same message needs to be consumed by multiple consumers, exchange needs to be configured to send the message to multiple queues, and each queue stores a complete message data, which can provide consumption service for a consumer.
This can also realize the function of “one message data can be consumed by multiple subscribers” in the new publish subscribe model in disguise.
Message model of rocketmq
The message model used by rocketmq is the standard publish subscribe model
The confirmation mechanism ensures the reliability of message delivery, but it brings a big problem in the consumer side. In order to ensure the order of the message, before a message is successfully consumed, the next message can not be consumed, otherwise there will be a message hole, which violates the principle of order.
In other words, each topic can only have one consumer instance at most at any time, so it is impossible to improve the overall consumption performance of the consumer side by expanding the number of consumers horizontally. To solve this problem,Rocketmq adds the concept of queue under the topic。
- Each topic contains multiple queues, through which multi instance parallel production and consumption can be realized
- Rocketmq only guarantees the order of messages on the queue, but it can’t guarantee the strict order of messages on the topic level (the same queue is in order, and the queues are out of order)
In rocketmq, the concept of subscriber is embodied by consumer group. Each consumer group consumes a complete message in the topic, and the consumption progress of different consumer groups is not affected by each other. In other words, a message that has been consumed by consumer group 1 will be consumed by consumer group 2。
There are many consumers in the consumption group. The consumers in the same group are competitive consumers, and each consumer is responsible for some messages in the consumption group. If a message is consumed by consumer 1, other consumers in the same group will not receive it again.
In the process of topic consumption, because messages need to be consumed by different groups for many times, the consumed messages will not be deleted immediately. This requires rocketmq to maintain a consumer location on each queue for each consumption group The previous messages of this location have been consumed, and the subsequent messages have not been consumed. For each successful consumption of a message, the consumption location will be increased by one. This consumption location is a very important concept. When we use message queue, most of the reasons for losing messages are due to improper processing of consumption location.
Kafka’s message model
Kafka’s message model is exactly the same as rocketmq’s
The only difference is that in Kafka, the name of queue is different. The corresponding name in Kafka is partition
- Subject: publish subscribe
- Queue: first in first out
The business model is not the same as the implementation level model. For example, MySQL and HBase are databases that support SQL. In their business models, the data cells are all “tables”. However, at the implementation level, no database stores data in the form of two-dimensional tables. MySQL uses B + Tree to store data, while HBase uses kV structure to store data. Similarly, the business models of Kafka and rocketmq are basically the same, not that their implementation is the same. In fact, the implementation of these two message queues is completely different.
- What are pessimistic lock and optimistic lock in MySQL?
- Into the black box: how SQL is executed in the database?
- Principle analysis of hash algorithm
- Consistent hash design idea
- Interpretation of redis cache penetration, cache breakdown and cache avalanche problems, with solutions
- In the face of massive data, how to search faster?