Comprehensive understanding of message queuing (1)


Let’s think about message queuing

1. Why is MQ used in the system? Must it be used in distributed systems?

2. What middleware does MQ have? What are their characteristics?

3. Does MQ bring benefits to the system, but does it bring any problems? How to solve it?

Generally, during our interview, the interviewer will ask the following questions:

1. What is the role of MQ in your project?

2. Why choose this MQ as message middleware?

3. What about repeated consumption?

4. How to ensure that messages are consumed?

Well, next, with these questions, I’d like to share some knowledge about MQ with you.

1、 The role of message middleware in system

What is the role of MQ in the system? Apart from basic message publishing and subscription, there are the following points:

1. Decoupling of distributed system

2. Asynchronous business processing without immediate return

3. Peak shaving and valley filling, no direct access to services, relieving service pressure and increasing performance

4. Logging

Distributed system decoupling:



In a distributed system, it is called either by rest or by RPC such as Dubbo. However, some scenarios need to be decoupled and cannot be called directly. For example, in a message driven system, the message sender completes the local business and sends the message. The message consumer service of multi platform needs to receive the push message and then continues to process other business.

Looking at these two architecture diagrams, the first kind of BC is directly dependent on a service. If the interface in a is modified, the BC must be modified along with it, and the coupling is high. The second way is to receive messages through MQ as a middleware, and BC only depends on the received messages rather than specific interfaces. In this way, to make a service modify or add other services, just subscribe to MQ.

No real-time business asynchronous processing is required

Take the user registration business process as an example

1. User registration and warehousing

2. User verification mail sending

3. User verification SMS sending

In the original system design, such a service process would be processed serially, i.e. 1-2-3 first. However, we can think about this: if a single machine is served, and there are many registered users, can the system resist?

Suppose that the time of elder brother stage is 1 = 50ms, 2 = 50ms, 3 = 50ms, then all = 150ms for a request. Then suppose that the CPU of the server is 1 and can only handle single thread. If the QPS of single server and single thread can be calculated, QPS = 1000 / 150 ≈ 7. Now I let this QPS * 3, improve the escort, and at this time introduce MQ service as the intermediate. As shown in the figure, I serve users in a After the group test is completed, it will return directly. At this time, MQ is used to send asynchronous processing messages, and B and C services handle them separately. A does not need to wait for the return results of B and C. in this way, the user experience is only 50ms waiting time. At the stage of e-mail and SMS, users can receive a certain waiting time due to network delay.

Peak shaving and valley filling

For general services, our request access system is a direct request. In this mode, the problem is not very big when the user’s access is small. However, if the user’s request hits a certain bottleneck or some problems arise, we need to consider optimizing our system architecture, which is one of the formal solutions of MQ middleware.

Next, take the seckill system as an example to analyze the problem. How to deal with the instant million concurrency of the seckill system? General seckill system will filter requests, invalid and repeated will be filtered once, and the rest will really enter the seckill service and order service. But even so, the concurrency is still very high. If the gateway forwards all the requests to the order service, the downstream system will be crushed and the service will not be available or even avalanched






The real seckill system is more complex, including nginx, gateway, registry, redis cache, database cluster, message queue cluster

The solution is to add the faster tasks processed by the upstream to the queue processing, and consume the queue one by one in the downstream. Until all queue consumption is completed. The number of requests processed by adding the seckill service is 1000 / s, and the processing request of downstream order service is 10 / S. in order not to cause pressure on the downstream order service, the information after the second kill is sent to the queue, and the order service can process the requests one by one at a calm speed of 10 / s, instead of directly blocking the 1000 requests, regardless of whether people want to or not.

Here, we can summarize the filtering methods of the seckill system:

1. Click the page button once to set the gray

2. By limiting the number of requests per second, such as 100 / s, nginx, sentinel can be used

3. Filter the repeated requests of the same user through the user’s unique identification and commodity information.

4. The successful seckill information is stored in message queue and processed by downstream order system


All services send the date to the MQ service to store the log. MQ serves as the middleware to persist and forward the log, and the big data service reads and analyzes the MQ log.

2、 How to choose MQ

Some people come up with a performance comparison, and then say that rabbitmq is the best MQ in the world. You compare choosing MQ to choosing a wife. You need a full set, white and beautiful, convex and backward, sexy, hardworking and capable It’s really lack of social education, brother. Can you raise it? Can a set of maintenance package be maintained for 1W / month? Lao Wang next door often comes to your house for dinner. Can you take it? Jujube + medlar + kidney tonic tablets, I’m afraid the heart is more than enough but not enough.

To get to the point, in fact, I think this is a question of thinking. First of all, what conditions should we look at?

1. Use? For logging, decoupling, or asynchronous processing

2. What about the company? Whether the personnel are sufficient, the existing personnel technology stack situation, the personnel technical stack strength

3. Project status? Project cycle, personnel, user volume, architecture design, old project or not

4. Current situation of mainstream MQ? Stability and reliability, community activity, document comprehensiveness and cloud service support

In the example above, the log message is Kafka. Why Kafka? Kafka is an open source distributed publish subscribe message system of LinkedIn, which belongs to the top project of Apache, and the community is active. Kafka is mainly characterized by processing message consumption based on pull mode and pursuing high throughput. At the beginning, Kafka was used for log collection and transmission. Later, the version upgrade started to support replication and did not support transactions. There were no strict requirements on the repetition, loss and error of messages. It is suitable for the data collection business of Internet services that generate a lot of data. However, Kafka is relatively important and needs to rely on zookeeper. There are no problems in the use of Kafka in large companies, and special maintenance is indispensable.

Rocketmq is an open source reliable messaging system in the project, and has donated Apache as a top-level project. At the beginning, it was focused on reliable message transmission without logging. In fact, the performance of log processing is also good. Currently, the supported clients include Java, C + +, and go. The community is relatively active and the documents are comprehensive. However, it is difficult to modify the core ones. After all, Alibaba cloud makes money by buying this service. Therefore, if the company’s instance is not confident, it’s better to choose carefully. If it’s really not possible, you can directly purchase cloud services to save your heart and effort.

3、 Features of mainstream MQ:

4、 How to ensure that messages are not consumed repeatedly

Here is a brief introduction. I will elaborate on this issue later. It is generally due to some special reasons, such as network reasons. The message consumption is not recorded due to service restart, resulting in the possibility of repeated consumption. The general processing method is to ensure the idempotency of the interface design, and the main purpose is to judge whether there exists through the unique identification.

1. The redis cache is used, and the unique token saves redis, and the token is deleted after each consumption

2. The database judges whether the progressive record exists or not. If it exists, it will be updated; if not, it will be inserted