Architecture design | based on message middleware, flexible transaction consistency is illustrated

Time:2021-6-9

Source code:GitHub. Click here || Gitee. Click here

1、 Best effort notification

TCC segmented submission is suitable for business scenarios with high consistency and real-time requirements in distributed architecture. In actual business, there are also services with low real-time performance, such as common SMS notification, client message, operation system update and other services. In order to reduce the complexity and pressure of the core process at this time, Flexible transaction management can be realized by best effort notification.

Architecture design | based on message middleware, flexible transaction consistency is illustrated

For example, in a common third-party payment business, a message notification is generated after the local business and the payment end business are processed. The basic process is as follows:

  • After the completion of local business preprocessing;
  • Request for third party payment services;
  • The payment operation is successful, and the message is sent to the account;
  • Payment service callback local business;
  • Local service generation system notification message;

There are some basic characteristics in the message scenario of the above process. After the core business processing is completed, a message notification is sent to allow failure. In a specified period of time or after a specified number of retries, the message loss is allowed, that is, the unreliability of the message.

In the actual payment system, when the daily reconciliation verification is started, the daily flow of the day will be verified. If there is an unfinished process in the payment flow, there will be a state to make up for it, and the subsequent processing can continue. This method is very common in reconciliation.

2、 Reliable information

Since it is a reliable message, MQ must support transaction management to ensure the consistency of business.

1. Rocketmq transaction message

Rocketmq started to support distributed transaction messages in version 4.3. The idea of 2pc is adopted to submit transaction messages. At the same time, a compensation logic is added to deal with the two-stage timeout or failure messages, as shown in the figure below:

Architecture design | based on message middleware, flexible transaction consistency is illustrated

The figure above shows the general scheme of transaction message, which is divided into two processes: normal transaction message sending and submitting, transaction message compensation process.

1.1 sending and submitting

(1) Send a message (half message, that is, sent but not consumed);

(2) The server responds to the message and writes the result;

(3) If the write fails, the half message will not be visible to the service and the local logic will not execute;

(4) Execute commit or rollback according to the local transaction state (the commit operation generates the message index, and the message is visible to consumers)

1.1 compensation process

(1) For transaction messages without commit / rollback (messages with pending status), initiate a “check back” from the server;

(2) Producer receives the backcheck message and checks the status of the local transaction corresponding to the backcheck message;

(3) According to the local transaction status, recommit or rollback;

The compensation phase is used to solve the timeout or failure of message commit or rollback.

1.3 design principle

In the main process of rocketmq transaction message, how the message of one stage is invisible to the user. Among them, the biggest feature of transaction message is that the message sent in one stage is invisible to users. So, how to write a message but not visible to the user? The method of rocketmq transaction message is: if the message is half message, the subject and message consumption queue of the original message will be backed up, and then the subject will be changed to RMQ_ SYS_ TRANS_ HALF_ TOPIC。 Because the consumer group does not subscribe to the topic, the consumer can not consume half type messages. Then rocketmq will start a timing task from topic to RMQ_ SYS_ TRANS_ HALF_ In topic, pull messages for consumption. According to the producer group, get a service provider to send a request to check the transaction status. According to the transaction status, decide whether to submit or roll back the message.

2. Final consistency

Based on the above characteristics of rocketmq transaction message reliability, that is, it can achieve the final consistency of transactions under a certain type of business. Message sending consistency means that the business action that generates the message is consistent with the message sending. That is to say, if the business operation is successful, the asynchronous message generated by the business operation must be sent out. Otherwise, the business will fail and the message will be discarded.

The process is as follows:

Architecture design | based on message middleware, flexible transaction consistency is illustrated

  • Sending half transaction message can not be consumed;
  • The logic processing of local business code is completed;
  • Sending a confirmation message to indicate that the message can be consumed;
  • If the message producer is abnormal, cancel the whole action;

This process is mainly aimed at the message producer. In the actual development, the message consumer is also very difficult to deal with. To ensure the final consistency, it is bound to face a problem. The consumer is abnormal, and the message is constantly retrying. There may be some business processing success and some business processing failure. At this time, it is necessary to solve the idempotent problem of service interface.

3、 Idempotent interface

1. Introduction to idempotent

The characteristic of an idempotent operation in programming is that the influence of any multiple execution is the same as that of one execution. That is to say, one or more requests for a resource will have the same effect.

Architecture design | based on message middleware, flexible transaction consistency is illustrated

In the complex asynchronous process, we pay special attention to the problem of failure and retrial. Generally, in the payment process, every time the interface is requested and every step of data update operation is preceded by a step of status query process, which is used to determine whether the next step of data update should be executed.

2. Idempotent interface

In the system service interface request, any explicit interface response, such as failure or success, is easy to handle in the business process, but for example, in the payment scenario, if the request times out, how to judge the result status of the service: client request time-out, local service time-out, request payment time-out, payment callback time-out, client response time-out, etc, Or based on the continuous retrial mechanism of MQ, in some abnormal business States, if the message does not return success, the message will always be retried.

This requires the design of process based state management. Especially in the message retrial mechanism, it is rare to use heavy transaction control on the retrial business interface. When some businesses are executed, it is only necessary to judge a state and skip the next message retrial. It is only necessary to compensate the unprocessed business, Before all the services are successfully executed, the message will be retried until all the services are completed.

4、 Source code address

GitHub · address
https://github.com/cicadasmile/data-manage-parent
Gitee · address
https://gitee.com/cicadasmile/data-manage-parent

Architecture design | based on message middleware, flexible transaction consistency is illustrated

Recommended reading: Architecture Design Series

Serial number title
01 Architecture design: single service, cluster, distributed, basic difference and connection
02 Architecture design: Global ID generation strategy in distributed business system
03 Architecture design: distributed system scheduling, zookeeper cluster management
04 Architecture design: the principle of interface idempotency, prevent duplicate submission token management
05 Architecture design: cache management mode, monitoring and memory recovery strategy
06 Architecture design: asynchronous processing flow, detailed explanation of various implementation modes
07 Architecture design: high concurrent traffic peak clipping, shared resource locking mechanism
08 Architecture design: distributed services, library table splitting mode
09 Architecture design: concept introduction and basic theory of distributed transaction
10 Architecture design: Based on e-commerce transaction process, diagram TCC transaction subsection submission