Architecture design | based on message oriented middleware, illustrating flexible transaction consistency

Time:2021-9-23

Source code of this article:GitHub · click here || Gitee · point here

1、 Best effort notification

TCC segment submission is applicable to business scenarios with high consistency and real-time requirements in the distributed architecture. In the actual business, there are also businesses with low real-time performance, such as common SMS notification, client message, operation system update and other businesses. At this time, in order to reduce the complexity and pressure of the core process, Best effort notification can be used to realize the management of flexible transactions.

Architecture design | based on message oriented middleware, illustrating flexible transaction consistency

For example, in a common third-party payment business, a message notification will be generated after the local business and the payment end business are processed. The basic process is as follows:

  • After the local business preprocessing is completed;
  • Request third-party payment services;
  • The payment operation is successful and a message is sent to the account;
  • Payment service callback local business;
  • Local business generates system notification messages;

There are some basic features in the message scenario of the above process. After the core business processing is completed, the message notification is sent to allow failure. Within the specified time period or after the specified number of retries, the message loss is allowed, that is, the message is unreliable.

In the actual payment system, when daily reconciliation verification is started, the daily flow of the current day will be verified. If it is found that there are unfinished processes in the payment flow, there will be status compensation, and subsequent processing can be continued. This method is very common in reconciliation.

2、 Reliable information

Distributed transactions are based on the final consistency of reliable messages. Since they are reliable messages, MQ must support transaction management to ensure business consistency.

1. Rocketmq transaction message

In version 4.3, rocketmq began to support distributed transaction messages, adopted the idea of 2pc to submit transaction messages, and added a compensation logic to deal with two-stage timeout or failure messages, as shown in the following figure:

Architecture design | based on message oriented middleware, illustrating flexible transaction consistency

The figure above illustrates the general scheme of transaction messages, which is divided into two processes: sending and submitting normal transaction messages and compensating transaction messages.

1.1 sending and submission

(1) Send a message (half message, i.e. sent but not consumed);

(2) Writing result of server response message;

(3) Execute the local transaction according to the sending result. If the write fails, the half message is invisible to the business and the local logic will not execute;

(4) Execute commit or rollback according to the local transaction status (the commit operation generates a message index, and the message is visible to the consumer)

1.1 compensation process

(1) For transaction messages without commit / rollback (messages in pending status), initiate a “backcheck” from the server;

(2) The producer receives the callback message and checks the status of the local transaction corresponding to the callback message;

(3) Re commit or rollback according to the local transaction status;

The compensation phase is used to solve the timeout or failure of the message commit or rollback.

1.3 design principle

In the main flow of rocketmq transaction messages, how the messages in the first stage are invisible to users. Compared with ordinary messages, the biggest feature of transaction messages is that the messages sent in one stage are invisible to users. So, how to write a message but not visible to the user? The method of rocketmq transaction message is: if the message is a half message, the subject of the original message and the message consumption queue will be backed up, and then the subject will be changed to RMQ_ SYS_ TRANS_ HALF_ TOPIC。 Since the consumer group does not subscribe to this topic, the consumer cannot consume half type messages. Then rocketmq will start a scheduled task from topic to RMQ_ SYS_ TRANS_ HALF_ Pull messages in topic for consumption, obtain a service provider to send a request to check back the transaction status according to the producer group, and decide whether to commit or rollback messages according to the transaction status.

2. Final consistency

Based on the above characteristics of rocketmq transaction message reliability, that is, the final consistency of transactions under a certain type of business can be realized. Message sending consistency means that the business action generating the message is consistent with the message sending, that is, if the business operation is successful, the asynchronous message generated by the business operation must be sent, otherwise the business fails to roll back and the message will be discarded.

The process is basically as follows:

Architecture design | based on message oriented middleware, illustrating flexible transaction consistency

  • Send a half transaction message, which cannot be consumed;
  • Logical processing of local business code is completed;
  • Send a confirmation message to identify that the message can be consumed;
  • If the message producer is abnormal, cancel the overall action;

This process is mainly aimed at the message producer. In the actual development, the message consumer is also difficult to handle. To ensure the final consistency, it is bound to face a problem. The consumer is abnormal and the messages are constantly retried. There may be some business processing successes and some business processing failures. At this time, it is necessary to solve the idempotency problem of the service interface.

3、 Idempotent interface

1. Introduction to idempotent

The characteristic of an idempotent operation in programming is that the impact of any multiple execution is the same as that of one execution. In other words, one and multiple requests for a resource will have the same impact.

Architecture design | based on message oriented middleware, illustrating flexible transaction consistency

In complex asynchronous processes, special attention should be paid to failure retry. In general, in the payment process, every time the interface is requested, each step of data update will be preceded by a one-step status query process to judge whether the next step of data update should be executed.

2. Idempotent interface

In the system service interface request, any explicit interface response, such as failure or success, is easy to handle, but for example, in the payment scenario, if the request times out, how to judge the result status of the service: client request timeout, local service timeout, request payment timeout, payment callback timeout, client response timeout, etc, Or the MQ based constant retry mechanism, in some business abnormal states, the message will always be retried if it does not return success.

This requires the design of process state management. Especially under the message retry mechanism, heavy transaction control is rarely used for the retried business interface again. After some businesses are executed, only one state needs to be judged, and the next message retry can be skipped. Only unprocessed businesses need to be compensated. Under the retry mechanism, Before all the services are successfully executed, the message will be retried until it is finally completed.

4、 Source code address

GitHub · address
https://github.com/cicadasmile/data-manage-parent
Gitee · address
https://gitee.com/cicadasmile/data-manage-parent

Architecture design | based on message oriented middleware, illustrating flexible transaction consistency

Recommended reading: Architecture Design Series

Serial number title
01 Architecture design: single service, cluster, distributed, basic differences and connections
02 Architecture design: Global ID generation strategy in distributed business system
03 Architecture design: distributed system scheduling, zookeeper cluster management
04 Architecture design: interface idempotency principle, anti duplicate submission token management
05 Architecture design: cache management mode, monitoring and memory recycling strategy
06 Architecture design: asynchronous processing flow, detailed explanation of various implementation modes
07 Architecture design: peak shaving of high concurrent traffic and locking mechanism of shared resources
08 Architecture design: distributed service, detailed explanation of library table splitting mode
09 Architecture design: distributed transaction ① concept introduction and basic theory
10 Architecture design: Based on the e-commerce transaction process, illustrate the piecewise submission of TCC transactions

Recommended Today

Supervisor

Supervisor [note] Supervisor – H view supervisor command help Supervisorctl – H view supervisorctl command help Supervisorctl help view the action command of supervisorctl Supervisorctl help any action to view the use of this action 1. Introduction Supervisor is a process control system. Generally speaking, it can monitor your process. If the process exits abnormally, […]