1、 Best effort notification
TCC segment submission is applicable to business scenarios with high consistency and real-time requirements in the distributed architecture. In the actual business, there are also businesses with low real-time performance, such as common SMS notification, client message, operation system update and other businesses. At this time, in order to reduce the complexity and pressure of the core process, Best effort notification can be used to realize the management of flexible transactions.
For example, in a common third-party payment business, a message notification will be generated after the local business and the payment end business are processed. The basic process is as follows:
- After the local business preprocessing is completed;
- Request third-party payment services;
- The payment operation is successful and a message is sent to the account;
- Payment service callback local business;
- Local business generates system notification messages;
There are some basic features in the message scenario of the above process. After the core business processing is completed, the message notification is sent to allow failure. Within the specified time period or after the specified number of retries, the message loss is allowed, that is, the message is unreliable.
In the actual payment system, when daily reconciliation verification is started, the daily flow of the current day will be verified. If it is found that there are unfinished processes in the payment flow, there will be status compensation, and subsequent processing can be continued. This method is very common in reconciliation.
2、 Reliable information
Distributed transactions are based on the final consistency of reliable messages. Since they are reliable messages, MQ must support transaction management to ensure business consistency.
1. Rocketmq transaction message
In version 4.3, rocketmq began to support distributed transaction messages, adopted the idea of 2pc to submit transaction messages, and added a compensation logic to deal with two-stage timeout or failure messages, as shown in the following figure:
The figure above illustrates the general scheme of transaction messages, which is divided into two processes: sending and submitting normal transaction messages and compensating transaction messages.
1.1 sending and submission
(1) Send a message (half message, i.e. sent but not consumed);
(2) Writing result of server response message;
(3) Execute the local transaction according to the sending result. If the write fails, the half message is invisible to the business and the local logic will not execute;
(4) Execute commit or rollback according to the local transaction status (the commit operation generates a message index, and the message is visible to the consumer)
1.1 compensation process
(1) For transaction messages without commit / rollback (messages in pending status), initiate a “backcheck” from the server;
(2) The producer receives the callback message and checks the status of the local transaction corresponding to the callback message;
(3) Re commit or rollback according to the local transaction status;
The compensation phase is used to solve the timeout or failure of the message commit or rollback.
1.3 design principle
In the main flow of rocketmq transaction messages, how the messages in the first stage are invisible to users. Compared with ordinary messages, the biggest feature of transaction messages is that the messages sent in one stage are invisible to users. So, how to write a message but not visible to the user? The method of rocketmq transaction message is: if the message is a half message, the subject of the original message and the message consumption queue will be backed up, and then the subject will be changed to RMQ_ SYS_ TRANS_ HALF_ TOPIC。 Since the consumer group does not subscribe to this topic, the consumer cannot consume half type messages. Then rocketmq will start a scheduled task from topic to RMQ_ SYS_ TRANS_ HALF_ Pull messages in topic for consumption, obtain a service provider to send a request to check back the transaction status according to the producer group, and decide whether to commit or rollback messages according to the transaction status.
2. Final consistency
Based on the above characteristics of rocketmq transaction message reliability, that is, the final consistency of transactions under a certain type of business can be realized. Message sending consistency means that the business action generating the message is consistent with the message sending, that is, if the business operation is successful, the asynchronous message generated by the business operation must be sent, otherwise the business fails to roll back and the message will be discarded.
The process is basically as follows:
- Send a half transaction message, which cannot be consumed;
- Logical processing of local business code is completed;
- Send a confirmation message to identify that the message can be consumed;
- If the message producer is abnormal, cancel the overall action;
This process is mainly aimed at the message producer. In the actual development, the message consumer is also difficult to handle. To ensure the final consistency, it is bound to face a problem. The consumer is abnormal and the messages are constantly retried. There may be some business processing successes and some business processing failures. At this time, it is necessary to solve the idempotency problem of the service interface.
3、 Idempotent interface
1. Introduction to idempotent
The characteristic of an idempotent operation in programming is that the impact of any multiple execution is the same as that of one execution. In other words, one and multiple requests for a resource will have the same impact.
In complex asynchronous processes, special attention should be paid to failure retry. In general, in the payment process, every time the interface is requested, each step of data update will be preceded by a one-step status query process to judge whether the next step of data update should be executed.
2. Idempotent interface
In the system service interface request, any explicit interface response, such as failure or success, is easy to handle, but for example, in the payment scenario, if the request times out, how to judge the result status of the service: client request timeout, local service timeout, request payment timeout, payment callback timeout, client response timeout, etc, Or the MQ based constant retry mechanism, in some business abnormal states, the message will always be retried if it does not return success.
This requires the design of process state management. Especially under the message retry mechanism, heavy transaction control is rarely used for the retried business interface again. After some businesses are executed, only one state needs to be judged, and the next message retry can be skipped. Only unprocessed businesses need to be compensated. Under the retry mechanism, Before all the services are successfully executed, the message will be retried until it is finally completed.
4、 Source code address
GitHub · address https://github.com/cicadasmile/data-manage-parent Gitee · address https://gitee.com/cicadasmile/data-manage-parent
Recommended reading: Architecture Design Series