Implementation of distributed transaction in sharding sphere


Trainer Introduction

Zhao Jun

Jingdong Finance

Senior Java Developer

  • Years of Internet development experience, love open source technology, have a strong interest in distributed storage. Familiar with elasticsearch, HBase, presto, storm and other offline and real-time data processing
  • At present, I am mainly responsible for the development of distributed transactions in the sharding sphere team

Application scenarios of distributed transaction


It all starts with acid. Acid is the four characteristics of local firms

  • Atomicity: atomicity

    Transactions are executed as a whole, either all or none.

  • Consistency: consistency

    Transactions should ensure that data changes from one consistent state to another.

  • Isolation: isolation

    When multiple transactions are executed simultaneously, the execution of one transaction should not affect the execution of other transactions.

  • Durability: persistence

    Committed transaction modification data is persisted.

Local transaction of relational database perfectly provides native support for acid. But in the distributed scenario, it becomes the shackles of system performance. How to make the database meet the characteristics of acid or find the corresponding alternative in the distributed scenario is the topic of this paper.

Cap and base theory

For Internet applications, with the surge of access and data, the traditional single architecture model will not be able to meet the rapid development of business. At this time, developers need to split a single application into multiple independent small applications, and split a single database into multiple libraries and tables according to the fragmentation rules.

After data splitting, how to ensure the acid characteristics of local transactions among multiple database nodes becomes a technical problem, and the classical theories of cap and base are derived.

Cap theory points out that for distributed applications, it is impossible to satisfy C (consistency), a (availability) and P (partition fault tolerance) at the same time. Because network partition is the basic element of distributed applications, developers need to balance C and a.

Due to the mutual exclusion of C and a, the trade-off result is base theory.

For most distributed applications, as long as the data reaches the final consistency within the specified time. We can call the traditional acid rigid transaction and the final consistent transaction satisfying the base theory flexible transaction.

Blindly pursuing strong consistency is not the best solution. For distributed applications, the combination of rigid and flexible is a more reasonable design scheme, that is, strong consistent transactions are used in local services, and final consistency is used in cross system calls. How to balance the performance and consistency of the system is a great test for architects and developers.

Implementation of distributed transaction in sharding sphere

Industry approach

As for the implementation of distributed transaction, the industry mainly adopts the strong consistency specification of XA protocol and the final consistency specification of flexible transaction.


Xa is the communication interface between TM (transaction manager) and RM (Resource Manager) defined in X / open CAE specification (distributed transaction processing) model.

In Java javax.transaction.xa . xaresource defines XA interface, which depends on the concrete implementation of JDBC Driver by database manufacturer.

The implementation of mysql-connector-java-5.1.30 can be referred to:


In XA specification, the database acts as RM role, and the application needs to act as TM role, that is to generate global txid, call xaresource interface, and coordinate multiple local transactions into global unified distributed transactions.

One stage commit: weak XA

Implementation of distributed transaction in sharding sphere

Weak XA can reduce the scope of resource locking and improve concurrency performance by removing the Prepare phase of Xa. The typical implementation is to traverse all the database connections in a business thread, and do commit or rollback in turn. Compared with local transaction, weak XA has low performance loss, but in the process of transaction commit, if there are unexpected exceptions such as network failure and database downtime, it will cause data inconsistency and unable to roll back. Transactions based on weak XA do not require additional implementation cost, so sharding sphere is supported by default.

Phase 2 submission: 2pc

Implementation of distributed transaction in sharding sphere

Two phase commit is the standard implementation of Xa. It divides the commit of distributed transaction into two phases: prepare and commit / rollback.

After XA global transaction is started, all sub transactions will lock resources according to the local default isolation level, and record undo and redo logs. Then TM initiates prepare vote to ask whether all sub transactions can be submitted. When the feedback result of all sub transactions is “yes”, TM initiates commit. If the feedback result of any sub transaction is “no”, TM initiates rol If the feedback result in the Prepare phase is yes and there is an exception such as downtime in the commit process, after the node service is restarted, the commit compensation can be performed again according to XA recover to ensure the consistency of the data.

In 2pc model, it is necessary to wait for the feedback of all participating sub transactions in the prepare stage, so it may cause too long locking time of database resources, which is not suitable for business scenarios with high concurrency and long life cycle of sub transactions.

Sharding sphere supports strong consistency transaction solution based on Xa. It can inject different third-party components as transaction manager through SPI to implement XA protocol, such as atomikos and Narayana.

Flexible transaction

Flexible transaction is the compromise and compensation of XA protocol. It can reduce the locking time of database resources by reducing the requirement of strong consistency. There are many kinds of flexible transactions, which can be balanced by different strategies.

One stage submission + compensation: best effort delivery (bed)

Best effort delivery is a compensation strategy for weak Xa. It uses transaction table to record all transaction operation SQL. If the sub transaction is submitted successfully, the transaction log will be deleted; if the execution fails, it will try to submit again according to the configured number of retries, that is, try its best to submit to ensure the consistency of data as much as possible. According to different business scenarios, it can balance C and a, and adopt synchronous or asynchronous retries.

The advantages of this strategy are no lock resource time and low performance loss. The disadvantage is that it can’t be rolled back after several failed attempts. It is only applicable to the business scenario where the transaction will eventually succeed. Therefore, bed is a compromise of transaction rollback function in exchange for performance improvement.

Implementation of distributed transaction in sharding sphere

TCC: Try-Confirm-Cancel

In TCC model, the granularity of lock is completely handed over to business processing. It requires every sub business to implement try confirm / cancel interface.

  • Try:

    Try to execute the business;

    Complete all business checks (consistency);

    Reserve necessary business resources (quasi isolation);

  • Confirm:

    Confirm the execution of business;

    Truly carry out business without any business inspection;

    Only the business resources reserved in try phase are used;

    The confirm operation is idempotent;

  • Cancel:

    Cancel the execution of business;

    Release the business resources reserved in try phase;

    The cancel operation satisfies idempotency.

These three phases are executed in the form of local transactions. Unlike XA prepare, TCC does not need to suspend all the resources during the voting period of Xa, so the throughput is greatly improved.

The following is a detailed analysis of the transformation of the business by taking the remittance of 100 yuan from account a to account B under the TCC mode as an example

Implementation of distributed transaction in sharding sphere

Remittance service and collection service need to implement try confirm cancel interface respectively, and inject it into TCC transaction manager in business initialization phase.

Remittance service

  • Try:

    Check the validity of account a, that is, check whether the status of account a is “transferring” or “frozen”;

    Check whether the balance of account a is sufficient;

    Deduct 100 yuan from account a and set the status as “transferring”;

    Reserve deduction resources, and save the event of transferring 100 yuan from a to B into the message or log;

  • Confirm:

    Do not do any operation;

  • Cancel:

    Account a increased by 100 yuan;

    Release the deduction resource from the log or message.

Collection service

  • Try:

    Check whether account B is valid;

  • Confirm

    Read logs or messages, B account increased by 100 yuan;

    Release deduction resources from logs or messages;

  • Cancel:

    Do nothing.

From this, we can see that the TCC model has strong invasion to business, and the transformation is difficult.

Message driven

Implementation of distributed transaction in sharding sphere

Message consistency scheme is to ensure the consistency of upstream and downstream application data operations through message middleware. The basic idea is to put the local operation and sending message in one transaction. The downstream application subscribes to the message system and executes the corresponding operation after receiving the message. In essence, it relies on the retrial mechanism of messages to achieve the final consistency. The disadvantages of message driven are: high coupling, MQ needs to be introduced into the business system, resulting in increased system complexity.


Saga originated from the paper sagas published by Hector & Kenneth in 1987.

Reference address:…

How saga works

Saga model divides a distributed transaction into multiple local transactions, and each local transaction has corresponding execution module and compensation module (confirm and cancel in TCC). When any local transaction in Saga transaction fails, the previous transaction can be recovered by calling the relevant compensation method to achieve the final consistency of the transaction.

When each saga sub transaction T1, T2 , TN have corresponding compensation definitions C1, C2 Then saga system can guarantee that:

  • Sub transaction sequence T1, T2 And TN can be completed (in the best case);
  • Or sequence… T1, T2 , Tj, Cj, … , C2, C1, 0 < J < n, can be completed.

Because there is no prepare stage in Saga model, the isolation between transactions cannot be guaranteed. When multiple saga transactions operate on the same resource, there will be problems such as update loss and dirty data reading. At this time, concurrency control is needed in the business layer, such as:

  • Lock at the application level;
  • At the application level, resources are frozen in advance.

Saga recovery mode

Saga supports forward and backward recovery:

  • Backward recovery: compensates all completed transactions, if any child transaction fails;
  • Forward recovery: retry failed transactions, assuming that each child transaction will eventually succeed.

Obviously, there is no need to provide compensation transaction for forward recovery. If the sub transaction (eventually) always succeeds in your business, or the compensation transaction is difficult to define or impossible, forward recovery is more in line with your needs. In theory, compensation transactions never fail. However, in a distributed world, servers may be down, networks may fail, and even data centers may lose power. In this case, it is necessary to provide a fallback mechanism after failure recovery, such as manual intervention.

In general, both TCC and MQ process distributed transactions in the scope of service, while Xa, bed and saga process distributed transactions in the scope of database. We prefer the latter, which has less invasion and low transformation cost for business.

Sharding sphere support for distributed transactions

Sharding sphere is an ecosystem of open source distributed database middleware solutions. It consists of three independent products: sharding JDBC, sharding proxy and sharding sidecar. They all provide standardized data fragmentation, read-write separation, flexible transaction and data governance functions, which can be applied to various application scenarios such as Java isomorphism, heterogeneous languages, containers, cloud native, etc.

Project address:…

Sharding sphere supports both Xa and flexible transactions. It allows each access to the database and can freely choose the transaction type. Distributed transaction is completely transparent to business operation, which greatly reduces the cost of introducing distributed transaction.

Transaction model

Implementation of distributed transaction in sharding sphere

Sharding sphere transaction manager integrates Xa and flexible transaction model

  • For XA transactions, SPI is used to keep the weak Xa, atomikos and Narayana mutually exclusive;
  • For flexible transactions, according to the type of transactions in each connection, you can select an independent transaction manager to process. Each transaction manager will implement the standard shardingtransaction interface and perform the corresponding begin, commit and rollback operations when the transaction event arrives.

The following is a detailed description of how sharding sphere decouples transactions from sharding main processes in an event driven manner

Implementation of distributed transaction in sharding sphere

As can be seen from the figure, when sharding core calls the execution engine, it will generate events for distribution according to the type of SQL. After receiving the events that meet the requirements, the transaction monitoring thread calls the corresponding transaction processor for processing.

Sharding proxy transaction implementation

Sharding proxy is a database intermediate agent layer based on netty, which implements the standard MySQL protocol and can be regarded as a database that implements data fragmentation. Sharding proxy has implemented XA transaction based on atomikos. In order to ensure that all sub transactions are in the same thread, the thread model of the whole proxy is adjusted as follows:

Implementation of distributed transaction in sharding sphere

When the transaction is started, the SQL command execution engine of proxy back end will adopt the mode of one channel and one thread, and the life cycle of this transaction thread is consistent with that of the channel. The specific process of transaction processing is completely decoupled from proxy, that is, proxy will publish transaction type events, and then sharding sphere TM will select the specific TM to process according to the incoming transaction messages.

Pressure test results show that the performance of XA transaction insertion and update is basically linear with the number of cross database, and the query performance is not affected. It is suggested that XA can be used when the concurrency is small and the number of databases involved in each transaction is less than 10.

Implementation of distributed transaction in sharding sphere

Principle analysis of atomikos transaction manager

Implementation of distributed transaction in sharding sphere

The transaction manager of atomikos can be embedded in the business process when the application calls TransactionManager.begin This XA transaction will be created and associated with the current thread. At the same time, atomikos also encapsulates the connection in the datasource. The proxy connection contains the status of the transaction related information, and intercepts the JDBC operation of the connection.

When creating statement, call XAResource.start When closing, call XAResource.end Let XA transaction be in idel committable state; when commit or rollback, call prepare and commit in turn for two-phase commit.

Saga transaction implementation of sharding sphere

Sharding sphere uses the saga transaction engine of service comb as its distributed transaction implementation through cooperation with Apache service comb.

Apache service comb is Huawei’s open source microservice framework, in which the microservice transaction processing framework is divided into centralized and distributed coordinator. In the future, Saga centralized Coordinator will be integrated in sharding sphere to support distributed transactions between different services (local) in the same thread.

Reference link:…

Service comb centralized transaction coordinator

Implementation of distributed transaction in sharding sphere

Centralized coordinator, including the content of Saga call request receiving, analysis, execution and result query. The task agent module needs to know the saga transaction call diagram in advance, and the execution module generates the call task according to the generated call diagram, and calls the related micro service interface. If there is an error in the execution of the service call, the relevant compensation method of the service will be called to roll back.

Saga execution module builds a call graph by analyzing the requested JSON data. Sharding sphere describes saga transaction through JSON, and calls sub transaction serially or in parallel. The relational call graph is decomposed into an execution task by the task running module in Saga implementation. The execution task is obtained by the task consumer and generates related calls (both serial and parallel calls are supported). Saga tasks will record the key events of corresponding saga transactions in Saga log according to the execution status, and query the execution status through event viewer.

Sharding sphere embedded saga transaction manager

Implementation of distributed transaction in sharding sphere

Saga provides distributed transaction governance capability in the form of jar package.

For sharding sphere, confirm and cancel processes represent the normal execution and reverse execution of SQL in sub transactions (sharding sphere will provide the ability to generate reverse SQL automatically in the future). When saga flexible transaction is enabled, the physical data source after routing will start the local auto commit transaction, and each confirm and cancel will be directly committed.

In sharding sphere, after triggering the SQL execution engine, Saga transaction event will be generated. At this time, the sharding sphere transaction listener will register the confirm and cancel of the current sub transaction to the saga transaction manager queue. After the business thread triggers commit and rollback, Saga transaction manager will judge whether to confirm or cancel the process according to the execution result of the sub transaction.

Future plans

In the future, sharding sphere will gradually improve the entire transaction framework according to the sharding sphere TM described in this paper

  • Weak XA transaction (published)
  • Xa transaction based on atomikos (recently released)
  • Xa transaction based on Narayana (under planning)
  • Bed flexible transaction (published)
  • Saga (under development)
  • TCC (Planning)

If the previous sharing is too lengthy, then thousands of words gather into a table, welcome to read.

Implementation of distributed transaction in sharding sphere

In the future, we will continue to optimize the current features and launch more new features such as flexible transactions and data governance. If you have any ideas, opinions and suggestions, you are welcome to leave a message and exchange them. You are also welcome to join sharding sphere’s open source project



Q1:Can XA based things be applied to microservice architecture?

A1:At present, we embed the transaction manager into the JVM process. For businesses with small concurrency and short transactions, XA can be used.

Q2:What is the basic order of development plan for each transaction framework?

A2:Based on the degree of difficulty, we put TCC at the end.

Q3:Do you support multilingualism? Like golang?

A3:Sharding proxy can be used for multi language.

Q4:This time it’s proxy to implement distributed transactions, right? I remember that sharding JDBC was implemented before.

A4:This is the transaction implementation of the whole SS, including sharding JDBC and proxy. At present, SJ is implemented with weak Xa and bed (best effort delivery), and saga and TCC will be added in the future.

Q5:If I only want to use the transaction module in SS, can I?

A5:SS is an event driven architecture, and the future transaction module is only responsible for transaction related processing.

Q6:Saga doesn’t support I in acid. How do we consider that?

A6:At present, isolation is not supported. In the future, we have plans to increase I. in fact, all flexible transactions do not support I. TCC adds try phase, which can be understood as quasi isolation. When saga is used, concurrency can be controlled at the business level to prevent dirty reading.

Q7:What do you mean, the current version 3 can’t use the transaction module alone?

A7:Now in version 3.0, the transaction module relies on sharding JDBC module. The transaction module needs to listen to events in sharding JDBC and proxy, and then perform transaction operations. If you want to use the transaction module alone, you need to publish it in your business according to the events defined in the core.

Live playback…