SpringBoot multi-data source transaction solution

Time:2022-11-23

background

A previous article provided an integration solution for springboot multi-data source dynamic registration switching. In the subsequent use process, it was found that there were various bugs in transaction control, and it was decided to analyze and solve this problem.

Recap

The multi-data source switching process structure diagram is shown below, including several components

  • Customized data source configuration processing, dynamically registered to the system through the DruidDataSource object
  • Custom data source identification annotations and aspects
  • Context thread variable holder when data source is switched
  • Customize AbstractRoutingDataSource to realize data source routing switching

SpringBoot multi-data source transaction solution

problem analysis

After the Controller adds the @Transitional annotation, the data source switch will be invalid, and only the main library will be operated. After querying the data, the solution is to set the Order of the aspect to -1 so that the execution order is before the transaction control interception. After modification, it is confirmed to be valid, but Subsequent switching to other libraries or operations on the main library is invalid, and the connection obtained is always the connection corresponding to the library after the first switch

After analyzing the code, it is found that AbstractRoutingDataSource is only responsible for providing the level of getConnection, but the subsequent operations on the connection cannot be tracked. The project framework mybatis and jdbcTemplate are mixed. The subsequent operations at the spring level are the logical level operations of the transaction/data source/connection. Similarly, the jdbcTemplate code is relatively simple, so use this as an entry point for further analysis

Through breakpoint debugging, you will find that the execution of the sql statement will eventually fall into the execute method. The method starts with obtaining the connection through DataSourceUtils.getConnection. This is where we need to track. Click it and find that it jumps to the doGetConnection method, which is what we need The specific logic of the analysis

SpringBoot multi-data source transaction solution

SpringBoot multi-data source transaction solution

The ConnectionHolder obtained in the first line is the thread holding object corresponding to the current transaction, because we know that the essence of the transaction is that the SQL inside the method corresponds to the same database connection when executed. For different nested business methods, the only thing that is the same is The current thread ID is consistent, so we can achieve transaction control by binding the connection to the thread

SpringBoot multi-data source transaction solution

Click on the getResource method and find that the dataSource is used as a key to retrieve the corresponding contextHolder from a Map collection

SpringBoot multi-data source transaction solution

Here we seem to find something. Before instantiating jdbcTemplatechu and setting the data source, we directly assign a custom DynamicDataSource, so every time we get the connection in the thing, the DynamicDataSource object is used as the key, so it will be the same every time! !

    @Bean
    public JdbcTemplate jdbcTemplate(){
        JdbcTemplate jdbcTemplate = null;
        try{
            jdbcTemplate = new JdbcTemplate(dynamicDataSource());
        }catch (Exception e){
            e.printStackTrace();
        }
        return jdbcTemplate;
    }

Later, I found relevant information for mybatis. The default implementation of transaction control is SpringManagedTransaction. After viewing the source code, I found the familiar DataSourceUtils.getConnection, which proves that our analysis direction is correct.

SpringBoot multi-data source transaction solution

solution

jdbcTemplate

The custom operation class inherits jdbcTemplate and rewrites getDataSource, and assigns the corresponding key of the DataSource we obtained to the data source object that actually switches the library.

public class DynamicJdbcTemplate extends JdbcTemplate {
    @Override
    public DataSource getDataSource() {
        DynamicDataSource router =  (DynamicDataSource) super.getDataSource();
        DataSource acuallyDataSource = router.getAcuallyDataSource();
        return acuallyDataSource;
    }

    public DynamicJdbcTemplate(DataSource dataSource) {
        super(dataSource);
    }
}
    public DataSource getAcuallyDataSource() {
        Object lookupKey = determineCurrentLookupKey();
        if (null == lookupKey) {
            return this;
        }
        DataSource determineTargetDataSource = this.determineTargetDataSource();
        return determineTargetDataSource == null ? this : determineTargetDataSource;
    }

mybatis

Customize the transaction operation class, implement the Transaction interface, and replace the TransitionFactory. The implementation here is slightly different from the online solution. On the Internet, three variables are defined, datasource (dynamic data source object)/connection (main connection)/connections (slave library connection), but the framework requires mybatis and jdbctemplate to be unified, mybatis is controlled from the connection level, and jdbctemplate is controlled from the datasource level, so all use key-value pair storage

public class DynamicTransaction implements Transaction {
    private final DynamicDataSource dynamicDataSource;
    private ConcurrentHashMap<String, DataSource> dataSources;
    private ConcurrentHashMap<String, Connection> connections;
    private ConcurrentHashMap<String, Boolean> autoCommits;
    private ConcurrentHashMap<String, Boolean> isConnectionTransactionals;

    public DynamicTransaction(DataSource dataSource) {
        this.dynamicDataSource = (DynamicDataSource) dataSource;
        dataSources = new ConcurrentHashMap<>();
        connections = new ConcurrentHashMap<>();
        autoCommits = new ConcurrentHashMap<>();
        isConnectionTransactionals = new ConcurrentHashMap<>();
    }

    public Connection getConnection() throws SQLException {
        String dataBaseID = DBContextHolder.getDataSource();
        if (!dataSources.containsKey(dataBaseID)) {
            DataSource dataSource = dynamicDataSource.getAcuallyDataSource();
            dataSources.put(dataBaseID, dataSource);
        }
        if (!connections.containsKey(dataBaseID)) {
            Connection connection = DataSourceUtils.getConnection(dataSources.get(dataBaseID));
            connections.put(dataBaseID, connection);
        }
        if (!autoCommits.containsKey(dataBaseID)) {
            boolean autoCommit = connections.get(dataBaseID).getAutoCommit();
            autoCommits.put(dataBaseID, autoCommit);
        }
        if (!isConnectionTransactionals.containsKey(dataBaseID)) {
            boolean isConnectionTransactional = DataSourceUtils.isConnectionTransactional(connections.get(dataBaseID), dataSources.get(dataBaseID));
            isConnectionTransactionals.put(dataBaseID, isConnectionTransactional);
        }
        return connections.get(dataBaseID);
    }


    public void commit() throws SQLException {
        for (String dataBaseID : connections.keySet()) {
            Connection connection = connections.get(dataBaseID);
            boolean isConnectionTransactional = isConnectionTransactionals.get(dataBaseID);
            boolean autoCommit = autoCommits.get(dataBaseID);
            if (connection != null && !isConnectionTransactional && !autoCommit) {
                connection.commit();
            }
        }
    }

    public void rollback() throws SQLException {
        for (String dataBaseID : connections.keySet()) {
            Connection connection = connections.get(dataBaseID);
            boolean isConnectionTransactional = isConnectionTransactionals.get(dataBaseID);
            boolean autoCommit = autoCommits.get(dataBaseID);
            if (connection != null && !isConnectionTransactional && !autoCommit) {
                connection.rollback();
            }
        }
    }

    public void close() {
        for (String dataBaseID : connections.keySet()) {
            Connection connection = connections.get(dataBaseID);
            DataSource dataSource = dataSources.get(dataBaseID);
            DataSourceUtils.releaseConnection(connection, dataSource);
        }
    }

    public Integer getTimeout() {
        return null;
    }
}
public class DynamicTransactionFactory extends SpringManagedTransactionFactory {
    @Override
    public Transaction newTransaction(DataSource dataSource, TransactionIsolationLevel level, boolean autoCommit) {
        return new DynamicTransaction(dataSource);
    }
}
@Bean
    public SqlSessionFactory sqlSessionFactory() throws Exception {
        //SpringBootExecutableJarVFS.addImplClass(SpringBootVFS.class);
        final PackagesSqlSessionFactoryBean sessionFactory = new PackagesSqlSessionFactoryBean();
        sessionFactory.setDataSource(dynamicDataSource());
        sessionFactory.setTransactionFactory(new DynamicTransactionFactory());
        sessionFactory.setMapperLocations(new PathMatchingResourcePatternResolver()
                .getResources("classpath*:mybatis/**/*Mapper.xml"));
        //Turn off camel case conversion to prevent underlined fields from being mapped
        sessionFactory.getObject().getConfiguration().setMapUnderscoreToCamelCase(false);
        return sessionFactory.getObject();
    }

transaction manager

The problem of dynamic library switching in the transaction is solved, but only for the main library transaction. If the operation of the slave library also needs the characteristics of the transaction, how to operate it? Here, it is necessary to manually register a transaction management for each data source when registering the data source device

The main library is fixed, you can directly declare masterTransitionManage in the configuration bean and set it as the default

    @Bean("masterTransactionManager")
    @Primary
    public DataSourceTransactionManager MasterTransactionManager() {
        return new DataSourceTransactionManager(masterDataSource());
    }

From the transaction manager of the library, we can get the dataSource initialization object, and then register the singleton object with the Spring container

public static void registerSingletonBean(String beanName, Object singletonObject) {
        // Convert applicationContext to ConfigurableApplicationContext
        ConfigurableApplicationContext configurableApplicationContext = (ConfigurableApplicationContext) context;
        //Get BeanFactory
        DefaultListableBeanFactory defaultListableBeanFactory = (DefaultListableBeanFactory) configurableApplicationContext.getAutowireCapableBeanFactory();
        if(configurableApplicationContext.containsBean(beanName)) {
            defaultListableBeanFactory.destroySingleton(beanName);
        }
        //Dynamic registration bean.
        defaultListableBeanFactory.registerSingleton(beanName, singletonObject);

    }
 SpringBootBeanUtil.registerSingletonBean(key + "TransactionManager", new DataSourceTransactionManager(druidDataSource));

When using it, just specify the transitionFactory name to the @Transitional annotation

Summarize

It took three days to solve this problem, and checked a lot of information and solutions, many of which are only reference or specific, so it is still necessary to grasp the core of the problem and track some source codes, for example, a clear understanding is required in this article Only by going to the relationship between Transition-Connection-LocalThread can we find the right direction for troubleshooting

Later, the global transaction that integrated the XA two-stage submission based on JMS (atomikos) was implemented. Using DruidXADataSrouce, there was a leak in the thread pool interaction between druid and atomikos. I gave up and avoided a pit for my friends.