Springboot integrates multiple data sources of mybatisplus

Time:2022-6-13

It is believed that many small partners using the mybatisplus framework will encounter the configuration problem of multiple data sources, and the official website also gives recommendationsMultiple data sources (dynamic-datasource-spring-boot-starter)Component. Recently, the project is also using this component to realize multi data source switching. Therefore, I want to know how this component operates. After my own debugging, I will simply record the implementation of this component, so that I can have a reference when the component has problems or some places need to be developed in the future.

Simple data source switching

Database demo

This example uses the same MySQL service and different databases for debugging, as shown in the figure

Springboot integrates multiple data sources of mybatisplus

database01.jpg

SpringBoot demo

Add dependency

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>com.baomidou</groupId>
            <artifactId>mybatis-plus-boot-starter</artifactId>
            <version>3.4.0</version>
        </dependency>
        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>com.baomidou</groupId>
            <artifactId>dynamic-datasource-spring-boot-starter</artifactId>
            <version>3.4.0</version>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.20</version>
            <scope>provided</scope>
        </dependency>

Configuring YML files

spring:
  datasource:
    type: com.zaxxer.hikari.HikariDataSource
    hikari:
      minimum-idle: 5
      maximum-pool-size: 15
      idle-timeout: 30000
      max-lifetime: 1800000
      connection-timeout: 30000
      pool-name: OasisHikariCP
      connection-test-query: SELECT 1
    dynamic:
      Primary: db1 \\default primary data source
      datasource:
        Db1: \configuration master data source
          url: jdbc:mysql://ip:3306/demo_user?useSSL=true&requireSSL=false&serverTimezone=UTC&characterEncoding=UTF-8
          username: root
          password: root
          driver-class-name: com.mysql.cj.jdbc.Driver
        Db2: \configure other data sources
          url: jdbc:mysql://ip:3306/demo_class?useSSL=true&requireSSL=false&serverTimezone=UTC&characterEncoding=UTF-8
          username: root
          password: root
          driver-class-name: com.mysql.cj.jdbc.Driver

MybaitsPlus

Entity layer

UserEntity.class

/**
 *Description: entity in db1
 * date: 2021/7/13 13:38 <br>
 * author: Neal <br>
 * version: 1.0 <br>
 */
@TableName("user_t")
public class UserEntity {

    private long id;

    private String userName;

    private String userSex;

    public long getId() {
        return id;
    }

    public void setId(long id) {
        this.id = id;
    }

    public String getUserName() {
        return userName;
    }

    public void setUserName(String userName) {
        this.userName = userName;
    }

    public String getUserSex() {
        return userSex;
    }

    public void setUserSex(String userSex) {
        this.userSex = userSex;
    }
}

ClassEntity.class

/**
 *Description: entities in DB2
 * date: 2021/7/13 13:40 <br>
 * author: Neal <br>
 * version: 1.0 <br>
 */
@TableName("class_t")
public class ClassEntity {

    private String name;

    private String number;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getNumber() {
        return number;
    }

    public void setNumber(String number) {
        this.number = number;
    }
}

Mapper layer

UserMapper. Class (use default data source)

/**
 * description: UserMapper <br>
 * date: 2021/7/13 13:41 <br>
 * author: Neal <br>
 * version: 1.0 <br>
 */
public interface UserMapper extends BaseMapper<UserEntity> {
}

ClassMapper. Class (using another data source)

/**
 * description: ClassMapper <br>
 * date: 2021/7/13 13:41 <br>
 * author: Neal <br>
 * version: 1.0 <br>
 */
@DS ("DB2") // use another data source
public interface ClassMapper extends BaseMapper<ClassEntity> {
}

unit testing

Springboot integrates multiple data sources of mybatisplus

test01.jpg

The result has been that multiple data sources can run perfectly.

Source code analysis

In our projects, we should not only learn to use these components, but also know why and how they are implemented. In fact, the principle is the aspect based agent processing method that can be found on the Internet, but some of them are still worth learning.

automatic assembly

First we start withdynamic-datasourceStart of automatic assembly of components

Springboot integrates multiple data sources of mybatisplus

step1.jpg

Next, let’s take a look at this automatic assembly class, the assembled bean

@Slf4j
@Configuration
//Start springboot auto assemble dynamicdatasourceproperties externalized configuration
@EnableConfigurationProperties(DynamicDataSourceProperties.class)
//Declare the assembly loading order, and load it before datasourceautoconfiguration
@AutoConfigureBefore(value = DataSourceAutoConfiguration.class, name = "com.alibaba.druid.spring.boot.autoconfigure.DruidDataSourceAutoConfigure")
//When auto assembling, the following three auto assembly classes are introduced and automatically assembled
@Import(value = {DruidDynamicDataSourceConfiguration.class, DynamicDataSourceCreatorAutoConfiguration.class, DynamicDataSourceHealthCheckConfiguration.class})
//Automatically assemble loading conditions when spring datasource. When dynamic = true, the automatic assembly is loaded. The default value is true
@ConditionalOnProperty(prefix = DynamicDataSourceProperties.PREFIX, name = "enabled", havingValue = "true", matchIfMissing = true)
public class DynamicDataSourceAutoConfiguration implements InitializingBean {

    //Injection externalized configuration
    private final DynamicDataSourceProperties properties;
    
    
    private final List<DynamicDataSourcePropertiesCustomizer> dataSourcePropertiesCustomizers;

    //Constructor injection
    public DynamicDataSourceAutoConfiguration(
            DynamicDataSourceProperties properties,
            ObjectProvider<List<DynamicDataSourcePropertiesCustomizer>> dataSourcePropertiesCustomizers) {
        this.properties = properties;
        this.dataSourcePropertiesCustomizers = dataSourcePropertiesCustomizers.getIfAvailable();
    }

    //Multi data source loading interface. The default implementation is to load all data sources from the YML information 
    @Bean
    public DynamicDataSourceProvider ymlDynamicDataSourceProvider() {
        return new YmlDynamicDataSourceProvider(properties.getDatasource());
    }

    //After implementing datasource Java JNDI, all database connections in the spring container are obtained from the implementation bean
    @Bean
    @ConditionalOnMissingBean
    public DataSource dataSource() {
        DynamicRoutingDataSource dataSource = new DynamicRoutingDataSource();
        dataSource.setPrimary(properties.getPrimary());
        dataSource.setStrict(properties.getStrict());
        dataSource.setStrategy(properties.getStrategy());
        dataSource.setP6spy(properties.getP6spy());
        dataSource.setSeata(properties.getSeata());
        return dataSource;
    }

    //Set dynamic data source transformation switch Configurator
    @Role(value = BeanDefinition.ROLE_INFRASTRUCTURE)
    @Bean
    public Advisor dynamicDatasourceAnnotationAdvisor(DsProcessor dsProcessor) {
        DynamicDataSourceAnnotationInterceptor interceptor = new DynamicDataSourceAnnotationInterceptor(properties.isAllowedPublicOnly(), dsProcessor);
        DynamicDataSourceAnnotationAdvisor advisor = new DynamicDataSourceAnnotationAdvisor(interceptor);
        advisor.setOrder(properties.getOrder());
        return advisor;
    }

    //Facet configuration class of database transaction
    @Role(value = BeanDefinition.ROLE_INFRASTRUCTURE)
    @ConditionalOnProperty(prefix = DynamicDataSourceProperties.PREFIX, name = "seata", havingValue = "false", matchIfMissing = true)
    @Bean
    public Advisor dynamicTransactionAdvisor() {
        AspectJExpressionPointcut pointcut = new AspectJExpressionPointcut();
        pointcut.setExpression("@annotation(com.baomidou.dynamic.datasource.annotation.DSTransactional)");
        return new DefaultPointcutAdvisor(pointcut, new DynamicLocalTransactionAdvisor());
    }

    //The execution chain required in the dynamicdatasourceannotationinterceptor aspect configurator,
    //Mainly used to determine which data source to use
    @Bean
    @ConditionalOnMissingBean
    public DsProcessor dsProcessor(BeanFactory beanFactory) {
        DsHeaderProcessor headerProcessor = new DsHeaderProcessor();
        DsSessionProcessor sessionProcessor = new DsSessionProcessor();
        DsSpelExpressionProcessor spelExpressionProcessor = new DsSpelExpressionProcessor();
        spelExpressionProcessor.setBeanResolver(new BeanFactoryResolver(beanFactory));
        headerProcessor.setNextProcessor(sessionProcessor);
        sessionProcessor.setNextProcessor(spelExpressionProcessor);
        return headerProcessor;
    }

    //The method executed after bean injection is not currently used in this demo
    @Override
    public void afterPropertiesSet() {
        if (!CollectionUtils.isEmpty(dataSourcePropertiesCustomizers)) {
            for (DynamicDataSourcePropertiesCustomizer customizer : dataSourcePropertiesCustomizers) {
                customizer.customize(properties);
            }
        }
    }

}

The automatic assembly in general has been introduced. Next, we will explain the important code segments or classes one by one

Dynamicdatasourcecreatorautoconfiguration analysis

This class is mainly used to load data sources. The main code is as follows

@Slf4j
@Configuration
@AllArgsConstructor
@EnableConfigurationProperties(DynamicDataSourceProperties.class)
public class DynamicDataSourceCreatorAutoConfiguration {

    //Describe the injection sequence of beans
    public static final int JNDI_ORDER = 1000;
    public static final int DRUID_ORDER = 2000;
    public static final int HIKARI_ORDER = 3000;
    public static final int BEECP_ORDER = 4000;
    public static final int DBCP2_ORDER = 5000;
    public static final int DEFAULT_ORDER = 6000;

    private final DynamicDataSourceProperties properties;

    //Default data source Creator
    @Primary
    @Bean
    @ConditionalOnMissingBean
    public DefaultDataSourceCreator dataSourceCreator(List<DataSourceCreator> dataSourceCreators) {
        DefaultDataSourceCreator defaultDataSourceCreator = new DefaultDataSourceCreator();
        defaultDataSourceCreator.setProperties(properties);
        defaultDataSourceCreator.setCreators(dataSourceCreators);
        return defaultDataSourceCreator;
    }

    //Omit some codes

    /**
     *Add creator when Hikari data source exists
     */
    @ConditionalOnClass(HikariDataSource.class)
    @Configuration
    public class HikariDataSourceCreatorConfiguration {
        @Bean
        @Order(HIKARI_ORDER)
        @ConditionalOnMissingBean
        public HikariDataSourceCreator hikariDataSourceCreator() {
            return new HikariDataSourceCreator(properties.getHikari());
        }
    }

    //Omit some codes

}

After the spring container injects the defaultdatasourcecreator instance, it is then used by the dynamicdatasourceprovider class.

Dynamicdatasourceprovider analysis

@Slf4j
@AllArgsConstructor
public class YmlDynamicDataSourceProvider extends AbstractDataSourceProvider {

    /**
     *All data sources
     */
    private final Map<String, DataSourceProperty> dataSourcePropertiesMap;

    //Inject all data sources through the constructor, and then call the superclass method to create the data source collection
    @Override
    public Map<String, DataSource> loadDataSources() {
        return createDataSourceMap(dataSourcePropertiesMap);
    }
}
@Slf4j
public abstract class AbstractDataSourceProvider implements DynamicDataSourceProvider {

    //Get the injected defaultdatasourcecreator from the spring container 
    @Autowired
    private DefaultDataSourceCreator defaultDataSourceCreator;

    //Create data source collection
    protected Map<String, DataSource> createDataSourceMap(
            Map<String, DataSourceProperty> dataSourcePropertiesMap) {
        Map<String, DataSource> dataSourceMap = new HashMap<>(dataSourcePropertiesMap.size() * 2);
        for (Map.Entry<String, DataSourceProperty> item : dataSourcePropertiesMap.entrySet()) {
            DataSourceProperty dataSourceProperty = item.getValue();
            String poolName = dataSourceProperty.getPoolName();
            if (poolName == null || "".equals(poolName)) {
                poolName = item.getKey();
            }
            dataSourceProperty.setPoolName(poolName);
            dataSourceMap.put(poolName, defaultDataSourceCreator.createDataSource(dataSourceProperty));
        }
        return dataSourceMap;
    }
}

Dynamicdatasourceannotationadvisor analysis

This is actually the aspect configurator of spring AOP. The main code is as follows

public class DynamicDataSourceAnnotationAdvisor extends AbstractPointcutAdvisor implements BeanFactoryAware {

    //Section enhancement method
    private final Advice advice;

    private final Pointcut pointcut;

   //Construction method injection
    public DynamicDataSourceAnnotationAdvisor(@NonNull DynamicDataSourceAnnotationInterceptor dynamicDataSourceAnnotationInterceptor) {
        this.advice = dynamicDataSourceAnnotationInterceptor;
        this.pointcut = buildPointcut();
    }

    @Override
    public Pointcut getPointcut() {
        return this.pointcut;
    }

    
    @Override
    public Advice getAdvice() {
        return this.advice;
    }

    //Omit some codes
    
    
    //When there is DS in a class or method Section enhancement during class annotation
    private Pointcut buildPointcut() {
        Pointcut cpc = new AnnotationMatchingPointcut(DS.class, true);
        Pointcut mpc = new AnnotationMethodPoint(DS.class);
        return new ComposablePointcut(cpc).union(mpc);
    }

    //Omit some codes 
}

Dynamicdatasourceannotationinterceptor analysis

This class is an aspect enhancement, that is, when the dynamicdatasourceannotationadvisor above intercepts DS in the class or method Class annotation, call the enhanced class for processing

public class DynamicDataSourceAnnotationInterceptor implements MethodInterceptor {

    /**
     * The identification of SPEL.
     */
    private static final String DYNAMIC_PREFIX = "#";

    private final DataSourceClassResolver dataSourceClassResolver;
    private final DsProcessor dsProcessor;

    public DynamicDataSourceAnnotationInterceptor(Boolean allowedPublicOnly, DsProcessor dsProcessor) {
        dataSourceClassResolver = new DataSourceClassResolver(allowedPublicOnly);
        this.dsProcessor = dsProcessor;
    }

    //The method of enhancing the section after AOP interception
    @Override
    public Object invoke(MethodInvocation invocation) throws Throwable {
        //Select data source
        String dsKey = determineDatasourceKey(invocation);
        //Switching data sources using ThreadLocal based implementation
        DynamicDataSourceContextHolder.push(dsKey);
        try {
            return invocation.proceed();
        } finally {
            DynamicDataSourceContextHolder.poll();
        }
    }

    //Confirm the data source by calling dsprocessor through chain call
    private String determineDatasourceKey(MethodInvocation invocation) {
        String key = dataSourceClassResolver.findDSKey(invocation.getMethod(), invocation.getThis());
        return (!key.isEmpty() && key.startsWith(DYNAMIC_PREFIX)) ? dsProcessor.determineDatasource(invocation, key) : key;
    }
}

Defaultpointcutadvisor analysis

This aspect is enhanced as a transaction enhancement. After setting this enhanced class, it cannot be associated with spring source transactions or@TransactionalAnnotation sharing.

@Slf4j
public class DynamicLocalTransactionAdvisor implements MethodInterceptor {

    @Override
    public Object invoke(MethodInvocation methodInvocation) throws Throwable {
        if (!StringUtils.isEmpty(TransactionContext.getXID())) {
            return methodInvocation.proceed();
        }
        boolean state = true;
        Object o;
        String xid = UUID.randomUUID().toString();
        TransactionContext.bind(xid);
        try {
            o = methodInvocation.proceed();
        } catch (Exception e) {
            state = false;
            throw e;
        } finally {
            ConnectionFactory.notify(state);
            TransactionContext.remove();
        }
        return o;
    }
}

Dynamicdatasourcecontextholder core switch class

public final class DynamicDataSourceContextHolder {

    /**
     *Why use linked list to store (exactly stack)
     * <pre>
     *To support nested switching, for example, the three ABC services are different data sources
     *A business of a needs to call the method of B, and the method of B needs to call the method of C. Level by level call switching, forming a chain.
     *The traditional method of setting only the current thread cannot meet this business requirement. Stack must be used, last in first out.
     * </pre>
     */
    private static final ThreadLocal<Deque<String>> LOOKUP_KEY_HOLDER = new NamedThreadLocal<Deque<String>>("dynamic-datasource") {
        @Override
        protected Deque<String> initialValue() {
            return new ArrayDeque<>();
        }
    };

    private DynamicDataSourceContextHolder() {
    }

    /**
     *Get current thread data source
     *
     *@return data source name
     */
    public static String peek() {
        return LOOKUP_KEY_HOLDER.get().peek();
    }

    /**
     *Set current thread data source
     * <p>
     *If it is not necessary to call it manually, ensure that it is finally cleared after the call
     * </p>
     *
     *@param DS data source name
     */
    public static String push(String ds) {
        String dataSourceStr = StringUtils.isEmpty(ds) ? "" : ds;
        LOOKUP_KEY_HOLDER.get().push(dataSourceStr);
        return dataSourceStr;
    }

    /**
     *Clear current thread data source
     * <p>
     *If the current thread switches data sources continuously, only the data source name of the current thread will be removed
     * </p>
     */
    public static void poll() {
        Deque<String> deque = LOOKUP_KEY_HOLDER.get();
        deque.poll();
        if (deque.isEmpty()) {
            LOOKUP_KEY_HOLDER.remove();
        }
    }

    /**
     *Force empty local thread
     * <p>
     *Prevent memory leakage. For example, if push is called manually, this method can be called to ensure clearing
     * </p>
     */
    public static void clear() {
        LOOKUP_KEY_HOLDER.remove();
    }
}

The core code has been introduced. Next, let’s step by step debugger to find out its execution process.

Data source switching execution process

Now when we execute the call annotation in the springboot demo above@DS("db2")When mapper of queries the database, its order is as follows (only the relevant classes related to this component are given)

  1. ClassMapper#selectList(): execute the mybatis query operation

  2. DynamicDataSourceAnnotationInterceptor#invoke(): Spring AOP intercepts data with@DS("db2")And perform agent enhancement actions

  3. DataSourceClassResolver#findDSKey(): find annotated@DS()Get the corresponding data source key value, that is, DB2.

  4. DynamicDataSourceContextHolder#push(): set current thread data source

  5. DynamicRoutingDataSource#getConnection(): call the parent method to obtain the database connection. The two processing methods are as follows

    public Connection getConnection() throws SQLException {
            String xid = TransactionContext.getXID();
            //When there is no transaction, the current operation is query
            if (StringUtils.isEmpty(xid)) {
                return determineDataSource().getConnection();
            } else {
                //When there is something, first obtain the data source FIFO principle from the previous dynamicdatasourcecontextholder
                String ds = DynamicDataSourceContextHolder.peek();
                ds = StringUtils.isEmpty(ds) ? "default" : ds;
                ConnectionProxy connection = ConnectionFactory.getConnection(ds);
                 //Create data source
                return connection == null ? getConnectionProxy(ds, determineDataSource().getConnection()) : connection;
            }
        }
  1. DynamicRoutingDataSource#getDataSourceset up data sources

    public DataSource getDataSource(String ds) {
            //Use the default data source if there is currently no data source declaration
            if (StringUtils.isEmpty(ds)) {
                return determinePrimaryDataSource();
            } else if (!groupDataSources.isEmpty() && groupDataSources.containsKey(ds)) {
                log.debug("dynamic-datasource switch to the datasource named [{}]", ds);
                return groupDataSources.get(ds).determineDataSource();
            } else if (dataSourceMap.containsKey(ds)) {
                //If there is a current data source, take out the data source and return
                log.debug("dynamic-datasource switch to the datasource named [{}]", ds);
                return dataSourceMap.get(ds);
            }
            if (strict) {
                throw new CannotFindDataSourceException("dynamic-datasource could not find a datasource named" + ds);
            }
            return determinePrimaryDataSource();
        }
  1. Perform the remaining database operations until the end.

Summary

In general, it is a little confusing. However, as long as we know which beans are instantiated during automatic assembly, what these beans are for, what they are called appropriately, and gradually debug the debugger according to the execution process, we can understanddynamic-datasourceHow do components switch data sources? I think the source code has been marked in the classic and core parts of the process. We can learn from itDynamicDataSourceContextHolderThe idea of this public class extends and optimizes some cross resource calls in our existing projects.