As a microservice framework, the digital intelligence performance platform Choerodon needs to solve the problems and complexities of microservice data initialization, and also needs to meet the unique data initialization requirements of the framework. The design idea and implementation in this regard are introduced below.
Data initialization problems of microservices
Let’s take a look at some common problems in data initialization of microservices.
▍ 1.1 initialization and smooth upgrade of table structure
The definition of table structure is the top priority in database initialization. It involves the operation of the entire service and the way to use the database to realize functions. Generally speaking, the definition and upgrade of table structure involves the following operations: create a table, create a field, create an index, modify an index, modify a field, rename a table, delete an index, delete a field, and delete a table. If these operations need to be compatible with and smoothly upgraded to a variety of databases, the complexity will suddenly increase, and it is basically impossible to manage them through SQL scripts like the traditional ones, which is the case for the pig toothed fish.
▍ 1.2 automatic initialization of cross service data
In the microservice architecture, it is inevitable that the data needs to be initialized to other services. For example, most services of pig toothed fish need to initialize the menu data to the Iam service, and the request to process the menu list is processed by the Iam service. However, for the deployment of microservices, it is often impossible to connect multiple data sources when running the initialization data, resulting in problems. Moreover, the deployment of microservices may not be full. There are cases where the deployment does not need the service. In this case, the initialization needs to modify the initialization script or program, resulting in complexity.
▍ 1.3 automatic discovery of tedious encoded data
In data initialization, there is a kind of data that can be collected and extracted from code, documents, or other places, and this part of data is often cumbersome and huge. For example, in the pig toothed fish, permission authentication requires the mapping relationship between URL, controller, and method. If this part of data is initialized manually, it will generate a lot of work, And the initialization data may not be updated after the actual code is modified.
Initialization of this service data of pig toothed fish
After recognizing these problems, let’s introduce the solutions to these problems proposed by the pig toothed fish after many iterations. Let’s first look at the solution to the service data initialization. For the specific implementation of this part, please refer to the open source code:https://github.com/choerodon/choerodon-starters/tree/master/choerodon-liquibase
▍ 2.1 initialization of data table structure
For the initialization of the database table structure, the pig toothed fish adopts the liquibase open source project. Specifically, it uses the groovy DSL of liquibase, which enhances the flexibility of liquibase. Liquibase itself supports smooth upgrade and multi database support, which solves the problem of table structure initialization.
▍ 2.2 initialization of preset data of this service
For some preset data, including preset user roles and preset data required for automatic test execution, the pig toothed fish uses Excel to assist in initializing data for easy operation, filling and association.
Initialization of cross service data of pig toothed fish
Next, let’s take a look at the processing methods of pig toothed fish for cross service data initialization and automatic data discovery.
▍ 3.1 initialization of auto discovery data
After the service is started, the management service accesses the general interface of each service to obtain data from classpath and initialize through distributed transactions.
Specific code reference:https://github.com/choerodon/manager-service/blob/master/src/main/java/io/choerodon/manager/api/eventhandler/EurekaEventObserver.java
▍ 3.2 initialization of cross service preset data
Use Excel with the same format as the preset data of this service to fill in the data, convert excel into JSON data during compilation, and finally initialize through distributed transactions together with the auto discovery data. During compilation, excel is generated into JSON, and the Maven plug-in is combined through Maven dependency. For specific code reference:https://github.com/choerodon/choerodon-starters/tree/master/choerodon-maven-plugin
epilogue
The pig toothed fish data initialization method is from the early SQL script to liquibase, plus the independent Python initialization tool designed to meet the needs of menu initialization. In version 0.17.0, it is upgraded to liquibase groovy + excel, which solves all the problems encountered at present. The above is the whole iterative process and implementation idea of pig toothed fish data initialization. Thank you.
This article was originally created by the technical team of pig toothed fish. Please indicate the source for reprint