Data initialization design and analysis of Choerodon

Time:2022-6-20

As a microservice framework, the digital intelligence performance platform Choerodon needs to solve the problems and complexities of microservice data initialization, and also needs to meet the unique data initialization requirements of the framework. The design idea and implementation in this regard are introduced below.

Data initialization problems of microservices

Let’s take a look at some common problems in data initialization of microservices.

▍ 1.1 initialization and smooth upgrade of table structure

The definition of table structure is the top priority in database initialization. It involves the operation of the entire service and the way to use the database to realize functions. Generally speaking, the definition and upgrade of table structure involves the following operations: create a table, create a field, create an index, modify an index, modify a field, rename a table, delete an index, delete a field, and delete a table. If these operations need to be compatible with and smoothly upgraded to a variety of databases, the complexity will suddenly increase, and it is basically impossible to manage them through SQL scripts like the traditional ones, which is the case for the pig toothed fish.

▍ 1.2 automatic initialization of cross service data

In the microservice architecture, it is inevitable that the data needs to be initialized to other services. For example, most services of pig toothed fish need to initialize the menu data to the Iam service, and the request to process the menu list is processed by the Iam service. However, for the deployment of microservices, it is often impossible to connect multiple data sources when running the initialization data, resulting in problems. Moreover, the deployment of microservices may not be full. There are cases where the deployment does not need the service. In this case, the initialization needs to modify the initialization script or program, resulting in complexity.

▍ 1.3 automatic discovery of tedious encoded data

In data initialization, there is a kind of data that can be collected and extracted from code, documents, or other places, and this part of data is often cumbersome and huge. For example, in the pig toothed fish, permission authentication requires the mapping relationship between URL, controller, and method. If this part of data is initialized manually, it will generate a lot of work, And the initialization data may not be updated after the actual code is modified.

Initialization of this service data of pig toothed fish

After recognizing these problems, let’s introduce the solutions to these problems proposed by the pig toothed fish after many iterations. Let’s first look at the solution to the service data initialization. For the specific implementation of this part, please refer to the open source code:https://github.com/choerodon/choerodon-starters/tree/master/choerodon-liquibase

▍ 2.1 initialization of data table structure

For the initialization of the database table structure, the pig toothed fish adopts the liquibase open source project. Specifically, it uses the groovy DSL of liquibase, which enhances the flexibility of liquibase. Liquibase itself supports smooth upgrade and multi database support, which solves the problem of table structure initialization.

▍ 2.2 initialization of preset data of this service

For some preset data, including preset user roles and preset data required for automatic test execution, the pig toothed fish uses Excel to assist in initializing data for easy operation, filling and association.

Initialization of cross service data of pig toothed fish

Next, let’s take a look at the processing methods of pig toothed fish for cross service data initialization and automatic data discovery.

▍ 3.1 initialization of auto discovery data

After the service is started, the management service accesses the general interface of each service to obtain data from classpath and initialize through distributed transactions.

Specific code reference:https://github.com/choerodon/manager-service/blob/master/src/main/java/io/choerodon/manager/api/eventhandler/EurekaEventObserver.java

▍ 3.2 initialization of cross service preset data

Use Excel with the same format as the preset data of this service to fill in the data, convert excel into JSON data during compilation, and finally initialize through distributed transactions together with the auto discovery data. During compilation, excel is generated into JSON, and the Maven plug-in is combined through Maven dependency. For specific code reference:https://github.com/choerodon/choerodon-starters/tree/master/choerodon-maven-plugin

epilogue

The pig toothed fish data initialization method is from the early SQL script to liquibase, plus the independent Python initialization tool designed to meet the needs of menu initialization. In version 0.17.0, it is upgraded to liquibase groovy + excel, which solves all the problems encountered at present. The above is the whole iterative process and implementation idea of pig toothed fish data initialization. Thank you.


This article was originally created by the technical team of pig toothed fish. Please indicate the source for reprint

Recommended Today

A program for collecting information from unit computers written in Perl

Copy codeThe codes are as follows: It is mainly used to collect IP, MAC, name and room. Later, the maintenance record function is added. The server receives the data and stores it in the database. ############################# use strict; use Tk; use Encode; #Socke parameter my $PF_INET = 2; my $port = 2345; my $remote_addr = pack(‘SnC4x8’,$PF_INET,$port,192,168,138,228); my $SOCK_DGRAM = 2; #Frame my ($label_room, $label_name, $label_ctrl, $label_notice); #Confirm, cancel my ($enter, $cancel); #Room, name variable my ($room, $name); $room = ”; […]