“Modular installation”, define your own cloudquery

Time:2022-5-23

As we all know, “user experience” is not only highly interactive and easy to operate on the UI, but also the performance experience of the system. The decisive parameters of performance are system throughput and response speed. In the PC era, we often forcibly raise the system response level by improving the hardware of the system, but the hardware cannot be expanded endlessly. As early as 2005, the CEO of Intel, a large chip manufacturer, proposed that the era of relying only on hardware to vertically improve the system performance has long passed, and “distributed” has become the mainstream of the current efficiency reform.

“Distributed system” refers to the service disassembly of the application system and the use of more servers to complete common tasks to process a large amount of data when a single server cannot bear the pressure of access and data processing. Therefore, when facing the processing requirements of high concurrency and large amount of data, distributed applications often perform better than centralized services. At the same time, their own high scalability support level and elastic scalability make the application system handy in the face of peak traffic.

As an enterprise database management and control platform, cloudquery not only has frequent access requests from internal personnel, but also has great resource requirements for its own services. For example, while streaming data, audit services also need to output dimension analysis reports and early warning risk information in real time. The DTS service added in version 1.4.0 needs to process user data import, export and other operations at a higher speed. For the above scenarios, we adopt a distributed deployment scheme to independently deploy services with high resources, high consumption and high computing, which will not affect the request processing of the main standard services while improving the performance and response speed.

At the same time, the installation method is adjusted from the original “one size fits all” installation to “modular” installation. The so-called “modular installation” refers to the packaging details and service disassembly, which do not affect each other. Each module realizes its own specific functions, greatly reduces the service coupling, and quickly meets the personalized installation requirements with the least modules and parts.

Modular installation” mode has been adjusted to cloudquery v1 4.1 select the installation module in a page-based manner, download and unzip the installation package successfully, and then use the command line authorization script to execute the permission.

Chmod + X install // authorized installation

After successful authorization, execute the command to start the pre installation service:

./install 

After the pre installation service is started successfully, it will return to the pre installation page address. Copy the address to the browser to enter the “modular installation” interface for custom configuration installation. The custom configuration mainly includes four parts: basic module, optional module, optional data source and system configuration.

First, the basic module includes user, query, task center, notification, web, and platform persistence layer data storage services. You can customize the port configuration for all the current cloudquery basic images (as shown in the figure below).

For the “optional module”, users can choose according to their own scenarios, including audit, task center and terminal.

After completing the “optional module”, you can select the required data source type.

After all functional configurations are completed, “system configuration” can be carried out, including installation path and upper log limit.

After all the above configuration items are configured, the automatic installation process will be entered. The current installation progress can be viewed in real time in the progress box. Problems encountered in the process can be located and solved in time.

After this installation upgrade, cloudquery will continue to iterate over the existing standard services. Improve its own platform functions, continuously launch more data related modules, help the enterprise’s internal data operation more convenient and automated, and speed up the efficiency of development, operation and maintenance personnel.

Btquery will be conducted on cloud31, btw. 1 4.2 live Q & A to explain and demonstrate the core functions of the recent two versions, including OpenAPI, modular installation, data import, new data source polardb, etc. At the same time, we will answer more questions raised by the community students recently, and arrange the iteration plan.