From 1 day to 10 minutes, Huawei cloud DRS has done this behind the scenes

Time:2022-7-27

In the digital era, the data focus of enterprise it has shifted from supporting online transactions to operational analysis. Timely updating of data, efficient analysis and reducing manual intervention are the goals of enterprise management. Therefore, enterprises are no longer satisfied with “t+1”, but “t+0”, which requires real-time and personalization, and the automotive industry is no exception.

图片

FAW Volkswagen Automobile Co., Ltd. (hereinafter referred to as FAW Volkswagen) was established in 1991. It is the first modern passenger car industrial base in China to be built according to the economic scale. It has more than 20 products of Audi, Volkswagen and Jetta brands, covering a, B and C-class full range of passenger cars. In order to better serve customers, FAW Volkswagen meets users’ car purchase and experience needs in multiple dimensions by analyzing customers’ usage habits and preferences and collecting personalized product needs.

However, with the deepening of data mining, the requirements for Bi platform are also higher. In order to solve the problems such as the rapid growth of the amount of data in the reporting business system, the difficulty of data collection and interactive analysis,FAW Volkswagen joins hands with Huawei cloud databaseBreakthroughs and challenges have been made in the MEP (dealer multifunctional ecological platform) -bi platform project cooperation.

Only with real-time data update can we have faster market response ability

As we all know, the report system of Bi platform can connect to multidimensional database. Report is to help users master and understand data. Enterprise decision makers analyze and mine real-time data to make timely response strategies for business changes.

The Bi report business of FAW Volkswagen 4S stores needs to collect the data of all stores in real time, facing huge technical challenges: 60 + MySQL instances, 200 + databases, a total of 30000 tables need to be synchronized to the BI system in real time.

If we follow the traditional way of data entering the lake, we need to configure 30000 data synchronization links and a large amount of work comparison data consistency. The maintenance cost is very high and the synchronization efficiency is low. In addition, with the expansion of the amount of data, the existing database cannot support the amount of data at the level of several terabytes.

Therefore, it is urgent to build a bi platform that supports massive data storage and real-time data synchronization for business data intelligent analysis.

图片

Facing the business challenges of FAW Volkswagen, Huawei cloud’s main considerations in scheme selection are as follows:

  • Minimize the impact of extracted data on the source database: extracting data can not drag down the source database, nor cause great attenuation to the performance of the source business, or even the table is locked;
  • The deployment and implementation should be light and simple: there is no need to install an agent program in each data source, which is ready to use, and the configuration operation is simple and easy to use;
  • Guaranteed data consistency: logical data synchronization based on binlog parsing, with breakpoint continuation and strict point recognition technology, effectively ensuring data consistency.

To sum up, it has lightweight services with real-time accurate synchronization and data consistency,Huawei cloud data replication service DRSIt is the best choice to meet business challenges.

Help the real-time synchronization of data in the reporting system, and fully release the value of data

DRS (data replication service) is a cloud data replication service, which can collect data on the cloud into Huawei’s flagship cloud native database gaussdb (for MySQL) with writing ability through a many-to-one scheme to ensure data consistency and effectiveness, and then synchronize from gaussdb (for MySQL) to Huawei cloud data warehouse gaussdb (DWS) for analysis through DRS implementation. However, it is very difficult to achieve real-time data synchronization. Therefore, Huawei cloud database solves the following core problems:

  • Add data source columns to resolve primary key conflicts: through the many to one scheme from Mysql to gaussdb (for MySQL), add the data source column and record the server/database/table information to avoid the primary key conflict of multiple source tables;
  • The synchronous link supports dynamic addition and subtraction tables: new businesses need to add synchronized tables at any time. Huawei cloud DRS supports the link dynamic increase and decrease table of gaussdb (for MySQL), which is simpler and more convenient to configure;
  • Mark the time of data entering the lake to help real-time analysis: add additional columns of data row writing time and update time, mark the data insertion and update time, and lay a foundation for data entry and real-time data analysis;
  • Introduce buffered database for real-time calculation: for the continuous collection and writing of 18000 tables, gaussdb (for MySQL) with million level QPS throughput and massive data storage capacity is selected as the buffer layer database to solve the natural data processing capacity difference between OLTP and OLAP databases. After real-time and accurate data synchronization based on DRS, Huawei cloud gaussdb (DWS) realizes real-time calculation, which greatly shortens the execution time of complex reports;
  • Real time accurate synchronization of data: Based on the accurate data synchronization ability of Huawei cloud DRS, the delay is within 1s under normal circumstances, which provides basic conditions for real-time reports, and the accuracy of data synchronization is high, and the number of data rows at the source and target are the same.

After the business goes online, the number of links maintained by FAW Volkswagen business reporting system has increased from30000 to 65, work efficiency has been greatly improved; meanwhileNone of the 1billion level data is lost, greatly reducing the data comparison work; Help FAW Volkswagen realize the availability time of sales report dataReduced from 1 day to 10 minutesThe timeliness has been greatly improved.

In addition, * * the buffer layer database gaussdb (for MySQL) * * has massive data storage capacity, supports 128tb at most, and has over one million QPS throughput, which can support the rapid development of business; The computing storage separation architecture based on gaussdb (for MySQL) has strong consistency in the storage of three copies of data and zero data loss; RTO second level, fault second switching, fully ensuring data security.

FAW Volkswagen takes users as the center, builds an ecosystem with data analysis as the core, has insight into user needs, and provides high-value products, which are favored by customers. Huawei cloud database realizes real-time data synchronization through technical means, finds insights from massive data, and provides a data basis for enterprise business decisions. In the future, Huawei cloud database will continue to work with FAW Volkswagen to create differentiated and competitive digital services through digital technology to meet the growing personalized needs of customers for products and services.

[recommended]Huawei cloud database special offer

This article is written byHua Weiyunrelease.

Recommended Today

redis day2

persistence mechanism client redis[memory] —–> memory data-data persistence–>disk Redis officially provides two different persistence methods to store data in memory to the hard disk: Snapshot AOF (Append Only File) only appends log files   Snapshot Features:In this way, all data at a certain moment can be written to the hard disk, of course, this is […]