Extended reading: DBUS design idea and working principle of big data bus platform
How to achieve data consistency and real-time extraction synchronously based on logs?
Quickly deploy DBUS to experience real-time data flow calculation
The implementation principle and architecture disassembly of two kinds of data sources supported by DBUS.
In general, DBUS supports two types of data sources:
- RDBMS data source
- Log data source
I. implementation of rmdbms data source
Take MySQL as an example. It is divided into three parts:
- Log extraction module
- Incremental conversion module
- Full pull module
1.1 log extraction module (extractor)
MySQL log extraction module consists of two parts:
- Can server: responsible for extracting incremental logs from mysql.
- MySQL extractor storm program: responsible for outputting incremental logs to Kafka, filtering unnecessary table data, ensuring at least one and high availability.
We know that although MySQL InnoDB has its own log, the primary and secondary synchronization of MySQL is realized through binlog. There are three modes of binlog synchronization: row mode, statement mode and mixed mode. Because the statement mode has various limitations, the production environment usually uses row mode for replication, which makes it possible to read the full amount of logs.
Generally, our MySQL layout adopts the solution of two master master databases (VIP) + one slave database + one backup disaster recovery database. Because the disaster recovery database is usually used for remote disaster recovery, the real-time performance is not high and it is not easy to deploy.
In order to minimize the impact on the source side, we read the binlog logs from the slave database.
There are many schemes to read binlog. DBUS is also on the shoulders of giants. For MySQL data source, Alibaba’s open-source canal is used to read incremental logs. The benefits of doing so are:
- No need for repeated development and avoid repeated wheel building
- Enjoy the benefits of the canal upgrade
For the introduction of canal, please refer to: https://github.com/alibaba/ca… Because of the high extraction authority of canal users, the general canal server nodes can also be maintained by DBA groups.
The main goal of the log extraction module is to read out the data from the canal server and land it in the first level Kafka as soon as possible to avoid data loss (after all, if you do not read the log data for a long time, the log may roll to a long time ago and may be deleted by the DBA). Therefore, you need to avoid doing too many things, mainly to do data unpacking to prevent the data packet from being too large.
From the perspective of high availability, in the process of using canal extraction, the high availability mode of canal server based on zookeeper is adopted, without single point problem. The log extraction module extractor also uses storm program, which is also a high availability architecture.
Different data sources have different log extraction methods, such as Oracle, Mongo and so on.
The DBUS log extraction module is independent in order to be compatible with different implementations of these different data sources.
1.2 incremental conversion module (stream)
Incremental data processing module, according to different data source type format for conversion and processing.
- Distribute logs from data sources to different topics according to different schemas. The purpose of this
- It is for data isolation (because generally different shamas correspond to different databases)
- The purpose is to separate the calculation pressure of the conversion module, because the calculation amount of the conversion module is relatively large, multiple can be deployed, one for each schema to improve the efficiency.
2) conversion module appender
- Real time data format conversion: the canal data is in protobuf format, which needs to be converted to our agreed UMS format to generate unique identifiers such as UMS UUID and UMS UUTs;
- Capture metadata version change: such as table addition and subtraction column, field change, etc., maintain version information, send notice and trigger alarm
- Real time data desensitization: desensitize the specified column as required, for example, replace with*, MD5 salt, etc.
- Response to pull full event: when receiving the pull full request, in order to ensure the corresponding sequence of data lines, the pull incremental data will be suspended, and then continue after the full data is completed.
- Monitoring data: both the distribution module and the conversion module will respond to the heartbeat event, count the data and delay of each table in two heartbeats, and send it to statistic for monitoring data.
- The distribution module and the conversion module will load the configuration from the Mgr library and ZK according to the relevant reload notification events.
1.3 full puller
Full pull can be used for initial load and data reload. In implementation, we use the idea of sqoop for reference. The whole process is divided into two parts:
1) data fragmentation
Read the max, min, count and other information of the slice, calculate the number of slices according to the slice size, generate the slice information and save it in split topic. Here are the specific segmentation strategies:
Based on the practical experience, for MySQL indb, only using the primary key index to segment can it be efficient. Because the primary key columns of MySQL inndb are in the same order as the data storage.
2) actual pulling
Each partition represents a small task, which is pulled from the library by the pull conversion module through multiple concurrent degrees. The pull completion is written to zookeeper for monitoring.
Full pull has certain pressure on the source database. We do as follows:
- Pull data from the library from the slave
- Control concurrency 6-8
- Recommended in the low peak period of business
Full pull does not happen frequently. It is usually done once for initialization or triggered once when full pull is needed in some cases.
1.3 consistency of total and incremental
In the whole data transmission, in order to ensure the order of log messages, Kafka uses a partition method. In general, it is basically sequential and unique. However, if the asynchronous write part of Kafka fails, the storm also uses the redo mechanism. Therefore, we do not strictly guarantee the exact once and complete order, but we guarantee the at least once.
Therefore, UMS? ID? Becomes particularly important. For full-scale extraction, UMS UU ID is a value, which is the UMS UU ID number of the full-scale pull event, indicating that all the data of the batch is a batch, because the data are different and can share a UMS UU ID number. The UMS uid serial number is generated from ZK to ensure the uniqueness of data. For incremental extraction, we use MySQL log file number + log offset as the unique ID. ID is a 64 bit long integer, the high 6 bits are used for the log file number, and the low 13 bits are used as the log offset. For example: 000103000012345678. 103 is the log file number and 12345678 is the log offset. In this way, the log level guarantees the physical uniqueness (the ID number remains unchanged even if it is redone) and the order (the log can also be located). You can know which message is updated by comparing UMS? ID.
The value of UMS ﹐ is that it can accurately know the time of event from the time dimension. For example, if you want to get the snapshot data at a certain time. The truncation time point can be known through UMS Φ ts.
II. Implementation of log data source
There are many ways to collect, structure and analyze industry logs, such as logstash, filebeat, flume, fluentd, chukwa. Scribe, Splunk, etc. In structured log, regular expression template is mostly used to extract the fixed and general parts of log, such as log time, log type, line number, etc. For the real information related to business comparison, this part is the most important part, called message part. We want to use visual way to structure.
For example, for the log4j like log shown below:
If the user wants to convert the above data into the following structured data information:
We call such a log “data log”
The data log synchronization scheme designed by DBUS is as follows:
- The log grabbing end uses industry popular components (such as logstash, flume, filebeat, etc.). On the one hand, it is convenient for users and the industry to unify standards and integrate users; on the other hand, it is also convenient for users to avoid unnecessary reconstruction of wheels. The captured data is called raw data log and put into Kafka for processing.
- Provide visual interface and configure rules to structure logs.Users can configure the log source and target. The same log source can be output to multiple destinations. For each “log source target” line, users can freely define the rule processing of intermediate data according to their own needs. The final output data is structured, that is, there are schema constraints, which can be understood as similar tables in the database.
- The so-called rules, in DBUS, are“Regular operator”。 DBUS designs a variety of easy-to-use filtering, splitting, merging, replacement operators for users to use. Users can process data in multiple steps, and the data processing results of each step can be viewed and verified in real time; different operators can be used repeatedly until the data they need is transformed and clipped.
- The configured rule operator group is applied to the execution engine to preprocess the target log data to form structured data and output it to Kafka for use by downstream data users.
The system flow chart is as follows:
According to the configuration, we support the same original log, which can be extracted as one table data or multiple table data.
Each table is structured to meet the same schema.
- Each table is a set of regular operator groups, and one or more regular operator groups can be configured
- Each group of regular operators is composed of a group of regular operators
Get a raw data log, which table should it belong to?
Each log needs to match the rule operator group:
- The qualified ones who enter the rule operator group are finally transformed into structured table data by the rule group.
- Does not match the next set of regular operators attempted.
- If none of them match, enter the unknown table table.
2.1 regular operators
Rule operator is the basic unit of data filtering, processing and transformation. Common regular operators are as follows:
The operators are independent. By combining different operators to achieve more complex functions, the operators can be used iteratively to process arbitrary data.
We try to make operators as orthogonal or easy to use as possible (although regular expressions are very powerful, we still develop some simple operators such as trim operators to complete simple functions to meet the ease of use).
3. UMS unified message format
Whether it’s incremental, full or log, the message output to the result Kafka is the unified message format we agreed, which is called UMS (unified message schema) format. As shown in the figure below:
Type of data, version number of UMS
1) the namespace consists of: type. Data source name. Schema name. Table name. Table version number. Sub library number. Sub table number. It can describe all tables.
For example: MySQL. Db1. Schema1. Testable. 5.0.0
2) fields is the field name description.
- UMS ﹣ ID ﹣ the unique ID of the message to ensure that the message is unique
- UMS ﹐ TS ﹐ can al capture the time stamp of the event;
- UMS [OP] indicates that the data type is I (insert), u (update), B (before update), D (delete)
- UMS UUID data serial number, unique value
3) payload refers to specific data.
A JSON package can contain one or more pieces of data to improve the data payload.
IV. heartbeat monitoring and early warning
RDBMS system involves many modules such as database synchronization, log extraction, incremental conversion and so on.
The log system involves log extraction end, log conversion module and so on.
How to know whether the system is working healthily and whether the data can flow in real time? Therefore, the process monitoring and early warning is particularly important.
4.1 for RDBMS system
The heartbeat module obtains the list of tables to be monitored from the dbusmgr library, inserts heartbeat data (with sending time in the data) into the heartbeat table of the source DBUS library at a fixed frequency (for example, every minute), and the heartbeat table is synchronized in real time as incremental data, and goes the same logic and thread as the same step table (in order to ensure the order, sharding is used when encountering multiple concurrency degrees By table, the heartbeat data goes the same bolt as the table data), so when the heartbeat data is received, even if there is no added, deleted or modified data, the whole link can be proved to be connected.
After receiving the heartbeat packet data, the incremental conversion module and the heartbeat module will send the data to influxdb as monitoring data, which will be displayed through grafana. The heartbeat module also monitors the delay condition and gives an alarm according to the delay condition.
4.2 for log system
Heartbeat packets will be generated automatically from the source end. Similar to RDBMS system, heartbeat packets will be synchronized to the end through extraction module and operator conversion module. The heartbeat module is responsible for monitoring and early warning.