Detailed interpretation of distributed | dble release notes 2.19.09.0

Time:2020-12-1

Detailed interpretation of distributed | dble release notes 2.19.09.0

The following is a detailed interpretation of the release notes of dble 2.19.09.0. The article is mainly divided into the following three parts:

  • 1、 Introduction to dble project
  • 2、 New version of the main update interpretation
  • 3、 Complete release notes and Translation

1、 Introduction to dble project

DBLE — Enterprise level open source distributed middlewareThe nickname “MYCAT plus” was given by the people in the rivers and lakes. With its simple and stable, continuous maintenance, good community environment and broad mass base, it has been strongly supported by the community;
Detailed interpretation of distributed | dble release notes 2.19.09.0
Dble official project:
https://github.com/actiontech…
If you are interested in source code or need customized functions, you can install it through source code compilation

Dble official document (Updated)
https://actiontech.github.io/…
You can learn more about the background and application scenarios of dble. Details not covered in this article can be obtained in official documents

Download address and release notes of the new version of dble:
https://github.com/actiontech…

Official courses

Click to learn “dble fast start” (full 18 lessons, updated to lesson 2), practice Video + text draft!

Click to learn the advanced course of dble open course

Tips: it is recommended to download the latest releases version, and the tar package can be downloaded. If there is a need for source code compilation, you can download the source code package.

Main update interpretation

Overview of the new version

71 days after the release of 2.19.07.0, version 2.19.09.0 was released; 8 refactoring functions, 34 defects and 3 backward compatibility adjustments were added in the new version of dble;

Thank you first

  • Thank you for your contribution to the community:

    @yinyanghuafa, @CrookedPotato, @lirulei, @ssxlulu.

  • Thanks to GitHub users for the quality issue
  • Thank you for your effective feedback from QQ group (669663113)

1. Add external backend mysql-ha connection

Dble provides correspondingExternal interfaceThe design is based on the following basic points:

  • External ha components can communicate with dble actively through certain ways (including script, network communication, command, etc.)
  • The external ha component can make a clear judgment on the availability of MySQL (including which nodes stop traffic, which nodes switch between master and slave, and which nodes start traffic)

Based on the above two conditions, considering the generality and ease of use of external interfaces, it is decided to add the corresponding interactive interfaces by the way of user commands at the management end. Specifically, the following three interactive interfaces are provided:

  • Stop traffic command (datahost @ disable)
  • Start traffic command (datahost @ enable)
  • Master slave switch command (datahost @ swtich)

Secondly, in order to facilitate management and observation, a query command of current switching state is provided

  • Switch status query command (datahost @ events)

External ha components can interact with each other through MySQL command, script calling MySQL command, or program DB connection executing MySQL command.
At the same time, this function design takes into account the needs of ZK cluster users. When users use dble in ZK cluster state for high availability switching, there will be a cluster linkage mechanism, and the external ha component only needs to notify one of the nodes.

An interaction diagram between dble and the following is provided:
Detailed interpretation of distributed | dble release notes 2.19.09.0

Command instructions & command implementation details & simple ha interactive use cases, please see the following documents:

https://github.com/actiontech…

https://github.com/actiontech…

https://github.com/actiontech…

2. Dump file split command

In POC, field personnel often encounter various problems when importing data. The typical problem is that dble does not support some SQL statements when importing files. In addition, the historical data that is not partitioned is imported through dble, and the old data will be partitioned by route. When the amount of data is large, it will take a long time. If there is an error in the process, it is difficult to check. Based on the above reasons, version 2.19.09.0 provides a tool to split the source data file exported by mysqldump according to the back-end fragment node. The segmented data file can be imported into each back-end fragment, which adapts to the demand of data import according to fragmentation.

Dump file statement processing

  1. Create database: converts a logical database to a physical database.
  2. DDL statement: according to the partition node of the table, it is written to the dump file of the corresponding back-end node. For the create statement of the global table, the global check column will be added. For the auto increment column, the data type of the auto increment column will be changed to bigint.
  3. Insert: the global sequence column value will be replaced by the dble, and then the split column will be routed to the dump file of the corresponding back-end node according to the splitting algorithm.
  4. Some attribute setting statements will be issued to the dump file of the specific back-end node according to the last resolved DDL.
  5. Processing of the view is skipped
  6. Processing of child tables is skipped

Command syntax

mysql > split src dest [-r500] [-w500] [-l10000]

  • SRC: represents the original dump file name
  • Dest: indicates the directory where the generated dump file is stored
  • -R means to set the size of the read file queue. The default value is 500
  • -W means to set the size of the write file queue. The default value is 500
  • -L represents the maximum values contained in the next insert after split. It is only for the partitioned table, and the default value is 10000. For example, if an insert statement in the original dump file partition table contains 30000 values, which can be divided into two nodes on average, then after the statement is split, an insert containing 10000 values and an insert containing 5000 values will be generated on each data node.

The generated fragment file is in the format: source file name-Datanode name - timestamp.dump, the latest file timestamp is the largest.

For example: my original dump file is / TMP / MySQL_ dump.sql After splitting, I want to output it to / TMP / dump / directory

split /tmp/mysql_dump.sql /tmp/dump/

For instructions on how to use the command, please consult the following documents:

https://github.com/actiontech…

4、 Complete release notes

Update list:

  • [#] added external mysql-ha software calling interface, which supports calling dble to switch routes synchronously when mysql-ha software switches databases
  • The file exported by mysqldump can be split into multiple files according to datanode.
  • [ා 1364] childtable adds a configuration property “incrementcolumn”.
  • Add a new parameter “rulefile” to some partition algorithms
  • [ා 1402] upgrade the JWs version because sometimes the old version does not work properly.
  • [] adds a distinguished name to the heartbeat connection.
  • [ා 1405] reconstruct dbleserver and dbleserver ා startup.
  • [# 1440] management command thread_ Performance optimization of used.

Defect repair:

  • [#ාා#] realizes online restart status check of cluster.
  • When join SQL containsusing()Clause, explain will get the wrong result.
  • [# 1499] when balance! = 0 and all back-end MySQL are disabled, @ @ config is overloaded_ All reported an error.
  • The query feedback is not accurate when the user’s permission is limited.
  • After a transaction fails, the new transaction may obtain the result set of the previous transaction.
  • When the data host cannot be accessed,CommitYou might get a strange result set.
  • [#1325] [#1317] server.xml The maxpacketsize property in does not work.
  • [ා1394] [ා1434] executionSQL_SELECT_LIMIT=1000An error was returned with another unreachable data host. Thank you @ yinyanghuafa for reporting the error.
  • The [] small int type will cause an array index out of range exception in the preprocessing statement protocol.
  • [] management commandpause @@dataNode=...Even if the service is not actually stopped, it returns OK.
  • When slow logging is enabled, a very large select SQL will take a lot of time
  • The X of [ා 1456] [load data… Fields terminated by “X”…] can only be of type char, and it is expected that x can be of type string.
  • The interval between queries in a transaction is calculated as part of sqlexecutetimeout.
  • If the table uses reserved words as column names, and server.xml When useglobaltablecheck = 1 in the global table, there will be an error in inserting data into the global table.
  • When setting lower_ case_ table_ When names = 1 and the view name contains uppercase letters,select ... from view_nameAn error will be reported.
  • [ා 1458] an error was returned while executing DDL in an XA transaction.
  • Implicit insertion into different data nodes may not follow final consistency.
  • The [ා] function form is char [n], if it is in the part without [n], dble execution will be wrong.
  • The join query with duplicate columns failed and an error was reported, but it succeeded.
  • [ා 1365] when MySQL is set to lower_ case_ table_ When names = 1, join returns the error “unknown column ‘ ta.id ‘ in ‘field list'”。
  • The internal join query returned the error “column T1. Id not found”.
  • For global tables, dble will remove the back quotation marks on the table name when routing to the datanode.
  • [#1375] dble does not support back quotes.
  • [ා1269] use C_ An error occurred while the API driver was checking for multi query functionality.
  • [#] PMA was unable to connect to dble. Thank @ lirulei for reporting this error.
  • [] executionselect ... from where ... limit n“Table node build own exception! Error:null ” 。
  • [ා] If an error occurs during data loading, the ID of MySQL protocol packet is wrong. Thank @ ssxlulu for reporting and repairing the error.
  • Semicolons containing Chinese characters in [#] statements, will directly return OK.
  • If the table name in the query is added with back quotation markstable_name, the table will be scanned globally. Thank you @ crookedpotato for reporting this error.
  • [ා 1323] selects a specific fragment table, which is decorated with back quotation marks to indicate that all partitions will be routed.
  • [ා1385] inwhereWhen using regular expressions under the condition, the query results are incorrect.
  • [] when the database name in MySQL has “.”, an error is reported when executing SQL.
  • [ා] used when connecting dble--default-auth=sha256_passwordAnd wrong password, return the error: “lost connected”.
  • [#1277] administrative users can log in to the server port. Thank @ crookedpotato for reporting this error.

compatibility:

  • Changereload @@configLogic, now andreload @@config_allThe same.
  • [] adds a distinguished name to the heartbeat connection.
  • [#1446] reconstructionshow @@directmemory。“