Build canal and connect rabbitmq

Canal is a very useful MySQL data incremental synchronization tool

In fact, it is OK to install according to the official documents. Reading other blogs will inevitably lead to errors, which will only increase unnecessary time

Official installation document address:QuickStart · alibaba/canal Wiki (github.com)

target

1. Preparation (configuring MySQL and creating a new canal user)
2. Install canal
3. Configure canal
4. Run canal
5. Observe that canal has successfully accessed rabbitmq

Step by step

preparation

• For self built mysql, you need to enable binlog write function first, and configure binlog format to row mode, my CNF is configured as follows

  [mysqld]
  Log bin = MySQL bin \
  Binlog format=row \\select row mode
  server_ Id=1  \

  • Note: for Alibaba cloud RDS for MySQL, binlog is enabled by default, and the account has binlog dump permission by default. You do not need any permission or binlog settings. You can skip this step directly

• Authorize the canal linked MySQL account to have the permission to act as a MySQL slave. If there is an existing account, grant it directly

  CREATE USER canal IDENTIFIED BY 'canal';  
  GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
  -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
  FLUSH PRIVILEGES;
  

Install canal

• Download canal, visitRelease page, select the package to download, for example, version 1.0.17

  wget https://github.com/alibaba/canal/releases/download/canal-1.0.17/canal.deployer-1.0.17.tar.gz
  

• decompression

  mkdir /tmp/canal
  tar zxvf canal.deployer-$version.tar.gz  -C /tmp/canal
  

  • After decompression, enter the /tmp/canal directory, and you can see the following structure

    drwxr-xr-x 2 jianghang jianghang  136 2013-02-05 21:51 bin
    drwxr-xr-x 4 jianghang jianghang  160 2013-02-05 21:51 conf
    drwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib
    drwxr-xr-x 2 jianghang jianghang   48 2013-02-05 21:29 logs
    

Configure canal

• to configure/tmp/canal/canal.properties

  You need to obtain the connection information of MQ in advance and establish the corresponding exchange. Otherwise, an error will be reported when starting

  # tcp, kafka, rocketMQ, rabbitMQ
  canal. Servermode = rabbitmq \
  
  
  ##################################################
  #########                   RabbitMQ         #############
  ##################################################
  #Configure rabbitmq connection information
  rabbitmq.host = 192.168.1.1
  rabbitmq.virtual.host = /
  rabbitmq. exchange = exchange. fanout. The name of canal \exchange needs to be created in advance
  rabbitmq.username = admin
  rabbitmq.password = admin
  rabbitmq. Deliverymode = mode of fanout \exchange

• to configure/tmp/canal/instance.properties

  You can log in to MySQL and executeshow master status;, the fields are as follows:

  File	Position	Binlog_Do_DB	Binlog_Ignore_DB	Executed_Gtid_Set
  mysql-bin.004911	471834950		mysql,mysql,information_schema,performation_schema,sys

  canal.instance.master.journal.nameThe file field [MySQL bin.004911] binlog file name

  canal.instance.master.positionIs the offset position of binlog [471834950]

  ...
  #Fill in MySQL information
  # position info
  canal.instance.master.address=192.168.1.1:3306
  canal.instance.master.journal.name=mysql-bin.004911
  canal.instance.master.position=471834950
  canal.instance.master.timestamp=
  canal.instance.master.gtid=
  
  ...
  #Fill in MySQL user information
  # username/password
  canal.instance.dbUsername=canal
  canal.instance.dbPassword=canal
  
  #Specify to listen to a library or table
  canal. instance. filter. regex=.*\\..* #  [.*\.. *] listens to all database tables. [test\.. *] means to listen to all the tables under the test database.

Start canal

• functionsh /tmp/canal/bin/startup.sh

• View server logs

  tail -f /tmp/canal/logs/canal/canal.log</pre>
  

  2013-02-05 22:45:27.967 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
  2013-02-05 22:45:28.113 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
  2013-02-05 22:45:28.210 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
  

• View instance logs

  tail -f /tmp/canal/logs/example/example.log
  

  2013-02-05 22:50:45.636 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
  2013-02-05 22:50:45.641 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
  2013-02-05 22:50:45.803 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
  2013-02-05 22:50:45.810 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....
  

• close

  sh /tmp/canal/bin/stop.sh
  

  If you turn off the prompt:Canal pid, but the process does not. Can be deleted directlycanal.pid, just start it up

Observe rabbitmq

1. Observation logtail -f /tmp/canal/logs/example/example.log

2. Observe the MQ situation: create a new queue to connect to the exchange, and then check whether the data can be received

Q&A

1. Tipsfound canal.pid , Please run stop.sh first ,then startup.sh

   A: This file records the PID number of canal running, and the PID corresponding to kill when the user stops. It should not be implementedstop.sh, it needs to be executedstop.sh。 If implementedstop.sh, the file has not been deleted, but the corresponding process has not been deleted. You can delete the file directly.

2. Tips

   Unrecognized VM option 'UseCMSCompactAtFullCollection'
   Error: Could not create the Java Virtual Machine.
   Error: A fatal exception has occurred. Program will exit.
   

   A: Maybe the JDK version is too high [usecmscompactatfullcollection has been removed]. It is recommended to use jdk8.

3. Tips:

   2021-10-19 16:31:26.803 [pool-6-thread-1] ERROR c.a.o.c.c.rabbitmq.producer.CanalRabbitMQProducer - com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'exchange.trade.order' in vhost '/', class-id=60, method-id=40)
   java.lang.RuntimeException: com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'exchange.trade.order' in vhost '/', class-id=60, method-id=40)
   

   A: The corresponding exchange has not been established. Please go to the MQ management center to create it.

summary

Canal is still very simple to build, as long as you understand its basic functions: disguise as a slave node of MySQL, request master dump binlog data to him, and then push the data analysis to MQ. So we only need to configure MySQL and MQ. But all the problems I encountered were the problems of setting up MQ and JDK, as well as the problems of server dependency, certificates and so on. There were almost no problems with canal, and it was set up all at once.