Build canal and connect rabbitmq

Time:2022-6-27

Build canal and connect rabbitmq

Canal is a very useful MySQL data incremental synchronization tool

In fact, it is OK to install according to the official documents. Reading other blogs will inevitably lead to errors, which will only increase unnecessary time

Official installation document address:QuickStart · alibaba/canal Wiki (github.com)

target

  1. Preparation (configuring MySQL and creating a new canal user)
  2. Install canal
  3. Configure canal
  4. Run canal
  5. Observe that canal has successfully accessed rabbitmq

Step by step

preparation

  • For self built mysql, you need to enable binlog write function first, and configure binlog format to row mode, my CNF is configured as follows

    [mysqld]
    Log bin = MySQL bin \
    Binlog format=row \\select row mode
    server_ Id=1  \
    • Note: for Alibaba cloud RDS for MySQL, binlog is enabled by default, and the account has binlog dump permission by default. You do not need any permission or binlog settings. You can skip this step directly
  • Authorize the canal linked MySQL account to have the permission to act as a MySQL slave. If there is an existing account, grant it directly

    CREATE USER canal IDENTIFIED BY 'canal';  
    GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
    -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
    FLUSH PRIVILEGES;
    

Install canal

  • Download canal, visitRelease page, select the package to download, for example, version 1.0.17

    wget https://github.com/alibaba/canal/releases/download/canal-1.0.17/canal.deployer-1.0.17.tar.gz
    
  • decompression

    mkdir /tmp/canal
    tar zxvf canal.deployer-$version.tar.gz  -C /tmp/canal
    
    • After decompression, enter the /tmp/canal directory, and you can see the following structure

      drwxr-xr-x 2 jianghang jianghang  136 2013-02-05 21:51 bin
      drwxr-xr-x 4 jianghang jianghang  160 2013-02-05 21:51 conf
      drwxr-xr-x 2 jianghang jianghang 1.3K 2013-02-05 21:51 lib
      drwxr-xr-x 2 jianghang jianghang   48 2013-02-05 21:29 logs
      

Configure canal

  • to configure/tmp/canal/canal.properties

    You need to obtain the connection information of MQ in advance and establish the corresponding exchange. Otherwise, an error will be reported when starting

    # tcp, kafka, rocketMQ, rabbitMQ
    canal. Servermode = rabbitmq \
    
    
    ##################################################
    #########                   RabbitMQ         #############
    ##################################################
    #Configure rabbitmq connection information
    rabbitmq.host = 192.168.1.1
    rabbitmq.virtual.host = /
    rabbitmq. exchange = exchange. fanout. The name of canal \exchange needs to be created in advance
    rabbitmq.username = admin
    rabbitmq.password = admin
    rabbitmq. Deliverymode = mode of fanout \exchange
  • to configure/tmp/canal/instance.properties

    You can log in to MySQL and executeshow master status;, the fields are as follows:

    File Position Binlog_Do_DB Binlog_Ignore_DB Executed_Gtid_Set
    mysql-bin.004911 471834950 mysql,mysql,information_schema,performation_schema,sys

    canal.instance.master.journal.nameThe file field [MySQL bin.004911] binlog file name

    canal.instance.master.positionIs the offset position of binlog [471834950]

    ...
    #Fill in MySQL information
    # position info
    canal.instance.master.address=192.168.1.1:3306
    canal.instance.master.journal.name=mysql-bin.004911
    canal.instance.master.position=471834950
    canal.instance.master.timestamp=
    canal.instance.master.gtid=
    
    ...
    #Fill in MySQL user information
    # username/password
    canal.instance.dbUsername=canal
    canal.instance.dbPassword=canal
    
    #Specify to listen to a library or table
    canal. instance. filter. regex=.*\\..* #  [.*\.. *] listens to all database tables. [test\.. *] means to listen to all the tables under the test database.

Start canal

  • functionsh /tmp/canal/bin/startup.sh

  • View server logs

    tail -f /tmp/canal/logs/canal/canal.log</pre>
    
    2013-02-05 22:45:27.967 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
    2013-02-05 22:45:28.113 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[10.1.29.120:11111]
    2013-02-05 22:45:28.210 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
    
  • View instance logs

    tail -f /tmp/canal/logs/example/example.log
    
    2013-02-05 22:50:45.636 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
    2013-02-05 22:50:45.641 [main] INFO  c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
    2013-02-05 22:50:45.803 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 
    2013-02-05 22:50:45.810 [main] INFO  c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start successful....
    
  • close

    sh /tmp/canal/bin/stop.sh
    

    If you turn off the prompt:Canal pid, but the process does not. Can be deleted directlycanal.pid, just start it up

Observe rabbitmq

  1. Observation logtail -f /tmp/canal/logs/example/example.log

  2. Observe the MQ situation: create a new queue to connect to the exchange, and then check whether the data can be received

Q&A

  1. Tipsfound canal.pid , Please run stop.sh first ,then startup.sh

    A: This file records the PID number of canal running, and the PID corresponding to kill when the user stops. It should not be implementedstop.sh, it needs to be executedstop.sh。 If implementedstop.sh, the file has not been deleted, but the corresponding process has not been deleted. You can delete the file directly.

  2. Tips

    Unrecognized VM option 'UseCMSCompactAtFullCollection'
    Error: Could not create the Java Virtual Machine.
    Error: A fatal exception has occurred. Program will exit.
    

    A: Maybe the JDK version is too high [usecmscompactatfullcollection has been removed]. It is recommended to use jdk8.

  3. Tips:

    2021-10-19 16:31:26.803 [pool-6-thread-1] ERROR c.a.o.c.c.rabbitmq.producer.CanalRabbitMQProducer - com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'exchange.trade.order' in vhost '/', class-id=60, method-id=40)
    java.lang.RuntimeException: com.rabbitmq.client.AlreadyClosedException: channel is already closed due to channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'exchange.trade.order' in vhost '/', class-id=60, method-id=40)
    

    A: The corresponding exchange has not been established. Please go to the MQ management center to create it.

summary

Canal is still very simple to build, as long as you understand its basic functions: disguise as a slave node of MySQL, request master dump binlog data to him, and then push the data analysis to MQ. So we only need to configure MySQL and MQ. But all the problems I encountered were the problems of setting up MQ and JDK, as well as the problems of server dependency, certificates and so on. There were almost no problems with canal, and it was set up all at once.