Teach you how to build rocketmq high availability cluster!

Time:2021-8-27

Teach you how to build rocketmq high availability cluster!

Rocketmq development history

Rocketmq is a message middleware open source by Alibaba. It was open source in 2012 and became the top project of Apache in 2017. Rocketmq is widely used in Alibaba. Thousands of applications in Alibaba run on rocketmq. Hundreds of millions of messages need to be processed during the double 11 Festival, and the TPS can reach hundreds of thousands. At present, it supports access to Java, C / C + +, Python and go.

Rocketmq now has two versions, one is the community open source version and the other is the commercial cloud service version (aliwaremq). Latest version: 4.8.0 (demo version of this article).

Its core design draws lessons from Kafka, so when we study rocketmq, we will find many of the same features as Kafka, but they are quite different from Kafka in some functions. It has the following features:

  • Support cluster model, load balancing and horizontal expansion capability
  • 100 million level message stacking capacity
  • It adopts the principle of zero copy, sequential disk writing and random reading
  • The underlying communication framework adopts netty NiO
  • Nameserver replaces zookeeper to realize service addressing and service coordination
  • Message failure retry mechanism, message queryable
  • It emphasizes that the cluster has no single point, is scalable, is highly available at any point, and is horizontally scalable
  • After many double eleven tests

High availability architecture

Rocketmq naturally supports clusters very well. It has the following modes:

Single master

  • Advantages: there are no advantages except simple configuration
  • Disadvantages: unreliable. Restarting or shutting down the machine will make the whole service unavailable

Multi master

  • Advantages: simple configuration and highest performance
  • Disadvantages: a small number of messages may be lost (related to configuration). During the restart or downtime of a single machine, messages not consumed under the machine cannot be subscribed before the machine is restored, affecting the real-time performance of messages

Multi master multi slave

Each master is equipped with a slave. There are multiple pairs of master slave. The cluster adopts asynchronous replication. The active and standby have a short message delay of milliseconds

  • Advantages: the performance is almost the same as that of multiple masters, with high real-time performance. The switching between active and standby is transparent to applications without manual intervention
  • Disadvantages: a small amount of messages will be lost when the Master goes down or the disk is damaged

Multi master multi slave

Each master is equipped with a slave. There are multiple pairs of master slave. The cluster adopts synchronous double write mode. Both active and standby write successfully and return success to the application

  • Advantages: service availability and data availability are very high
  • Disadvantages: the performance is slightly lower than that of asynchronous clusters. The current version cannot automatically switch between primary and standby

Two master and two slave asynchronous cluster installation

This paper adopts the two master and two slave installation mode, that is, the fourth multi master and multi slave

Port planning

First of all, I bought two ECS servers. I usually use them to learn how to install some middleware, which is more convenient. If you have conditions, you can also buy them. Of course, you can also operate by installing virtual machines locally.

Teach you how to build rocketmq high availability cluster!

Next, I will install two master and two slave servers through these two servers. First, I will plan their ports

First machine    42.192.77.73
#Port planning:
9876     NameServer1
10910    BrokerA-master
10921    BrokerB-slave
Second machine    39.103.144.86
#Port planning:
9876     NameServer2
10920    BrokerB-master
10911    BrokerA-slave

The drawing looks clearer

Teach you how to build rocketmq high availability cluster!

download

Go to the official website rocketmq.apache.org/ to get the latest download address, which is marked in the red box

Teach you how to build rocketmq high availability cluster!

The address here can be downloaded

cd /jackxu
wget https://mirror.bit.edu.cn/apache/rocketmq/4.8.0/rocketmq-all-4.8.0-bin-release.zip

After downloading, unzip it and change its name

unzip rocketmq-all-4.8.0-bin-release.zip
mv unzip rocketmq-all-4.8.0-bin-release.zip rocketmq

Configure the first machine

Download and unzip on both machines. In the rocketmq / conf directory, there are three recommended configuration modes:

  • 2m-2s-async (2 master and 2 slave asynchronous) — this paper adopts this method
  • 2m-2s-sync (2 master and 2 slave synchronization)
  • 2m noslave (2 main)

Now you need to modify the files in 2m-2s-async on both machines. Modify broker-a.properties on 42.192.77.73 machine

cd /jackxu/rocketmq/conf/2m-2s-async
vim  broker-a.properties

Modify cluster name

brokerClusterName=jackxu-cluster

Added content

#Broker   Listening port for external services
listenPort=10910
#Allow   Broker   Automatically create topic. It is recommended to open offline and close online
autoCreateTopicEnable=true
#Allow   Broker   Automatically create subscription groups. It is recommended to open offline and close online
autoCreateSubscriptionGroup=true
#Nameserver address, semicolon split
namesrvAddr=39.103.144.86:9876;42.192.77.73:9876
#Storage path
storePathRootDir=/jackxu/rocketmq/store/broker-a
#commitLog   Storage path
storePathCommitLog=/jackxu/rocketmq/store/broker-a/commitlog
#Consumption queue storage path storage path
storePathConsumeQueue=/jackxu/rocketmq/store/broker-a/consumequeue
#Message index storage path
storePathIndex=/jackxu/rocketmq/store/broker-a/index
#checkpoint   File storage path
storeCheckpoint=/jackxu/rocketmq/store/checkpoint
#abort   File storage path
abortFile=/jackxu/rocketmq/store/abort

Modify broker-b-s.properties

vim  broker-b-s.properties

Modify cluster name

brokerClusterName=jackxu-cluster

Added content

#Broker   Listening port for external services
listenPort=10921
#Allow   Broker   Automatically create topic. It is recommended to open offline and close online
autoCreateTopicEnable=true
#Allow   Broker   Automatically create subscription groups. It is recommended to open offline and close online
autoCreateSubscriptionGroup=true
#Nameserver address, semicolon split
namesrvAddr=39.103.144.86:9876;42.192.77.73:9876
#Storage path
storePathRootDir=/jackxu/rocketmq/store/broker-b-s
#commitLog   Storage path
storePathCommitLog=/jackxu/rocketmq/store/broker-b-s/commitlog
#Consumption queue storage path storage path
storePathConsumeQueue=/jackxu/rocketmq/store/broker-b-s/consumequeue
#Message index storage path
storePathIndex=/jackxu/rocketmq/store/broker-b-s/index
#checkpoint   File storage path
storeCheckpoint=/jackxu/rocketmq/store/checkpoint
#abort   File storage path
abortFile=/jackxu/rocketmq/store/abort

Configure second machine

Modify broker-b.properties on 39.103.144.86 machine

cd /jackxu/rocketmq/conf/2m-2s-async
vim  broker-b.properties

Modify cluster name

brokerClusterName=jackxu-cluster

Added content

#Broker   Listening port for external services
listenPort=10920
#Allow   Broker   Automatically create topic. It is recommended to open offline and close online
autoCreateTopicEnable=true
#Allow   Broker   Automatically create subscription groups. It is recommended to open offline and close online
autoCreateSubscriptionGroup=true
#Nameserver address, semicolon split
namesrvAddr=39.103.144.86:9876;42.192.77.73:9876
#Storage path
storePathRootDir=/jackxu/rocketmq/store/broker-b
#commitLog   Storage path
storePathCommitLog=/jackxu/rocketmq/store/broker-b/commitlog
#Consumption queue storage path storage path
storePathConsumeQueue=/jackxu/rocketmq/store/broker-b/consumequeue
#Message index storage path
storePathIndex=/jackxu/rocketmq/store/broker-b/index
#checkpoint   File storage path
storeCheckpoint=/jackxu/rocketmq/store/checkpoint
#abort   File storage path
abortFile=/jackxu/rocketmq/store/abort

Modify broker-a-s.properties

vim  broker-a-s.properties

Modify cluster name

brokerClusterName=jackxu-cluster

Added content

#Broker   Listening port for external services
listenPort=10911
#Allow   Broker   Automatically create topic. It is recommended to open offline and close online
autoCreateTopicEnable=true
#Allow   Broker   Automatically create subscription groups. It is recommended to open offline and close online
autoCreateSubscriptionGroup=true
#Nameserver address, semicolon split
namesrvAddr=39.103.144.86:9876;42.192.77.73:9876
#Storage path
storePathRootDir=/jackxu/rocketmq/store/broker-a-s
#commitLog   Storage path
storePathCommitLog=/jackxu/rocketmq/store/broker-a-s/commitlog
#Consumption queue storage path storage path
storePathConsumeQueue=/jackxu/rocketmq/store/broker-a-s/consumequeue
#Message index storage path
storePathIndex=/jackxu/rocketmq/store/broker-a-s/index
#checkpoint   File storage path
storeCheckpoint=/jackxu/rocketmq/store/checkpoint
#abort   File storage path
abortFile=/jackxu/rocketmq/store/abort

Create data directory

The first machine 42.192.77.73 is executed and only needs to be executed once

mkdir -p /jackxu/rocketmq/store/broker-a /jackxu/rocketmq/store/broker-a/consumequeue /jackxu/rocketmq/store/broker-a/commitlog /jackxu/rocketmq/store/broker-a/index /jackxu/rocketmq/logs /jackxu/rocketmq/store/broker-b-s /jackxu/rocketmq/store/broker-b-s/consumequeue /jackxu/rocketmq/store/broker-b-s/commitlog /jackxu/rocketmq/store/broker-b-s/index

The second machine is 39.103.144.86 and only needs to be executed once

mkdir -p /jackxu/rocketmq/store/broker-a-s /jackxu/rocketmq/store/broker-a-s/consumequeue /jackxu/rocketmq/store/broker-a-s/commitlog /jackxu/rocketmq/store/broker-a-s/index /jackxu/rocketmq/logs /jackxu/rocketmq/store/broker-b /jackxu/rocketmq/store/broker-b/consumequeue /jackxu/rocketmq/store/broker-b/commitlog /jackxu/rocketmq/store/broker-b/index

Start both nameservers

It is executed separately on two machines, & which means it is running in the background. By default, nameserver listens to port 9876

nohup sh /jackxu/rocketmq/bin/mqnamesrv >/jackxu/rocketmq/logs/mqnamesrv.log 2>&1 &

view log

tail -f /jackxu/rocketmq/logs/mqnamesrv.log

Start broker

Start up in the following order, – C is the configuration file of the specified broker

1. Start main a of 73

nohup sh /jackxu/rocketmq/bin/mqbroker -c /jackxu/rocketmq/conf/2m-2s-async/broker-a.properties > /jackxu/rocketmq/logs/broker-a.log 2>&1 &

2. Starting a from 86

nohup sh /jackxu/rocketmq/bin/mqbroker -c /jackxu/rocketmq/conf/2m-2s-async/broker-a-s.properties > /jackxu/rocketmq/logs/broker-a-s.log 2>&1 &

3. Start main B of 86

nohup sh /jackxu/rocketmq/bin/mqbroker -c /jackxu/rocketmq/conf/2m-2s-async/broker-b.properties > /jackxu/rocketmq/logs/broker-b.log 2>&1 &

4. Starting B from 73

nohup sh /jackxu/rocketmq/bin/mqbroker -c /jackxu/rocketmq/conf/2m-2s-async/broker-b-s.properties > /jackxu/rocketmq/logs/broker-b-s.log 2>&1 &

view log

tail -f /jackxu/rocketmq/conf/2m-2s-async/broker-a.properties
tail -f /jackxu/rocketmq/conf/2m-2s-async/broker-a-s.properties
tail -f /jackxu/rocketmq/conf/2m-2s-async/broker-b.properties
tail -f /jackxu/rocketmq/conf/2m-2s-async/broker-b-s.properties

Check whether the startup is successful

Enter the JPS command. There are four processes in total. No problem

Teach you how to build rocketmq high availability cluster!

However, it will not be so smooth in most cases. I have some pits in the installation:

  • 1. The JDK is not installed because this is my new server and there is no JDK environment on it. As we all know, rocketmq is written in Java, so it needs a Java environment to run. The JPS command is to view the java process

Teach you how to build rocketmq high availability cluster!

  • 2. There is not enough memory, because the default configuration file sets a lot of memory, but my machine has only 1 core 2G, which is certainly not enough, so it will report that there is not enough memory at startup

Teach you how to build rocketmq high availability cluster!

The solution is to modify the configuration file

vim /jackxu/rocketmq/bin/runserver.sh
vim /jackxu/rocketmq/bin/runbroker.sh

Reduce the memory size of nameserver and broker, and then restart

Teach you how to build rocketmq high availability cluster!

close command

When stopping the service, you should first stop the broker and then stop the nameserver.

cd /jackxu/rocketmq/bin
sh mqshutdown namesrv
sh mqshutdown broker

Web console installation

Unlike Kafka, rocket officially provides a visual console, which you can download here

https://github.com/apache/rocketmq-externals

This is an extension of rocketmq, which includes not only the console extension, but also the docking and extension of big data flume, HBase and other components.

download

Download the source code. Generally, the download speed will be slow. If you can’t wait, you can go to the online disk to download the link:

cd /jackxu
wget https://github.com/apache/rocketmq-externals/archive/master.zip

decompression

unzip master.zip

Modify profile

cd /jackxu/rocketmq-externals-master/rocketmq-console/src/main/resources
vim application.properties
#This is a spring   Boot project, we need to modify the parameters inside
#Modify port number:
server.port=7298
#Modify name   Server address (multiple addresses are separated by English semicolons)
rocketmq.config.namesrvAddr=39.103.144.86:9876;42.192.77.73:9876

pack

To make a jar package, you need to install Maven first. If the compilation is slow, you can replace it with the image of Taobao

cd /jackxu/rocketmq-externals-master/rocketmq-console
mvn clean package -Dmaven.test.skip=true

Start the jar package

cd target
java -jar rocketmq-console-ng-2.0.0.jar

visit

Visit  http://42.192.77.73:7298/  , You can see that it has been started successfully

Teach you how to build rocketmq high availability cluster!

Pit point

There are two pit stepping points here, which will report an error when starting

  • (1) 39.103.144.86:9876 when spring boot starts; 42.192.77.73:9876, my first reaction here is to close the firewall
systemctl disable firewalld.service 

I still can’t start it again, and then my second reaction is that the port is not open, telnet

telnet 42.192.77.73 9876

Sure enough, telnet doesn’t work. The reason is that I install it on the ECS. The default port is not open. I need to open it on the console. This is also an experience.

Teach you how to build rocketmq high availability cluster!

  • (2) Another error was reported when I started up again. I couldn’t connect to 172.26.182.88:9876. At that time, I was surprised that there was no such address in the address I wrote. How could I connect it, and the IP looked like an intranet address. Later, I opened the Alibaba cloud console and found that it was really the intranet address of the machine, and then I Baidu again, The probable reason is that rocketmq is an open source Alibaba machine, and that machine is an Alibaba cloud machine. Its source code should give priority to the intranet address of Alibaba cloud. The online method is to explicitly add the public network address to the configuration file.
#Add public IP
brokerIP1=39.103.144.86

Console introduction

  • Operation and maintenance: mainly setting nameserver and configuring vipchannel.
  • Cockpit: dashboard on the console. You can view the number and trend of messages by broker and subject respectively.
  • Cluster: the cluster situation of the entire rocketmq, including fragmentation, number, address, version, TPS for message production and message consumption, which can be used as a data indicator during performance test.
  • Topic: that is, topic. You can add / update topic and view the information of topic, such as status, routing, consumer management and sending messages.
  • Consumer: you can view / create a new consumer group in the current broker, including consumer information and consumption progress.
  • Producer: you can view the producer group under the production group in the current broker, including producer information and producer status.
  • Message: you can query specific messages according to topic, messageid and messagekey.
  • User center: language switching is related to login (login requires opening the corresponding configuration in the console configuration, and login is not required by default).

The most commonly used are cluster, theme, consumer and message.

Profile description

The following describes the properties in the installation rocketmq configuration file. First, the cluster name is the same. The cluster name of the above four machines is brokerclustername = jackxu cluster. Second, connect to the same nameserver, namesrvaddr = 39.103.144.86:9876; 42.192.77.73:9876。 In the configuration file, brokerid = 0 represents master and brokerid = 1 represents slave.

There are also these two properties in the configuration file

Teach you how to build rocketmq high availability cluster!

The broker role can be configured as sync in the master broker_ Master or Async_ Master is uniformly configured as slave in the slave broker

Teach you how to build rocketmq high availability cluster!

Teach you how to build rocketmq high availability cluster!

flow chart

Teach you how to build rocketmq high availability cluster!

The recommended configuration is asynchronous disk brushing + synchronous replication.

Ha and failover

In previous versions, rocketmq only had a master / slave deployment mode. There was one master in a group of brokers, with 0 to more than one slave. This mode provides a certain degree of high availability.

When the master is suspended, the slave can still provide read services. By default, all reads and writes are on the master. If slavereadenable = true is enabled, the slave can also participate in the read load. However, the default switch to read from the server with brokerid 1 is only when the backlog of messages on the master exceeds 40% of the physical memory. The specific switch to which machine is controlled by the which brokerwhenconsumeslowly parameter.

org.apache.rocketmq.common.subscription.SubscriptionGroupConfig
private long whichBrokerWhenConsumeSlowly=1;

Due to the existence of multiple masters, when one master hangs, it can be written to other masters.

In version 4.5.0 released in March 2019, rocketmq uses the dledger technology to solve the problem of automatic master selection. Dledger is a commitlog repository based on raft protocol. It is also the key to rocketmq’s new high availability multi replica architecture. Its advantage is that it does not need to introduce external components, the automatic master selection logic is integrated into the process of each node, and the master selection can be completed through communication between nodes.

Architecture diagram

Teach you how to build rocketmq high availability cluster!

In this case, the commitlog is managed by dledger and has the function of selecting the master. It is not enabled by default. If it needs to be enabled, the following configuration needs to be added to the configuration file:

#   Enable dledger
    enableDLegerCommitLog=true
    #   DLedger   Raft   Group name
    dLegerGroup=broker-a
    #   DLedger   The address and port of each node in the group. At least 3 nodes are required
    dLegerPeers=n0-192.168.44.163:10911;n1-192.168.44.164:10911;n2-192.168.44.165:10911
    #   Current node ID
    dLegerSelfId=n0

epilogue

When learning a middleware, children must practice the installation. First, they should experience the atmosphere of the installation process and understand some parameter configurations. Although we may not touch it at ordinary times, we still need to learn, not necessarily fine, at least.

Source: Jack_ xu |https://juejin.cn/post/692950…

Teach you how to build rocketmq high availability cluster!