Rocketmq 4.7.1 environment construction, cluster and springboot integration MQ

Time:2021-6-22

Reading guide

I’ve learned ActiveMQ before, but the amount of concurrency is not very largeI’ll take you there directlySo I learned from rocketmq, which is open source of Alibaba. It is said that queues can be stackedBillion level. The following is a comparison chart of message queues found on the Internet, for reference only

deploy

Official website

I’ll take you there directly

Preconditions

  1. 64 bit operating system and Linux / Unix / Mac are recommended;
  2. 64 bit JDK 1.8 +;
  3. Maven 3.2.x;
  4. Git;
  5. 4G memory + available disk for broker server

download

Address:https://downloads.apache.org/rocketmq/4.7.1/rocketmq-all-4.7.1-source-release.zip 

Baidu cloud disk:

Link: https://pan.baidu.com/s/1luq_ MwxSn8k_ Bugrnqsjwg password: varj

Install dependencies

  1. jdk:I’ll take you there directly
  2. maven:I’ll take you there directly
  3. Git installation:yum install -y git
export JAVA_HOME=/opt/soft/jdk1.8.0_202
export PATH=$JAVA_HOME/bin:$PATH
export CLASPATH=.:$JAVA_home/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME PATH CLASSPATH
export MAVEN_HOME=/opt/soft/apache-maven-3.6.3
export PATH=$PATH:$MAVEN_HOME/bin

MQ upload to Linux

decompression

 

Maven compilation

 

Start nameserver

Background startup mode

nohup sh bin/mqnamesrv &

Insufficient memory when nameserver starts (problem solved)

Find runserver.sh and modify Java_ OPT

VIM / bin / runserver.sh configuration

Start broker

nohup sh bin/mqbroker -n localhost:9876 &

Syntax: nohup sh bin / mqbroker - N nameserver service IP address

 

Broker out of memory (problem solving)

Find runbroker.sh and modify Java_ OPT

VIM / bin / runbroker.sh configuration

All services started successfully

 

Simulated consumption

export NAMESRV_ADDR=localhost:9876

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer

sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer

Open two console, connect one Linux

be careful

  Nameserver default port number: 9876; Broker default port number: 10911

Visual console

Official website address

I’ll take you there directly

Baidu cloud disk

Link: https://pan.baidu.com/s/1mdEGkq-JBTy1wtNmFPkmDg   Password: v6bq

decompression

Installation and compilation

Enter / opt / soft / rocketmq externals master / rocketmq console
Compile: MVN clean package - dmaven. Test. Skip = true

Modify rocketmq.config.namesrvaddr of application.properties

Compile and package

start-up

Enter the target directory and start Java jar

Daemons start: nohup Java - jar rocketmq-console-ng-2.0.0.jar&

Springboot integrates rocketmq (producer)

Create a springboot project

I’ll take you there directly

Project structure

Join dependency

pom.xml

4.0.0
    
        org.springframework.boot
        spring-boot-starter-parent
        2.3.1.RELEASE
         
    
    com.ybchen
    ybchen-mq
    0.0.1-SNAPSHOT
    ybchen-mq
    Demo project for Spring Boot

    
        1.8
    

    
        
            org.springframework.boot
            spring-boot-starter-web
        

        
            org.springframework.boot
            spring-boot-starter-test
            test
            
                
                    org.junit.vintage
                    junit-vintage-engine
                
            
        
        
        
            org.apache.rocketmq
            rocketmq-client
            4.7.1
        
        
            org.apache.rocketmq
            rocketmq-common
            4.7.1
        
    

    
        
            
                org.springframework.boot
                spring-boot-maven-plugin

PayProducer.java

package com.ybchen.ybchenmq.jms;

import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.DefaultMQProducer;
import org.springframework.stereotype.Component;

/**
 *Message producer
 */
@Component
public class PayProducer {
    /**
     *The group to which the producer belongs
     */
    private String producerGroup = "pay_group";
    /**
     *MQ address, pay attention to open the port number or close the firewall
     */
    private String nameServerAddr = "192.168.199.100:9876";
    private DefaultMQProducer producer;

    public PayProducer() {
        producer = new DefaultMQProducer(producerGroup);
        //Specify the nameserver address, and multiple addresses can be used as the address; separate
        //For example, producer. Setnamesrvaddr ("192.168.199.100:9876; 192.168.199.101:9876; 192.168.199.102:9876")
        producer.setNamesrvAddr(nameServerAddr);
        start();
    }

    /**
     *Access to producers
     * @return
     */
    public DefaultMQProducer getProducer() {
        return this.producer;
    }

    /**
     *Open, the object must be called once before use, and can only be initialized once
     */
    public void start() {
        try {
            this.producer.start();
        } catch (MQClientException e) {
            e.printStackTrace();
        }
    }

    /**
     *Close, generally in the application context, use the context listener to close
     */
    public void shutdown() {
        this.producer.shutdown();
    }
}

PayController.java

package com.ybchen.ybchenmq.controller;

import com.ybchen.ybchenmq.jms.PayProducer;
import org.apache.rocketmq.client.exception.MQBrokerException;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.client.producer.SendResult;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.remoting.exception.RemotingException;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

/**
 * @ClassName:PayController
 *@ Description: payment
 * @Author:chenyb
 *@ date: 2020 / 10 / 18 2:47 PM
 * @Versiion:1.0
 */
@RestController
@RequestMapping("/api/v1")
public class PayController {
    @Autowired
    private PayProducer payProducer;

    private static final String TOPIC = "ybchen_pay_topic";

    /**
     *Payment callback
     *
     * @param text
     * @return
     */
    @RequestMapping("pay_cb")
    public Object callback(String text) {
        /**
         *String topic: topic
         *String Tags: secondary classification
         *Byte [] body: body message byte array
         */
        Message message = new Message(TOPIC,"tag_a",("hello ybchen ==>"+text).getBytes());
        try {
            SendResult send = payProducer.getProducer().send(message);
            System.out.println("send------>"+send);
        } catch (MQClientException e) {
            e.printStackTrace();
        } catch (RemotingException e) {
            e.printStackTrace();
        } catch (MQBrokerException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        return "ok";
    }
}

test

Common mistakes

Error one

org.apache.rocketmq.remoting.exception.RemotingTooMuchRequestException:
sendDefaultImpl call timeout
Reason: there are multiple network cards in alicloud. Rocketmq will choose an IP to use according to the current network card. When your machine has multiple network cards, there may be problems. For example, there are two IP, one public IP and one private IP on the machine. Therefore, you need to configure broker.conf to specify the IP of the current public network, and then restart the broker


Modify configuration / opt / soft / rocketmq-all-4.7.1-source-release / distribution / target / rocketmq-4.7.1/rocketmq-4.7.1/conf/broker.conf
Add this configuration: brokerip1 = XXX. XXX. XXX. XXX

Start command: nohup sh bin / mqbroker - n localhost:9876 -c  ./conf/broker.conf &

Error 2

MQClientException: No route info of this topic, TopicTest1

Reason: the topic is created automatically by the broker, and the user does not create the topic manually, or the network between the broker and the nameserver is disconnected

solve:
    View configuration through sh bin / mqbroker - M
    If autocreatetopicenable = true, the topic will be created automatically

CentOS 7: systemctl stop firewalld

Error 3

The console can't view the data, indicating 10909 connection error

Reason: Rocket turns on VIP channel by default, and the port number of VPI channel is 10911-2 = 10909

Solution: add a port to alicloud security group: 10909

Error 4

Unable to create topic automatically: the client version should be consistent with the server version

4.7.1 is installed on the server

When introducing dependencies
        
        
            org.apache.rocketmq
            rocketmq-client
            4.7.1
        
        
            org.apache.rocketmq
            rocketmq-common
            4.7.1

Retrieve message sending

Springboot integrates rocketmq (consumer)

Create a springboot project

 

Project structure

Join dependency

4.0.0
    
        org.springframework.boot
        spring-boot-starter-parent
        2.3.1.RELEASE
         
    
    com.ybchen
    ybchen-mq
    0.0.1-SNAPSHOT
    ybchen-mq
    Demo project for Spring Boot

    
        1.8
    

    
        
            org.springframework.boot
            spring-boot-starter-web
        

        
            org.springframework.boot
            spring-boot-starter-test
            test
            
                
                    org.junit.vintage
                    junit-vintage-engine
                
            
        
        
        
            org.apache.rocketmq
            rocketmq-client
            4.7.1
        
        
            org.apache.rocketmq
            rocketmq-common
            4.7.1
        
    

    
        
            
                org.springframework.boot
                spring-boot-maven-plugin

PayConsumer.java

package com.ybchen.ybchenmqconsumer.jms;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyContext;
import org.apache.rocketmq.client.consumer.listener.ConsumeConcurrentlyStatus;
import org.apache.rocketmq.client.consumer.listener.MessageListenerConcurrently;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.Message;
import org.apache.rocketmq.common.message.MessageExt;
import org.springframework.stereotype.Component;

import java.io.UnsupportedEncodingException;
import java.util.List;

/**
 * @ClassName:PayConsumer
 *@ Description: Consumer
 * @Author:chenyb
 *@ date: 2020 / 10 / 18 4:13 PM
 * @Versiion:1.0
 */
@Component
public class PayConsumer {
    /**
     *The group to which the producer belongs
     */
    private String producerGroup = "pay_consumer_group";
    /**
     *MQ address, pay attention to open the port number or close the firewall
     */
    private String nameServerAddr = "192.168.199.100:9876";
    /**
     *Subscribe to topics
     */
    private String topic = "ybchen_pay_topic";
    private DefaultMQPushConsumer consumer;

    public PayConsumer() throws MQClientException {
        consumer = new DefaultMQPushConsumer(producerGroup);
        //Specify the nameserver address, and multiple addresses can be used as the address; separate
        //For example, producer. Setnamesrvaddr ("192.168.199.100:9876; 192.168.199.101:9876; 192.168.199.102:9876")
        consumer.setNamesrvAddr(nameServerAddr);
        //Set the place of consumption and start from the last one
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
        //Subscribe to the topic and listen to the tags under the topic
        consumer.subscribe(topic, "*");
        //Annotate a listener
        //Lambda mode
//        consumer.registerMessageListener((MessageListenerConcurrently) (msg, context) -> {
//            try {
//                Message message = msg.get(0);
//                System.out.printf("%s Receive New Messages: %s %n",
//                        Thread.currentThread().getName(), new String(msg.get(0).getBody()));
//// theme
//                String topic = message.getTopic();
//// message content
//                String body = null;
//                body = new String(message.getBody(), "utf-8");
//// secondary classification
//                String tags = message.getTags();
//// key
//                String keys = message.getKeys();
//                System.out.println("topic=" + topic + ", tags=" + tags + ", keys=" + keys + ", msg=" + body);
//                return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
//            } catch (UnsupportedEncodingException e) {
//                e.printStackTrace();
//                return ConsumeConcurrentlyStatus.RECONSUME_LATER;
//            }
//        });

        //General way
        consumer.registerMessageListener(new MessageListenerConcurrently() {
            @Override
            public ConsumeConcurrentlyStatus consumeMessage(List list, ConsumeConcurrentlyContext consumeConcurrentlyContext) {
                try {
                    Message message = list.get(0);
                    System.out.printf("%s Receive New Messages: %s %n",
                            Thread.currentThread().getName(), new String(list.get(0).getBody(),"utf-8"));
                    //Themes
                    String topic = message.getTopic();
                    //Message content
                    String body = null;
                    body = new String(message.getBody(), "utf-8");
                    //Secondary classification
                    String tags = message.getTags();
                    //Key
                    String keys = message.getKeys();
                    System.out.println("topic=" + topic + ", tags=" + tags + ", keys=" + keys + ", msg=" + body);
                    return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
                } catch (UnsupportedEncodingException e) {
                    e.printStackTrace();
                    return ConsumeConcurrentlyStatus.RECONSUME_LATER;
                }
            }
        });
        consumer.start();
        System.out.println("consumer start ..........");
    }
}

application.properties

server.port=8081

Test producer consumer

Analysis of MQ cluster architecture pattern

Single node

advantage

Local development test, simple configuration, synchronous brush disk message will not be lost

shortcoming

Unreliable. If it goes down, the service will be unavailable

Master slave (asynchronous, synchronous dual write)

advantage

Synchronous double write messages are not lost, asynchronous replication has a small amount of loss, the master node is down, the slave node can provide external consumption of messages, but does not support writing

shortcoming

The master / standby system has short message delay and millisecond level. At present, it does not support automatic switching. It needs script or other programs to detect, then stop the broker and restart the slave node to become the master node

Double master

advantage

The configuration is simple, and the message reliability can be guaranteed by configuring RAID disk array, and a small amount of messages can be lost by asynchronous disk brushing

shortcoming

During the outage of the master, messages that are not consumed cannot be sent before the machine is restored, and the real-time performance will be affected

Dual master, dual slave, multi master, multi slave mode (asynchronous replication)

advantage

The disk is damaged, the message loss is very small, and the real-time performance of the message will not be affected. After the Master goes down, consumers can still consume from slave

shortcoming

The master / standby system has a short message delay of milliseconds. If the Master goes down and the disk is damaged, a small amount of messages will be lost

Double master, double slave, multi master and multi slave mode (synchronous double write)

advantage

In the synchronous dual write mode, only when the master and standby write successfully, can they return success to the application. The service availability and data availability are very high

shortcoming

The performance is slightly lower than that of asynchronous replication mode. After the primary machine is down, the standby machine cannot be automatically switched to the host machine

recommend

  1. Master slave (asynchronous, synchronous dual write)
  2. Dual master, dual slave, multi master, multi slave mode (asynchronous replication)
  3. Double master, double slave, multi master and multi slave mode (synchronous double write)

Construction of master-slave cluster

preparation

Prepare two machines with IP addresses of 192.168.199.100; 192.168.199.101;

Environment: rocketmq4.7.1 + jdk8 + Maven + CentOS 7

Start two nameservers

Start the nameservers of both machines

Path: opt / soft / rocketmq-all-4.7.1-source-release / distribution / target / rocketmq-4.7.1 / rocketmq-4.7.1

Start: nohup sh bin / mqnamesrc&

Edit and start roccketmq

Master node

Enter / opt / soft / rocketmq-all-4.7.1-source-release / distribution / target / rocketmq-4.7.1/rocketmq-4.7.1/conf/2m-2s-async


Edit and modify as follows: VIM broker-a.properties 
namesrvAddr=192.168.199.100:9876;192.168.199.101:9876
brokerClusterName=YbChenCluster
brokerName=broker-a
brokerId=0
deleteWhen=04
fileReservedTime=48
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH


Start: nohup sh bin / mqbroker - C conf / 2m-2s-async / broker / broker-a.properties&

Slave node

Enter / opt / soft / rocketmq-all-4.7.1-source-release / distribution / target / rocketmq-4.7.1/rocketmq-4.7.1/conf/2m-2s-async


Edit and modify as follows: VIM broker-a-s.properties 
namesrvAddr=192.168.199.100:9876;192.168.199.101:9876
brokerClusterName=YbChenCluster
brokerName=broker-a
brokerId=1
deleteWhen=04
fileReservedTime=48
brokerRole=SLAVE
flushDiskType=ASYNC_FLUSH


Start: nohup sh bin / mqbroker - C conf / 2m-2s-async / broker / broker-a-s.properties&

matters needing attention

  1. Namesrvaddr: same
  2. Brokerclustername: same
  3. Brokername: same
  4. brokerId:Different, 0 is the master node
  5. Delete when: same
  6. Filereservedtime: same
  7. brokerRole:Different, divided into Async_ MASTER、SLAVE
  8. Flushdisktype: same

Start broker

Use control desk

Use 192.168.199.100 to modify the configuration

192.168.199.100

Enter / opt / soft / rocketmq externals master / rocketmq console / SRC / main / resources


Modify the configuration file: VIM application.properties

rocketmq.config.namesrvAddr=192.168.199.100:9876;192.168.199.101:9876


compile

Switch to / opt / soft / rocketmq externals master / rocketmq console
pack:
mvn clean
mvn install -Dmaven.test.skip=true


start-up

Enter / opt / soft / rocketmq externals master / rocketmq console / target
守护进程方式start-up:nohup java -jar rocketmq-console-ng-2.0.0.jar &

Cluster testing

Fault drill

The simulation master is suspended, but the slave can still be consumed. At this time, it cannot be written. After the master restarts, it can continue to write (the data will not be consumed repeatedly). The following contents are continuous

summary

Well, so far, the master-slave has been built.

  Broker scorebyMaster and slaveA master can correspond to multiple slaves, butA slave can only correspond to one master, master and slaveMatch with the same broker name, differentMaster or slave when defined by broker ID

Broker establishes a long connection to all nameserver nodes, registers topics regularly and sends metadata information

Nameserver scans the connections of all surviving brokers regularly (the default is 2 minutes). If it fails to respond after a certain time, the connection will be disconnected (heartbeat detection). However, the consumer client cannot perceive it. The consumer gets the latest information of topic from nameserver regularly (30 seconds). Therefore, when the broker is unavailable, the consumer needs at most 30 seconds to discover it

  Only master can writeSlave does not allow writing and can only synchronizeThe synchronization policy depends on the master configuration

  The client can consume from master and slave. By default, consumers consume from masterIf the client perceives the broker outage from nameserver after the master hangs up, it will consume from slave, perceives non real time, and has a certain lag. Slave can’t guarantee 100% synchronization of the master, and there will be a small amount of message loss. Once the master is restored, the unsynchronized messages will be consumed.

If the number of consumer instances is more than the total number of message queues, the extra consumer instances will not be assigned to the queue, and the messages will not be consumed, and the load will not be shared. Therefore, it is necessary to control the total number of queues to be greater than the number of consumers.

Scene simulation

Production and consumption and treatment

Producer retrying

  • Message retrying (to ensure high reliability of data). It supports retrying internally. The default number is 2
  • If the network condition is poor or cross cluster, it is recommended to modify it several times

The producer sets the number of retries and sets a unique key (general unique identifier)

Consumer retrying

  • Causes: abnormal message processing, problems from broker to consumer, such as network failure, consumption processing failure, ACK return failure, etc
  • be careful
    • Retry interval configuration, default to each messageUp to 16 retries
    • Manual compensation for exceeding the number of retries
    • De emphasis on the consumer side
    • No matter how many times a message is retryed, the number of retrying messages is zeroMessage ID, key will not change
    • Consumption retrial is only effective for cluster consumption mode; Broadcast mode does not provide the feature of failure retrial, that is, after consumption failure, the failure message will not be retried and continue to consume new messages

Set broadcast mode

Simulated message retransmission

Asynchronous message sending and callback

Application scenarios

For example, 12306 issues tickets asynchronously after payment, which requires high performance and supports higher concurrency. After successful callback, the corresponding business (onsuccess) is triggered

Official examples

I’ll take you there directly

Transforming producers

demonstration

Onsuccess: because it is asynchronous, you can record logs here
Onexception: compensation mechanism, according to the actual situation, to see whether to try again

Oneway (no waiting)

Application scenarios

Mainly do log collection, suitable for high performance requirements, but not high reliability scenarios.

Delayed message combat

What is delayed message

  • Producer sends a message to the message queue rocketmq server, but it does not expect the message to be delivered immediately. Instead, it delays the message to be delivered to the consumer at a certain time after the current time point for consumption. The message is a timed message, and currently supports fixed precision messages
  • Delay message level, 1… 18
1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h

Application scenarios

  • Some timing tasks are triggered by messages, such as sending reminder messages to users at a fixed time point
  • There are time window requirements for message production and consumption: for example, in the scenario of closing an order without payment due to overtime in tmall e-commerce transaction, a delay message will be sent when the order is created. This message will be delivered to the consumer in 30 minutes. After receiving this message, the consumer needs to judge whether the corresponding order has been paid. If the payment is not completed, close the order. Ignore if payment has been completed.

Changing producers

Producer message queue selector

brief introduction

The production message is delivered to the specified queue under topic using the message queue selector

Application scenarios

  • Sequential message
  • Load sharing

By default, the number of queues under topic is 4, which can be configured

Support synchronous and asynchronous sending of specified message queue

The number of selected queues must be less than the configured number, otherwise an error will occur

benefit

If the traffic of a product in the queue increases sharply and is randomly allocated, the whole topic will not be used. If it is assigned to the queue, other queues will not affect the use if the queue is broken.

Transforming producers

Synchronous sending

Send result = send_ OK,msg=SendResult [sendStatus=SEND_ OK, msgId=AC1068013E3F18B4AAC276723EAC0000, offsetMsgId=C0A8C76400002A9F000000000009B536, messageQueue=MessageQueue [topic=ybchen_ pay_ topic, brokerName=broker-a, queueId=0], queueOffset=1]
Send result = send_ OK,msg=SendResult [sendStatus=SEND_ OK, msgId=AC1068013E3F18B4AAC27672BCD50001, offsetMsgId=C0A8C76400002A9F000000000009B602, messageQueue=MessageQueue [topic=ybchen_ pay_ topic, brokerName=broker-a, queueId=0], queueOffset=2]
Send result = send_ OK,msg=SendResult [sendStatus=SEND_ OK, msgId=AC1068013E3F18B4AAC27672CAA20002, offsetMsgId=C0A8C76400002A9F000000000009B6CF, messageQueue=MessageQueue [topic=ybchen_ pay_ topic, brokerName=broker-a, queueId=0], queueOffset=3]

You can see it printed, queueid = 0

Asynchronous sending

Producer code modification

@Autowired
    private PayProducer payProducer;

    private static final String TOPIC = "ybchen_pay_topic";

    /**
     *Payment callback
     *
     * @param text
     * @return
     */
    @RequestMapping("pay_cb")
    public Object callback(String text) {
        /**
         *String topic: topic
         *String Tags: secondary classification
         *Byte [] body: body message byte array
         */
        Message message = new Message(TOPIC, "tag_a", text.getBytes());
        //The producer uses the message queue selector to post to the specified queue under topic. Arg can only be less than or equal to 4
//        try {
//            SendResult sendResult = payProducer.getProducer().send(message, new MessageQueueSelector() {
//                @Override
//                public MessageQueue select(List list, Message message, Object o) {
//                    int queueNum=Integer.parseInt(o.toString());
//                    return list.get(queueNum);
//                }
//            }, 0);
//System. Out. Printf ("send result =% s, MSG =% s", sendresult. Getsendstatus(), sendresult) ";
//        } catch (MQClientException e) {
//            e.printStackTrace();
//        } catch (RemotingException e) {
//            e.printStackTrace();
//        } catch (MQBrokerException e) {
//            e.printStackTrace();
//        } catch (InterruptedException e) {
//            e.printStackTrace();
//        }
        //Send asynchronously to the specified queue
        try {
            payProducer.getProducer().send(message, new MessageQueueSelector() {
                @Override
                public MessageQueue select(List list, Message message, Object o) {
                    int queueNum = Integer.parseInt(o.toString());
                    return list.get(queueNum);
                }
            }, 3, new SendCallback() {
                @Override
                public void onSuccess(SendResult sendResult) {
                    System. Out. Printf ("send result =% s, MSG =% s", sendresult. Getsendstatus(), sendresult) ";
                }

                @Override
                public void onException(Throwable e) {
                    e.printStackTrace();
                }
            });
        } catch (MQClientException e) {
            e.printStackTrace();
        } catch (RemotingException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        //1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
        //message.setDelayTimeLevel(2);
//        try {
//            SendResult send = payProducer.getProducer().send(message);
//            System.out.println("send------>"+send);
//        } catch (MQClientException e) {
//            e.printStackTrace();
//        } catch (RemotingException e) {
//            e.printStackTrace();
//        } catch (MQBrokerException e) {
//            e.printStackTrace();
//        } catch (InterruptedException e) {
//            e.printStackTrace();
//        }
        //Asynchronous sending
//        try {
//            payProducer.getProducer().send(message, new SendCallback() {
//                @Override
//                public void onSuccess(SendResult sendResult) {
//System. Out. Printf ("send result =% s, MSG =% s", sendresult. Getsendstatus(), sendresult) ";
//                }
//
//                @Override
//                public void onException(Throwable e) {
//                    e.printStackTrace();
//// the compensation mechanism is used according to the actual situation to see whether to try again
//                }
//            });
//        } catch (MQClientException e) {
//            e.printStackTrace();
//        } catch (RemotingException e) {
//            e.printStackTrace();
//        } catch (InterruptedException e) {
//            e.printStackTrace();
//        }
        return "ok";
    }

Application scenarios of sequential message

brief introduction

Sequential messages can be applied to e-commerce, securities system and order system.

What is a sequential system?

The production and consumption of messages are in the same order

Global order

All messages under topic should be ordered (rarely used)

  1. The performance requirement is not high. All messages are released and consumed in strict accordance with FIFO (first in first out) principle. The parallelism becomes the bottleneck of the message system and the throughput is not enough
  2. In securities processing, taking RMB to us dollar as an example, in the case of the same price, the first bidder has priority, and then it can publish and consume in a global order

Local order

Just ensure that a group of messages are consumed sequentially (used in rocketmq)

  1. High performance requirements
  2. For e-commerce order creation, the creation order message, order payment message, order refund message, order logistics message and order transaction success message related to the same order will be released and consumed in sequence

Sequential release

For a given topic, the client sends messages in a certain order

Sequential consumption

For a given topic, messages are received in a certain order, that is, the messages sent first will be received by the client first

matters needing attention

  1. Asynchronous sending is not supported for sequential messagesOtherwise, there will be no guarantee of sequential consumption
  2. Broadcast mode is not supported for sequential messages

Official examples

I’ll take you there directly

Transform producer code

Create productorder.java

 

package com.ybchen.ybchenmq.entity;

import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;

/**
 * @ClassName:ProductOrder
 *@ Description: order
 * @Author:chenyb
 *@ date: 2020 / 10 / 25 12:56 PM
 * @Versiion:1.0
 */
public class ProductOrder implements Serializable {
    /**
     *Order ID
     */
    private long orderIdl;
    /**
     *Order operation type
     */
    private String type;

    public long getOrderIdl() {
        return orderIdl;
    }

    public void setOrderIdl(long orderIdl) {
        this.orderIdl = orderIdl;
    }

    public String getType() {
        return type;
    }

    public void setType(String type) {
        this.type = type;
    }

    public ProductOrder() {

    }

    public ProductOrder(long orderIdl, String type) {
        this.orderIdl = orderIdl;
        this.type = type;
    }

    @Override
    public String toString() {
        return "ProductOrder{" +
                "orderIdl=" + orderIdl +
                ", type='" + type + '\'' +
                '}';
    }

    /**
     *Simulating batch creation of entity classes
     * @return
     */
    public static List getOrderList(){
        List list=new ArrayList<>();
        List. Add (new product order (111l, "create order");
        List. Add (new product order (222l, "create order");
        List. Add (new product order (333l, "create order");
        List. Add (new product order (111l, "payment order");
        List. Add (new product order (222l, "payment order");
        List. Add (new product order (111l, "complete order");
        List. Add (new product order (222l, "complete order");
        List. Add (new product order (333l, "payment order");
        List. Add (new product order (333l, "complete order");
        return list;
    }
}

Control layer: paycontroller.java

 

@Autowired
    private PayProducer payProducer;

    private static final String TOPIC = "ybchen_pay_topic";
    private static final String TOPIC_ORDER = "ybchen_pay_order_topic";

    @RequestMapping("pay_order")
    public Object payOrder() throws Exception{
        //Get order number
        List list=ProductOrder.getOrderList();
        for (int i = 0; i < list.size(); i++) {
            ProductOrder order=list.get(i);
            Message message=new Message(TOPIC_ORDER,
                    "",
                    order.getOrderIdl()+"",
                    order.toString().getBytes());
            //Send, the same order ID into the same queue
           SendResult sendResult =payProducer.getProducer().send(message, new MessageQueueSelector() {
                @Override
                public MessageQueue select(List mqs, Message message, Object arg) {
                    Long id=(Long)arg;
                    long index=id%mqs.size();
                    return mqs.get((int) index);
                }
            },order.getOrderIdl());
           //Printout results
            System. Out. Printf ("send result =% s, sendresult =% s, OrderID =% s, type =% s'",
                    sendResult.getSendStatus(),
                    sendResult.toString(),
                    order.getOrderIdl(),
                    order.getType());

        }
        return "ok";
    }

Transform consumers

 

package com.ybchen.ybchenmqconsumer.jms;

import org.apache.rocketmq.client.consumer.DefaultMQPushConsumer;
import org.apache.rocketmq.client.consumer.listener.*;
import org.apache.rocketmq.client.exception.MQClientException;
import org.apache.rocketmq.common.consumer.ConsumeFromWhere;
import org.apache.rocketmq.common.message.MessageExt;
import org.apache.rocketmq.common.protocol.heartbeat.MessageModel;
import org.springframework.stereotype.Component;

import java.util.List;

/**
 * @ClassName:PayOrderConsumer
 *@ Description: consumer order
 * @Author:chenyb
 *@ date: 2020 / 10 / 18 4:13 PM
 * @Versiion:1.0
 */
@Component
public class PayOrderConsumer {
    /**
     *The group to which the producer belongs
     */
    private String producerGroup = "pay_order_consumer_group";
    /**
     *MQ address, pay attention to open the port number or close the firewall
     */
    private String nameServerAddr = "192.168.199.100:9876;192.168.199.101:9876";
    /**
     *Subscription topic, order
     */
    private static final String TOPIC_ORDER = "ybchen_pay_order_topic";
    private DefaultMQPushConsumer consumer;

    public PayOrderConsumer() throws MQClientException {
        consumer = new DefaultMQPushConsumer(producerGroup);
        //Specify the nameserver address, and multiple addresses can be used as the address; separate
        //For example, producer. Setnamesrvaddr ("192.168.199.100:9876; 192.168.199.101:9876; 192.168.199.102:9876")
        consumer.setNamesrvAddr(nameServerAddr);
        //Set the place of consumption and start from the last one
        consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_LAST_OFFSET);
        //Subscribe to the topic and listen to the tags under the topic
        consumer.subscribe(TOPIC_ORDER, "*");
        //The default mode is cluster mode. The broadcast mode does not support retrying
        consumer.setMessageModel(MessageModel.CLUSTERING);
        //Annotate a listener
        consumer.registerMessageListener(new MessageListenerOrderly() {
            @Override
            public ConsumeOrderlyStatus consumeMessage(List list,
                                                       ConsumeOrderlyContext consumeOrderlyContext) {
                MessageExt msg=list.get(0);
                System.out.printf("%s Receive New Messages: %s %n",Thread.currentThread().getName(),
                        new String(msg.getBody()));
                return ConsumeOrderlyStatus.SUCCESS;
            }
        });
        consumer.start();
        System.out.println("consumer order start ..........");
    }
}

Test sequence message

One producer, one consumer

You can see that the consumption is a little slow, because I have installed two virtual machines locally to be one master and one slave. The consumption order is correct. They are all in the following order: create order, pay order and complete order

2020-10-25 13:52:31.822  INFO 1473 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-10-25 13:52:31.822  INFO 1473 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2020-10-25 13:52:31.825  INFO 1473 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 3 ms
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D46F0000, offsetMsgId=C0A8C76400002A9F000000000009C8B2, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 3], queueoffset = 6], OrderID = 111, type = create order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4930001, offsetMsgId=C0A8C76400002A9F000000000009C9A5, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 2], queueoffset = 6], OrderID = 222, type = create order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4A90002, offsetMsgId=C0A8C76400002A9F000000000009CA98, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 1], queueoffset = 6], OrderID = 333, type = create order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4C00003, offsetMsgId=C0A8C76400002A9F000000000009CB8B, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 3], queueoffset = 7], OrderID = 111, type = payment order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4CC0004, offsetMsgId=C0A8C76400002A9F000000000009CC7E, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 2], queueoffset = 7], OrderID = 222, type = payment order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4D00005, offsetMsgId=C0A8C76400002A9F000000000009CD71, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 3], queueoffset = 8], OrderID = 111, type = complete order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4D30006, offsetMsgId=C0A8C76400002A9F000000000009CE64, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 2], queueoffset = 8], OrderID = 222, type = complete order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4DE0007, offsetMsgId=C0A8C76400002A9F000000000009CF57, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 1], queueoffset = 7], OrderID = 333, type = payment order
Send result = send_ OK,sendResult=SendResult [sendStatus=SEND_ OK, msgId=AC10680105C118B4AAC27E92D4F80008, offsetMsgId=C0A8C76400002A9F000000000009D04A, messageQueue=MessageQueue [topic=ybchen_ pay_ order_ Topic, brokername = broker-a, queueid = 1], queueoffset = 8], OrderID = 333, type = complete order
ConsumeMessageThread_ 1 receive new messages: productorder {orderidl = 333, type ='create order '} 
ConsumeMessageThread_ 1 receive new messages: productorder {orderidl = 333, type ='pay order '} 
ConsumeMessageThread_ 1 receive new messages: productorder {orderidl = 333, type ='complete order '} 
ConsumeMessageThread_ 2 receive new messages: productorder {orderidl = 222, type ='create order '} 
ConsumeMessageThread_ 2 receive new messages: productorder {orderidl = 222, type ='pay order '} 
ConsumeMessageThread_ 2 receive new messages: productorder {orderidl = 222, type ='complete order '} 
ConsumeMessageThread_ 3 receive new messages: productorder {orderidl = 111, type ='create order '} 
ConsumeMessageThread_ 3 receive new messages: productorder {orderidl = 111, type ='pay order '} 
ConsumeMessageThread_ 3 receive new messages: productorder {orderidl = 111, type ='complete order '}

One producer and three consumers

  The number of consumers will be equal to or less than 4!!!

Local online simulation, a producer and three consumers scenario, to see the order of consumption, the content is long, divided into three pieces

Consumer core configuration

setConsumeFromWhere

  1. CONSUME_ FORM_ FIRST_ Offset: consumption starts from the header of message queue for the first time, that is, all historical messages (also stored in the broker) are consumed once, and then consumption starts from the progress of last consumption
  2. CONSUME_ FROM_ LAST_ Offset: the default policy, starting consumption at the end of the queue for the first time, that is, skipping the history carefully, and starting consumption at the start of the next consumption
  3. CCONSUME_ FROM_ Time step: start consumption at a certain time point. By default, it is half an hour ago. The next step is to start consumption after the last consumption progress

setAllocateMessageQueueStrategy

  • Load balancing strategy algorithm, that is, the algorithm for consumers to allocate to the queue. The default value of allocatemessagequeue average is the average allocation

setOffsetStore

  • Message consumption progress memory, 2 strategies
    • Localfileoffsetstore (broadcast mode is used by default)
    • Remote broker offsetstore (cluster mode is used by default)

setConsumeThreadMin

  • Minimum number of consumption thread pools

setConsumeThreadMax

  • Maximum number of consumption thread pools

setPullBatchSize

  • How many messages do consumers pull at a time when they go to the broker

setConsumeMessageBatchMaxSize

  • How many messages are consumed at one time in a single consumption

setMessageModel

  • Consumer consumption pattern
    • Clustering: the default is cluster mode
    • Broadcasting: broadcast mode

The odd and even number of queues under topic will affect the consumption quantity in the number of customers

  • If there are 4 queues (the default queue is 4), 8 messages and 4 nodes, they will consume 2 messages each. If they are not equal, the load balancing will be unevenly distributed
  • If the number of consumer instances is more than the total number of message queues, the extra consumer instances will not be allocated to the queue, and the messages will not be consumed, and the load will not be shared. Therefore, it is necessary to control the total number of queues to be greater than the number of consumers

Cluster mode (default)

  • The consumer instance allocates the messages sent by the consumer producers equally
  • For example, order messages can only be consumed once

Broadcast mode

  • In broadcast mode, the message delivered to broker will be consumed by each consumer, and a message will be consumed by multiple consumers. In broadcast consumption, the consumergroup is temporarily useless
  • For example: QQ group, group leader sends a message, everyone can see

Message store

ConsumeQueue

Logical queue, default storage location: root / store / consumequeue

CommitLog

The default storage location for message files is / root / store / commitlog

Common interview questions

Why message queuing?

advantage

  1. Asynchronous: for example, seckill can be used,I’ll take you there directly
  2. decoupling
  3. Peak cutting: in the case of second kill, one by one into the team, one by one out of the team, orderly

shortcoming

  1. The lower the system availability: the more external dependence, the more dependence, the greater the risk of problems
  2. System complexity improvement: need to consider a variety of scenarios, such as message repeat consumption, message loss
  3. More machines and manpower are needed: Message Queuing is generally deployed in clusters and needs operation, maintenance and monitoring

How to avoid repeated consumption?

Rocketmq does not guarantee that messages will not be duplicated. If the business guarantees that it cannot be consumed repeatedly, it needs to go to the business side to duplicate

Database table de duplication

Specifies a unique value for a field

setNX

Using redis’s feature of distributed lock, the following is my previous code to be modified

package com.cyb.redis.utils;

import redis.clients.jedis.Jedis;
import redis.clients.jedis.JedisPool;

public class jedisUtils {
    private static String ip = "192.168.31.200";
    private static int port = 6379;
    private static JedisPool pool;
    static {
        pool = new JedisPool(ip, port);
    }
    public static Jedis getJedis() {
        return pool.getResource();
    }
    public static boolean getLock(String lockKey, String requestId, int timeout) {
        //Get the jedis object to connect with the remote redis server
        Jedis je=getJedis();
        //Parameter 3: NX and XX
        //Parameter 4: ex and PX
        String result = je.set(lockKey, requestId, "NX", "EX", timeout);
        if (result=="ok") {
            return true;
        }
        return false;
    }

    public static synchronized boolean getLock2(String lockKey, String requestId, int timeout) {
        //Get the jedis object to connect with the remote redis server
        Jedis je=getJedis();
        //Parameter 3: NX and XX
        //Parameter 4: ex and PX
        Long result = je.setnx(lockKey, requestId);
        if (result==1) {
            je.expire(lockKey, timeout); // Set validity period
            return true;
        }
        return false;
    }
}

Redis atomic increment

Using the incr feature of redis, if it is greater than 0, it means that it has consumed too much(The expiration time needs to be set)

How to ensure the reliable transmission of messages?

Producer side

  1. Do not send in oneway, send in synchronous or partial way, and try again, but the message key must be unique
  2. The delivery log needs to be saved, including key fields, delivery time, delivery status, number of retries, request body, response body, etc

Broker side

  1. Double master and double slave architecture, nameserver needs multiple nodes
  2. Synchronous dual write, asynchronous disk brushing

Consumer side

  1. Message consumption is saved in the log file

How to deal with the massive accumulation in the broker?

  1. Expansion of temporary topic queue to improve consumers’ ability
  2. Write a temporary processing and distribution program to quickly read from the old topic to the temporary new topic. The number of queues of the new topic is multiplied, and then start more consumers to consume the temporary new topic

What are the reasons for rocketmq’s high performance?

MQ architecture configuration

  1. Write in order
  2. random block read
  3. Zero-copy

High availability of sender

  1. Dual master and dual slave architecture: when creating a topic, the message queue is created on multiple brokers, that is, the same broker name and different broker IDs; When one master is not available, other masters in the group are still available

Consumer high availability

  1. Master slave architecture: broker role, master provides read and write, slave only supports read and write
  2. There is no need to configure the consumer. When the master is unavailable or busy, the consumer will automatically switch to the slave node for reading

Improve the consumption power of news

  1. Add multiple consumers
  2. Modify the minimum / large number of thread pools for consumers

Project source code

Case source code

Link: https://pan.baidu.com/s/1Q8iL0lH-bdFEycYGq61hQg   Password: rww2

Rocketmq installation package under Linux

Link: https://pan.baidu.com/s/1dkE7sAs9E4TjwDQ38Pv4_ A password: mkjm

Construction of dual master and dual slave cluster

I’ll take you there directly