Development and application of rabbitmq

Time:2021-1-14

1. Introduction

RabbitMQIt is a complete and reusable enterprise message system based on AMQP and written by Erlang. Support a variety of languages, including Java, python, ruby, PHP, C / C + + and so on.

AMQP model

AMQP: Advanced message queuing protocol, an application layer standard for unified messaging service, is an open standard of application layer protocol, which is designed for message oriented middleware. Based on this protocol, the client and message middleware can deliver messages, which is not limited by different products and development languages of client / middleware.

AMQP model diagram
Development and application of rabbitmq

1.1.1. Working process

Publisher(publisher) publishingnews(message), viaSwitch(Exchange)。

According to the routing rules, the switch distributes the received messages to the users bound to the switchqueue(Queue)。

Finally, the AMQP agent will deliver the message to the consumers who subscribe to the queue, or the consumers can get the message according to their needs.

1. Publishers, switches, queues, and consumers can have more than one. At the same time, because AMQP is a network protocol, publishers, consumers and message agents in this process can exist on different devices.

2. When a publisher publishes a message, it can specify various message meta data. Some properties may be used by message brokers, while others are completely opaque and can only be used by the application receiving the message.

3. From the security point of view, the network is unreliable, or the consumers hang up in the process of processing messages, so the messages that are not processed successfully will be lost. For this reason, AMQP module includes a message acknowledgements mechanism: when a message is delivered from the queue to the consumer, it will not be deleted from the queue immediately until it receives the acknowledgement from the consumer.

4. In some cases, for example, when a message cannot be successfully routed (cannot be distributed from the switch to the queue), the message may be returned to the publisher and discarded. Or, if the message broker performs a delay operation, the message is put into a so-called dead letter queue. At this time, the message publisher can select some parameters to deal with these special cases.

1.1.2.exchange switch

Switches are AMQP entities used to send messages. The switch takes a message and routes it to one or zero queues. Which routing algorithm it uses is determined by switch type and binding rules. Common switches are as follows:

  1. Direct connection switch: routing key = = binding key, strictly matching.
  2. Fanout fan switch: route messages sent to the exchange to all the queues bound to it.
  3. Topic switch: routing key = = binding key, fuzzy matching.
  4. Headers header switch: matches according to the headers attribute in the content of the sent message.
    Specific instructions and usage of these five switches will be described in detail in the following chapters.

1.1.3.queue

Queues in AMQP are very similar to queues in other message queues or task queues: they store messages that will be consumed by applications. Queues share some properties with switches, but queues also have some other properties.

  • Durable (queue still exists after message agent restarts)
  • Exclusive (used by only one connection and the queue is deleted when the connection is closed)
  • Auto delete (delete when the last consumer unsubscribes)
  • Arguments (used by some message agents to perform some additional functions similar to TTL)

1.2. Comparison of rabbitmq and Kafka

Rabbitmq follows AMQP protocol and is used for real-time message delivery with high reliability requirements. Kafka is mainly used to process active streaming data and large amount of data. It is mainly reflected in:

1.2.1. Architecture

  1. rabbitmq: rabbitmq followsAMQP protocolRabbitmq’s broker is managed by exchange,Binding,queueAmong them, exchange and binding constitute the routing key of the message; the client producer communicates with the server through connecting the channel, and the consumer obtains the message from the queue for consumption (for long connection, the message from the queue will be pushed to the consumer, and the consumer loop reads the data from the input stream). Rabbitmq is broker centric.
  2. kafka: Kafka complianceGeneral MQ structure, producer, broker, consumer, with consumer as the center, the consumption information of the message is saved on the client consumer, and the consumer will batch pull the data from the broker according to the consumption point.

1.2.2. Message confirmation

  1. rabbitmq: there is a message confirmation mechanism.
  2. kafka: no message confirmation mechanism.

1.2.3. Throughput

  1. rabbitmqRabbitmq is slightly inferior to Kafka in terms of throughput. Their starting point is different. Rabbitmq supports reliable transmission of messages, supports transactions, and does not support batch operations. Based on the requirements of storage reliability, memory or hard disk can be used for storage.
  2. kafkaKafka has high throughput, batch processing of messages, zero copy mechanism, local disk sequential batch operation of data storage and acquisition, O (1) complexity and high efficiency of message processing.
    (Note:Kafka zero copy, bysendfileThe way. (1) Common data reading: disk, kernel buffer (pagecache) – > user buffer, kernel buffer, network card output; (2) Kafka data reading: disk, kernel buffer (pagecache) – > network card output.

1.2.4. Availability

  1. rabbitmq(1) General cluster: start multiple rabbitmq instances on multiple machines, one for each machine. However, the queue you create will only be placed on one rabbtimq instance, but each instance synchronizes the metadata of the queue. When you are finished consuming, if you are connected to another instance, that instance will pull data from the instance where the queue is located.(2) Image cluster: different from the common cluster mode, the queue created by you, no matter the metadata or the message in the queue, will exist in multiple instances, and then every time you write a message to the queue, the message will be automatically synchronized to the queue of multiple instances. In this case, the advantage is that if one machine goes down, other machines can be used. The disadvantages are: first, the performance overhead is too large. Messages synchronize all machines, resulting in heavy network bandwidth pressure and consumption. Second, if you play like this, there will be no scalability. If a queue is heavily loaded and you add a machine, the new machine also contains all the data of the queue. There is no way to expand your queue linearly
  2. kafka: Kafka is composed of multiple brokers, each broker is a node; each topic created can be divided intoMultiple partitions, each partition can exist in different brokers, and each partition puts part of the data. This is the natural distributed message queue, that is to say, the data of a topic is distributed on multiple machines, and each machine puts part of the data. The data of each partition will be synchronized to other machines to form its own partitionMultiple replica copiesAnd then all replicates will elect a leader, the master-slave structure.

1.2.5. Cluster load balancing

  1. rabbitmq: the load balancing of rabbitmq needs a separate loadbalancer, such as haproxy and keepalived.
  2. kafka: Kafka uses zookeeper to manage brokers and consumers in the cluster, and can register topics to zookeeper; through zookeeper’s coordination mechanism, producer saves broker information of corresponding topics, which can be sent to broker randomly or by polling; and producer can specify partitions based on semantics, and send messages to a certain partition of broker.

2. Structure

2.1. Switch mode

Rabbitmq commonly uses four exchange types: fanout, direct, topic and headers.

2.1.1.Direct Exchange

The direct type exchange routing rule is very simple. It routes messages to the queues whose binding key and routing key exactly match.

2.1.2.Topic Exchange

As mentioned earlier, the direct type of exchange routing rules exactly match the binding key and routing key, but this strict matching method can not meet the actual business requirements in many cases. Exchange of topic type is similar to exception of direct type in that it routes messages to the queue where the binding key matches the routing key, but supports fuzzy matching

  • The routing key is a string separated by a sentence dot “.” (we call each independent string separated by a sentence dot “.” a word)“ stock.usd.nyse ”、“ nyse.vmw ”、“ quick.orange.rabbit “
  • Binding key, like routing key, is also a string separated by a dot “.”
  • There are two special characters “*” and “#” in the binding key, which are used for fuzzy matching, in which “*” is used to match a word and “#” is used to match multiple words (can be zero)

2.1.3.Fanout Exchange

Fanout type exchange routing rules are very simple. It will forward all messages sent to fanout exchange to all queues bound to the exchange.
Fanout exchange does not need to handle routekey. Simply bind the queue to exchange. In this way, messages sent to exchange will be forwarded to all queues bound to the switch. Similar to subnet broadcast, the host in each subnet gets a copy of the message. Therefore, fanout exchange is the fastest to forward messages.

2.1.4.Headers Exchange

The headers type exchange does not rely on the matching rules of routing key and binding key to route messages, but matches them according to the headers attribute in the message content.
Specify a set of key value pairs when binding the queue to exchange. When a message is sent to exchange, rabbitmq will get the headers (also in the form of a key value pair) of the message and compare whether the key value pairs match exactly the key value pairs specified when binding the queue to exchange. If they match exactly, the message will be routed to the queue, otherwise it will not be.

2.1.5.default exchange default

Strictly speaking, default exchange should not be combined with the above four switches, because it is not a separate switch type, but a direct exchange direct connection switch.

The default exchange is actually a direct exchange with no name (the name is an empty string) declared in advance by the message broker.

It has a special property that makes it particularly useful for simple applications: each new queue is automatically bound to the default switch, and the routing key name is the same as the queue name.

For example: when you declare a queue named “search indexing online”, the AMQP agent will automatically bind it to the default switch, and the binding routing key name is also “search indexing online”. So when you want to post a message to the “search indexing online” queue, you can specify that the delivery information includes: the switch name is an empty string, and the routing key is “search indexing online”.

Therefore, the usage of default exchange in direct exchange reflects the point to point of message queue, which feels like directly delivering messages to the queue with a specified name.

2.2. Persistence

Although we want to avoid system downtime, this kind of “force majeure” is always possible. If rabbitmq goes down, restart it. It’s a big deal that it won’t be available for a short time. But if you start it up and find that the rabbitmq server seems to have been reset, and the previous exchange, queue and message data are all gone, it’s too crashing. Not only the business system is affected by the lack of corresponding exchange and queue, but also the loss of a lot of message data is fatal. Therefore, how to ensure the persistence of rabbitmq must be considered before using the service.

Persistence can improve the reliability of rabbitmq to prevent data loss under abnormal conditions (restart, shutdown, downtime, etc.). Rabbitmq persistence is divided into three parts: exchange persistence, queue persistence and message persistence.

2.2.1. Exchange persistence

The persistence of exchange switches is that when the switch is declared, theDurable is set to true

If the switch does not set persistence, after the rabbitmq switch service is restarted, the relevantExchange information will be lost, but messages will not be lost, but messages cannot be sent to this exchange

When exchange is created in spring, the construction method is persistent by default.

2.2.2.queue persistence

When a queue is declared, the persistence of the queue willDurable is set to true

If the queue does not set persistence, after the rabbitmq switch service is restarted, the relevantThe queue information will be lost, and the messages in the queue will also be lost

If one of exchange and queue is non persistent and the other is persistent, an error will be reported in bind.

When exchange is created in spring, the construction method is persistent by default.

2.2.3. Message persistence

To ensure that messages are not lost, in addition to setting the persistence of the queue, you also need to set the message to be persistent. adoptMessage persistence can be realized by setting the delivery mode (deliverymode property in basic properties) to 2

  • Persistent messages are written to disk when they arrive at the queue, and if they can, persistent messages will also be saved as a backup in memory, which can improve the performance to a certain extent. Only when the memory is tight will they be cleared from memory.
  • Non persistent messages are usually only stored in memory. When memory is tight, they will be swapped to disk to save memory space.

If all messages are persisted, the performance of rabbitmq will be affected. Writing to disk is much slower than writing to memory, so there is a trade-off between reliability and throughput.

In spring, the deliverymode property in basic properties corresponds to the deliverymode property in message properties. Usually used RabbitTemplate.convertAndSend () method is persistent by default, deliverymode = 2. If you need to set non persistent sending message, you need to set it manually:

messageProperties.setDeliveryMode(MessageDeliveryMode.NON_PERSISTENT);

2.2.4. Complete scheme

Here is a complete solution for message persistence.

1、 Exchange, queue, message

To ensure message persistence, we need to implement the following on the structure of rabbitmq itself:

  • The durable of the exchange switch is set to true.
  • The durable of the queue is set to true.
  • The delivery mode of message delivery mode is set to 2.

2、 Release confirmation
The front is how to ensure the persistence of messages in rabbit when messages are delivered to rabbit MQ.
Then it is also necessary to ensure that the producer can successfully publish messages, such as the name of the switch is wrongly written, etc. You can set the successful delivery callback when publishing a message to ensure that the message can be successfully delivered to the target queue.

3、 Receiving confirmation
For consumers, if autoack is set to true when subscribing to a message, an exception will hang up before the consumer receives the message. At this time, the message has been deleted from the queue, and the consumer cannot receive the message.

In this case, you can set autoack to false for manual confirmation.

4、 Image queue cluster
After persistent messages are stored in rabbitmq, it will take some time for them to be stored in disk. Rabbitmq does not save every message synchronously. It may just save it to the operating system cache instead of the physical disk. If in this period of time, the server downtime or restart, the message has not yet been saved to the disk, thus lost. In this case, rabiitmq image queue mechanism can be introduced.

It is emphasized hereIt’s a mirror queue cluster, not a normal cluster。 Because for the sake of synchronization efficiency, ordinary cluster will only synchronize the metadata of the queue, not the messages in the queue. Only after upgrading to a mirror queue cluster can messages also be synchronized.

Each mirror queue consists of a master and one or more mirrors. The master node is located on a node commonly known as the master node. Each queue has its own master node. All operations of a given queue are first applied to the primary node of the queue and then propagated to the mirror. This includes enqueueing publications, passing messages to consumers, tracking consumer confirmation, and so on.

Messages published to the queue are copied to all mirrors. No matter which node the consumer connects to, it will connect to the master, and the mirror will delete the confirmed message on the master. As a result, queue mirroring improves availability, but does not distribute load between nodes. If the node hosting the queue master fails, the oldest mirror will be upgraded to the new master as long as it is synchronized. According to the queue mirror parameters, the unsynchronized mirror can also be upgraded.

3. Development

Java development, here tospring-boot-starter-amqpFor example, record some concerns about using rabbitmq in springboot. pom.xml It is quoted as:

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-amqp</artifactId>
        </dependency>

3.1. Simple example

A simple example, only limited to the text message publishing and receiving.

3.1.1. Producers

ProducerController.java

@RestController
public class ProducerController {
    private static final String HEADER_KEY_UID="uid";
    @Autowired
    private ProducerService producerService;

    @PostMapping("/sendText")
    public void sendText(@RequestParam("uid")String uid,@RequestParam("msg")String msg){
        MessageProperties messageProperties=new MessageProperties();
        messageProperties.setHeader(HEADER_KEY_UID,uid);
        producerService.sendText(msg,messageProperties);
    }
}

ProducerService.java

@Service
public class ProducerService {
    private static final String EXCHANGE_NAME="direct.exchange.a";
    private static final String ROUTING_KEY_NAME="direct.routingKey.a";
    @Resource
    private RabbitTemplate rabbitTemplate;


    /**
     *Send message text
     *@ param data text message
     *@ param message properties
     */
    public void sendText(String data, MessageProperties messageProperties) {
        Message message = rabbitTemplate.getMessageConverter().toMessage(data, messageProperties);
        rabbitTemplate.convertAndSend(EXCHANGE_NAME, ROUTING_KEY_NAME, message);
    }
}

Common methods of message sending:

  • rabbitTemplate.send (message); / / send a message. The parameter type is org.springframework.amqp . core.Message
  • rabbitTemplate.convertAndSend (object); / / transform and send the message. Convert parameter object to org.springframework.amqp . core.Message Send after
  • rabbitTemplate.convertSendAndReceive (message) / / transform and send the message, and wait for the sender to return the response message.

3.1.2. Consumers

MessageListener.java

import com.rabbitmq.client.Channel;
import lombok.extern.slf4j.Slf4j;
import org.springframework.amqp.core.ExchangeTypes;
import org.springframework.amqp.core.Message;
import org.springframework.amqp.rabbit.annotation.*;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.support.converter.MessageConverter;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

@Component
@Slf4j
public class MessageListener {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    @RabbitListener(bindings = @QueueBinding(
            value = @Queue(value = "direct.queue.d",
                    durable = "true"),
            exchange = @Exchange(value = "direct.exchange.a",
                    durable = "true",
                    type = ExchangeTypes.DIRECT,
                    ignoreDeclarationExceptions = "true"),
            key = "direct.routingKey.a"
    )
    )
    @RabbitHandler
    public void onMessage(Message message, Channel channel) throws Exception {
        MessageConverter messageConverter = rabbitTemplate.getMessageConverter();
        String msg = (String) messageConverter.fromMessage(message);
        log.info (consumer body: + MSG);
    }
}
  • @Rabbitlistener can be annotated on a class, which needs to be used with @ rabbithandler annotation
  • @Rabbitlistener is marked on the class to indicate that when a message is received, it will be handled by the method of @ rabbithandler. The specific method to be used depends on the parameter type after the message converter conversion

3.2. Message serialization

Message serialization in rabbitmq relies on message convert, which is an interface for serializing message content.

  • Message is divided into two parts: body and message properties. The serialization of rabbitmq refers to the body property of a message, that is, the content we really need to transmit. Rabbitmq abstracts a message convert interface to process the serialization of messages. In fact, there are simplemessage converter (default), Jackson 2J son message converter, etc.
  • When the convertandsend method is called, message convert is used internally to serialize the message.
  • Message convert is a property defined in rabbittemplate. If you need to use more than one message convert in your project. Because the rabbittemplate in spring is singleton injection, it is recommended to define a rabbittemplate for each message convert.

3.2.1. Producers

RabbitConfig.java

public class RabbitConfig {
    
    @Bean("jsonRabbitTemplate")
    public RabbitTemplate jsonRabbitTemplate(ConnectionFactory connectionFactory){
        RabbitTemplate rabbitTemplate=new RabbitTemplate(connectionFactory);
        rabbitTemplate.setMessageConverter(new Jackson2JsonMessageConverter());
        return rabbitTemplate;
    }

    @Bean("defaultRabbitTemplate")
    public RabbitTemplate defaultRabbitTemplate(ConnectionFactory connectionFactory){
        RabbitTemplate rabbitTemplate=new RabbitTemplate(connectionFactory);
        return rabbitTemplate;
    }
}

ProducerService.java

@Service
public class ProducerService {
    private static final String EXCHANGE_NAME="direct.exchange.a";
    private static final String ROUTING_KEY_NAME="direct.routingKey.a";

    @Resource(name = "defaultRabbitTemplate")
    private RabbitTemplate defaultRabbitTemplate;
    @Resource(name = "jsonRabbitTemplate")
    private RabbitTemplate jsonRabbitTemplate;

    /**
     *Send message object JSON
     *
     * @param data
     * @param messageProperties
     */
    public void sendObject(Object data, MessageProperties messageProperties) {
        Message message = jsonRabbitTemplate.getMessageConverter().toMessage(data, messageProperties);
        jsonRabbitTemplate.convertAndSend(EXCHANGE_NAME, ROUTING_KEY_NAME, message);
    }

    /**
     *Send message text
     *
     * @param data
     * @param messageProperties
     */
    public void sendText(String data, MessageProperties messageProperties) {
        Message message = defaultRabbitTemplate.getMessageConverter().toMessage(data, messageProperties);
        defaultRabbitTemplate.convertAndSend(EXCHANGE_NAME, ROUTING_KEY_NAME, message);
    }
}

3.2.2. Consumers

MessageListener.java

@Component
@Slf4j
public class MessageListener {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    @Autowired
    private ObjectMapper objectMapper;

    @RabbitListener(bindings = @QueueBinding(
            value = @Queue(value = "direct.queue.d",
                    durable = "false"),
            exchange = @Exchange(value = "direct.exchange.a",
                    durable = "true",
                    type = ExchangeTypes.DIRECT,
                    ignoreDeclarationExceptions = "true"),
            key = "direct.routingKey.a"
    )
    )
    @RabbitHandler
    public void onMessage(Message message, Channel channel) throws Exception {
        String contentType = message.getMessageProperties().getContentType();
        String bodyText = null;
        System.out.println(contentType);
        switch (contentType) {
            //String
            case MessageProperties.CONTENT_TYPE_TEXT_PLAIN:
                bodyText = (String) rabbitTemplate.getMessageConverter().fromMessage(message);
                break;
            //JSON object
            case MessageProperties.CONTENT_TYPE_JSON:
                User user = objectMapper.readValue(message.getBody(), User.class);
                bodyText = user.toString();
                break;
        }
        log.info (the "consumer side payload:" and "bodytext");
    }  
}

When the producer sends the object message, we use Jackson 2J son message converter and encapsulate it with its tomessage method. However, when consumers receive object messages, instead of using the frommessage method of Jackson 2jsonmessageconverter, we use objectmapper to deserialize the JSON object. Because as like as two peas, rabbitmq sends the package name information containing the class when sending serialized objects of Jackson2JsonMessageConverter, and consumers must create a class exactly the same as the package name in the producer when they are using fromMessage to serialize. Obviously not very realistic.

3.3. Release confirmation (producer)

3.3.1.ConfirmCallback

The confirmcallback interface is used to receive the ACK callback after the message is sent to the rabbitmq switch.

  • Delivery object: Exchange
  • Callback trigger: a callback will be triggered regardless of success or failure.
  • Delivery success: ack = true
  • Delivery failed: ack = false

The usage mode is as follows:

  • Set publisher confirm type to related.
  • realization RabbitTemplate.ReturnCallback And use.

ProducerService.java

@Slf4j
@Service
public class ProducerService {
    private static final String EXCHANGE_NAME = "direct.exchange.a";
    private static final String ROUTING_KEY_NAME = "direct.routingKey.ab";

    @Resource(name = "defaultRabbitTemplate")
    private RabbitTemplate defaultRabbitTemplate;

    /**
     * ConfirmCallback
     *
     *Delivery object: Exchange
     *Callback trigger: a callback will be triggered regardless of success or failure.
     *Delivery success: ack = true
     *Delivery failed: ack = false
     */
    RabbitTemplate.ConfirmCallback confirmCallback = (CorrelationData correlationData, boolean ack, String cause) -> {
        log.info("ack: " + ack);
        if (!ack) {
            log.info (failed to post exchange! .... Log recording, exception handling, compensation handling, etc.);
        } else {
            log.info ("exchange successfully delivered!");
        }
    };

 
    /**
     *Send message text
     *
     * @param data
     * @param messageProperties
     */
    public void sendText(String data, MessageProperties messageProperties) {
        Message message = defaultRabbitTemplate.getMessageConverter().toMessage(data, messageProperties);
        
        //confirmCallback
        defaultRabbitTemplate.setConfirmCallback(confirmCallback);

        defaultRabbitTemplate.convertAndSend(EXCHANGE_NAME, ROUTING_KEY_NAME, message);
    }
}

Configuration file needs to be set:

spring.rabbitmq.publisher-confirm-type = correlated

3.3.2.ReturnCallback

The returncallback interface is used to implement the callback when messages are sent to the rabbitmq switch, but there is no corresponding queue bound to the switch.

  • Delivery object: queue
  • Callback trigger: only when the delivery fails, the callback will be triggered.

The usage mode is as follows:

  • Set publisher returns to true.
  • Set mandatory to true.
  • realization RabbitTemplate.ReturnCallback And use.

ProducerService.java

@Slf4j
@Service
public class ProducerService {
    private static final String EXCHANGE_NAME = "direct.exchange.a";
    private static final String ROUTING_KEY_NAME = "direct.routingKey.ab";

    @Resource(name = "defaultRabbitTemplate")
    private RabbitTemplate defaultRabbitTemplate;

    /**
     * ReturnCallback
     *
     *Delivery object: queue
     *Callback trigger: only when the delivery fails, the callback will be triggered.
     */
    RabbitTemplate.ReturnCallback returnCallback = (Message message, int replyCode, String replyText,
                                                    String exchange, String routingKey) -> {
        log.info "Delivery to queue failed! exchange: " + exchange + ", routingKey: "
                + routingKey + ", replyCode: " + replyCode + ", replyText: " + replyText);
    };

    /**
     *Send message text
     *
     * @param data
     * @param messageProperties
     */
    public void sendText(String data, MessageProperties messageProperties) {
        Message message = defaultRabbitTemplate.getMessageConverter().toMessage(data, messageProperties);
        //returnCallback
        defaultRabbitTemplate.setMandatory(true);
        defaultRabbitTemplate.setReturnCallback(returnCallback);

        defaultRabbitTemplate.convertAndSend(EXCHANGE_NAME, ROUTING_KEY_NAME, message);
    }
}

You need to configure in the configuration file:

spring.rabbitmq.publisher-returns = true

3.4. Acceptance confirmation (consumer)

The previous section explained how to confirm that the message is published to the switch and queue of rabbitmq when the producer publishes the message. So this section explains how to ensure that consumers can completely “consume” the news.

Usually, rabbitmq, as a message middleware, pushes the message to the consumer to complete its mission, and the message is automatically “signed in”. After receiving the message, the consumer implements the business logic of the message. However, if an error occurs in the process of implementing the business logic and needs to be re executed, it will be difficult to do. Once the message is “signed in”, it is deleted from rabbitmq and cannot be sent again.

If the consumer can manually control the “sign in” operation of the message, the message can only be “signed in” after the execution of the business logic about the message, and the message can be deleted from rabbitmq. Otherwise, the message can be sent again. That’s all for this section.

3.4.1.AcknowledgeMode

“Acknowledge” means “confirm”. A message is received correctly by ack. Every message has to be acknowledged. You can go to ack manually or automatically.

When you use manual response message, you should pay special attention to that you can’t forget to respond message, because for rabbitmq, there is no time-out for processing message. As long as you don’t respond message, it will think that it is still processing message normally, causing message queue blocking and affecting business execution. If you don’t want to process it, reject discards the message.

The message confirmation modes are as follows:

  • AcknowledgeMode.NONE : automatic confirmation
  • AcknowledgeMode.AUTO : confirm according to the situation
  • AcknowledgeMode.MANUAL : manual confirmation

The default is automatic confirmation, which can be opened manually in the rabbitlistener containerfactory or in the configuration file

spring.rabbitmq.listener.simple.acknowledge-mode = manual

MessageListener.java

@Component
@Slf4j
public class MessageListener {
    @Autowired
    private RabbitTemplate rabbitTemplate;
    @Autowired
    private ObjectMapper objectMapper;

    @RabbitListener(bindings = @QueueBinding(
            value = @Queue(value = "direct.queue.d",
                    durable = "false"),
            exchange = @Exchange(value = "direct.exchange.a",
                    durable = "true",
                    type = ExchangeTypes.DIRECT,
                    ignoreDeclarationExceptions = "true"),
            key = "direct.routingKey.a"
    )
    )
    @RabbitHandler
    public void onMessage(Message message, Channel channel) throws Exception {
        String contentType = message.getMessageProperties().getContentType();
        String bodyText = null;
        System.out.println(contentType);
        switch (contentType) {
            //String
            case MessageProperties.CONTENT_TYPE_TEXT_PLAIN:
                bodyText = (String) rabbitTemplate.getMessageConverter().fromMessage(message);
                break;
            //JSON object
            case MessageProperties.CONTENT_TYPE_JSON:
                User user = objectMapper.readValue(message.getBody(), User.class);
                bodyText = user.toString();
                break;
        }
        log.info (the "consumer side payload:" and "bodytext");
        channel.basicAck(message.getMessageProperties().getDeliveryTag(), false);
    }

}

3.4.2.Ack/Nack/Reject

After setting to manual confirmation, there are three confirmation operations:

  • ACK: acknowledge the receipt of the message and remove it from the queue.
  • NACK: confirm that no message is received, and the message is sent back to the queue.
  • Reject: rejects the message, discards it directly, and does not return to the queue.

For example, in the basic ack method of the sample code, it should be noted that two parameters should be passed:

  • deliveryTag(unique ID): when a consumer registers with rabbitmq, a channel will be established and rabbitmq will use it basic.deliver Method to push a message to a consumer. This method carries a delivery tag, which represents the unique ID of the message delivered by rabbitmq to the channel. It is a monotonically increasing positive integer, delivery tag The scope of is limited to channel
  • multiple: in order to reduce network traffic, manual confirmation can be batch processed. When this parameter is true, delivery can be confirmed at one time_ All messages whose tag is less than or equal to the incoming value

3.4.3. Abnormal retrying

In addition to the above manual confirmation method, there is a less commonly used method, which can realize the repeated sending of messages. If an exception is thrown in the consumer code, the message will be automatically retransmitted.

application.properties

spring.rabbitmq.listener . simple.retry.enabled=true  Do you want to turn on the consumer and try again
spring.rabbitmq.listener . simple.retry.max -Attempts = 5 maximum attempts
spring.rabbitmq.listener . simple.retry.initial -Interval = 5000 retry interval (in milliseconds)
spring.rabbitmq.listener . simple.default -Request rejected = false whether to discard after the number of retries exceeds the above setting

MessageListener.java

@RabbitListener(bindings = @QueueBinding(
            value = @Queue(value = "direct.queue.d",
                    durable = "false"),
            exchange = @Exchange(value = "direct.exchange.a",
                    durable = "true",
                    type = ExchangeTypes.DIRECT,
                    ignoreDeclarationExceptions = "true"),
            key = "direct.routingKey.a"
    )
    )
    @RabbitHandler
    public void onMessage(Message message, Channel channel) throws Exception {
        String contentType = message.getMessageProperties().getContentType();
        String bodyText = null;
        System.out.println(contentType);
        switch (contentType) {
            //String
            case MessageProperties.CONTENT_TYPE_TEXT_PLAIN:
                bodyText = (String) rabbitTemplate.getMessageConverter().fromMessage(message);
                break;
            //JSON object
            case MessageProperties.CONTENT_TYPE_JSON:
                User user = objectMapper.readValue(message.getBody(), User.class);
                bodyText = user.toString();
                break;
        }
        log.info (the "consumer side payload:" and "bodytext");
        Throw new runtimeException ("try again");
    }

3.5. Consumption pattern

In rabbitmq, there are two ways for consumers to get messages in the queue

  • push: basic.consume Command to subscribe to a message in a queue. Channel will automatically receive the next message after processing the previous message. (the same channel message processing is serial). The client will always receive messages from the queue unless the channel is closed or unsubscribed.
  • pull: basic.get The command actively obtains the message in the queue, but it can never be called through the loop basic.get Instead basic.consume This is because basic.get When rabbitmq is actually executed, it first consults a queue, then retrieves the first message, and then unsubscribes. If it’s a high throughput consumer, it’s better to use it basic.consume 。

In contrast, if there is a demand for continuous consumption, it is recommended to use push mode and subscribe through the listener. If you only need to get some data from the queue at a specific time, you can use the pull mode.

4. Noun concept

4.1.channel

We know that both producers and consumers need to establish a connection with rabbitmq broker, which is a TCP connection, that is, a connection. Once the TCP connection is established, the client can then create an AMQP channel, and each channel will be assigned a unique ID.

Channel is a virtual connection based on connection. Every AMQP instruction processed by rabbitmq is completed through channel.

We can use connection to complete the work of channel. Why introduce channel? Imagine a scenario in which many threads in an application need to consume or produce messages from rabbitmq, so it is necessary to establish many connections, that is, multiple TCP connections. However, for the operating system, it is very expensive to establish and destroy TCP connections. If the peak usage occurs, the performance bottleneck will appear.

Rabbitmq adopts the method similar to NiO (non blocking I / O) and chooses TCP connection reuse, which can not only reduce the performance overhead, but also facilitate the management.

Each thread controls a channel, so the channel multiplexes the TCP connection of connection. At the same time, rabbitmq can ensure the privacy of each thread, just like having a separate connection. When the traffic of each channel is not very large, multiplexing a single connection can effectively save TCP connection resources in the case of performance bottleneck. However, when the traffic of the channel itself is very large, when multiple channels reuse a connection, there will be a performance bottleneck, and then the overall traffic will be limited. At this time, it is necessary to open up multiple connections and share these channels equally among these connections. As for these relevant tuning strategies, they need to be adjusted according to the actual situation of the service itself.

Channel is a very important concept in AMQP. Most operations are carried out at the channel level. For example channel.exchangeDeclare , channel.queueDeclare , channel.basicPublish , channel.basicConsume And so on. Rabbitmq related APIs are closely linked to AMQP, such as channel.basicPublish Corresponding to AMQP Basic.Publish Orders.

4.2.QoS

For push mode, rabbitmq can set basic QoS (consumer prefetch) to control the flow of consumers, so as to limit the number of unack messages.

The premise is that the message confirmation mode must be manual confirmation.

basicQos(int var1, boolean var2)
  • The first parameter is to limit the maximum number of unack messages.
  • The second parameter is a Boolean value, (1) when it is true, it indicates that the flow control restriction is made for the channel; (2) when it is false, it indicates that the flow control restriction is made for the whole consumer.