How does the rabbitmq series ensure that messages are not lost

Time:2021-10-17

1. The previous article introduced how to use rabbitmq to realize RPC function in springboot, and shared possible pits and solutions;
2. This chapter mainly introduces scenarios where messages may be lost and solutions, and basically covers all possible scenarios.

1、 By setting persistence

Persistence can improve the reliability of rabbitmq to prevent data loss under abnormal conditions (such as restart, shutdown, downtime, etc.). Rabbitmq persistence is divided into three parts: switch persistence, queue persistence and message persistence.

1. What is switch persistence

Switch persistence means that the attribute data of the switch is stored on the disk. When the MQ server is accidentally or shut down, there is no need to manually or execute code to create the switch when restarting rabbitmq. The exchange opportunity is automatically created, which is equivalent to always existing.

2. How to persist the switch

When creating a switchdurableParameter set totrueJust. For example, I declare a switch of type direct:

/**

 *Set the switch as direct

 * @return DirectExchange

 */

@Bean

DirectExchange myExchange() {

return new DirectExchange(QueueConstants.QUEUE\_EXCHANGE\_NAME, true, false);

}
explain
  • By willdurableParameter set totrue, the metadata of the switch will be stored on disk. For a long-term switch, it is recommended to set it as persistent.

3. Queue persistence

If the queue is not set to persistent, the metadata and data of the relevant queue will be lost after the rabbitmq service is restarted. The queues are gone, and there is no place to store messages.

4. How to persist queues

Similarly, when creating a queue, thedurableParameter set totrueJust.

/**
 *Create queue
 */
@Bean
public Queue myQueue() {
    return new Queue(QueueConstants.RPC_QUEUE1);
}
explain
  • The durable parameter is false by default and is only valid for the current connection. Data will be lost after the rabbitmq service is restarted;
  • The persistence of the queue can ensure that its own metadata will not be lost due to abnormal conditions, but it can not guarantee that the messages stored inside will not be lost;
  • If you want to ensure that messages are not lost, you need to set message persistence.

5. Message persistence

Rabbitmq messages are attached to the queue, so if you want to persist messages, the premise is that the queue must also set persistence.

6. How to persist messages

When creating a message, add an attribute of the persistent messagedelivery_modeSet to 2).

How to set message persistence in springboot

Messages sent using rabbittemplate in spring boot are persistent by default, because the default is set todelivery_mode = 2, let’s verify it by looking at the source code.

Source code analysis

1> sendAndReceive
When sending messages, the producer will use the sendandreceive interface of rabbittemplate to send messages:

@Nullable
    public Message sendAndReceive(String exchange, String routingKey, Message message) throws AmqpException {
        return this.sendAndReceive(exchange, routingKey, message, (CorrelationData)null);
    }

2> Message
The third parameter message has a messageproperties property

Open message.class:

public class Message implements Serializable {
    private static final long serialVersionUID = -7177590352110605597L;
    private static final String ENCODING = Charset.defaultCharset().name();
    private static final Set<String> whiteListPatterns = new LinkedHashSet(Arrays.asList("java.util.*", "java.lang.*"));
    private final MessageProperties messageProperties;
    private final byte[] body;

3> MessageProperties
Open messageproperties.class:

static {
        DEFAULT_DELIVERY_MODE = MessageDeliveryMode.PERSISTENT;
        DEFAULT_PRIORITY = 0;
    }

MessageDeliveryMode.class:

public enum MessageDeliveryMode {
    NON_PERSISTENT,
    PERSISTENT;

    private MessageDeliveryMode() {
    }

    public static int toInt(MessageDeliveryMode mode) {
        switch(mode) {
        case NON_PERSISTENT:
            return 1;
        case PERSISTENT:
            return 2;
        default:
            return -1;
        }
    }
conclusion

Check the source code. The messages sent using rabbimqtemplate in spring boot are persistent messages by default.

7. Summary

  1. Queue and message persistence are set. After rabbitmq service is restarted, messages will still exist;
  2. Only queue persistence is set, and messages will be lost after restart;
  3. If you only set message persistence, the queue will disappear after restart, so the message will be lost. Therefore, it is meaningless to set message persistence instead of queue persistence;
  4. Setting all messages to persistent (the speed of writing to disk is much slower than that of writing to memory) may affect the performance of rabbitmq. For messages with low reliability, persistence can not be used to improve the throughput of rabbitmq.

2、 The producer turns on sending confirmation

scene

1. I don’t know whether the message sent by the producer has reached the rabbitmq server; > 2. It is not known whether the messages sent by the producer have been successfully allocated to the queue.

terms of settlement

Enable message sending confirmation, which is guaranteed through the confirmcallback interface and returncallback interface.

remarks
For specific operation methods, please refer to the message confirmation mechanism of rabbitmq series.

3、 Consumer open message acknowledgement (ACK)

scene

The service went down before the consumer could handle the message.

terms of settlement

The consumer starts the message confirmation (ACK) and sets the message to manual confirmation:

#Enable ack (manually confirm when the consumer receives the message) 

spring.rabbitmq.listener.simple.acknowledge-mode=manual

In this way, although the service is down, consumers will still consume the data after restart.
remarks
For specific operation methods, please refer to the message confirmation mechanism of rabbitmq series.

4、 Deploy ##### scenarios using rabbitmq’s mirror set group image mode

After the persistent messages are successfully stored in rabbitmq, if the rabbitmq service node goes down and restarts abnormally during the storage process, the messages have not been stored in time.

terms of settlement

The mirror set group image mode of rabbitmq can be used for deployment. If the master node dies within this special time period, it will automatically switch to the slave node, so as to ensure high availability unless the whole cluster dies.

remarks
Learn moreAbout rabbitmq mirror cluster modeYou can refer to the deployment patterns of rabbitmq series.

5、 Message compensation mechanism

scene

For example, for messages set to be persistent, during the process of saving to disk, the current queue node hangs and the disk of the storage node hangs.

terms of settlement

  1. Because the system functions are complex and there are too many network uncertainties, the message compensation mechanism needs to be established on the premise that the system records detailed logs, such as message sending log, message receiving log, stored in database log, etc;
  2. Through manual triggering or regular scanning, the data meeting the message compensation requirements are extracted from these logs for message compensation.

WeChat official account

Welcome to my WeChat official account for more articles:
How does the rabbitmq series ensure that messages are not lost