An in-depth explanation of redis master-slave replication principle

Time:2019-9-10

Preface

Redis persistence guarantees that data will not be lost even if redis service is restarted, because redis service will restore persistent data on hard disk to memory after restart, but when redis server’s hard disk is damaged, it may lead to data loss. This single point of failure can be avoided through redis master-slave replication mechanism.

This paper mainly explains the principle of redis master-slave replication and shares it for your reference and study. Let’s see the detailed introduction together.

1. Duplication process

2. Synchronization between data

3. Full replication

4. Partial replication

5. Heart beat

6. Asynchronous replication

1. Duplication process

  1. Execute the slaveof command from the node.
  2. The slave node only saves the information of the master node in the slaveof command, and does not initiate replication immediately.
  3. The information of the primary node is found from the timing task inside the node, and socket is used to connect the primary node.
  4. After the connection is successfully established, send the Ping command, hoping to get the pong command response, otherwise the connection will be reconnected.
  5. If the primary node sets permissions, permission validation is required, and if validation fails, replication terminates.
  6. Data synchronization is the longest time-consuming operation after permission validation. The master node will send all the data to the slave node.
  7. When the master node synchronizes the current data to the slave node, the establishment process of replication is completed. Next, the master node will continue to send write commands to the slave node to ensure the consistency of master-slave data.

2. Synchronization between data

One of the steps in the replication process mentioned above is “synchronizing data sets”, which is now referred to as “synchronization between data”.

There are two commands for redis synchronization: sync and psync. The former is the synchronization command before redis 2.8, and the latter is the command designed by redis 2.8 to optimize sync. We will focus on the 2.8 PSYNC command.

The PSYNC command requires three component support:

  1. Replication offsets of master and slave nodes
  2. Master node replication backlog buffer
  3. Running ID of primary node

Master and slave nodes replicate offsets respectively:

  1. Both master and slave nodes participating in replication maintain their own replication offset.
  2. After processing the written command, the master node will record the byte length of the command accumulatively, and the statistics are in the master reploffset index of info replication.
  3. The slave node reports its own copy offset to the master node every second, so the master node also saves the copy offset of the slave node.
  4. After receiving the command sent by the master node, the slave node also accumulates its own offset, and the statistics are in the info replication.
  5. By comparing the replication offset of master and slave nodes, we can judge whether the data of master and slave nodes are consistent.

Primary node replicates backlog buffer:

  1. The replication backlog buffer is a fixed-length first-in-first-out queue stored in the primary node with a default size of 1MB.
  2. This queue is created in the slave connection. When the master node responds to the write command, it not only sends the command to the slave node, but also writes the replication buffer.
  3. His function is to remedy data lost in partial replication and replication commands. Relevant information can be seen through info replication.

The primary node runs ID:

  1. Each redis starts with a 40-bit running ID.
  2. The main function of running ID is to identify Redis nodes. If the way of IP + port is used, then if the master node restarts and modifies the RDB / AOF data, it will be unsafe for the slave node to replicate based on the offset. So, when the running ID changes, the slave node replicates in full. That is to say, when redis restarts, the default slave node replicates in full.

What if you don’t change the running ID on reboot?

  1. The debug reload command can be used to reload RDB and keep the running ID unchanged, thus effectively avoiding unnecessary full replication.
  2. The disadvantage is that the debug reload command will block the main thread of the current Redis node, so it needs to be used cautiously for large data volume primary nodes or nodes that cannot tolerate blocking. Generally, this problem can be solved through the failover mechanism.

How to use the PSYNC command:

The command format ispsync{runId}{offset}

RunId: Running ID of master node copied from slave node

Offset: Current data offset duplicated from a node

PSYNC execution process:

Process description:

RunId is the ID of the target primary node. If it does not default to – 1, offset is the copy offset saved from the node. If it is the first copy, it is – 1.

The primary node returns the result based on runid and offset:

  1. If the reply + FULLRESYNC {runId} {offset}, then the slave node triggers the full replication process.
  2. If reply + CONTINUE, the slave node will trigger partial replication.
  3. If the reply + ERR indicates that the master node does not support the 2.8 PSYNC command, it will use sync to perform full replication.

So far, the synchronization between data is almost the same, and the length is still relatively long. Mainly for PSYNC commands related to the introduction.

3. Full replication

Full replication is the earliest replication method supported by Redis, and it is also the stage that master and slave must go through when establishing replication for the first time. The commands that trigger full replication are sync and psync. As I said before, the watershed version of these two commands is 2.8. Before redis 2.8, sync can only be used to execute different quantities. After 2.8, both full synchronization and partial synchronization are supported.

The process is as follows:

Send the PSYNC command (spync? -1)

The primary node returns FULLRESYNC according to the command

Slave node records master node ID and offset

  1. Send the PSYNC command (spync? -1)
  2. The primary node returns FULLRESYNC according to the command
  3. Slave node records master node ID and offset
  4. Main node bgsave and save RDB to local
  5. The master node sends the RBD file to the slave node
  6. Receive RDB files from nodes and load them into memory
  7. During receiving data from the slave node, the master node saves the new data to the “replication client buffer”. When loading RDB from the slave node, it sends it back. (If it takes too long from the node, it will cause a buffer overflow and eventually a full synchronization failure)
  8. Loading RDB files after clearing data from the node, if the RDB files are large, this step is still time-consuming. If client access at this time will lead to inconsistent data, you can use the configuration slave-server-stale-data to close.
  9. Once the RBD is loaded successfully from the node, if AOF is turned on, it will do bgrewriteaof immediately.

The rough part above is the time-consuming part of the whole synchronization.

Be careful:

If the RDB file is larger than 6GB and is a gigabit network card, Redis’s default timeout mechanism (60 seconds) will lead to full replication failure. This problem can be solved by adjusting the repl-timeout parameter. Although Redis supports diskless replication, i.e. sending it directly to slave nodes through the network, its function is not perfect and its production environment is cautious.

4. Partial replication

When the slave node is replicating the master node, if there are network flashes and other anomalies, the slave node will send the lost command data to the master node. The master node only needs to send the replicated buffer data to the slave node to ensure the consistency of the data. Compared with full replication, the cost is much lower.

  1. When the slave node interrupts the network beyond the repl-timeout time, the master node interrupts the replication connection.
  2. The master node writes the requested data to the “replication backlog buffer” by default of 1MB.
  3. When the slave node restores and reconnects the upper master node, the slave node sends offset and the master node ID to the master node.
  4. After the primary node checks, if the offset data is in the buffer, the cuntinue response is sent, indicating that partial replication can be made.
  5. The master node sends the data of the buffer to the slave node to ensure the normal state of master-slave replication.

5. Heart beat

After establishing replication, the master and slave nodes maintain long connections and send heartbeat commands to each other.

The key mechanisms of heartbeat are as follows:

  1. Each slave has heartbeat detection mechanism, which simulates each other’s client to communicate. The client list command is used to view and replicate relevant client information. The connection status of the master node is flags = M, and the connection status of the slave node is flags = S.
  2. By default, the master node sends Ping commands to the slave node every 10 seconds, and the repl-ping-slave-period configuration can be modified to control the sending frequency.
  3. The slave node sends the replconf ACK {offset} command every second to the main thread to report its current replication offset.
  4. When the master node receives replconf information, it judges the time-out of the slave node, and if it exceeds 60 seconds of repl-timeout, it judges that the slave node is offline.

Be careful:

In order to reduce the master-slave delay, redis master-slave nodes are usually deployed in the same computer room/same City computer room to avoid the heartbeat interruption caused by network partition caused by network delay.

6. Asynchronous replication

The master node not only reads and writes data, but also synchronizes the write command to the slave node. The sending process of the write command is asynchronous. That is to say, the master node returns to the client degree immediately after processing the write command, and does not wait for the slave node to replicate.

The steps for asynchronous replication are simple, as follows:

  1. The primary node accepts processing commands.
  2. The primary node returns the response result after processing.
  3. For modification commands, they are sent asynchronously to the slave node, which executes replicated commands in the main thread.

summary

This paper mainly analyses the principle of Redis replication, including replication process, data synchronization, full replication process, partial replication process, heartbeat design, asynchronous replication process. Among them, it can be seen that synchronization between RDB data is very time-consuming. So Redis exits the PSYNC command similar to incremental replication in version 2.8. When Redis master and slave interrupt directly, they will not replicate in full, but put the data into the buffer (default 1MB). By maintaining the replicated offset between the master and slave, they can judge whether the data in the buffer overflows or not, if not. If there is an overflow, it only needs to send buffer data. The cost is very small. On the contrary, it needs to replicate in full. Therefore, it is very important to control the size of buffer.

All right, that’s the whole content of this article. I hope the content of this article has some reference and learning value for everyone’s study or work. Thank you for your support for developpaer.