Redis series (V) publish subscribe mode, master-slave replication and sentinel mode

Time:2021-9-17

It is more or less used in NoSQL development, and it is also a necessary knowledge point for interview. I’ve asked every interview in recent days. However, I feel that the answer is not good, and there are still many knowledge points that need to be sorted out. Here, I’ll comb through several redis notes, and then add the above test questions.

Redis series:

  1. Redis series (I) introduction to redis
  2. Redis series (II) 8 data types of redis
  3. Redis series (III) integration of redis transactions and spring boot
  4. Redis series (IV) redis profile and persistence
  5. Redis series (V) publish subscribe mode, master-slave replication and sentinel mode
  6. Redis series (VI) redis’s cache penetration, cache breakdown and cache avalanche
  7. Redis series (VII) redis interview questions
  8. Redis Command Reference

1. Redis subscription publication

Redis publish / subscribe (Pub / sub) is a message communication mode: the sender sends (PUB) messages and the subscriber (sub) receives them.

Redis client can subscribe to any number of channels.

Subscribe / publish message diagram:

The following figure shows channel 1. Three clients have subscribed to this channel.

When a new message is sent to channel 1 through the publish command, the message will be sent to the three clients subscribing to it.

command

These commands are widely used to build instant messaging applications, such as network chat rooms, real-time broadcasting, real-time reminders and so on.

Serial number Command and description
1 [PSUBSCRIBE pattern pattern …]Subscribe to one or more channels that match the given pattern.
2 [PUBSUB subcommand argument [argument …]]View subscription and publishing system status.
3 PUBLISH channel messageSends information to the specified channel.
4 [PUNSUBSCRIBE pattern [pattern …]]Unsubscribe from all channels in a given mode.
5 [SUBSCRIBE channel channel …]Subscribe to information for a given channel or channels.
6 [UNSUBSCRIBE channel [channel …]]Unsubscribe from a given channel.

test

The following example demonstrates how publish subscribe works. In our example, we created a subscription channel namedredisChat:

redis 127.0.0.1:6379> SUBSCRIBE redisChat

Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "redisChat"
3) (integer) 1

Now, let’s restart a redis client, and then publish the message twice on the same channel redischat, and the subscriber can receive the message.

redis 127.0.0.1:6379> PUBLISH redisChat "Redis is a great caching technique"

(integer) 1

redis 127.0.0.1:6379> PUBLISH redisChat "Learn redis by runoob.com"

(integer) 1

#The subscriber's client displays the following message
1) "message"
2) "redisChat"
3) "Redis is a great caching technique"
1) "message"
2) "redisChat"
3) "Learn redis by runoob.com"

principle

Redis is implemented in C. by analyzing the public. C file in redis source code, we can understand the underlying implementation of publish and subscribe mechanism, so as to deepen our understanding of redis. Redis implements publish and subscribe functions through public, subscribe, psubscribe and other commands.

WeChat:

After subscribing to a channel through the subscribe command, a dictionary is maintained in redis = server, and the keys of the dictionary are channels! The value of the dictionary is a linked list, which saves all clients subscribing to the channel. The key of the subscribe command is to add the client to the subscription chain of a given channel.

Send a message to subscribers through the publish command. Redis server will use the given channel as the key, find the linked list of all clients subscribing to the channel in the channel dictionary maintained by it, traverse the linked list, and publish the message to all subscribers.

Usage scenario:

1. Real time message system

2. Real time chat

3. You can subscribe and follow the system

Slightly more complex scenarios use message oriented middleware MQ.


2. Master slave replication

1. Concept

Master slave replication refers to copying data from one redis server to other redis servers. The former is called master / leader, and the latter is called slave / flower; Data replication is unidirectional and can only be from master node to slave node. Master mainly writes, slave mainly reads.

By default, each redis server is the master node. A master node can have multiple slave nodes or no slave nodes, but a slave node can only have one master node.

2. Role of master-slave replication

1. Data redundancy: master-slave replication realizes the hot backup of data, which is a data redundancy method other than persistence.

2. Fault recovery: when the master node has problems, the slave node can provide services to achieve rapid fault recovery. In fact, it is also a kind of service redundancy.

3. Load balancing: on the basis of master-slave replication, combined with read-write separation, the master node can provide write services, and the slave node can provide read services (that is, the application connects to the master node when writing redis data, and the application connects to the slave node when reading redis), sharing the server load; Especially in the scenario of less write and more read, the concurrency of redis server can be greatly improved by sharing the read load among multiple nodes.

4. Cornerstone of high availability (cluster): in addition to the above functions, master-slave replication is also the basis for the implementation of sentinel mode and cluster. Therefore, master-slave replication is the basis of redis high availability.

Generally speaking, to apply redis to engineering projects, it is absolutely impossible to use only one redis (it may be down) for the following reasons:

1. Structurally, a single redis server will have a single point of failure, and one server needs to handle all request loads, which is very stressful;

2. In terms of capacity, the memory capacity of a single redis server is limited. Even if the memory capacity of a redis server is 265g, all memory can not be used as redis storage memory. Generally speaking,The maximum memory used by a single redis should not exceed 20g

Commodities on e-commerce websites are generally uploaded once and browsed countless times. It is said that the professional point is “read more and write less”.

For this scenario, we can use the following architecture:

Master-slave copy, read-write separation! In 80% of cases, read operations are performed. This architecture can reduce server pressure and often use the configuration of “one master and two slave” in the actual production environment. It is impossible to use stand-alone redis in a real environment.

3. Environment configuration

Configure only slave libraries, not master libraries.

[[email protected] bin]# redis-cli -p 6379
127.0.0.1:6379> ping
PONG
127.0.0.1:6379> info replication 			#  View information about the current library
# Replication
role:master 									#  role
connected_ slaves:0 							#  There are currently no from the library
master_replid:2467dd9bd1c252ce80df280c925187b3417055ad
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
127.0.0.1:6379>

Copy the three configuration files and modify the corresponding information

1. Port

2. PID name

3. Log file name

4. Dump.rdb name

port 6381
pidfile /var/run/redis_6381.pid
logfile "6381.log"
dbfilename dump6381.rdb

After modification, start our three redis servers, which can be queried through process information.

[[email protected] ~]# ps -ef|grep redis
root       426     1  0 16:53 ?        00:00:00 redis-server *:6379
root       446     1  0 16:54 ?        00:00:00 redis-server *:6380
root       457     1  0 16:54 ?        00:00:00 redis-server *:6381
root       464   304  0 16:54 pts/3    00:00:00 grep --color=auto redis

4. One master and two slaves

By default, each redis server is the master node. Generally, we only need to configure the slave.

Master: 6379, Slave: 6380 and 6381

There are two ways to configure: one is to use the command directly. In this way, the configuration will fail after redis is restarted. Another way is to use a configuration file. Here’s a demonstration using the command.

Next, configure 80 and 81 as slave computers.

127.0.0.1:6380> SLAVEOF 127.0.0.1 6379		# SLAVEOF host  port
OK
127.0.0.1:6380> info replication
# Replication
role:slave 			#  The role is already slave
master_ host:127.0.0.1 	#  Master node address
master_ port:6379 			#  Primary node port
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:0
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:907bcdf00c69d361ede43f4f6181004e2148efb7
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:0
127.0.0.1:6380>

After configuration, look at the host:

127.0.0.1:6379> info replication
# Replication
role:master
connected_ slaves:2 		#  There are two slave nodes under the master node
slave0:ip=127.0.0.1,port=6380,state=online,offset=420,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=420,lag=1
master_replid:907bcdf00c69d361ede43f4f6181004e2148efb7
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:420
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:420
127.0.0.1:6379>

The real master-slave configuration should be configured in the configuration file, which is permanent. The commands used here are temporary.

Configuration file redis.conf

################################# REPLICATION #################################

# Master-Replica replication. Use replicaof to make a Redis instance a copy of
# another Redis server. A few things to understand ASAP about Redis replication.
#
#   +------------------+      +---------------+
#   |      Master      | ---> |    Replica    |
#   | (receive writes) |      |  (exact copy) |
#   +------------------+      +---------------+
#
# 1) Redis replication is asynchronous, but you can configure a master to
#    stop accepting writes if it appears to be not connected with at least
#    a given number of replicas.
# 2) Redis replicas are able to perform a partial resynchronization with the
#    master if the replication link is lost for a relatively small amount of
#    time. You may want to configure the replication backlog size (see the next
#    sections of this file) with a sensible value depending on your needs.
# 3) Replication is automatic and does not need user intervention. After a
#    network partition replicas automatically try to reconnect to masters
#    and resynchronize with them.
#
# replicaof   			#  Configuration here

# If the master is password protected (using the "requirepass" configuration
# directive below) it is possible to tell the replica to authenticate before
# starting the replication synchronization process, otherwise the master will
# refuse the replica request.
#
# masterauth

The configuration method is the same.

5. Several questions

1. The host can write, the slave can’t write, only read. All information and data in the master will be saved in the slave. If the slave tries to write, an error will be reported.

127.0.0.1:6381> get k1 			#  The value of K1 is written in the host and can be read from the host.
"v1"
127.0.0.1:6381> set k2 v2 			#  An error is reported when the slave tries to write
(error) READONLY You can't write against a read only replica.
127.0.0.1:6381>

2. If the host is disconnected, the slave is still linked to the host and can read, but there is still no write. At this time, if the host is restored, the slave can still synchronize information directly from the host.

3. The master-slave configured with the command line will change back to the host if the slave is restarted. If you change back to the slave through the command, you can get the value from the host immediately. This is determined by the principle of replication.

6. Replication principle

After slave is successfully started and connected to the master, it will send a sync synchronization command.

After receiving the command, the master starts the background save process and collects all the received commands for modifying the data set. After the background process is executed, the master will transfer the whole data file to the slave and complete a complete synchronization.

Full copy:After receiving the database file, the slave service saves it and loads it into memory.

Incremental replication:The master continues to transmit all the new collected modification commands to the slave at one time to complete the synchronization.

However, as long as the master is reconnected, a full synchronization (full replication) will be performed automatically. Our data must be visible in the slave.

Schematic diagram of this mode:

Second mode

In this mode, the master node of 6381 is configured as 6380. Master node 6379 has only one slave.

If node 6379 goes down now, nodes 6380 and 6381 are slave nodes, which can only read and will not automatically become the master node. You need to manually change one of them into the master node, and use the following command:

SLAVEOF no one

3. Sentinel mode

1. Overview

The way of master-slave switching technology is: after the host server goes down, it is necessary to manually switch a server to the master server, which requires manual intervention, takes time and effort, and will cause the service to be unavailable for a period of time. This is not a recommended way. More often, we give priority to sentinel mode. Redis has officially provided sentinel architecture since 2.8 to solve this problem.

Sentinel mode can monitor whether the host fails in the background. If it fails, it will automatically convert from the library to the main library according to the number of votes.

Sentinel mode is a special mode. Firstly, redis provides sentinel commands. Sentinel is an independent process. As a process, it runs independently. The principle isThe sentinel sends a command and waits for a response from the redis server to monitor multiple running redis instances

The sentry here has two functions

  • Send a command to let redis server return to monitor its running status, including master server and slave server
  • When the sentinel detects that the master is down, it will automatically switch the slave to the master, and then notify other slave servers through publish subscribe mode to modify the configuration file and let them switch hosts.

However, there may be problems when a sentinel process monitors the redis server. Therefore, we can use multiple sentinels for monitoring. Each sentinel will also be monitored, which forms a multi sentinel mode.

Assuming that the main server is down, sentry 1 detects this result first, and the system will not fail immediately. Sentry 1 only subjectively thinks that the main server is unavailable. This phenomenon is calledSubjective Downline。 When the following sentinels also detect that the primary server is unavailable and the number reaches a certain value, a vote will be held between sentinels. The voting result is initiated by a sentinel to perform failover. After the switch is successful, each sentinel will switch its monitored from the server to the host through the publish subscribe mode. This process is calledObjective offline

2. Configure a sentinel mode of one master and two slave

1. Configure sentinel mode profile, new file/usr/local/bin/kconfig/sentinel.conf

#Sentinel monitor monitored name (optional) host 1
sentinel monitor myredis 127.0.0.1 1

The following number 1 means that after the host goes down, slave votes to decide who will become the new host, and the largest number of votes will become the host.

2. Start the sentry

[[email protected] bin]# ls
6379.log  6381.log      dump6380.rdb  dump.rdb  redis-benchmark  redis-check-rdb  redis-sentinel
6380.log  dump6379.rdb  dump6381.rdb  kconfig   redis-check-aof  redis-cli        redis-server
[[email protected] bin]# redis-sentinel kconfig/sentinel.conf 
2421:X 15 May 2020 20:24:06.847 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2421:X 15 May 2020 20:24:06.847 # Redis version=5.0.5, bits=64, commit=00000000, modified=0, pid=2421, just started
2421:X 15 May 2020 20:24:06.847 # Configuration loaded
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 5.0.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 2421
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

2421:X 15 May 2020 20:24:06.848 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2421:X 15 May 2020 20:24:06.851 # Sentinel ID is 100430af0018d23bd1ae2fe57e71e0d45f64d9a5
2421:X 15 May 2020 20:24:06.851 # +monitor master myredis 127.0.0.1 6379 quorum 1
2421:X 15 May 2020 20:24:06.852 * +slave slave 127.0.0.1:6381 127.0.0.1 6381 @ myredis 127.0.0.1 6379

Start successfully ~!

If the master node goes down now, one of the slave nodes will be selected as the host according to the voting algorithm.

If the original host resumes operation, it can only be returned to the new host as a slave. This is the rule of sentinel mode.

Advantages of sentinel mode

1. Sentinel cluster is based on master-slave replication mode. It has all the advantages of master-slave configuration

2. The master-slave can be switched and the fault can be transferred, so the system availability will be better

3. Sentinel mode is the upgrade of master-slave mode. It is more robust from manual to automatic.

Disadvantages of sentinel mode

1. Redis is not convenient for online capacity expansion. If the cluster reaches a certain upper limit, online capacity expansion will be very troublesome;

2. The configuration of sentinel mode is actually troublesome. There are many configuration items.

The next note will introduce redis’s cache penetration, cache breakdown and cache avalanche.

Recommended Today

A detailed explanation of the differences between Perl and strawberry Perl and ActivePerl

Perl is the abbreviation of practical extraction and report language “practical report extraction language”. Application of activestateperl and strawberry PERL on Windows platformcompiler。 Perl   The relationship between the latter two is that C language and Linux system have their own GCC. The biggest difference between activestate Perl and strawberry Perl is that strawberry Perl […]