Details of sentinel, a high availability component of redis service

Time:2020-10-7

In the previous article, we learned about the use and description of the commonly used data types of redis. Please refer to the https://www.jb51.net/article/120364.htm Today, let’s talk about sentinel, a highly available component of redis. First of all, we will review the master-slave synchronization of redis. The main function of master-slave synchronization is to make the master’s data have a real-time copy on other servers, which has the effect of backup Results: for the reading and writing of redis, the master-slave architecture can disperse the read requests to multiple slave servers, thus reducing the IO pressure of a single redis read request and improving the concurrency of reading requests of redis. Generally, for data consistency, once the slave server becomes a slave of a certain redis, the previous data on the slave server will be cleared, and then the master will be deleted The data sent by the ER is applied to the memory, so that it is consistent with the master data. In addition, the slave is usually read-only. In other words, the slave can only perform read-only operations, and the write operation will be rejected, so the write request is always completed by the master;

In this master-slave replication environment, if the master is down, it means that the whole system will not be able to write data to redis. Obviously, this situation should be solved in time. How to solve it? Is there such a component to help us monitor the master in real time? Once the master is found to be down, a slave will be promoted and selected as a new master. If the original master has other slaves, all the other slaves will be subordinate to the new master. In addition, it should let the system trigger an alarm when the master switch occurs, so that the administrator can repair the broken master online as soon as possible. Yes, sentine L has the above functions. It can monitor the master node in the master-slave synchronous cluster. After the master fails, it can automatically fail over. It will promote a slave as a new master, and then notify the administrator;

Sentinel is a distributed system. We can run multiple sentinel processes in one architecture. These sentinel processes use gossip protocols to receive information about whether the master is offline, and uses agreement protocols to decide whether to perform automatic fault migration and which slave to choose as the new master. Each sentinel process will regularly send messages to other sentinel processes, master, and slave to ensure that the other party is “alive”. If it is found that the other party has not received a response within the specified configuration time (configurable), it will temporarily consider that the other party has been offline, that is, the so-called “subjective down”, English Name: subjective down, referred to as sdown. If there is subjective downtime, there must be objective downtime.

When the majority of sentinel processes make sdown judgment on master and communicate with each other through sentinel is master down by addr command, this method is called “objective down”, and its English name is objective down. Through a certain vote algorithm, select one of the remaining slave servers to upgrade to the master server node, and then automatically modify the relevant configuration, and start the fail over.

Configure to use Sentinel

Environmental description

role IP address port
master 192.168.0.41 6379
slave01 192.168.0.42 6379
slave02 192.168.0.43 6379
sentinel01 192.168.0.41 26379
sentinel02 192.168.0.42 26379
sentinel03 192.168.0.43 26379

Architecture diagram

Tip: as can be seen from the above architecture diagram, we must first have a master-slave architecture cluster, and then deploy sentinel to monitor the master-slave synchronous cluster;

Construction of redis master slave replication cluster

1. To install redis on 192.168.0.41/42/43, you can use either Yum or compile. Please refer to https://www.jb51.net/article/79096.htm

2. Configure redis on 192.168.0.41/42/43 to listen on non-native 127.0.0.1 and configure that redis on 42 / 43 belongs to 192.168.0.41

  master

  slave01

  slave02

Tips: redis supports online modification of configuration and saving the configuration to the configuration file; the slave of instruction is used to specify the IP address and port of the redis master, indicating that the redis is configured as the slave role of the corresponding master; config rewrite is to save our configuration to the configuration file;

Check whether there are two slave nodes connected to the master on the master

Verification: write data on the master to see if it can be synchronized to the two slaves in time?

Tip: you can see that when you write data on the master database, the slave database can synchronize the data on the master database in time. By now, the master-slave cluster of redis has been set up;

Configure sentinel to monitor the master

Note: the configuration of the three sentinels is the same. Here, it is necessary to specify the IP address and port of the master-slave synchronous cluster, as well as the number of valid legal votes. The valid quorum refers to the number of sentinel subjective masters If there are three sentinels, at least two are considered to be down before triggering the election of a new master. If the master is configured with an authentication password, we need to specify the authentication password in the sentinel;

Description of sentinel configuration file

Bind: this instruction is the same as bind in the redis configuration file. It is used to specify the listening address of sentinel. It is not specified by default and listens to all available addresses of the local machine;

Protected mode: Specifies whether to turn on the protected mode;

Port: used to specify the listening port of sentinel; the default is 26379

Daemonize: used to specify whether sentinel runs as a daemons, yes means it runs as a background daemons; no means it does not run as a daemons, but runs directly in the foreground;

Pidfile: specify the PID file path;

Logfile: specify the log file path;

Dir: Specifies the working path of sentinel;

sentinel monitor <master-name> <ip> <redis-port> < quorum >: used to specify the IP address and port of the monitoring master node as well as the number of valid legal votes; where < master name > is a name given to the monitored master node, which can be arbitrarily written to identify the function of identification; < quorum > indicates the quorum mechanism of sentinel cluster, that is, when there are at least quorum sentinel nodes judging the master node failure at the same time, it will be considered as a real failure;

Sentinel auth pass < master name > < password >: Specifies the authentication password of the master. Usually, the password needs to be set, and the password of the master and slave should be the same;

Sentinel down after milliseconds < master name > < milliseconds >: configure how long the abnormal state of the primary node of the specified cluster lasts before it is marked as “failure”;

Sentinel parallel syncs < master name > < numslaves >: refers to the number of slave nodes that can be configured in parallel by sentinel during the failure process;

Sentinel failure timeout < master name > < milliseconds >: sentinel must complete the failover operation within the specified time, otherwise, it will be regarded as a failure of the failover operation;

Sentinel notification script < master name > < script path >: a notification script, which is automatically passed multiple parameters;

After knowing the configuration file of sentinel, we will start all three sentinel

  master

  slave01

  slave02

Tip: from the above information, we can see that all three sentinel monitor the IP address and port of the master. In fact, their configuration files are the same;

View sentinel log

Tip: from the above log information, it can be seen that the master monitored by sentinel is 192.168.0.41:6379, and there are two slaves 192.168.0.42:6379 and 192.168.0.43:6379, respectively;

View sentinel status

Tip: it prompts us to turn on the protection mode;

Turn off protected mode

Restart sentinel and check the sentinel status again


[[email protected] ~]# systemctl restart redis-sentinel.service
[[email protected] ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port 
LISTEN 0 511 *:26379 *:* 
LISTEN 0 511 *:6379 *:* 
LISTEN 0 128 *:22 *:* 
LISTEN 0 100 127.0.0.1:25 *:* 
LISTEN 0 511 :::26379 :::* 
LISTEN 0 128 :::22 :::* 
LISTEN 0 100 ::1:25 :::* 
[[email protected] ~]# redis-cli -h 192.168.0.41 -p 26379 
192.168.0.41:26379> info sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=192.168.0.41:6379,slaves=2,sentinels=3
192.168.0.41:26379> info clients
# Clients
connected_clients:3
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
192.168.0.41:26379> CLIENT LIST
id=2 addr=192.168.0.42:59048 fd=14 name=sentinel-f60b324b-cmd age=38 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=3 addr=192.168.0.43:37480 fd=15 name=sentinel-eada229c-cmd age=38 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=publish
id=4 addr=192.168.0.41:36706 fd=16 name= age=32 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
192.168.0.41:26379>

Tip: from the above status information, it can be seen that the current sentinel monitoring master is in normal OK state, with two slaves and three sentinels; for 192.168.0.41:26379, there are currently three client connections, two sentinel and one local machine; the three sentinel building and starting are completed;

Verification: shut down the master to see if sentinel will elect one of the two slave nodes as a new master? Do you want to point another slave to the new master?

View master slave synchronization information on slave01

Tip: the first view only tells us that the master is down. The second view tells us that the current node is the master and has a slave node. This indicates that the failover has been completed and slave01 has been promoted to a new master;

Check the master-slave information on 192.168.0.43 to see if it points to the new master?

Note: from the master-slave synchronization information on slave02, you can see that slave02 is subordinate to the new master;

Viewing the sentinel log at failover

Tip: from the above log information, after sdown to odown, the vote algorithm will be triggered to elect the leader; then the original master will be demoted to slave, and then the original save attribute of the elected leader will be removed (slave of no one); then the new master will be promoted, and then the remaining slave will be reconfigured as the new master; finally, the master will be switched to start new monitoring;

View redis configuration file after failover

Tip: after failover redis.conf The master IP of the slaveof line in will be modified, sentinel.conf The sentinel monitor IP in will be modified. At the same time, information such as known slave and known sentinel will be added at the end of sentinel configuration file;

Fix the old master and bring it back online

Tip: after the original master is started, it will automatically become the slave of the new master. This is mainly because sentinel has changed the slaveof in its configuration file to the new master address during the failover;

View master slave synchronization information on new master

Tip: when the original master is not restored, only one salve can be seen when viewing the master-slave synchronization information on the new master. After starting the original master, two slaves are online;

summary

This article introduces sentinel, a highly available component of redis service. For more content about sentinel, a highly available component of redis service, please search the previous articles of developeppaer or continue to browse the related articles below. I hope you can support developeppaer more in the future!