[redis we used in those years] it was so simple to build a redis cluster

Time:2021-8-17

The IDS service cluster mode can be divided into master-slave, sentinel and cluster modes. This blog mainly introduces the master-slave and cluster modes

Introduction to three cluster modes

Master-slave mode: master-slave mode is the simplest of the three modes. Generally, there is onemasterServices and multiple (at least one)slaveService composition. It mainly realizes read-write separation and reduces the pressure of single redis service.masterBe responsible for writing data, and then copy the data toslaveslaveThe service is responsible for reading data.

Sentinel modeOnce the master-slave mode has a disadvantage, that is:masterWhen the service hangs up, the redis service can no longer write data,slaveDoes not upgrade tomaster。 Another type of service is neededSentinel(sentry).SentinelMainly responsible formonitornoticeAutomatic failoverConfiguration provider。 In short,SentinelCan monitor data nodes(masterandslave)Health status and notify DBA (database administrator) ifmasterHang up, from manyslaveElect a new one in the electionmasterAnd tell the client (CLI) the information of the data node.

ClusterSentinel mode is much friendlier than master-slave mode, butmasterIt is not easy to load, and the storage is limited by a single machine. Another type of service is introduced, which is a little cumbersome. soClusterIt usually has the characteristics of high availability, scalability, distributed, fault tolerance and so on. of course,ClusterandSentinel modeThere is no clear who is good and who is bad. It still depends on your own needs. Generally distributed, usingClusterBetter.

Master-slave mode

BuildingMaster-slave modeandClusterPreviously, I default that you will install stand-alone redis, your environment is Linux (centos7), and your redis version is 5.0.9. If you will not install stand-alone redis, please stamp.

First, the nativeredis.confCopy two copies to/usr/localUnder path:

cp redis.conf /usr/local/redis_master.conf
cp redis.conf /usr/local/redis_replic.conf

Then usevimModify the two configuration files. The main parameters are as follows:
redis_master.conf

#The host address is 127.0.0.1 by default and changed to 0.0.0.0
bind 0.0.0.0
#Port
port 16379
#Whether to enable it as a daemon thread. The default is no and changed to yes
daemonize yes
#Specify the PID file storage location of the redis process and rename the file
pidfile /var/run/redis_16379.pid
#Rename the log storage location to the file
logfile /var/log/redis_16379.log
#RDB pattern persistent snapshot
dbfilename dump-16379.rdb
#Persistent file storage location
dir ./redis-workir
#Redis connection password
requirepass 123456
#The password to connect to the master server can be omitted here
masterauth 123456

redis_replic.conf

#The host address is 127.0.0.1 by default and changed to 0.0.0.0
bind 0.0.0.0
#Port
port 26379
#Whether to enable it as a daemon thread. The default is no and changed to yes
daemonize yes
#Specify the PID file storage location of the redis process and rename the file
pidfile /var/run/redis_26379.pid
#Rename the log storage location to the file
logfile /var/log/redis_26379.log
#RDB pattern persistent snapshot
dbfilename dump-26379.rdb
#Persistent file storage location
dir ./redis-workir
#Redis connection password
requirepass 123456
#Password to connect to the master server
masterauth 123456
#Master node address of this replication node
Replicaof server IP 16379

Note above that in previous versionsreplicaofbe calledslaveof 。 Here, we simulate that there are two hosts, so the host IP is not written127.0.0.1After that, write the IP of your server’s external network. It’s also convenient to talk about the gateway later. Then start two services.

redis-server /usr/local/redis_master.conf
redis-server /usr/local/redis_replic.conf

At this point, check the two service logs and both are started. But at this time, the master writes data and slave
Unable to copy data. Check the log and report an error

Error condition on socket for SYNC: No route to host

You need to open the firewall of the serveriptables, please move on to my last blog (please stamp).

iptables -I INPUT -p tcp -m tcp --dport 16379 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 26379 -j ACCEPT
service iptables save
#Check it out
iptables -L -n -v
#These two ports are indeed exposed
   66  3528 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:26379
 196K   17M ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:16379

Then use it

[[email protected]_0_10_centos ~]# redis-cli -h 127.0.0.1 -p 16379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:16379> set today 1106
OK
127.0.0.1:16379> 
[[email protected]_0_10_centos ~]# redis-cli -h 127.0.0.1 -p 26379 -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:26379> get today
"1106"

Cluster mode

First we create a folder/usr/local/redis-cluster。 And create it under itlogdataconf, and indataCreate folders for different services in the file. BasicallycdGo to the location and usemkdir, it’s relatively simple. Let’s look at the directory structure

[[email protected]_0_10_centos redis-cluster]# tree -d
.
|-- conf
|-- data
|   |-- redis-6378
|   |-- redis-6380
|   |-- redis-6381
|   |-- redis-6389
|   |-- redis-6390
|   |-- redis-6391
`-- log

Then inconfCreate the configuration files of these services under file,Exactly the same, just replace the port number in the key position. You know, the following is6378For example.

cd /usr/local/redis-cluster/conf
#Copy an original file
cp /usr/local/redis.conf /usr/local/redis-cluster/conf/redis-6378.conf
vim redis-6378.conf

Modify the key configuration as follows, and other defaults remain unchanged
redis-6378.conf

#The host address is 127.0.0.1 by default and changed to 0.0.0.0
bind 0.0.0.0
#Port
port 6378
#Whether to enable it as a daemon thread. The default is no and changed to yes
daemonize yes
#Specify the PID file storage location of the redis process and rename the file
pidfile /var/run/redis-cluster/redis-6378.pid
#Rename the log storage location to the file
logfile /usr/local/redis-cluster/log/redis-6378.log
#RDB pattern persistent snapshot
dbfilename dump-6378.rdb
#Persistent file storage location
dir /usr/local/redis-cluster/data/redis-6378
#Redis connection password
requirepass 123456
#Password to connect to the master server
masterauth 123456
#Enable AOF persistence mode
appendonly yes
#Turn on cluster mode
cluster-enabled yes
#For the configuration of the cluster, the configuration file is automatically generated for the first time
cluster-config-file /usr/local/redis-cluster/conf/node-6378.conf
#The request timed out. The default is fine
cluster-node-timeout 15000

cluster-enabledopenclusterCluster mode,appendonlyEnable AOF persistence mode, a real-time ratioRDBPatterns are better persistence patterns. GeneratedaofFormat files are also availabledirThe priority ratio under the path and read into memorydump-6378.rdbThe file should be high.cluster-config-fileIt is the place where the cluster configuration file is stored. It is generated automatically. With the same configuration, replace the port number with the other five to generateredis-6380.conf redis-6381.conf redis-6389.conf redis-6390.conf redis-6391.conf 。 Then start these services.

redis-server redis-6378.conf 
redis-server redis-6380.conf 
redis-server redis-6381.conf 
redis-server redis-6389.conf 
redis-server redis-6390.conf 
redis-server redis-6391.conf 
#Check whether it is turned on
ps -ef | grep redis
#Process already exists
root      3009     1  0 Nov05 ?        00:01:49 redis-server 0.0.0.0:6378 [cluster]
root      3047     1  0 Nov05 ?        00:01:47 redis-server 0.0.0.0:6381 [cluster]
root      3056     1  0 Nov05 ?        00:01:50 redis-server 0.0.0.0:6389 [cluster]
root      3073     1  0 Nov05 ?        00:01:49 redis-server 0.0.0.0:6390 [cluster]
root      3083     1  0 Nov05 ?        00:01:50 redis-server 0.0.0.0:6391 [cluster]
root     26443     1  0 Nov05 ?        00:01:39 redis-server 0.0.0.0:6380 [cluster]

Note that they are shown behind them asclusterIt is started in the form of cluster. Start the cluster below. If it is a version before redis5.0, you need to download ruby. Start it through ruby. You can start it directly after redis5.0. The command is as follows:

#Start cluster
redis-cli -a 123456--cluster create 127.0.0.1:6378 127.0.0.1:6380 127.0.0.1:6381 127.0.0.1:6389 127.0.0.1:6390 127.0.0.1:6391 --cluster-replicas 1
#One more step will be asked, and enter yes
Can I set the above configuration? (type 'yes' to accept): yes
#Turn on proxy firewall
iptables -I INPUT -p tcp -m tcp --dport 6378 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6380 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6381 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6389 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6390 -j ACCEPT
iptables -I INPUT -p tcp -m tcp --dport 6391 -j ACCEPT
service iptables save

The command to start the cluster will automatically generate the master-slave relationship and allocatesolt, will the terminal ask whether this allocation method is OK? If the operator thinks it is OK, it needs to enteryes。 Connect to a redis service by cli.

redis-cli -c -h 127.0.0.1 -p 6378 -a 123456
127.0.0.1:6378> CLUSTER INFO
#Some information is omitted here
127.0.0.1:6378> CLUSTER NODES
114d082bb43c67042c806e0060f5fb8157736b98 127.0.0.1:[email protected] slave ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 0 1604672507350 5 connected
1efc0139a5422c68f1c43428d1d6f566ce361fed 127.0.0.1:[email protected] myself,master - 0 1604672504000 1 connected 0-5460
f10fc263b1b887fe175307b2fb7600370d39c0e1 127.0.0.1:[email protected] master - 0 1604672505000 8 connected 5461-10922
e1eb98ead97d9b22fe4538601cbcd193e998baad 127.0.0.1:[email protected] slave f10fc263b1b887fe175307b2fb7600370d39c0e1 0 1604672505347 8 connected
ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 127.0.0.1:[email protected] master - 0 1604672506000 3 connected 10923-16383
0f75c7f994a43551671e955a9f72c28a9c6edb56 127.0.0.1:[email protected] slave 1efc0139a5422c68f1c43428d1d6f566ce361fed 0 1604672504344 6 connected

It can be seen that637863896381yesmasterNode to manage card slots respectively0-54605461-1092210923-16383。 And three can be determined according to the ID valueslaveWho is the primary node of the node, for example6390The primary node ID value isca998854f1d96e2b64b7304aea5e5b4c96a9d85a, so6381Yes.

It should be explained here that when we save the value, the key will be calculated according to the hash function and fall on the card slot.

127.0.0.1:6378> set haha 123
OK
127.0.0.1:6378> set name pjjlt
-> Redirected to slot [5798] located at 127.0.0.1:6389
OK
127.0.0.1:6389> 

It can be seen that after hashing, the card slot of haha falls in the range of 0-5460, and the card slot of name falls in the range of 5461-10922. Therefore, the service terminal will automatically redirect to the 6389 service. Get also causes redirection of the service terminal.

127.0.0.1:6389> get name
"pjjlt"
127.0.0.1:6389> get haha
-> Redirected to slot [3662] located at 127.0.0.1:6378
"123"
127.0.0.1:6378> 

Add and remove nodes

Cluster clusters can be expanded at a friendly level. Next, add two nodes. As above, add a configuration fileredis-6395.confredis-6396.confAnd start the two services

redis-server redis-6395.conf 
redis-server redis-6396.conf 

Join cluster

#Connect to any service
redis-cli -c -h 127.0.0.1 -p 6378 -a 123456
#Join cluster
CLUSTER MEET 127.0.0.1 6395
CLUSTER MEET 127.0.0.1 6395
#View status
CLUSTER NODES
114d082bb43c67042c806e0060f5fb8157736b98 127.0.0.1:[email protected] slave ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 0 1604673986000 5 connected
1efc0139a5422c68f1c43428d1d6f566ce361fed 127.0.0.1:[email protected] myself,master - 0 1604673985000 1 connected 0-5460
2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 127.0.0.1:[email protected] master - 0 1604673985000 7 connected
f10fc263b1b887fe175307b2fb7600370d39c0e1 127.0.0.1:[email protected] master - 0 1604673988720 8 connected 5461-10922
e1eb98ead97d9b22fe4538601cbcd193e998baad 127.0.0.1:[email protected] slave f10fc263b1b887fe175307b2fb7600370d39c0e1 0 1604673987717 8 connected
f8c884f07375cb524236341c4fcb963f58f82622 127.0.0.1:[email protected] master - 0 1604673987517 7 connected
ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 127.0.0.1:[email protected] master - 0 1604673986000 3 connected 10923-16383
0f75c7f994a43551671e955a9f72c28a9c6edb56 127.0.0.1:[email protected] slave 1efc0139a5422c68f1c43428d1d6f566ce361fed 0 1604673986716 6 connected

It can be seen that the two nodes added at this time are allmasterService, now set 6396 as the slave node of 6395.

redis-cli -c -h 127.0.0.1 -p 6396 -a 123456 cluster replicate 2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e
#Connect to any service
redis-cli -c -h 127.0.0.1 -p 6378 -a 123456
CLUSTER NODES
114d082bb43c67042c806e0060f5fb8157736b98 127.0.0.1:[email protected] slave ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 0 1604673986000 5 connected
1efc0139a5422c68f1c43428d1d6f566ce361fed 127.0.0.1:[email protected] myself,master - 0 1604673985000 1 connected 0-5460
2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 127.0.0.1:[email protected] master - 0 1604673985000 7 connected
f10fc263b1b887fe175307b2fb7600370d39c0e1 127.0.0.1:[email protected] master - 0 1604673988720 8 connected 5461-10922
e1eb98ead97d9b22fe4538601cbcd193e998baad 127.0.0.1:[email protected] slave f10fc263b1b887fe175307b2fb7600370d39c0e1 0 1604673987717 8 connected
f8c884f07375cb524236341c4fcb963f58f82622 127.0.0.1:[email protected] slave 2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 0 1604673987517 7 connected
ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 127.0.0.1:[email protected] master - 0 1604673986000 3 connected 10923-16383
0f75c7f994a43551671e955a9f72c28a9c6edb56 127.0.0.1:[email protected] slave 1efc0139a5422c68f1c43428d1d6f566ce361fed 0 1604673986716 6 connected

It can be seen that at this time, the 6396 node has become the 6395 slave node.

Remove cluster

Rule: you cannot remove the login node or your own master node. You can delete other master or slave nodes

#As can be seen from the above, 1efc0139a5422c68f1c43428d1d6f566ce361fed is node 6378
127.0.0.1:6378> CLUSTER FORGET 1efc0139a5422c68f1c43428d1d6f566ce361fed
(error) ERR I tried hard but I can't forget myself...
#Log in to the slave node 6391 of 6378 and delete 6378. Failed.
[[email protected]_0_10_centos conf]# redis-cli -c -h 127.0.0.1 -p 6391 -a pjjlt1126
127.0.0.1:6391> CLUSTER FORGET 1efc0139a5422c68f1c43428d1d6f566ce361fed
(error) ERR Can't forget my master!
#Add the newly added node 6396
127.0.0.1:6378> CLUSTER FORGET f8c884f07375cb524236341c4fcb963f58f82622
OK
127.0.0.1:6378> CLUSTER NODES
114d082bb43c67042c806e0060f5fb8157736b98 127.0.0.1:[email protected] slave ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 0 1604675461284 5 connected
1efc0139a5422c68f1c43428d1d6f566ce361fed 127.0.0.1:[email protected] myself,master - 0 1604675460000 1 connected 0-5460
2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 127.0.0.1:[email protected] master - 0 1604675460283 7 connected
f10fc263b1b887fe175307b2fb7600370d39c0e1 127.0.0.1:[email protected] master - 0 1604675462284 8 connected 5461-10922
e1eb98ead97d9b22fe4538601cbcd193e998baad 127.0.0.1:[email protected] slave f10fc263b1b887fe175307b2fb7600370d39c0e1 0 1604675462000 8 connected
ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 127.0.0.1:[email protected] master - 0 1604675460000 3 connected 10923-16383
0f75c7f994a43551671e955a9f72c28a9c6edb56 127.0.0.1:[email protected] slave 1efc0139a5422c68f1c43428d1d6f566ce361fed 0 1604675463287 6 connected

It was found that the was removed because the node.conf file still exists(cluster-config-fileThe configured file) can also be restored or removed

#Resume removal
127.0.0.1:6378> CLUSTER SAVECONFIG 
OK
#Found 6396 it's back
127.0.0.1:6378> CLUSTER NODES
114d082bb43c67042c806e0060f5fb8157736b98 127.0.0.1:[email protected] slave ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 0 1604675573000 5 connected
1efc0139a5422c68f1c43428d1d6f566ce361fed 127.0.0.1:[email protected] myself,master - 0 1604675569000 1 connected 0-5460
2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 127.0.0.1:[email protected] master - 0 1604675572000 7 connected
f10fc263b1b887fe175307b2fb7600370d39c0e1 127.0.0.1:[email protected] master - 0 1604675572000 8 connected 5461-10922
e1eb98ead97d9b22fe4538601cbcd193e998baad 127.0.0.1:[email protected] slave f10fc263b1b887fe175307b2fb7600370d39c0e1 0 1604675571000 8 connected
f8c884f07375cb524236341c4fcb963f58f82622 127.0.0.1:[email protected] slave 2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 0 1604675572509 7 connected
ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 127.0.0.1:[email protected] master - 0 1604675574514 3 connected 10923-16383
0f75c7f994a43551671e955a9f72c28a9c6edb56 127.0.0.1:[email protected] slave 1efc0139a5422c68f1c43428d1d6f566ce361fed 0 1604675573511 6 connected

Simulated downtime

FirstA pit: in the cluster mode, you cannot directly kill -9 a node. You can only use kill (of course, we use the cluster forget command to simulate downtime). Kill – 9. An error will be reported when restarting the cluster. You need to delete pidfile, data, log and other things to restart the cluster successfully, so try not to kill – 9.

Just kill 6378

[[email protected]_0_10_centos conf]# redis-cli -c -h 127.0.0.1 -p 6391 -a 123456
127.0.0.1:6391> CLUSTER NODES
0f75c7f994a43551671e955a9f72c28a9c6edb56 127.0.0.1:[email protected] myself,master - 0 1604675774000 9 connected 0-5460
ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 127.0.0.1:[email protected] master - 0 1604675779737 3 connected 10923-16383
f8c884f07375cb524236341c4fcb963f58f82622 127.0.0.1:[email protected] slave 2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 0 1604675778000 7 connected
f10fc263b1b887fe175307b2fb7600370d39c0e1 127.0.0.1:[email protected] master - 0 1604675777000 8 connected 5461-10922
2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 127.0.0.1:[email protected] master - 0 1604675776000 7 connected
114d082bb43c67042c806e0060f5fb8157736b98 127.0.0.1:[email protected] slave ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 0 1604675779000 5 connected
1efc0139a5422c68f1c43428d1d6f566ce361fed 127.0.0.1:[email protected] master,fail - 1604675757260 1604675756000 1 disconnected
e1eb98ead97d9b22fe4538601cbcd193e998baad 127.0.0.1:[email protected] slave f10fc263b1b887fe175307b2fb7600370d39c0e1 0 1604675778000 8 connected

It is found that 6378 ismaster,fail, and its original slave node ismaster。 Restart node 6378 again.

[[email protected]_0_10_centos conf]# redis-server redis-6378.conf 
[[email protected]_0_10_centos conf]# redis-cli -c -h 127.0.0.1 -p 6391 -a 123456
127.0.0.1:6391> CLUSTER NODES
0f75c7f994a43551671e955a9f72c28a9c6edb56 127.0.0.1:[email protected] myself,master - 0 1604675996000 9 connected 0-5460
ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 127.0.0.1:[email protected] master - 0 1604675993000 3 connected 10923-16383
f8c884f07375cb524236341c4fcb963f58f82622 127.0.0.1:[email protected] slave 2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 0 1604675992368 7 connected
f10fc263b1b887fe175307b2fb7600370d39c0e1 127.0.0.1:[email protected] master - 0 1604675997000 8 connected 5461-10922
2eefbdc56047282fc80baf3cc7bc08ff86dbdf9e 127.0.0.1:[email protected] master - 0 1604675996376 7 connected
114d082bb43c67042c806e0060f5fb8157736b98 127.0.0.1:[email protected] slave ca998854f1d96e2b64b7304aea5e5b4c96a9d85a 0 1604675997376 5 connected
1efc0139a5422c68f1c43428d1d6f566ce361fed 127.0.0.1:[email protected] slave 0f75c7f994a43551671e955a9f72c28a9c6edb56 0 1604675995000 9 connected
e1eb98ead97d9b22fe4538601cbcd193e998baad 127.0.0.1:[email protected] slave f10fc263b1b887fe175307b2fb7600370d39c0e1 0 1604675998378 8 connected

Find the originalmasterNode 6378 becomes node 6391slaveNode.

However, the master-slave node conversion occurs only when the master node has a slot. When the master node has no slot, the conversion will not occur.For example, delete 6395 and restart it. It is found that 6395 is stillmaster, 6396 orslave

In addition, the node.conf mentioned above can not only help remove and restore (cluster saveconfig), but also record the data that has just been deletedCLUSTERCommand log.

The above isCluster modeThank you for watching.