5000 + words hard core dry goods! Redis distributed cluster deployment

Time:2021-1-28

5000 + words hard core dry goods! Redis distributed cluster deployment

Principle:

  • Redis cluster uses consistent hash slot to assign a hash slot to each master node in the cluster, and the data written is hashed and then allocated to a master node for storage.
  • The cluster uses the formula (CRC16 key) & 16384 to calculate the key data slot.
  • 16384 slots are evenly distributed on each node.
  • Each master node in the cluster will be responsible for the maintenance of some slot points, while the slot points store data, and each master node has at least one slave node for high availability.
Node communication mode:
  • Open a port with the port number of + 10000, which is used for communication and information exchange between nodes in the cluster.
  • By default, each node selects five random nodes to send Ping messages 10 times per second, and transmits its own information and the cluster information it knows. After receiving the Ping message, it returns the pong message as a reply. Finally, through this random message exchange, each node will get all the information.
  • When a master node fails, all nodes will find that the master node fails. As a slave node of the master node, it will take over the work of the master node, and then tell all other nodes that it has become the master node. In this way, other surviving nodes will update the information table they maintain, and the slave node will take over as the master. If they all hang up, the cluster will report an error. When a slave node operates, it will be stored in one of the master nodes according to the consistent hash calculation, and the slave node will synchronize the data of the master node.

5000 + words hard core dry goods! Redis distributed cluster deployment

  • Redis cluster is decentralized, each node in the cluster is equal, and each node saves its own data and the state of the whole cluster. Each node is connected to all other nodes, and these connections remain active.
  • When a cluster is built, a slave node is set up for each master node. At the same time, when the master node is down, the sentinel Sentry can automatically fail over

Redis distributed cluster (deployment)

Port number: 7000-7005

This time, the distributed partitioned cluster can be installed in one Linux system, and only multiple instances need to be installed as the cluster configuration.

Install Ruby environment support:
yum -y install ruby rubygems

Yum installs version 2.0.0, but gem needs more than version 2.2.2, so you choose to compile

Download and install the ruby environment:
wget https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.1.tar.gz
tar xf ruby-2.6.1.tar.gz && cd ruby-2.6.1/
./configure --prefix=/usr/local/ruby
make && make install && echo $?
echo 'export PATH=$PATH:/usr/local/ruby/bin' >> /etc/profile
source /etc/profile
Modify the gem tool
#View the source address of gem tool
gem sources -l
#Add an alicloud gem tool source
gem sources -a http://mirrors.aliyun.com/rubygems/
#Delete the default value of gem tool
gem sources --remove https://rubygems.org/
#Download the latest version of cluster plug-in
gem install redis -v 4.1.0
Cluster node preparation:
mkdir /data/700{0..5}
Configure port 7000 instance:
vim /data/7000/redis.conf
port 7000
daemonize yes
pidfile /data/7000/redis.pid
loglevel notice
logfile "/data/7000/redis.log"
dbfilename dump.rdb
dir /data/7000
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
Copy other port instances:
#Copy configuration file CP / data / 7000/ redis.conf  /data/7001/
cp /data/7000/redis.conf /data/7002/
cp /data/7000/redis.conf /data/7003/
cp /data/7000/redis.conf /data/7004/
cp /data/7000/redis.conf /data/7005/
#Modify configuration file content
sed -i 's#7000#7001#g' /data/7001/redis.conf
sed -i 's#7000#7002#g' /data/7002/redis.conf
sed -i 's#7000#7003#g' /data/7003/redis.conf
sed -i 's#7000#7004#g' /data/7004/redis.conf
sed -i 's#7000#7005#g' /data/7005/redis.conf
Start all instances:
redis-server /data/7000/redis.conf
redis-server /data/7001/redis.conf
redis-server /data/7002/redis.conf
redis-server /data/7003/redis.conf
redis-server /data/7004/redis.conf
redis-server /data/7005/redis.conf
Create command soft link:
(this command has expired, now use redis cli command) (optional execution command)
ln -s /usr/local/redis-5.0.2/src/redis-trib.rb /usr/sbin/
View process:
ps -ef |grep redis-server

5000 + words hard core dry goods! Redis distributed cluster deployment

Join all instance nodes to cluster management:
#-- replicas 1 ", 1 means that each master has a slave, followed by the address and port information of all nodes
redis-cli --cluster create --cluster-replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

The distributed master-slave rule is that the first three instance nodes are master nodes, and the corresponding last three instance nodes are slave nodes. If replicas 2 is used, three more instance nodes will be added

View master node status:
redis-cli -p 7000 cluster nodes|grep master

5000 + words hard core dry goods! Redis distributed cluster deployment

To view the slave node status:
redis-cli -p 7000 cluster nodes|grep slave

5000 + words hard core dry goods! Redis distributed cluster deployment

Redis distributed cluster (Management)

Cluster node increase preparation:

mkdir /data/700{6..7}

Copy other port instances:
#Copy configuration file CP / data / 7000/ redis.conf  /data/7006/
cp /data/7000/redis.conf /data/7007/

#Modify configuration file content
sed -i 's#7000#7006#g' /data/7006/redis.conf
sed -i 's#7000#7007#g' /data/7007/redis.conf

Start new node instance:

redis-server /data/7006/redis.conf
redis-server /data/7007/redis.conf
View process:
ps -ef |grep redis-server

5000 + words hard core dry goods! Redis distributed cluster deployment

Add master node: (7000 instances are management nodes)
#'add the 7006 instance to the cluster where the master node of the 7000 instance is located (there are already four master nodes)
redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000
View master node status:
redis-cli -p 7000 cluster nodes|grep master

5000 + words hard core dry goods! Redis distributed cluster deployment

Transfer slot:
#'operate the cluster management node to re allocate, specify the partition size in the interactive interface, and select the node ID to receive the partition
redis-cli --cluster reshard 127.0.0.1:7000

How many slots do you want to move (from 1 to 16384)? 4096
#Manually calculate the number of the total size of data slices divided by the master node

What is the receiving node ID? 2129d28f0a86fc89571e49a59a0739812cff7953
#Select the node ID to receive the data fragment (this is the ID number of the new node 7006 instance)

Source node #1: all
#Select which source master nodes to reparticate to the new master (all is all nodes)
)

Do you want to proceed with the proposed reshard plan (yes/no)? yes           
#Confirm to modify the above operation
Re view the status of the primary node: (you can see the re fragmentation of the cluster data)
redis-cli -p 7000 cluster nodes|grep master

5000 + words hard core dry goods! Redis distributed cluster deployment

Add a slave node:
#'add the 7007 instance node to the 7006 instance master node, and specify the corresponding 7006 instance master node to sit on the management node of the cluster
redis-cli --cluster add-node 127.0.0.1:7007 127.0.0.1:7000 --cluster-slave --cluster-master-id 2129d28f0a86fc89571e49a59a0739812cff7953
To view the slave node status:
redis-cli -p 7000 cluster nodes|grep slave

5000 + words hard core dry goods! Redis distributed cluster deployment

Cluster node deletion preparation:

Move the data fragment of the node to be deleted:
#'operate the cluster management node to re allocate, specify the partition size in the interactive interface, and select the node ID to receive the partition
redis-cli --cluster reshard 127.0.0.1:7000

#The method is to delete the master node according to the location of the partition, and then a component of a node, you can also directly move all the data slices to a node
How many slots do you want to move (from 1 to 16384)? 1365                    

#Manually view the total size of data slices
What is the receiving node ID? e64f9074a3733fff7baa9a4848190e56831d5447

#Select the node ID to receive the data fragment (this is the ID number of the new node 7006 instance)
Source node #1: 2129d28f0a86fc89571e49a59a0739812cff7953

#Select which source master node to re fragment to the new master node (this is the ID number of the master node to be deleted) 

done
#This is the end order

Do you want to proceed with the proposed reshard plan (yes/no)? yes           
#Confirm to modify the above operation
Re view the status of the primary node: (you can see the re fragmentation of the cluster data)
redis-cli -p 7000 cluster nodes|grep master

5000 + words hard core dry goods! Redis distributed cluster deployment

Continue to move slices:
#'operate the cluster management node to re allocate, specify the partition size in the interactive interface, and select the node ID to receive the partition
redis-cli --cluster reshard 127.0.0.1:7000

#The method is to delete the master node according to the location of the partition, and then a component of a node, you can also directly move all the data slices to a node
How many slots do you want to move (from 1 to 16384)? 1366                    

#Manually view the total size of data slices
What is the receiving node ID? f6c1aaea3a8c56e0c7dee8ad7ae17e26dd04244c

#Select the node ID to receive the data fragment (this is the ID number of the new node 7006 instance)
Source node #1: 2129d28f0a86fc89571e49a59a0739812cff7953

#Select which source master node to re fragment to the new master node (this is the ID number of the master node to be deleted) 

done
#This is the end order

Do you want to proceed with the proposed reshard plan (yes/no)? yes           
#Confirm to modify the above operation
Re view the status of the primary node: (you can see the re fragmentation of the cluster data)
redis-cli -p 7000 cluster nodes|grep master

5000 + words hard core dry goods! Redis distributed cluster deployment

Last move data slice:
#'operate the cluster management node to re allocate, specify the partition size in the interactive interface, and select the node ID to receive the partition
redis-cli --cluster reshard 127.0.0.1:7000

#The method is to delete the master node according to the location of the partition, and then a component of a node, you can also directly move all the data slices to a node
How many slots do you want to move (from 1 to 16384)? 1365
Re view the status of the primary node: (you can see the re fragmentation of the cluster data)
redis-cli -p 7000 cluster nodes|grep master

5000 + words hard core dry goods! Redis distributed cluster deployment

Delete the master node of the empty data slice:
#'delete 7006 instances that have been cleared
redis-cli --cluster del-node 127.0.0.1:7006 2129d28f0a86fc89571e49a59a0739812cff7953
#Delete 7007 instances without a master library
redis-cli --cluster del-node 127.0.0.1:7007 821bcf78c5e4c0b08aa7a5d514214b657ebec4ab

5000 + words hard core dry goods! Redis distributed cluster deployment

Other configuration management:

#View memory information
redis-cli -p 6382 -a redhat info memory

#Set a maximum of 100MB of memory
redis-cli -p 6382 -a redhat config set maxmemory 102400000

If there are errors or other problems, welcome to comment and correct. If you have any help, please click like + forward to share.

Author: brother migrant workers
Link: https://segmentfault.com/a/11…
Source: segmentfault
The copyright belongs to the author. For commercial reprint, please contact the author for authorization. For non-commercial reprint, please indicate the source.

Recommended Today

020_CSS3

catalog How to learn CSS What is CSS History of development quick get start Advantages of CSS Three ways to import CSS Expansion: two ways of writing external style selector Basic selector Hierarchy selector Structure pseudo class selector attribute selectors Beautify web page elements Why beautify web pages Span label: for the text that needs […]