Redis cluster (document arrangement)

Time:2021-3-9

Redis cluster

·Redis cluster provides a way to run redis installation. In this installation, data will be automatically divided among multiple redis nodes.

Redis cluster also provides a certain degree of availability during partition, which is actually the ability to continue working when some nodes fail or cannot communicate. However, if a more serious failure occurs (for example, when most of the primary nodes are unavailable), the cluster will stop running.

In fact, what can redis cluster bring to you?

  • The ability to automatically split data sets between multiple nodes.
  • When some nodes fail or cannot communicate with other nodes in the cluster, they can continue to operate.

Redis cluster TCP port

Each redis cluster node needs to open two TCP connections. General redistcp port for service clients, such as6379And add10000For example, in the example16379

The second port with a larger value is used for the cluster bus, which is the node to node communication channel using the binary protocol. Nodes use cluster bus for fault detection, configuration update, fail over authorization and so on. The client should never try to communicate with the cluster bus port, but should always communicate with the common redis command port. However, please make sure that you open both ports in the firewall, otherwise the redis cluster node will not be able to communicate.

The offset between the command port and the cluster bus port is fixed and is always zero10000

Please note that for each node, to make the redis cluster work properly, you need to:

  1. The normal communication port (usually 6379) used to communicate with clients is open to all clients who need to access the cluster and all other cluster nodes (using client port for key migration).
  2. The cluster bus port (client port + 10000) must be accessible from all other cluster nodes.

If you do not open both TCP ports at the same time, the cluster will not work properly.

Cluster bus uses different binary protocols for node to node data exchange, which is more suitable for exchanging information between nodes with less bandwidth and processing time.

Redis cluster data fragmentation

Redis cluster does not use consistent hash, but uses a different form of fragmentation. Conceptually, each key is a part of what we call hash slot.

There are 16384 hash slots in the redis cluster. To calculate the hash slot of a given key, you need to modulus the CRC16 value of the key with 16384.

Each node in the redis cluster is responsible for a subset of hash slots. For example, you may have a cluster with three nodes, where:

  • Node a contains hash slots from 0 to 5500.
  • Node B contains hash slots from 5501 to 11000.
  • Node C contains hash slots from 11001 to 16383.

This makes it easy to add and remove nodes from the cluster. For example, if I want to add a new node D, I need to move some hash slots from nodes a, B, C to d. Similarly, if I want to remove node a from the cluster, I just need to move the hash slots served by a to B and C. When node a is empty, I can completely remove it from the cluster.

Because there is no need to stop moving hash slots from one node to another, there is no need to stop adding or deleting nodes or changing the percentage of hash slots served by nodes.

As long as all the keys involved in a single command execution (or the whole transaction or Lua script execution) belong to the same hash slot, redis cluster supports multi key operation. Users can force multiple keys to be part of the same hash slot by using a concept called hash tag.

The hash tag is recorded in the redis cluster specification. Note that if there is a substring between the brackets in {} of the key, only the contents in the string will be hashed. For example, a key named {foo} and another key named {foo} are guaranteed to be in the same hash slot, and can be used together in commands with multiple keys as parameters.

Redis cluster active standby mode

In order to remain available when the primary node subset fails or fails to communicate with most nodes, the redis cluster uses the primary standby model, in which each hash slot has from 1 (the primary node itself) to N copies (n – 1 other backup nodes).

In a cluster containing nodes a, B and C, if node B fails, the cluster will not be able to continue because we cannot provide services for hash slots in the range of 5501-11000. However, when creating the cluster (or later), we add a backup node to each primary node, so that the final cluster is composed of a, B, C as the primary node and A1, B1, C1 as the backup node. If node B fails, the system will be able to run continuously. If node B1 replicates B and B fails, the cluster will promote node B1 to a new master node and continue to operate normally.

However, please note that if node B and B1 fail at the same time, the redis cluster cannot continue to run.

Consistency guarantee of redis cluster

Redis cluster cannot guarantee strong consistency. In fact, this means that in some cases, the redis cluster may lose the confirmed write operation from the system to the client.

The first reason the redis cluster may lose writes is because it uses asynchronous replication. This means that in the write period
The following will happen between the two times:

  • Your client writes data to the B master node.
  • Master B replies “OK” to your client.
  • The B primary node propagates write operations to its backup nodes B1, B2, and B3.

As you can see, B will not wait for the confirmation of B1, B2, B3 before replying to the client, because this will delay redis. Therefore, if your client has a write operation, then B will confirm, but it crashes before it sends the write operation to the backup node. At this time, one of the backup nodes (not receiving the write operation) can be upgraded to the primary node, so it will be lost forever This is a write operation. This is very similar to what happens to most databases that are configured to refresh data to disk every second, so you can reasonably infer this due to past experience with traditional database systems that do not involve distributed systems. Similarly, you can improve consistency by forcing the database to refresh the data on disk before replying to the client, but this often leads to low performance. In a redis cluster, this is equivalent to synchronous replication.

Basically, a trade-off between performance and consistency is necessary.

Redis cluster can realize synchronous write through wait command when needed, which greatly reduces the possibility of lost write. However, please note that even if synchronous replication is used, redis cluster does not realize strong consistency: in more complex cases, there is always a scenario where a backup node that cannot receive data is selected as the primary node.

Another noteworthy situation is that the redis cluster will also lose the write operation. This situation occurs during the network partition, in which the client is isolated from a few instances (including at least the primary node).

Take our six node cluster as an example, which is composed of a, B, C, A1, B1 and C1, and has three main nodes
And three backup nodes. There’s another customer, which we call Z1.

After partition, there may be a, C, A1, B1, C1 on one side of the partition, and B and Z1 on the other side.

Z1 can still write to B, and B will accept it. If the partition returns to normal in a short time, the cluster will continue to run normally. However, if the partition lasts long enough for B1 to be upgraded to the master node of the majority end of the partition, the write operation sent by Z1 to B will be lost.

Note that Z1 will be able to send a maximum number of writes to B: if there is enough time for the large partition to elect a master node, each master node in the small partition will stop accepting writes. This time is a very important configuration instruction of redis cluster, which is called node timeout.

After a node times out, the primary node is considered to have failed and can be replaced with one of its replicas. Similarly, after a specified period of time, the master node is still unable to perceive most of the other master nodes. The master node enters an error state and stops accepting writes.

Redis cluster configuration parameters

We will create a cluster deployment as an example. Before we go on, let’s talk about the redis.conf Configuration parameters introduced in the file.
Read on and you’ll get more clear points.

  • cluster-enabled : if set to yes, the redis instance will enable cluster support. Otherwise, the instance will start as a stand-alone instance as usual.
  • cluster-config-file Note that despite this option, it is a redis set that does not allow users to edit
    Each time a cluster node changes (basically state), it will automatically persist the cluster configuration file so that it can be read again at startup.
    This file lists the contents, States, persistent variables, etc. of other nodes in the cluster. Usually, when some message is received, this file will be deleted
    Rewrite and refresh to disk.
  • Cluster node timeout: the longest interval before a redis cluster node is considered to have failed. If the primary node cannot be accessed for more than a specified length of time, its backup node will
    It fails over. This parameter also controls other important things in the redis cluster. It is worth noting that most of the primary nodes cannot be connected within the specified time
    Each node of will stop accepting query requests.
  • cluster-slave-validity-factor : if set to zero, the backup node will always attempt to fail over the primary node regardless of the link between the primary node and the backup node
    The duration of the state. If the value is positive, the maximum disconnection time is calculated as the node timeout times the factor provided by this option if the node is a backup
    Node, and the primary link has been broken for more than the specified time, it will not attempt to initiate a failover. For example, if the node timeout is set to 5 seconds and there is a
    If the availability factor is set to 10, the backup node will not attempt to fail over its primary node if it is disconnected from the primary node for more than 50 seconds. Please note that if not
    If the backup node can fail over, any non-zero value may cause the redis cluster to be unavailable after the failure of the primary node.
    In this case, the cluster will return to the available state only when the original master node rejoins the cluster.
  • cluster-migration-barrier : the primary node needs to keep the minimum number of connected backup nodes so that another backup node can migrate to a primary node that is not covered by any backup nodes
    Node. For more information, see the appropriate section on replica migration in this tutorial.
  • cluster-require-full-coverage : if it is set to yes, by default, if a node does not cover a certain proportion of the key space, the cluster will stop accepting writes. If this option is set to
    Set to no, even if this node can only process requests about a partial subset of keys, the cluster will still provide queries.

Creating and using a redis cluster

Note: it is very important to understand some operations of redis cluster manually. However, if you want to set up a cluster and run it as soon as possible, please skip this section and the next section and go directly to create a redis cluster using the create cluster script.

To create a cluster, the first thing we need to do is to let some empty redis instances run in cluster mode. Basically, this means that you can’t use a regular redis instance to create a cluster, because you need to configure a special mode so that the redis instance can enable cluster specific functions and commands.

The following is the smallest redis cluster configuration file:

Port 7000 # port number
Cluster enabled yes # is to become a cluster node
cluster-config-file  nodes.conf   #Node profile
Cluster node timeout 5000
Appendonly yes # use AOF

To enable cluster mode, you only need to directly open the cluster enabled command. Each instance also contains the file path of the node’s configuration storage location. By default, it is nodes.conf . The document will not be touched. It is only generated by the redis cluster instance at startup and updated when needed.

Note that the minimum cluster that works as expected requires at least three primary nodes. For your first test, it is strongly recommended to start a six node cluster with three primary nodes and three backup nodes. To do this, enter a new directory and create the following directory, named after the port number of the instance we will run in the given directory. It’s like:

mkdir cluster-test
cd cluster-test
mkdir 7000 7001 7002 7003 7004 7005

Create one in each directory from 7000 to 7005 redis.conf Documents. As a template for the configuration file, just use the small example above, but make sure to replace port number 7000 with the correct port number based on the directory name. Now, copy your redis server executable (compiled from the latest resources in the GitHub unstable branch) to the cluster test directory, and finally open six terminal tabs in your favorite terminal application.

Start each instance like this, one for each tab:

cd 7000
../redis-server ./redis.conf

From the log of each instance, it can be seen that the nodes.conf File, so each node is assigned a new ID.

[82462] 26 Nov 11:56:55.329 * No cluster configuration found, I'm 97a3a64667477371c4479320d683e4c8db5858b1

The ID will be permanently used by the instance so that the instance has a unique name in the context of the cluster. Each node uses the ID instead of the IP or port to remember each other node. The IP address and port may change, but the unique node identifier does not change throughout the lifetime of the node. We call this identifier node ID for short.

Creating clusters

Now, we have many instances running, and then we need to create the cluster by writing some meaningful configuration to the nodes.

If you are using redis 5, this is easy to complete. This is because redis cluster command line utility is embedded in redis cli. We can use it to create new clusters, check or reparticate existing clusters, etc.

For redis version 3 or 4, there is one called redis- trib.rb It’s very similar to the older tool. You can find it in the SRC directory of redis source distribution. You need to install redis gem to run redis trib.

gem install redis

The first example, cluster creation, will be displayed using redis cli in redis 5 and redis trib in redis 3 and 4. However, all of the following examples will only use redis cli, because you can see that their syntax is very similar, and you can also use redis- trib.rb Help to get information about the syntax to simply change one command line to another. Important: Please note that redis 5 redis cli can be used for redis 4 cluster if necessary.

To create a cluster for redis 5 using redis cli, simply type:

redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
--cluster-replicas 1

For redis 4 or 3, use redis- trib.rb Tools:

./redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 \
127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005

The command used here is create, because we want to create a new cluster. option
–Cluster replicas 1 means that we want each created primary node to have a backup node. The other parameters are the instances to be used to create a new cluster
List of addresses for.

Obviously, the only setting that meets our requirements is to create a cluster with three master nodes and three slave nodes.

Redis cli will provide configuration for you. Type yes to accept the recommended configuration. The cluster will be configured and joined, which means that the instances will be started and connected to each other
You can talk to each other. Finally, if all goes well, you will see the following message:

[OK] All 16384 slots covered

This means that at least one master instance serves one of 16384 slots

Use the create cluster script to create a redis cluster

If you don’t want to create a redis cluster by manually configuring and executing a single instance as described above, you can use a simpler system (but you won’t learn as much about it).

Just check the utils / create cluster directory in the redis distribution. Inside there is a script named create cluster (with the same name as the directory contained in it), which is a simple bash script. To start a 6-node cluster with 3 primary nodes and 3 backup nodes, just type the following command:

  1. create-cluster start
  2. create-cluster create

In step 2, when redis cli wants you to accept the cluster layout, reply yes.

Now you can interact with the cluster, and by default the first node will start at port 30001. When finished, stop the cluster with the following command:

create-cluster stop.

For more information on how to run this script, read readme in the directory.

Cluster operation

So far, one of the problems of redis cluster is the lack of implementation of client library.

As far as I know, there are the following implementations:

  • redis-rb-clusterIt is a ruby implementation written by me (@ antirez), which can be used as a reference for other languages. It is a simple wrapper of the original redis Rb, which realizes the minimum semantics to communicate with the cluster effectively.
  • redis-py-clusterThe python implementation of redis RB cluster. Most redis py functions are supported. It is developing actively.
  • fashionablePredisIt supports redis cluster, which has been updated recently and is under active development.
  • The most widely used java client,JedisRecently, support for redis cluster has been added. Please refer to jedis cluster in readme.
  • StackExchange.RedisProvide support for C # (and should be compatible with most. Net languages, VB, f #)
  • thunk-redisProvide information on Node.js and io.js It is a redis client based on thunk / promise, with pipeline and cluster functions.
    redis-go-clusterIt’s the go language
    Implementation of redis cluster, which usesRedigo library clientAs a basic client, mget / Mset is implemented through result aggregation.
  • ioredisIt’s popular Node.js The client provides strong support for redis cluster.
  • When the – C switch is used to start, the redis cli program realizes the basic cluster support.

A simple way to test the redis cluster is to try any of the above clients, or just try the redis cli command. Here is an example of using the latter for interaction:

$ redis-cli -c -p 7000
redis 127.0.0.1:7000> set foo bar
-> Redirected to slot [12182] located at 127.0.0.1:7002
OK
redis 127.0.0.1:7002> set hello world
-> Redirected to slot [866] located at 127.0.0.1:7000
OK
redis 127.0.0.1:7000> get foo
-> Redirected to slot [12182] located at 127.0.0.1:7002
"bar"
redis 127.0.0.1:7000> get hello
-> Redirected to slot [866] located at 127.0.0.1:7000
"world"

be careful: if you use a script to create a cluster, the node may listen on different ports, starting from 30001 by default.

The support of redis cli is very basic, so it is always based on the fact that redis cluster nodes can redirect clients to the right nodes. A strict client can do better and cache the mapping between hash slot and node address to directly use the correct connection with the node. The mapping is refreshed only when certain changes occur in the cluster configuration (for example, after a failover or after the system administrator changes the cluster layout by adding or removing nodes).

Use redis RB cluster to write a simple application

Before continuing to show how to operate the redis cluster, such as performing operations such as fail over or re partitioning, we need to create some sample applications, or at least be able to understand the semantics of simple redis cluster client interaction.

In this way, we can run an example and try to make the nodes fail or start to re slice at the same time to understand the behavior of redis cluster in real environment. It’s not helpful to just look at a cluster that doesn’t write any data.

This section describes some basic usage of redis RB cluster, and shows two examples. The first is the following, which is from the redis RB cluster distribution example.rb File:

1  require './cluster'
   2
   3  if ARGV.length != 2
   4      startup_nodes = [
   5          {:host => "127.0.0.1", :port => 7000},
   6          {:host => "127.0.0.1", :port => 7001}
   7      ]
   8  else
   9      startup_nodes = [
  10          {:host => ARGV[0], :port => ARGV[1].to_i}
  11      ]
  12  end
  13
  14  rc = RedisCluster.new(startup_nodes,32,:timeout => 0.1)
  15
  16  last = false
  17
  18  while not last
  19      begin
  20          last = rc.get("__last__")
  21          last = 0 if !last
  22      rescue => e
  23          puts "error #{e.to_s}"
  24          sleep 1
  25      end
  26  end
  27
  28  ((last.to_i+1)..1000000000).each{|x|
  29      begin
  30          rc.set("foo#{x}",x)
  31          puts rc.get("foo#{x}")
  32          rc.set("__last__",x)
  33      rescue => e
  34          puts "error #{e.to_s}"
  35      end
  36      sleep 0.1
  37  }

The program does a very simple thing, it will fooThe value of the form key is set to a number and is incremented one by one.
Therefore, if you run the program, the result will be the same as the following command:

  • SET foo0 0
  • SET foo1 1
  • SET foo2 2

The program looks more complex because it needs to display errors on the screen rather than exit abnormally, so every operation performed on the cluster should be wrapped by error handling.

Line 14 is the first interesting line in the program. It creates a redis cluster object, uses the start node list as a parameter, and allows the object to establish the maximum number of connections with different nodes. The last is the timeout. For a given operation, how many times is considered a failure.

The boot node does not need to be all nodes of the cluster. But at least one node is reachable. Also note that redis RB cluster will update the list of boot nodes as long as it can connect to the first node. You should expect this behavior from any other strict client.

Now that we have stored the redis cluster object instance in the RC variable, we can use the object just like the ordinary redis object instance.

This happens on lines 18 to 26: when we restart the example, we don’t want to restart with foo0, so we store the counter in redis itself. The purpose of the above code is to read this counter or assign it a value of zero if it does not exist.

But note that this is a while loop, because even if the cluster is shut down and an error is returned, we will try again and again. Ordinary applications don’t need to be that careful.

Start the main loop between 28 and 37, in which the key is set or an error is displayed.

Note the sleep call at the end of the loop. In the test, if you want to write to the cluster as quickly as possible, you can delete the sleep (relatively speaking, it’s just a busy circular operation, it doesn’t have real parallelism, so under the best conditions, you usually get 10K operations per second).

Usually, in order to make the sample program easier to understand, the writing speed will slow down. Starting the application produces the following output:

ruby ./example.rb
1
2
3
4
5
6
7
8
9
^C (I stopped the program here)

It’s not a very interesting program. We’ll use a better program later, but we can see what happens when the program is running, during the re slicing.

Cluster re fragmentation

Now, we are ready to try to reparticate the cluster. For this, please keep example.rb Program running, so that you can see if it has an impact on the running of the program. In addition, you may want to comment on the sleep call so that some more severe write load occurs during repartitioning. Repartitioning basically means moving hash slots from one group of nodes to another, and it is done using redis cli program just like cluster creation.

To start repartitioning, just type:

redis-cli --cluster reshard 127.0.0.1:7000

You only need to specify one node, and redis cli will automatically find other nodes.

Currently, redis cli can only be repartitioned with the support of the administrator. You can’t just move 5% of the slots from this node to another node (of course, it’s very easy to implement). So it will start with a question. First of all, how many slices do you want to do

How many slots do you want to move (from 1 to 16384)?

We can try to reassign 1000 hash slots. If the example is still running and there is no sleep call, the hash slot should already contain a small number of keys.

Then redis cli needs to know what the target of re slicing is, that is, the node that will receive the hash slot. I will use the first master node, 127.0.0.1:7000, but I need to specify the node ID of the instance. Redis cli has printed it in the list, but if necessary, I can also use the following command to find the ID of the node:

$ redis-cli -p 7000 cluster nodes | grep myself
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5460

So my target node should be 97a3a6466777371c4479320d683e4c8db58b1.

Now, it will ask which nodes you want to get the keys from. I just type all to get some hash slots from all the other master nodes.

After the final confirmation, you will see a message indicating that redis cli will be moved from one node to another, and each actual key that will be moved from one side to the other will be printed.

During the reparting process, you should be able to see that the running of the sample program is not affected. If necessary, you can also stop and restart it multiple times during the refactoring. At the end of repartitioning, you can test the health of the cluster using the following command:

redis-cli --cluster check 127.0.0.1:7000

All slots will be covered, but this time the 127.0.0.1:7000 master node will have more hash slots
About 6461.

A more interesting example application

The sample program we wrote before is not very good. It writes to the cluster in a simple way, without even checking whether the writing is correct. From our point of view, the cluster receiving write operations can always write the key named foo to the hash slot 42 in each operation, which we will not notice at all. Therefore, in the redis RB cluster code warehouse, there is a more interesting program called consistency- test.rb . It uses a set of counters, 1000 by default, and sends the incr command to increase the value of the counter. However, the application can not only write data, but also do two things:

  • When the counter is updated with incr, the application remembers the write.
  • It also reads a random counter before each write, checks that the value meets our expectations, and compares it with the value in memory.

This means that the program is a simple consistency checker, which can tell you whether the cluster has lost some writes or whether it has accepted writes that we have not received confirmation. In the first case, we’ll see that a counter’s value is smaller than the one we remember earlier, and in the second case, it’s larger.

Running the conformance test application produces one line of output per second:

$ ruby consistency-test.rb
925 R (0 err) | 925 W (0 err) |
5030 R (0 err) | 5030 W (0 err) |
9261 R (0 err) | 9261 W (0 err) |
13517 R (0 err) | 13517 W (0 err) |
17780 R (0 err) | 17780 W (0 err) |
22025 R (0 err) | 22025 W (0 err) |
25818 R (0 err) | 25818 W (0 err) |

This line shows the number of reads and writes performed, as well as the number of errors (queries cannot be accepted due to errors because the system is unavailable). If inconsistencies are found, new rows are added to the output. For example, if I reset the counter manually while the program is running, this happens:

$ redis-cli -h 127.0.0.1 -p 7000 set key_217 0
OK

(in the other tab I see...)

94774 R (0 err) | 94774 W (0 err) |
98821 R (0 err) | 98821 W (0 err) |
102886 R (0 err) | 102886 W (0 err) | 114 lost |
107046 R (0 err) | 107046 W (0 err) | 114 lost |

When I set the counter to 0, the actual value is 114, so the program will report that the write of 114 is lost (incr command that the cluster cannot remember). This program is more interesting as a test case, so we will use it to test redis cluster failover.

Test failover

Note: during this test, you should open a tab and run the conformance test application on it.

The simplest thing we can do to trigger a failover (that is, the most semantically simple failure that can occur in a distributed system) is to crash a single process, in our case a single host.

We can use the following command to identify and crash the master node:

$ redis-cli -p 7000 cluster nodes | grep master
3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385482984082 0 connected 5960-10921
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 master - 0 1385482983582 0 connected 11423-16383
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5959 10922-11422

OK, now 700070017002 are all master nodes. Let’s crash the 7002 machine with the debug segfail command.

$ redis-cli -p 7002 debug segfault
Error: Server closed the connection

Now we can see what the output report of this conformance test is.

18849 R (0 err) | 18849 W (0 err) |
23151 R (0 err) | 23151 W (0 err) |
27302 R (0 err) | 27302 W (0 err) |

... many error warnings here ...

29659 R (578 err) | 29660 W (577 err) |
33749 R (578 err) | 33750 W (577 err) |
37918 R (578 err) | 37919 W (577 err) |
42077 R (578 err) | 42078 W (577 err) |

As you can see, 578 reads and 577 writes were not accepted during the failover, but no inconsistent data was created in the database.

This may sound like an accident, because in the first part of this tutorial, we said that the redis cluster will lose write operations during a failover because it uses asynchronous replication. What we didn’t say is that this is unlikely to happen, because redis will send a response to the client, and the same command will be copied to the backup node almost at the same time, so the window of data loss is very small. However, the fact that it is difficult to trigger does not mean that it is impossible, so it will not change the consistency guarantee provided by redis cluster.

Now, we can check what the cluster settings are after the failover (note that I restarted the crashed instance so that it rejoins the cluster as a backup node):

$ redis-cli -p 7000 cluster nodes
3fc783611028b1707fd65345e763befb36454d73 127.0.0.1:7004 slave 3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 0 1385503418521 0 connected
a211e242fc6b22a9427fed61285e85892fa04e08 127.0.0.1:7003 slave 97a3a64667477371c4479320d683e4c8db5858b1 0 1385503419023 0 connected
97a3a64667477371c4479320d683e4c8db5858b1 :0 myself,master - 0 0 0 connected 0-5959 10922-11422
3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 127.0.0.1:7005 master - 0 1385503419023 3 connected 11423-16383
3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385503417005 0 connected 5960-10921
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385503418016 3 connected

The master node is now running on ports 7000, 7001, and 7005. The former primary node (the redis instance running on port 7002) is now the backup node of 7005.

CLUSTER NODESThe output of a command may look complex, but it’s actually very simple, consisting of the following tags:

  • Node ID
  • IP: Port
  • Flag bits: primary node, backup node, myself, failure status
  • If you are a backup node, it is the ID of its primary node
  • The duration of time that a reply has not been received since the last Ping was sent
  • Time of last received pong
  • The configuration of epoch (see cluster specification)
  • The link status of this node
  • Slot for service

Manual fail over

Sometimes it’s useful to force a failover without actually causing any problems for the primary node. For example, in order to upgrade the redis process of one of the primary nodes, it is better to fail over it to turn it into a backup node with the least impact on availability.

Redis cluster usageCLUSTER FAILOVERThe command supports manual fail over, which must be performed in one of the backup nodes of the primary node to fail over.

Compared with the actual primary server failure caused by the fail over, manual fail over is not the same, but it is more secure, because they trigger the way to avoid the data loss in this process. Only after the system determines that the new primary node has been running and replaced the data replication function of the old primary node, can the client switch from the original primary node to the new primary node Point.

When performing manual fail over, you can see in the backup node log:

# Manual failover user request accepted.
# Received replication offset for paused master manual failover: 347540
# All master replication stream processed, manual failover can start.
# Start of election delayed for 0 milliseconds (rank #0, offset 347540).
# Starting a failover election for epoch 7545.
# Failover election won: I'm the new master.

Basically, all the clients connected to the master node that we are going to fail over are stopped. At the same time, the primary node sends its copy offset to the backup node, and the backup node will wait for the offset to be received. When the replication offset is complete, the failover begins and the old master node is notified of the configuration switch. When clients are unlocked on the old master node, they will be redirected to the new master node.

Add new node

The basic process of adding a new node is to first add an empty node, and then move some data into the node (if it is a new primary node), or tell it to set it as a copy of a known node (if it is a backup node). Starting with adding a new master node, we’ll show both. In both cases, the first step to perform is to add an empty node. This is as simple as starting a new node in port 7006 (the existing six nodes have used the new node from 7000 to 7005). Except for the port number, all other nodes use the same configuration, so you should operate in order to conform to the settings used by our previous nodes:

  • Open a new tab on your terminal application.
  • Enter the cluster test directory
  • Create a folder named 7006
  • Create in folder redis.conf File, just like other nodes that are already in use, is just changed to port 7006
  • Finally, through the command.. / redis server/ redis.conf Start the service,

At this point, the service should be running. Now we can use redis cli to add a node to an existing cluster.

redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000

As you can see, I use the add node command to specify the address of the new node as the first parameter and the address of the node that randomly exists in the cluster as the second parameter. In fact, redis cli is of no use to us here. It just sends a cluster message to the node, which can also be done manually. However, redis cli will check the status of the cluster before running, so even if you know how the internal structure works, it is still a good idea to perform cluster operations through redis cli.

Now we can connect to the new node to see if it really joined the cluster:

redis 127.0.0.1:7006> cluster nodes
3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385543178575 0 connected 5960-10921
3fc783611028b1707fd65345e763befb36454d73 127.0.0.1:7004 slave 3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 0 1385543179583 0 connected
f093c80dde814da99c5cf72a7dd01590792b783b :0 myself,master - 0 0 0 connected
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385543178072 3 connected
a211e242fc6b22a9427fed61285e85892fa04e08 127.0.0.1:7003 slave 97a3a64667477371c4479320d683e4c8db5858b1 0 1385543178575 0 connected
97a3a64667477371c4479320d683e4c8db5858b1 127.0.0.1:7000 master - 0 1385543179080 0 connected 0-5959 10922-11422
3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 127.0.0.1:7005 master - 0 1385543177568 3 connected 11423-16383

Note that since this node is already connected to the cluster, it has been able to redirect client queries correctly, and it is generally part of the cluster. However, compared with other master nodes, it has two characteristics

  • Since there is no hash slot assigned, no data is saved.
  • Because it is a host with no slot assigned, it will not participate in the selection when the backup node wants to be the primary node
    It’s a process.

Now you can use redis cli’s repartition function to assign hash slots to this node. As shown in the previous section, I will not show them here. There is no difference in their operation. They just re slice the empty node as the target.

Add a node as a replica (backup node)

There are two ways to add a backup node. The most common is to use redis cli, but use--cluster-slaveOptions, like this:

redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000 --cluster-slave

Note that the command line here is exactly the same as the one we used to add a new master, so we did not specify the master to which to add the replica. In this case, what redis cli needs to do is to add the new node to the replica of the random master node in the master node with fewer replicas. However, you can use the following command line to specify the master node that you want to use with the new replica:

redis-cli --cluster add-node 127.0.0.1:7006 127.0.0.1:7000 --cluster-slave --cluster-master-id 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

In this way, we assign the new replica to a specific master node.

A more manual way to add a replica to a specific master is to add the new node as an empty master, and then use theCLUSTER REPLICATECommand to convert it to a copy. The same applies if you add the node as a backup node, but you want to move it as a copy of another primary node.

For example, in order to add a replica to the node 127.0.0.1:7005, the hash slot currently served by this node is in the range of 11423-16383, and the node ID is 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e. All I have to do is connect to the new node (which has been added to the cluster as an empty primary node) and send the command on the new node:

redis 127.0.0.1:7006> cluster replicate 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

this is it. Now, a new copy of this set of hash slots is available, and all other nodes in the cluster are aware of it (their configuration needs to be updated in a few seconds). We can use the following command to verify:

$ redis-cli -p 7000 cluster nodes | grep slave | grep 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e
f093c80dde814da99c5cf72a7dd01590792b783b 127.0.0.1:7006 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385543617702 3 connected
2938205e12de373867bf38f1ca29d31d0ddb3e46 127.0.0.1:7002 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385543617198 3 connected

Node 3c3a0c… Now has two backup nodes running on port 7002 (existing) and port 7006 (New)

Remove node

To remove a backup node, you only need to use the Del node command on redis cli

redis-cli --cluster del-node 127.0.0.1:7000 ``

The first parameter is just a random node in the cluster, and the second parameter is the ID of the node you want to delete. You can delete the master in the same way, but to delete the master, it must be empty. If the master node is not empty, the data needs to be repartitioned to all other master nodes.

Another way to delete the primary node is to perform a manual fail over on one of its backup nodes and delete the node after it becomes the backup node of the new primary node. Obviously, this doesn’t work when you want to reduce the actual number of primary nodes in the cluster, in which case you need to reparticate.

Replica migration

At any time in a redis cluster, you can reconfigure a backup node to be a slave of another master node. Use the following command:

CLUSTER REPLICATE

However, there is a special case where you want the replica to automatically move from one master node to another without the help of the system administrator. Automatic replica reconfiguration is called replica migration, which can improve the reliability of redis cluster.

Note: you can read the details of replica migration in redis cluster specification. Here we only provide some general ideas and information that you should benefit from.

In some cases, you may want to move your cluster replicas from one master node to another because redis clusters usually have the same fault tolerance as the number of replicas for a given master node.

For example, if a master node and its replica fail at the same time, the cluster with one replica for each master node will not continue to work, just because no other instance has a replica of the same hash slot that the master node serves. However, although network disruption may isolate multiple nodes at the same time, many other types of failures (such as local hardware or software failures of a single node) are very noteworthy. This kind of failure is unlikely to occur at the same time. Therefore, in a cluster where each primary node has a backup node, if the backup node is shut down at 4 a.m., and the primary node is shut down It was closed at 6 a.m. This still causes the cluster to fail to run.

In order to improve the reliability of the system, we can choose to add a replica to each primary node, but the cost is very high. Replica migration allows more backup nodes to be added to a few primary nodes. So you have 10 nodes, each with one backup node, for a total of 20 instances. However, for example, if you add three instances as backup nodes of some primary nodes, some primary nodes will have multiple copies.

When using replica migration, if a primary node does not contain a backup node, the replica of the primary node with multiple backup nodes will be migrated to an isolated primary node. Therefore, when your backup node is shut down at 4 a.m. in the above example, another backup node will replace it; when the primary node also fails at 5 a.m., another backup node will be elected as the primary node so that the cluster can continue to operate.

So what should you know about replica migration?

  • At some point, the cluster will try to select a replica from the master node with the largest number of replicas for migration.
  • To benefit from replica migration, you only need to add some replicas for a single primary node in the cluster, no matter which primary node it is
  • There is a configuration parameter to control the replica migration function, which is called cluster migration barrier
    Take the example that comes with the redis cluster redis.conf Learn more about this feature in the.
    reference resources

Recommended Today

Rust and python: why rust can replace Python

In this guide, we compare the rust and python programming languages. We will discuss the applicable use cases in each case, review the advantages and disadvantages of using rust and python, and explain why rust might replace python. I will introduce the following: What is rust? What is Python? When to use rust When to […]