Redis getting started to giving up series – redis cluster cluster principle


Redis Cluster

Redis cluster is redis’s distributed solution. After the release of redis version 3.0, it effectively solves the distributed requirements of reids. The main features of redis cluster are:

  • The data is segmented and part of the data is saved on each master
  • The built-in high availability support can still provide services when some masters are unavailable
  • N master nodes are supported. Each master node can mount multiple slave nodes
  • Automatic election mechanism is supported. When the master node fails, a new master will be elected from the slave node and switched
  • Each master node is responsible for reading and writing data, and each node communicates with each other

Virtual slot partition of redis cluster (hash slot)

Virtual slot partition is the partition method adopted by redis cluster.

  • Redis cluster preset 16384 hash slots (range 0-16383) for us, and each hash slot maps a data subset.
  • The key in the data subset confirms the corresponding hash slot by calculating the value of CRC16 and taking the modulus of 16384.
  • Each master manages a part of slots, which are evenly distributed according to the data of the master node.
  • Since the data is saved in the slot, when the node expands or shrinks, the slot can be reassigned and migrated without data loss.
  • The cost of allocating migration slots is very low (the total number of slots is only 16384)
  • Redis cluster nodes share messages. Each node knows which node is responsible for which slot in which range

Internal communication mechanism between nodes

Basic communication principle

  1. The redis cluster nodes communicate with each other using the gossip protocol

Unlike centralized, cluster metadata (node information, faults, etc.) is not stored on a node,
Instead, they communicate with each other continuously to keep the data of all nodes in the whole cluster complete

  • Centralized: the advantage is that metadata updating and reading are very timely. Once metadata changes,
    Update to the centralized storage immediately, and other nodes can immediately perceive it when reading;
    The disadvantage is that the pressure of updating all metadata is concentrated in one place, which may lead to pressure on metadata storage
  • Mission: the advantage is that metadata updates are scattered, not centralized in one place, and update requests will continue,
    To update all nodes, there is a certain delay, which reduces the pressure; Disadvantages: metadata update is delayed, which may cause some lag in some cluster operations
  1. 10000 ports
  • Each node has a port dedicated to inter node communication, that is, the port number of its own service + 10000, such as 7001, so the port used for inter node communication is 17001
  • Each node sends a ping message to several other nodes at regular intervals. At the same time, the other nodes return a Pong after receiving the Ping
  1. Information exchanged

Fault information, node addition and removal, hash slot information, etc

gossip protocol

The gossip protocol contains a variety of messages, including Ping, pong, meet, and fail

  • Meet: a node sends a meet to the newly added node to join the cluster, and then the new node will start to communicate with other nodes
    redis-trib.rb add-node
    In fact, a mission meet message is sent internally to the newly added node to inform that node to join our cluster
  • Ping: each node frequently sends pings to other nodes, including its own status and its own maintained cluster metadata. They exchange metadata with each other through Ping
    Each node will frequently send pings to other clusters every second to exchange data and update metadata with each other
  • Pong: returns Ping and meet, including their own status and other information. It can also be used for information broadcasting and updating
  • Fail: after a node judges that another node fails, it sends a fail to other nodes to notify other nodes that the specified node is down

Ping message

  • Ping is very frequent and carries some metadata, which may increase the burden on the network
    Each node will ping 10 times per second, and 5 other nodes that have not communicated for the longest time will be selected each time
  • Of course, if it is found that the communication delay of a node reaches the cluster_ node_ Timeout / 2, then send Ping immediately to avoid too long data exchange delay and too long lag time
  • For example, if there is no data exchange between the two nodes for 10 minutes, there will be a problem if the metadata of the whole cluster is seriously inconsistent
    So cluster_ node_ The timeout can be adjusted. If the adjustment is large, the transmission frequency will be reduced
  • For each Ping, one is to bring the information of its own node, and the other is to bring the information of 1 / 10 other nodes, and send it out for data exchange
    It contains the information of at least 3 other nodes, and the information of at most 2 other nodes

Message header

Ping and Pong packets contain headers common to all types of packets (for example, packets requesting failover voting).
The general message header has the following information:

  • The node ID, the 160 bit pseudo-random string allocated when the node is first created, remains unchanged throughout the life cycle of the redis cluster node
  • Node flag, indicating whether the node is a slave node, master node or other node information
  • Bitmap of the hash slot of the sending node
  • TCP port of sending node
  • Status of sending node
  • The master node ID of the sending node (if it is a slave node)

Client routing

Moved redirection

  1. Each node will share the relationship between the slot in redis cluster and the corresponding node in the cluster through communication
  2. The client sends a command to any node of redis cluster. The node receiving the command will perform hash operation and 16384 modulo calculation according to CRC16 rules to calculate its own slot and corresponding node
  3. If the slot storing data is assigned to the current node, execute the command in the slot and return the command execution result to the client
  4. If the slot where the data is saved is not within the management range of the current node, a moved redirection exception is returned to the client
  5. The client receives the result returned by the node. If it is a moved exception, it obtains the information of the target node from the moved exception
  6. The client sends a command to the target node to obtain the command execution result

Ask redirection

What is ask redirection?
When expanding and shrinking the cluster, you need to migrate the slot and the data in the slot
When the client sends a command to a node, the node returns a moved exception to the client, telling the client the node information of the slot corresponding to the data
If the cluster expansion or capacity reduction operation is in progress at this time, when the client sends a command to the node being migrated, the data to be operated has been migrated to another node, and an ask will be returned. This is the ask redirection mechanism

  1. The client sends a command to the target node. The slot in the target node is migrating. At this time, the target node will return ask to redirect to the client
  2. After receiving the ask, the client initiates a redirection operation to the new node by sending the asking command.
  3. The new node executes the command and returns the command execution result to the client
  4. The client receives the ask and does not update the local mapping table (the corresponding relationship between the slot and the nodes in the cluster)

Multi key operation (hash tags)

In redis cluster, the hash slots corresponding to each key are different, and the data storage nodes are also different. So how to batch operate multiple keys?
Hash tags is used to solve this problem. Using hash tags can ensure that multiple keys are mapped to the same hash slot.
In redis cluster, some special calculations are made to implement hash tags.
If a key contains a pattern like “{…}”, only the string between {and} will be used for hashing to obtain the hash slot
However, since there may be multiple {or}, the algorithm is as follows:

  • If the key contains one {character.
  • Then there will be a} to the right of {.
  • There will be one or more characters between {and}, and the first} must appear after the first {.
    For example:
  1. abc{hello} yhn{hello}
  2. {abc}}hello {abc}}nihao

Multi key operation:
MSET {user:1000}.name Angela {user:1000}.surname White

reference material: