[original] Qianglu. Net redis cluster access component

Time:2021-3-5

Hello, everyone. I’m tanzame. We meet again. Today, let’s talk about how to build a redis cluster client by hand. It’s purely manual and has dry goods. You can make fine products.

With the growth of business and the surge of QPS in online environment, it is natural to switch the current stand-alone redis to cluster mode. Burning goose, we find tragically, ServiceStack.Redis This officially recommended. Net client does not support cluster mode. A pass degree Niang over the wall fruitless, decided to force a based on their own ServiceStack.Redis Redis cluster access component of.

Let’s not say too much. First, let’s go to the operation rendering

[original] Qianglu. Net redis cluster access component

 

Redis cluster uses hash slot algorithm to calculate CRC16 value for each key, and then modulus 16383 to obtain hash slot corresponding to key. Each master in redis cluster will hold some slots. When accessing the key, it will find the specific master node according to the calculated hash slot, and then the node currently found will execute the specific redis cluster command (please refer toOfficial documentation)。

because ServiceStack.Redis The redis command of a single instance has been implemented, so we can regard the redis cluster client to be implemented as a proxy, which is only responsible for calculating which specific node (addressing) the key is located in, and then forwarding the redis command to the corresponding node for execution.

ServiceStack.Redis Redisclient is non thread safe, ServiceStack.Redis Use the pooled redisclient manager to improve the performance and concurrency. Our redis cluster client should also integrate the pooled redisclient manager to obtain the redisclient instance.

At the same time, redis cluster supports online dynamic expansion and slot migration. Our redis cluster client should also have the ability to automatically and intelligently discover new nodes and automatically refresh the slot distribution.

  To sum up, to implement a redis cluster client, we need to implement the following key points:

As shown in the following class diagram, next we analyze the specific code implementation in detail.

  

1、 CRC16 

CRC, namely cyclic redundancy check code, is a common error detection code in information system. Different organizations have different standards for CRC check codes. The standard that redis follows here is crc-16-ccitt, which is also the CRC standard used by XMODEM protocol. Therefore, XMODEM CRC is also commonly used, which is a classic “CRC check code generation algorithm based on byte lookup table”.

1 /// 
 2 // calculate the corresponding hash slot according to the key
 3 /// 
 4 public static int GetSlot(string key)
 5 {
 6     key = CRC16.ExtractHashTag(key);
 7     // optimization with modulo operator with power of 2 equivalent to getCRC16(key) % 16384
 8     return GetCRC16(key) & (16384 - 1);
 9 }
10 
11 /// 
12 // / calculate the CRC16 check code of a given byte group
13 /// 
14 public static int GetCRC16(byte[] bytes, int s, int e)
15 {
16     int crc = 0x0000;
17 
18     for (int i = s; i < e; i++)
19     {
20         crc = ((crc << 8) ^ LOOKUP_TABLE[((crc >> 8) ^ (bytes[i] & 0xFF)) & 0xFF]);
21     }
22     return crc & 0xFFFF;
23 }

 

  2、 Read cluster node

Using cluster nodes command from any node in the cluster, you can read all the node information in the cluster, including the connection status, their flags, attributes and allocated slots, etc. Cluster nodes provides all of these information in serial format

d99b65a25ef726c64c565901e345f98c496a1a47 127.0.0.1:7007 master - 0 1592288083308 8 connected
2d71879d6529d1edbfeed546443051986245c58e 127.0.0.1:7003 master - 0 1592288084311 11 connected 10923-16383
654cdc25a5fa11bd44b5b716cdf07d4ce176efcd 127.0.0.1:7005 slave 484e73948d8aacd8327bf90b89469b52bff464c5 0 1592288085313 10 connected
ed65d52dad7ef6854e0e261433b56a551e5e11cb 127.0.0.1:7004 slave 754d0ec7a7f5c7765f784a6a2c370ea38ea0c089 0 1592288081304 9 connected
754d0ec7a7f5c7765f784a6a2c370ea38ea0c089 127.0.0.1:7001 master - 0 1592288080300 9 connected 0-5460
484e73948d8aacd8327bf90b89469b52bff464c5 127.0.0.1:7002 master - 0 1592288082306 10 connected 5461-10922
2223bc6d099bd9838e5d2f1fbd9a758f64c554c4 127.0.0.1:7006 myself,slave 2d71879d6529d1edbfeed546443051986245c58e 0 0 6 connected

The meaning of each field is as follows:

  1. id: node ID, a random string of 40 characters that will not change when a node is created (unlessCLUSTER RESET HARDIs used).

  2. ip:port: the address of the node that the client should contact to run the query.

  3. flags: comma list separated flag:myselfmasterslavefail?failhandshakenoaddrnoflags. Signs are explained in detail in the next section.

  4. master: if the node is a slave node and the master node is known, then the node ID is the master node, otherwise it is a ‘-‘ character.

  5. ping-sent: the UNIX time in milliseconds for the currently active Ping to be sent. If there is no pending Ping, it is zero.

  6. pong-recv: Ms UNIX time to receive the last ping pong ball.

  7. config-epoch: the configuration period (or version) of the current node (or the current master node, if it is a slave node). Each time a failure occurs, a new, unique, monotonically increasing configuration period is created. If multiple nodes claim to serve the same hash slot, the node with higher configuration period will win.

  8. link-state: used for link state of node to node cluster bus. We use this link to communicate with the node. It could beconnectedordisconnected

  9. slot: hash slot number or range. Start with parameter 9, but there may be a total of 16384 entries (limit never reached). This is the list of hash slots provided by this node. If the entry is just a number, it is parsed like this. If it’s a scope, it’s in formstart-endAnd it means that the node is responsible for all hash slotsstartreachendIncludes start and end values.

Meaning of logo (field number 3)

  • myself: the node you are contacting.
  • master: the node is the master.
  • slave: the node is subordinate.
  • fail?: node inPFAILStatus. It is not accessible to the node being contacted, but it is still logically accessible (not in theFAILStatus).
  • fail: node inFAILStatus. For willPFAILUpgrade status toFAILThis is not accessible to multiple nodes of.
  • handshake: untrusted node, we shake hands.
  • noaddr: this node has no known address.
  • noflags: no sign at all.
[original] Qianglu. Net redis cluster access component[original] Qianglu. Net redis cluster access component

1 // read the node information on the cluster
  2 static IList ReadClusterNodes(IEnumerable source)
  3 {
  4     RedisClient c = null;
  5     StringReader reader = null;
  6     IList result = null;
  7 
  8     int index = 0;
  9     int rowCount = source.Count();
 10 
 11     foreach (var node in source)
 12     {
 13         try
 14         {
 15 // read the redis cluster node information from the current node
 16             index += 1;
 17             c = new RedisClient(node.Host, node.Port, node.Password);
 18             RedisData data = c.RawCommand("CLUSTER".ToUtf8Bytes(), "NODES".ToUtf8Bytes());
 19             string info = Encoding.UTF8.GetString(data.Data);
 20 
 21 // convert the read back character text to a strongly typed node entity
 22             reader = new StringReader(info);
 23             string line = reader.ReadLine();
 24             while (line != null)
 25             {
 26                 if (result == null) result = new List();
 27                 InternalClusterNode n = InternalClusterNode.Parse(line);
 28                 n.Password = node.Password;
 29                 result.Add(n);
 30 
 31                 line = reader.ReadLine();
 32             }
 33 
 34 // as long as any node gets the cluster information, it will exit directly
 35             if (result != null && result.Count > 0) break;
 36         }
 37         catch (Exception ex)
 38         {
 39 // an exception occurs. If the last node is not reached, continue to use the next node to read the cluster information
 40 // otherwise, an exception is thrown
 41             if (index < rowCount)
 42                 Thread.Sleep(100);
 43             else
 44                 throw new RedisClusterException(ex.Message, c != null ? c.GetHostString() : string.Empty, ex);
 45         }
 46         finally
 47         {
 48             if (reader != null) reader.Dispose();
 49             if (c != null) c.Dispose();
 50         }
 51     }
 52 
 53 
 54     if (result == null)
 55         result = new List(0);
 56     return result;
 57 }
 58 
 59 /// 
 60 // read the relevant information of cluster nodes from each command line of cluster nodes
 61 /// 
 62 // cluster command
 63 /// 
 64 public static InternalClusterNode Parse(string line)
 65 {
 66     if (string.IsNullOrEmpty(line))
 67         throw new ArgumentException("line");
 68 
 69     InternalClusterNode node = new InternalClusterNode();
 70     node._nodeDescription = line;
 71     string[] segs = line.Split(' ');
 72 
 73     node.NodeId = segs[0];
 74     node.Host = segs[1].Split(':')[0];
 75     node.Port = int.Parse(segs[1].Split(':')[1]);
 76     node.MasterNodeId = segs[3] == "-" ? null : segs[3];
 77     node.PingSent = long.Parse(segs[4]);
 78     node.PongRecv = long.Parse(segs[5]);
 79     node.ConfigEpoch = int.Parse(segs[6]);
 80     node.LinkState = segs[7];
 81 
 82     string[] flags = segs[2].Split(',');
 83     node.IsMater = flags[0] == MYSELF ? flags[1] == MASTER : flags[0] == MASTER;
 84     node.IsSlave = !node.IsMater;
 85     int start = 0;
 86     if (flags[start] == MYSELF)
 87         start = 1;
 88     if (flags[start] == SLAVE || flags[start] == MASTER)
 89         start += 1;
 90     node.NodeFlag = string.Join(",", flags.Skip(start));
 91 
 92     if (segs.Length > 8)
 93     {
 94         string[] slots = segs[8].Split('-');
 95         node.Slot.Start = int.Parse(slots[0]);
 96         if (slots.Length > 1) node.Slot.End = int.Parse(slots[1]);
 97 
 98         for (int index = 9; index < segs.Length; index++)
 99         {
100             if (node.RestSlots == null)
101                 node.RestSlots = new List();
102 
103             slots = segs[index].Split('-');
104 
105             int s1 = 0;
106             int s2 = 0;
107             bool b1 = int.TryParse(slots[0], out s1);
108             bool b2 = int.TryParse(slots[1], out s2);
109             if (!b1 || !b2)
110                 continue;
111             else
112                 node.RestSlots.Add(new HashSlot(s1, slots.Length > 1 ? new Nullable(s2) : null));
113         }
114     }
115 
116     return node;
117 }

View Code

 

  3、 Assigning cache client managers to nodes

In single instance redis, we use pooled redisclient manager to get redisclient. Using this idea for reference, in the redis cluster, we instantiate a “pooledredisclientmanager” for each master node, and the slot “held by the master node shares a” pooledredisclientmanager “instance. Take slot as the key to map and cache slot and pooled redisclient manager one by one.

[original] Qianglu. Net redis cluster access component[original] Qianglu. Net redis cluster access component

1 // initialize cluster management
 2 void Initialize(IList clusterNodes = null)
 3 {
 4 // read cluster information from redis
 5     IList nodes = clusterNodes == null ? RedisCluster.ReadClusterNodes(_source) : clusterNodes;
 6 
 7 // the master node is generated, and the slot of each master node corresponds to a redis client buffer pool manager
 8     IList masters = null;
 9     IDictionary managers = null;
10     foreach (var n in nodes)
11     {
12 // the node is invalid or
13         if (!(n.IsMater &&
14             !string.IsNullOrEmpty(n.Host) &&
15             string.IsNullOrEmpty(n.NodeFlag) &&
16             (string.IsNullOrEmpty(n.LinkState) || n.LinkState == InternalClusterNode.CONNECTED))) continue;
17 
18         n.SlaveNodes = nodes.Where(x => x.MasterNodeId == n.NodeId);
19         if (masters == null)
20             masters = new List();
21         masters.Add(n);
22 
23 // use the hash slot of each master node as the key to import it into the redis client buffer pool manager
24 // then, the method table pointer (also known as the type object pointer) appears, occupying four bytes. 4 * 16384 / 1024 = 64KB
25         if (managers == null)
26             managers = new Dictionary();
27 
28         string[] writeHosts = new[] { n.HostString };
29         string[] readHosts = n.SlaveNodes.Where(n => false).Select(n => n.HostString).ToArray();
30         var pool = new PooledRedisClientManager(writeHosts, readHosts, _config);
31         managers.Add(n.Slot.Start, pool);
32         if (n.Slot.End != null)
33         {
34 // all hash slots in this range use the same buffer pool
35             for (int s = n.Slot.Start + 1; s <= n.Slot.End.Value; s++)
36                 managers.Add(s, pool);
37         }
38         if (n.RestSlots != null)
39         {
40             foreach (var slot in n.RestSlots)
41             {
42                 managers.Add(slot.Start, pool);
43                 if (slot.End != null)
44                 {
45 // all hash slots in this range use the same buffer pool
46                     for (int s = slot.Start + 1; s <= slot.End.Value; s++)
47                         managers.Add(s, pool);
48                 }
49             }
50         }
51     }
52 
53     _masters = masters;
54     _redisClientManagers = managers;
55     _clusterNodes = nodes != null ? nodes : null;
56 
57     if (_masters == null) _masters = new List(0);
58     if (_clusterNodes == null) _clusterNodes = new List(0);
59     if (_redisClientManagers == null) _redisClientManagers = new Dictionary(0);
60 
61     if (_masters.Count > 0)
62         _source = _masters.Select(n => new ClusterNode(n.Host, n.Port, n.Password)).ToList();
63 }

View Code

 

  4、 Route hash slot to the correct node

When accessing a key, the hash slot value calculated by the key can be used to quickly find the corresponding “pooledredisclientmanager” instance of the key according to the “pooledredisclientmanager” cached in the third step, and call the “pooledredisclientmanager” instance edRedisClientManager.GetClient () to route hash slot to the correct master node.

1 // execute the specified action and return the value
 2 private T DoExecute(string key, Func action) => this.DoExecute(() => this.GetRedisClient(key), action);
 3 
 4 // execute the specified action and return the value
 5 private T DoExecute(Func slot, Func action, int tryTimes = 1)
 6 {
 7     RedisClient c = null;
 8     try
 9     {
10         c = slot();
11         return action(c);
12     }
13     catch (Exception ex)
14     {
15 // omit here
16     }
17     finally
18     {
19         if (c != null)
20             c.Dispose();
21     }
22 }
23 
24 // get the master node corresponding to the specified key
25 private RedisClient GetRedisClient(string key)
26 {
27     if (string.IsNullOrEmpty(key))
28         throw new ArgumentNullException("key");
29 
30     int slot = CRC16.GetSlot(key);
31     if (!_redisClientManagers.ContainsKey(slot))
32         throw new SlotNotFoundException(string.Format("No reachable node in cluster for slot {{{0}}}", slot), slot, key);
33 
34     var pool = _redisClientManagers[slot];
35     return (RedisClient)pool.GetClient();
36 }

   

  5、 Automatic discovery of new nodes and automatic refresh of slot distribution

In the actual production environment, the redis cluster often has add / delete nodes, migration slots, master node downtime, from node to master node, and so on. In view of these situations, our redis cluster component must have the ability to automatically discover nodes and refresh nodesThe third stepThe ability of the cached slot. My implementation idea here is to force refresh the cluster node information and re cache the mapping between slot and node when the node returns redisexception when executing redis command.

[original] Qianglu. Net redis cluster access component[original] Qianglu. Net redis cluster access component

1 // execute the specified action and return the value
  2 private T DoExecute(Func slot, Func action, int tryTimes = 1)
  3 {
  4     RedisClient c = null;
  5     try
  6     {
  7         c = slot();
  8         return action(c);
  9     }
 10     catch (Exception ex)
 11     {
 12         if (!(ex is RedisException) || tryTimes == 0) throw new RedisClusterException(ex.Message, c != null ? c.GetHostString() : string.Empty, ex);
 13         else
 14         {
 15             tryTimes -= 1;
 16 // try refreshing the cluster information again
 17             bool isRefresh = DiscoveryNodes(_source, _config);
 18             if (isRefresh)
 19 // the cluster node has been updated, re execute
 20                 return this.DoExecute(slot, action, tryTimes);
 21             else
 22 // if the cluster node has not been updated, an exception will be thrown directly
 23                 throw new RedisClusterException(ex.Message, c != null ? c.GetHostString() : string.Empty, ex);
 24         }
 25     }
 26     finally
 27     {
 28         if (c != null)
 29             c.Dispose();
 30     }
 31 }
 32 
 33 // refresh cluster information again
 34 private bool DiscoveryNodes(IEnumerable source, RedisClientManagerConfig config)
 35 {
 36     bool lockTaken = false;
 37     try
 38     {
 39         // noop
 40         if (_isDiscoverying) { }
 41 
 42         Monitor.Enter(_objLock, ref lockTaken);
 43 
 44         _source = source;
 45         _config = config;
 46         _isDiscoverying = true;
 47 
 48 // the last synchronization time should be more than {monitorinterval} seconds before synchronization
 49         if ((DateTime.Now - _lastDiscoveryTime).TotalMilliseconds >= MONITORINTERVAL)
 50         {
 51             bool isRefresh = false;
 52             IList newNodes = RedisCluster.ReadClusterNodes(_source);
 53             foreach (var node in newNodes)
 54             {
 55                 var n = _clusterNodes.FirstOrDefault(x => x.HostString == node.HostString);
 56                 isRefresh =
 57 n = = null | // new node                                                                
 58                     n.Password !=  node.Password  ||// the password has changed                                                                
 59                     n.IsMater !=  node.IsMater  ||// master to slave or slave to master                                                                
 60                     n.IsSlave !=  node.IsSlave  ||// master to slave or slave to master                                                                
 61                     n.NodeFlag !=  node.NodeFlag  ||// the node flag bit has changed                                                                
 62                     n.LinkState !=  node.LinkState  ||// the node status bit has changed                                                                
 63                     n. Slot.Start  !=  node.Slot.Start  ||// hash slot changed                                                                
 64                     n. Slot.End  !=  node.Slot.End  ||// hash slot changed
 65                     (n.RestSlots == null && node.RestSlots != null) ||
 66                     (n.RestSlots != null && node.RestSlots == null);
 67                 if (!isRefresh && n.RestSlots != null && node.RestSlots != null)
 68                 {
 69                     var slots1 = n.RestSlots.OrderBy(x => x.Start).ToList();
 70                     var slots2 = node.RestSlots.OrderBy(x => x.Start).ToList();
 71                     for (int index = 0; index < slots1.Count; index++)
 72                     {
 73                         isRefresh =
 74 slots1 [index]. Start! = slots2 [index]. Start | // hash slot changed                                                                
 75 slots1 [index]. End! = slots2 [index]. End; // hash slot changed
 76                         if (isRefresh) break;
 77                     }
 78                 }
 79 
 80                 if (isRefresh) break;
 81             }
 82 
 83             if (isRefresh)
 84             {
 85 // reinitialize the cluster
 86                 this.Dispose();
 87                 this.Initialize(newNodes);
 88                 this._lastDiscoveryTime = DateTime.Now;
 89             }
 90         }
 91 
 92 // the last refresh time is within {monitorinterval}, indicating that it is the latest cluster information
 93         return (DateTime.Now - _lastDiscoveryTime).TotalMilliseconds < MONITORINTERVAL;
 94     }
 95     finally
 96     {
 97         if (lockTaken)
 98         {
 99             _isDiscoverying = false;
100             Monitor.Exit(_objLock);
101         }
102     }
103 }

View Code

 

  6、 Configure access component call entry

Finally, we need to provide access to components. We use rediscluster class to implement the basic operations of string, list, hash, set, ordered set and keys, and use redisclusterfactory factory class to provide singleton operations. In this way, we can call redis cluster like single instance rediscluster. Call example:

var node = new ClusterNode("127.0.0.1", 7001);
var redisCluster = RedisClusterFactory.Configure(node, config);
string key = "B070x14668";
redisCluster.Set(key, key);
string value = redisCluster.Get(key);
redisCluster.Del(key);
[original] Qianglu. Net redis cluster access component[original] Qianglu. Net redis cluster access component

1 /// 
 2 // redis cluster factory
 3 /// 
 4 public class RedisClusterFactory
 5 {
 6     static RedisClusterFactory _factory = new RedisClusterFactory();
 7     static RedisCluster _cluster = null;
 8 
 9     /// 
10 // redis cluster
11     /// 
12     public static RedisCluster Cluster
13     {
14         get
15         {
16             if (_cluster == null)
17                 throw new Exception("You should call RedisClusterFactory.Configure to config cluster first.");
18             else
19                 return _cluster;
20         }
21     }
22 
23     /// 
24 // initializes a new instance of the class
25     /// 
26     private RedisClusterFactory()
27     {
28     }
29 
30     /// 
31 // configure redis cluster
32 // if there are nodes with specified password in the cluster, you must use IEnumerable < clusternode > overload to list these nodes
33     /// 
34 // cluster node
35     /// 
36     public static RedisCluster Configure(ClusterNode node)
37     {
38         return RedisClusterFactory.Configure(node, null);
39     }
40 
41     /// 
42 // configure redis cluster
43 // if there are nodes with specified password in the cluster, you must use IEnumerable < clusternode > overload to list these nodes
44     /// 
45 // cluster node
46 // client buffer pool configuration
47     /// 
48     public static RedisCluster Configure(ClusterNode node, RedisClientManagerConfig config)
49     {
50         return RedisClusterFactory.Configure(new List { node }, config);
51     }
52 
53     /// 
54 // configure redis cluster
55     /// 
56 // cluster node
57 // client buffer pool configuration
58     /// 
59     public static RedisCluster Configure(IEnumerable nodes, RedisClientManagerConfig config)
60     {
61         if (nodes == null)
62             throw new ArgumentNullException("nodes");
63 
64         if (nodes == null || nodes.Count() == 0)
65             throw new ArgumentException("There is no nodes to configure cluster.");
66 
67         if (_cluster == null)
68         {
69             lock (_factory)
70             {
71                 if (_cluster == null)
72                 {
73                     RedisCluster c = new RedisCluster(nodes, config);
74                     _cluster = c;
75                 }
76             }
77         }
78 
79         return _cluster;
80     }
81 }

View Code

 

  summary

Today, we introduced in detail how to write a redis cluster client access component from 0. I believe it will inspire students who are also looking for similar solutions. If you like, please click star. In the absence of the same case can be referred to, the author through consulting the official documentation and learning from the implementation idea of Java’s jediscluster, although bumpy, but also the initial completion of this component and put into use, must give yourself a drumstick!! I have a little question here. When the students of. Net are using redis cluster, what components do you use? Why are there almost no related introduction and ready-made components on the Internet? Welcome to the discussion.

GitHub? Code Hosting:https://github.com/TANZAME/ServiceStack.Redis.Cluster

QQ group of technical exchange: 816425449

 

Recommended Today

Rust and python: why rust can replace Python

In this guide, we compare the rust and python programming languages. We will discuss the applicable use cases in each case, review the advantages and disadvantages of using rust and python, and explain why rust might replace python. I will introduce the following: What is rust? What is Python? When to use rust When to […]