Redis foundation — analyzing the basic data structure and its usage

Time:2021-11-25

This is a series of articles. It is intended toBasic data structureAdvanced data structurePersistent modeas well asHighly available wayOnce again, the official account will be updated ahead of other platforms, and those who are interested can be concerned about it in advance. “SH’s full stack notes“, the text begins below.

If you are an experienced backend or server developer, you must have heard of redis, its full name isRemote Dictionary Server。 It is a key value based storage system written in C language. To put it bluntly, the point is oneMemory database, since it is an in memory database, it will encounter the problem of data inconsistency caused by unexpected server downtime.

This is the same as many game servers. If you are interested, please refer to my previous article on the difference between game server and web server. Its data will first flow to memory, and high performance will be achieved based on fast memory reading and writing, and then the data in the memory data will be landed regularly. Redis is actually such a process. Based on fast memory read and write operations, a single redis can even carry 100000 QPS.

In addition to its high performance, redis also has rich data structures and supports most business scenarios. This is one of the reasons why it is so popular. Let’s take a look at redisBasic data typeAnd how they are implemented at the bottom.

1. Data type

The basic data types areStringListHashSetSorted Set, these are common basic data types. You can see that they are very rich and can almost meet most requirements. In fact, there are some advanced data structures. We will not mention them in this chapter for the time being, but only talk about the basic data structures.

2. String

String can be said to be the most basic data structure. In terms of usage, it can be directly linked to string in Java. You can use string type to store a flag bit, a counter, or even worse. The serialized JSON string is OK, and its single key is limited to 512M. The common commands aregetsetincrdecrmget

2.1 use

  • getGet a key. If the key does not exist, a null pointer will be returned
  • setAssign a value to the key and set the key to the specified value. If the key has a value before, it will be overwritten by the new value
  • incrGive the value of the current key + 1. If the key does not exist, it will be called firstsetAssign a value of 0, and then callincr。 Of course, if the type of the key cannot be added, such as a string, an error will be thrown
  • decrThe value given to the current key is – 1, and the rest are the same as above
  • mgetThe same as get, only multiple pieces of data are returned at one time, and the nonexistent key will return a null pointer

Redis foundation -- analyzing the basic data structure and its usage

It is understandable that most people only use it for one purpose, but if it is a development that pursues technology, or if you want to be close to a large factory, you must have the spirit of digging to the bottom. Only when you really know the underlying principle of a thing can you have more ideas to solve problems when you encounter problems. Next, let’s talk about how the underlying string in redis is implemented.

2.2 principle

2.2.1 structure

We know that redis is written in C language, but redis does not use it directly. Instead, it implements a structure called SDS (simple dynamic string) to implement the string. The structure is as follows.

struct sdshdr {
  //Record the number of bytes used in the buf
  int len;
  //Record the number of unused bytes in the buf
  int free;
  //Byte array to hold string
  char buf[];
}

2.2.2 advantages

Why should redis implement SDS instead of using C strings directly? Mainly because of the following points.

  • Reduce the cost of getting string lengthTo get the length of a string in C language, you need to traverse the whole string until you encounter the end flag bit\0, the time complexity is O (n), while SDS directly maintains the variable of length, and the time complexity of length is O (1)
  • Avoid buffer overflowIn C language, if you insert more bytes than its capacity into a byte array, it will causeout of bufferAnd SDS through maintenancefreeVariables solve this problem. When writing data to the buf array, you will first judge whether the remaining space is enough to insert new data. If not, SDS willReassignBuffer, increase the previous buffer. And the increased length is equal to the length of the new data
  • Space pre allocation & Space inertia releaseIn C language, memory space will be reallocated every time the string is modified. If the string is modified n times, memory reallocation will occur n times. SDS optimizes this problem because it redundancies part of the spaceinevitableReassign n times tomostAllocate n times, and when the data is removed from the buf, the free memory will not be recycled immediately, so as to prevent the re allocation of memory caused by newly written data
  • Ensure binary securityIn C language, string encountered\0Will be truncated, and SDS will not appear in the data\0The truncated string, in other words, will not affect the actual operation results because of some special characters

SDS can be understood in combination with the following figure.

Redis foundation -- analyzing the basic data structure and its usage

To sum up, there are four subheadings in the above list. In order to reduce the overhead of obtaining string length, avoid buffer overflow, space pre allocation & Space inertia release and ensure binary security.

3. List

List is also a frequently used data structure. It has too many designed commands. It is not like string demonstrating one by one. You can search it if you are interested. The commands include lpush, lpushx, rpush, rpushx, lpop, rpop, lindex, linsert, lrange, len, lrem, lset, ltrim, rpolpush, brpoppush, blpop and brpop, all of which are operations on elements in the array.

3.1 use

I think the use of list mainly focuses on the following two aspects.

  1. Store data as a normal list (similar to Java’s ArrayList)
  2. Used as asynchronous queue

Naturally, there is no need to say more about the general list. The data stored in it must be needed in the business. Let’s focus on itAsynchronous queue

What, can list still be used as a queue?

In addition to being used as a queue, list can also be used as a stack. A lot of operation list commands are introduced above. When we userpush/lpopWhen we combine commands, we are actually using a queuerpush/rpopWhen commands are combined, a stack is used. Lpush / rpop and lpush / lpop are the same.

Suppose we userpushTo produce messages. When our program needs to consume messages, it uses themlpopfromAsynchronous queueConsumer news. However, if you use this method, when the queue is empty, you may need to keep asking whether there is data in the queue, which will waste the CPU resources of the machine.

So you can take the current threadSleepFor a period of time, this can indeed save some CPU resources. However, you may need to consider the sleep time. The interval is too short and the CPU is too slowContext switchingIt may also be a lot of overhead. If the interval is too long, this message may be delayed for consumption (but asynchronous queues are used, and this problem should be ignored).

Is there any other way besides sleep?

Yes, the answer isblpop。 When we use blpop to consume, if the current queue is empty, the thread will block until the following two conditions.

  1. The set timeout time has been reached
  2. There are messages in the queue that can be consumed

Compared toSleepFor a period of time, the real-time performance will be better; It is more CPU friendly than polling.

3.2 principle

Before redis 3.2, redis used ziplist (compressed list) or LinkedList (linked list). When an element in a listmeanwhilesatisfyThe size of each element is less than 64 bytesandThe number of list elements is less than 512The storage method isZipList。 If one condition is not met, it will be converted toLinkedList

After 3.2, it is implemented as a QuickList. Because LinkedList is a relatively basic thing, it will not be repeated here.

3.2.1 ZipList

Ziplist uses continuous memory compact storage. Unlike the linked list, it needs additional space to store the pointers of precursor nodes and subsequent nodes. According to the storage area, it can be roughly divided into three parts, and each part also has its own division. Its detailed structure is as follows.

  • Header information of ziplost

    • Zlbytes identifies the number of memory bytes occupied by the ziplost
    • The offset from zltail to the tail node of the ziplost
    • Number of storage nodes in zllen ziplost
  • Entries stores information about the actual node

    • pre_ entry_ Length records the length of the previous node. Through this value, you can quickly jump to the previous node
    • Encoding, as the name suggests, is the encoding format of storage element
    • Length the length of the stored data
    • Content saves the content of the node
  • End identifies the end of the ziplost

If the storage mode of linked list is adopted, the elements in the linked list are connected by pointers, which may cause some problemsMemory fragmentation。 The pointer also takes up additional storage space. Ziplist does not have these conditions. Ziplist occupies a continuous memory space.

Accordingly, however, the modification operation of ziplist is inefficient. The insertion and deletion operations will be designed to frequent memory space application and release (a bit similar to the re expansion of ArrayList), and the query efficiency will also be affected. In essence, ziplist query elements traverse the linked list.

3.2.2 QuickList

After version 3.2,listThe implementation of is replaced by QuickList. QuickList divides the list into multiple nodes, and each node adoptsZipListStore data.

4. Hash

Its usage is the same as that in HashMap in Java, which is to throw key value pairs into the map.

4.1 use

The basic commands are as follows:

  • hsetSet key value pairs in hash
  • hgetGet a key value in the hash
  • hdelDelete a key in the hash
  • hlenCount the number of elements in the hash
  • hmgetGet the value of the key in the hash in batch
  • Hmset key and value in batch setting hash
  • hexistsDetermine whether a key exists in the hash
  • hkeysReturns all keys (excluding values) in the hash
  • hvalsReturns all values in the hash (excluding keys)
  • hgetallGet all key value pairs, including keys and values

In fact, in most cases, the use of HashMap is similar to that of HashMap, and there is nothing special.

4.2 principle

There are also two underlying implementations of hash, ziplist and hashtable. However, the specific one is not related to the redis version, but to the elements stored in the current hash. First, when we create a hash, we use ziplist to store it. As the number of elements in the hash increases and reaches the threshold set by redis, it will be converted to hashtable.

The thresholds set are as follows:

  • The length of a stored key or value is greater than the default value (64)
  • The number of elements stored in ziplist is greater than the default value (512)

We specifically analyzed ziplist, and it should be easier to understand this setting. When there are too many elements in ziplist, its query efficiency will become inefficient. The underlying design of hashtable is actually similar to that in JavaHashMapAlmost all hash conflicts are solved by zipper method. For details, please refer to this article on digging HashMap from the use of foundation.

5. Set

The concept of set can be equated with set in Java to store a set that does not contain duplicate elements.

5.1 use

The main commands are as follows: key represents the set in redis, and member represents the elements in the set.

  • sadd sadd key member [...]Add one or more elements to the collection. If there are existing elements, they will be ignored.
  • srem srem key member [...]Remove one or more elements from the collection. Elements that do not exist are ignored
  • smembers smembers keyReturns all members in the collection
  • sismember dismember key memberJudge whether the member exists in the key. If it exists, it returns 1. If it does not exist, it returns 0
  • scard scard keyReturns the number of elements in the set key
  • smove move source destination memberMove the element from the source collection to the destination collection. If the source does not contain a member, no operation will be performed, and it will be removed from the collection if and only if it exists. If the destination element already exists, no action will be taken on the destination. This command is an atomic operation.
  • spop spop keyRandomly delete and return an element in the collection
  • srandmember srandmember keyLike spop, it just does not delete elements, which can be understood as randomly selecting an element from the collection.
  • sinterFind the intersection of one or more sets
  • sinterstore sinterstore destination key [...]Similar to sinter, but the result is saved in destination.
  • sunionFind the union of one or more sets
  • sunionstore sunionstore destination key [...]
  • sdiffFind the difference set of one or more sets
  • sdiffstore sdiffstore destination key [...]Similar to sdiff, but the resulting results are saved in the destination.

5.2 principle

We know that there are many implementations of set in Java. It is also true in redisIntSetandHashTableThe two implementations are first initialized usingIntSet, and when the following conditions are met, it will be converted toHashTable

  • When all elements saved in the collection are integers
  • The number of elements saved by the collection object does not exceed 512

Hashtable has been briefly introduced above, so we only talk about intset here.

5.2.1 IntSet

The bottom layer of intset is an array. Since the data structure is an array, the stored data can be orderly, which also makes the bottom query of intset realized through binary search. Its structure is as follows.

struct intset {
  //Coding mode
  uint32_t encoding;
  //The number of elements contained in the collection
  uint32_t length;
  //An array of storage elements
  int8_t contents[];
}

Similar to ziplist, intset also uses a series of memory spaces, but the difference is that ziplist can store binary contents, while intset can only store integers; Moreover, ziplist storage is unordered, while intset is ordered. In this way, the query efficiency of intset will be higher on the premise of the same number of elements.

6. Sorted Set

Its function is roughly similar to that of set, but on this basis, each element can be given a weight. You can understand it as JavaTreeSet。 Like list, hash and set, there are two underlying implementations, namelyZipListandSkipList(skip table).

When initializing the sorted set, ziplist will be used as its implementation. In fact, it is easy to understand that there are few elements at this time. Compact storage with ziplist will save more space. When the following conditions are met in the current period, it will be converted to skiplist:

  • The number of elements saved is less than 128
  • The length of all elements saved is less than 64 bytes

6.1 use

In the following command, key represents the name of Zset; 4 stands for score, that is, weight; And member is the name of the key in Zset.

  • zadd zadd key 4 memberUsed to add elements
  • zcard zcard keyUsed to get the number of elements in the Zset
  • zrem zrem key member [...]Delete one or more keys in Zset
  • zincrby zincrby key 1 memberAdd the value of score (i.e. 1) to the weight value of key
  • zscore zscore key memberUsed to obtain the weight value of the specified key
  • zrange zrange key 0 -1Get all elements in Zset,zrange key 0 -1 withscoresGet all elements and weights,withscoresParameter determines whether to return the weight value together. The elements returned arefrom small to largeSort. If the elements have the same weight, they are sorted in dictionary order.
  • zrevrange zrevrange key 0 -1 withscores, which is similar tozrangeSimilar, butzrevrangeaccording toFrom big to smallSort.
  • zrangebyscore zrangebyscore key 1 5, return the elements in the key whose weight is within the range (1, 5). Of course, it can also be usedwithscoresTo return the weight values together. Its elements followfrom small to largeSort. 1 for min, 5 for max, they can also be-infandinf, you can use this when you don’t know the score interval in the key. There is also an optional parameter similar to limit in SQL, which will not be repeated here.

In addition to being able to add weights to the elements, it can also be implemented using ZsetDelay queue

Delay queue is used to store delayed tasks. What is delay queue?

For a very simple example, if you place an order in an e-commerce app but do not pay, it will remind you that “if the order is not paid for more than 1 hour, it will be closed automatically”; Another example is to push messages to users one hour before the end of an activity; Another example is how many days after the order is completed, the receipt is automatically confirmed, etc.

Explain it in human words. That’s what we have to do after a meeting.

How does Zset realize this function?

In fact, it is very simple, that is, set the execution time of the task to the element weight in the Zset, and then regularly query the element with the smallest weight from the Zset through a background thread, and then judge with the current timestamp. If it is greater than the current timestamp (that is, it is time to execute), it will be taken out of the Zset.

Well, how?

In fact, I read many blogs about redis’s implementation of delay queue, but they didn’t explain clearly how to get this, what commands to use and what parameters to pass. We usezrangebyscoreCommand, remember – inf and inf, whose full name is infinity, representing infinitesimal and infinity respectively.

Since we do not know the range of score (i.e. task execution time) in the delay queue, we can directly use – inf and inf. the complete command is as follows.

zrangescore key -inf inf limt 0 1 withscores

It’s still a little useful. How is the bottom layer of Zset implemented?

Ziplist has been mentioned above, so I won’t repeat it. Let’s talk about skiplist.

6.2 principle

6.2.1 SkipList

Skiplist exists in the structure of Zset (sorted set), as follows:

struct zset {
  //Dictionary
  dict *dict;
  //Skip table
  zskiplist *zsl;
}

The skiplist structure is as follows:

struct zskiplist {
  //Header node and footer node
  struct zskiplistNode *header, *tail;
  //Number of nodes in the table
  unsigned long length;
  //The number of layers of the node with the largest number of layers in the table
  int level;
}

I wonder if you have thought about why redis uses skiplist to implement Zset instead of array?

First, if Zset is stored in an array, because the elements stored in Zset are orderly, the elements need to be placed in the corresponding position in the array. In this way, the array will be frequently added and deleted, and the efficiency of frequent addition and deletion in the array is not high, because it involves the movement of array elements. If the insertion position of elements is the first, then all subsequent elements will be moved.

Therefore, in order to cope with frequent additions and deletions, we need to use linked lists. However, with the increase of elements in the linked list, the same problems will occur. Although the efficiency of addition and deletion is improved, the efficiency of query becomes lower, because the query elements will traverse the linked list from beginning to end. All if there is any way to improve the query efficiency of the linked list.

So the jump watch was born. Based on the single linked list, the index is established every other node from the first node. In fact, it is also a single linked list. Only the node is not omitted in the middle.

For example, there is a single linked list 1 3 4 5 7 8 9 10 13 16 17 18

The index after abstraction is 1 4 7 9 13 17

If you want to query 16, you only need to traverse 13 in the index layer, and then according to the lower node stored in 13 (the address of the real linked list node), you only need to traverse two more nodes to find the node with the value of 16.

So you can redefine the jump table,The structure of linked list and multi-level index is jump table

In the jump table, the time complexity of querying any data isO(logn)。 The time complexity is the same as binary search. In other words, it is implemented with a single linked listBinary search。 But this is also a kind of useSpace for timeThe idea is not free.

End

The basic data structure of Redis and its underlying principle simply chat here. After several articles, we should talk about the high availability of Redis and its corresponding solutions. Interested parties can continue to pay attention to it, and the official account will be updated first than other platforms.

Previous articles:

  • Talk about microservices in vernacular — the evolution process that everyone can understand
  • Simply understand the underlying principle of InnoDB
  • On JVM and garbage collection
  • Learn more about concurrenthashmap
  • [simple understanding series] dig deep into HashMap from the use of foundation

If you think this article is helpful to you, it’s troublesomeLike itClose a noteShareLeave a message

WeChat can also search official account.SH’s full stack notes】Of course, you can also scan the QR code directly

Redis foundation -- analyzing the basic data structure and its usage

Recommended Today

Apache sqoop

Source: dark horse big data 1.png From the standpoint of Apache, data flow can be divided into data import and export: Import: data import. RDBMS—–>Hadoop Export: data export. Hadoop—->RDBMS 1.2 sqoop installation The prerequisite for installing sqoop is that you already have a Java and Hadoop environment. Latest stable version: 1.4.6 Download the sqoop installation […]