Redis core principle and practice — QuickList structure of list implementation principle

Time:2022-5-8

In the last articleZiplist structure of redis list implementation principle, we analyzed how the ziplost structure uses a complete piece of memory to store list data.
At the same time, it also raises a problem: if the linked list is very long, a large number of memory copies are required every time nodes are inserted or deleted in the ziplost, and this performance is unacceptable.
This paper analyzes how the QuickList structure solves this problem and implements the list type of redis.

The design idea of QuickList is very simple. A long zip list is divided into multiple short zip lists to avoid a large number of memory copies when inserting or deleting elements.
The form of ziplost storing data is more similar to array, and QuickList is a real linked list structure. It is linked by quicklistnode node, in which ziplost is used to store data.

Tip: the following codes in this article are located on quicklist.com unless otherwise specified h/quicklist. C.
The “node” mentioned below in this article refers to the quicklistnode node, rather than the node in the ziplost, unless otherwise specified.

definition

Quicklistnode is defined as follows:

typedef struct quicklistNode {
    struct quicklistNode *prev;
    struct quicklistNode *next;
    unsigned char *zl;
    unsigned int sz;             
    unsigned int count : 16;  
    unsigned int encoding : 2; 
    unsigned int container : 2; 
    unsigned int recompress : 1;
    unsigned int attempted_compress : 1;
    unsigned int extra : 10; 
} quicklistNode;
  • Prev and next: point to the predecessor node and the successor node.
  • ZL: ziplist, which is responsible for storing data.
  • SZ: number of bytes occupied by ziplist.
  • Count: the number of elements in the ziplost.
  • Encoding: 2 means that the node has been compressed, and 1 means that it has not been compressed.
  • Container: currently fixed to 2, which means ziplost is used to store data.
  • Recompress: 1 stands for temporary decompression (for reading data, etc.), and then compress it when necessary.
  • Extra: reserved attribute, not used yet.

When the linked list is very long, the data access frequency of intermediate nodes is low. At this time, redis will compress the intermediate node data to further save memory space. Redis adopts lossless compression algorithm – lzf algorithm.
The compressed nodes are defined as follows:

typedef struct quicklistLZF {
    unsigned int sz;
    char compressed[];
} quicklistLZF;
  • SZ: zip list size after compression.
  • Compressed: store the compressed ziplost byte array.

QuickList is defined as follows:

typedef struct quicklist {
    quicklistNode *head;
    quicklistNode *tail;
    unsigned long count;        
    unsigned long len;          
    int fill : QL_FILL_BITS;              
    unsigned int compress : QL_COMP_BITS;
    unsigned int bookmark_count: QL_BM_BITS;
    quicklistBookmark bookmarks[];
} quicklist;
  • Head and tail: point to the head node and tail node.
  • Count: the sum of the number of ziplost elements of all nodes.
  • Len: number of nodes.
  • Fill: 16bit, used to judge whether the node ziplost is full.
  • Compress: 16bit, storing node compression configuration.

The structure of QuickList is shown in Figure 2-5.
Redis core principle and practice -- QuickList structure of list implementation principle

Operation analysis

Insert element into QuickList header:

int quicklistPushHead(quicklist *quicklist, void *value, size_t sz) {
    quicklistNode *orig_head = quicklist->head;
    // [1]
    if (likely(
            _quicklistNodeAllowInsert(quicklist->head, quicklist->fill, sz))) {
        // [2]
        quicklist->head->zl =
            ziplistPush(quicklist->head->zl, value, sz, ZIPLIST_HEAD);
        // [3]
        quicklistNodeUpdateSz(quicklist->head);
    } else {
        // [4]
        quicklistNode *node = quicklistCreateNode();
        node->zl = ziplistPush(ziplistNew(), value, sz, ZIPLIST_HEAD);

        quicklistNodeUpdateSz(node);
        _quicklistInsertNodeBefore(quicklist, quicklist->head, node);
    }
    quicklist->count++;
    quicklist->head->count++;
    return (orig_head != quicklist->head);
}

Parameter Description:

  • Value, SZ: the content and size of the inserted element.

【1】 Judge whether the ziplost of the head node is full_ Quicklistnodeallowinsert function The fill attribute determines whether the node is full.
【2】 If the head node is not full, directly call the ziplistpush function and insert the element into the ziplost.
【3】 Update quicklistnode SZ property.
【4】 When the head node is full, create a new node, insert the element into the ziplost of the new node, and then insert the node head into the QuickList.

You can also insert elements in the specified position of the QuickList:

REDIS_STATIC void _quicklistInsert(quicklist *quicklist, quicklistEntry *entry,
                                   void *value, const size_t sz, int after) {
    int full = 0, at_tail = 0, at_head = 0, full_next = 0, full_prev = 0;
    int fill = quicklist->fill;
    quicklistNode *node = entry->node;
    quicklistNode *new_node = NULL;
    ...
    // [1]
    if (!_quicklistNodeAllowInsert(node, fill, sz)) {
        full = 1;
    }

    if (after && (entry->offset == node->count)) {
        at_tail = 1;
        if (!_quicklistNodeAllowInsert(node->next, fill, sz)) {
            full_next = 1;
        }
    }

    if (!after && (entry->offset == 0)) {
        at_head = 1;
        if (!_quicklistNodeAllowInsert(node->prev, fill, sz)) {
            full_prev = 1;
        }
    }
    // [2]
    ...
}

Parameter Description:

  • Entry: quicklistentry structure, quicklistentry Node specifies the quicklistnode node into which the element is inserted, quicklistentry Offset specifies the index location where the ziplost is inserted.
  • After: whether it is in quicklistentry Insert after offset.

【1】 Set the following flags according to the parameters.

  • Full: whether the ziplost of the node to be inserted is full.
  • at_ Tail: whether to insert the tail of ziplost.
  • at_ Head: whether to insert the ziplist header.
  • full_ Next: whether the rear drive node is full.
  • full_ Prev: whether the precursor node is full.

    Tip: insert the head finger into the head of the linked list, and insert the tail finger into the tail of the linked list.

【2】 According to the above flag, the code is cumbersome and will not be listed here.
The execution logic here is shown in table 2-2.

Redis core principle and practice -- QuickList structure of list implementation principle

Let’s just look at the implementation of the last scenario:

    // [1]
    quicklistDecompressNodeForUse(node);
    // [2]
    new_node = _quicklistSplitNode(node, entry->offset, after);
    new_node->zl = ziplistPush(new_node->zl, value, sz,
                                after ? ZIPLIST_HEAD : ZIPLIST_TAIL);
    new_node->count++;
    quicklistNodeUpdateSz(new_node);
    // [3]
    __quicklistInsertNode(quicklist, node, new_node, after);
    // [4]
    _quicklistMergeNodes(quicklist, node);

【1】 If the node is compressed, unzip the node.
【2】 Split a new node from the inserted node and insert the element into the new node.
【3】 Insert the new node into the QuickList.
【4】 Attempt to merge nodes_ Quicklistmergenodes attempts to do the following:

  • Merge node – > prev – > prev into node – > prev.
  • Merge node – > next into node – > next – > next.
  • Merge node – > prev into node.
  • Merge node into node – > next.
    Merge condition: if the size of the merged node still meets the QuickList If the fill parameter requires, merge nodes.
    This scenario processing is a little similar to the node splitting and merging of B + tree.

The commonly used functions of QuickList are shown in table 2-3.

function effect
quicklistCreate、quicklistNew Create an empty QuickList
quicklistPushHead,quicklistPushTail Insert elements at the head and tail of QuickList
quicklistIndex Finds the quicklistentry node for the given index
quicklistDelEntry Delete the given element

Configuration description

  • List Max ziplist size: configure server list_ max_ ziplist_ Size attribute, which will be assigned to QuickList fill。 Take a positive value to indicate the maximum number of elements that can be stored in the ziplost of the QuickList node. For example, if the configuration is 5, it means that the ziplost of each QuickList node contains up to 5 elements. Take a negative value, indicating the maximum number of bytes occupied by the ziplost of the QuickList node. At this time, it can only take five values from – 1 to – 5 (the default value is – 2). The meaning of each value is as follows:
    -5: The zip list size on each QuickList node cannot exceed 64 kb.
    -4: The zip list size on each QuickList node cannot exceed 32 KB.
    -3: The zip list size on each QuickList node cannot exceed 16 KB.
    -2: The zip list size on each QuickList node cannot exceed 8 KB.
    -1: The zip list size on each QuickList node cannot exceed 4 KB.
  • List compress depth: configure server list_ compress_ Depth attribute, which will be assigned to QuickList compress。
    0: indicates that nodes are not compressed. Redis is the default configuration.
    1: Indicates that one node at each end of the QuickList is not compressed, and the middle node is compressed.
    2: It means that two nodes at both ends of the QuickList are not compressed, and the middle node is compressed.
    3: It means that there are three nodes at both ends of the QuickList that are not compressed, and the middle node is compressed.
    and so on.

code

Ziplist is widely used in redis because of its compact structure and efficient use of memory. It can be used to save user lists, hashes, ordered collections and other data.
The list type has only one encoding format obj_ ENCODING_ QuickList: use QuickList to store data (redisobject.ptr points to QuickList structure). The implementation code of list type is in t_ list. C, readers can check the source code for more details.

summary

  • Ziplist is a compact data structure, which uses a piece of complete memory to store all the data of the linked list.
  • The elements in the zip list support different encoding formats to maximize memory savings.
  • QuickList improves the performance of operations such as inserting and deleting elements by splitting ziplost.
  • The coding format of linked list is only obj_ ENCODING_ QUICKLIST。

This article is excerpted from the author’s new bookRedis core principles and practices, this book deeply analyzes the internal mechanism and implementation methods of common redis features. Most of the content comes from the analysis of redis source code, and summarizes the design ideas and implementation principles. By reading this book, readers can quickly and easily understand the internal operation mechanism of redis.
With the consent of the editor of the book, I will continue to publish some chapters of the book on the personal technology official account as the preview content of the book. Welcome to consult. Thank you.

JD link
Douban link