Redis QuickList source code analysis

Time:2021-5-31

1、 Introduction to QuickList

Redis list is a simple list of strings, sorted by insertion order. You can add an element to the top (left) or bottom (right) of the list.

A list can contain at most   two32-1 element (4294967295, more than 4 billion elements per list).

QuickList is the internal data structure that the underlying implementation relies on

1. List is a two-way linked list.

2. It is very convenient to add and delete data at both ends of the list, and the time complexity is O (1).

3. List also supports the access operation in any middle position, and the time complexity is O (n).

 

Before looking at the source code (version 3.2.2), let’s take a look at several main data structures in QuickList:

A QuickList consists of multiple quicklistnodes. Each quicklistnode points to a ziplist. A ziplist contains multiple entry elements, and each entry element is an element of a list. The schematic diagram is as follows:

 

 

Figure 1: QuickList

2、 QuickList data structure source code

Let’s take a look at the source code of QuickList and quicklistnode (the code file is QuickList. H, which will be analyzed later in the article)

quicklist:

/* 
   The QuickList structure takes up 32 bytes (64 bit system), in which the fields are:
   Head: points to the first quicklistnode.
   Tail: points to the last quicklistnode.
   Count: the total number of entries in all ziplists.
   Len: the number of quicklistnodes.
   Fill: the size of ziplist is limited by server.list_ max_ ziplist_ Size given.
   Compress: node compression depth setting, given by server.list-compress-depth, 0 means to turn off compression.
 */
typedef struct quicklist {
    quicklistNode *head;
    quicklistNode *tail;
    unsigned long count;        /* total count of all entries in all ziplists */
    unsigned int len;           /* number of quicklistNodes */
    int fill : 16;              /* fill factor for individual nodes */
    unsigned int compress : 16; /* depth of end nodes not to compress;0=off */
} quicklist;

quicklistNode:

/*
 Prev: points to the previous quicklistnode.
 Next: points to the next quicklistnode.
 ZL: ziplist pointing to the current node.
 SZ: the number of bytes occupied by ziplist.
 Count: the number of elements in the ziplist.
 Encoding: encoding type, raw = = 1 or lzf = = 2.
 Container: container type, none = = 1 or zip = = 2
 Recompress: bool type, true indicates that the node data has been decompressed temporarily.
 attempted_ Compress: bool type, used in the test phase.
 Extra: fill in the dictionary, which may be used in the future.
 */
typedef struct quicklistNode {
    struct quicklistNode *prev;
    struct quicklistNode *next;
    unsigned char *zl;
    unsigned int sz;             /* ziplist size in bytes */
    unsigned int count : 16;     /* count of items in ziplist */
    unsigned int encoding : 2;   /* RAW==1 or LZF==2 */
    unsigned int container : 2;  /* NONE==1 or ZIPLIST==2 */
    unsigned int recompress : 1; /* was this node previous compressed? */
    unsigned int attempted_compress : 1; /* node can't compress; too small */
    unsigned int extra : 10; /* more bits to steal for future usage */
} quicklistNode;

 

 

3、 Adding, deleting, modifying and querying QuickList

1. Create QuickList

When the push command is executed (for example, lpush), the QuickList will be created and initialized when no such key is found, as follows:

 

void pushGenericCommand(client *c, int where) {
    int j, waiting = 0, pushed = 0;
    robj *lobj = lookupKeyWrite(c->db,c->argv[1]);

    if (lobj && lobj->type != OBJ_LIST) {
        addReply(c,shared.wrongtypeerr);
        return;
    }

    for (j = 2; j < c->argc; j++) {
        c->argv[j] = tryObjectEncoding(c->argv[j]);
        if (! Lobj) {// if the key does not exist, create the key object and add it to DB
            lobj = createQuicklistObject(); //  Initialize QuickList object
            quicklistSetOptions(lobj->ptr, server.list_max_ziplist_size,
                                server.list_ compress_ depth); //  Use the configuration item of redis server to initialize
            dbAdd(c->db,c->argv[1],lobj); //  Add QuickList to redis DB
        }
        //Add new elements to the list
        listTypePush(lobj,c->argv[j],where);
        pushed++;
    }
    addReplyLongLong(c, waiting + (lobj ? listTypeLength(lobj) : 0));
    if (pushed) {
        char *event = (where == LIST_HEAD) ? "lpush" : "rpush";

        signalModifiedKey(c->db,c->argv[1]);
        notifyKeyspaceEvent(NOTIFY_LIST,event,c->argv[1],c->db->id);
    }
    server.dirty += pushed;
}

 

Let’s look at the create quicklistobject:

/* Create a new quicklist.
 * Free with quicklistRelease(). */
quicklist *quicklistCreate(void) {
    struct quicklist *quicklist;

    quicklist = zmalloc(sizeof(*quicklist));
    quicklist->head = quicklist->tail = NULL;
    quicklist->len = 0;
    quicklist->count = 0;
    quicklist->compress = 0;
    quicklist->fill = -2;
    return quicklist;
}

 

2. Add elements

Continue to see the listtypepush method in the above source code:

void listTypePush(robj *subject, robj *value, int where) {
    if (subject->encoding == OBJ_ENCODING_QUICKLIST) {
        int pos = (where == LIST_HEAD) ? QUICKLIST_HEAD : QUICKLIST_TAIL;
        value = getDecodedObject(value);
        size_ t len = sdslen(value->ptr);//  Calculate new element length
        quicklistPush(subject->ptr, value->ptr, len, pos); //  Join QuickList
        decrRefCount(value); 
    } else {
        serverPanic("Unknown list encoding");
    }
}

Continue with quicklistpush:

/* Wrapper to allow argument-based switching between HEAD/TAIL pop */
void quicklistPush(quicklist *quicklist, void *value, const size_t sz,
                   int where) {
    if (where == QUICKLIST_ Head) {// add to the list header
        quicklistPushHead(quicklist, value, sz);
    } else if (where == QUICKLIST_ Tail) {// add to the end of the list
        quicklistPushTail(quicklist, value, sz);
    }
}

/* Add new entry to head node of quicklist.
 *
 * Returns 0 if used existing head.
 * Returns 1 if new head created. 
 Add a new element in the head node of QuickList:
 If the new element is added to the head, return 0, otherwise return 1
 */
int quicklistPushHead(quicklist *quicklist, void *value, size_t sz) {
    quicklistNode *orig_head = quicklist->head;
    //If the head is not empty and the space size meets the storage requirements of the new element, the new element is added to the head, otherwise a quicklistnode is added
    if (likely(
            _quicklistNodeAllowInsert(quicklist->head, quicklist->fill, sz))) {
        quicklist->head->zl =
            ziplistPush(quicklist->head->zl, value, sz, ZIPLIST_HEAD);
        quicklistNodeUpdateSz(quicklist->head);
    } else {
        //Create a new quicklistnode
        quicklistNode *node = quicklistCreateNode();
        //Add the new element to the new ziplist
        node->zl = ziplistPush(ziplistNew(), value, sz, ZIPLIST_HEAD);
        //Update the length of ziplist to SZ field of quicklistnode, and then add the new node to QuickList, that is, add it in front of the original head
        quicklistNodeUpdateSz(node);
        _quicklistInsertNodeBefore(quicklist, quicklist->head, node);
    }
    quicklist->count++;
    quicklist->head->count++;
    return (orig_head != quicklist->head);
}

Ziplistbush will add new elements to ziplist, which will be analyzed later.

3. Get elements

The method to get elements is quicklistpop method (QuickList. C), as follows:

/* Default pop function
 *
 * Returns malloc'd value from quicklist */
int quicklistPop(quicklist *quicklist, int where, unsigned char **data,
                 unsigned int *sz, long long *slong) {
    unsigned char *vstr;
    unsigned int vlen;
    long long vlong;
    if (quicklist->count == 0)
        return 0;
    //Pop is an element
    int ret = quicklistPopCustom(quicklist, where, &vstr, &vlen, &vlong,
                                 _quicklistSaver);
    if (data)
        *data = vstr;
    if (slong)
        *slong = vlong;
    if (sz)
        *sz = vlen;
    return ret;
}

It is implemented in quicklistpopcustom

/* pop from quicklist and return result in 'data' ptr.  Value of 'data'
 * is the return value of 'saver' function pointer if the data is NOT a number.
 *
 * If the quicklist element is a long long, then the return value is returned in
 * 'sval'.
 *
 * Return value of 0 means no elements available.
 * Return value of 1 means check 'data' and 'sval' for values.
 * If 'data' is set, use 'data' and 'sz'.  Otherwise, use 'sval'. 
 If there is no element in QuickList, return 0, otherwise return 1
 When 1 is returned, the data and sval fields need to be checked
 1. If it is string type, save the result address in the data pointer and the length in SZ.
 2. If it is of long long type, save the value in the sval field.
 */
int quicklistPopCustom(quicklist *quicklist, int where, unsigned char **data,
                       unsigned int *sz, long long *sval,
                       void *(*saver)(unsigned char *data, unsigned int sz)) {
    unsigned char *p;
    unsigned char *vstr;
    unsigned int vlen;
    long long vlong;
    int pos = (where == QUICKLIST_HEAD) ? 0 : -1;

    if (quicklist->count == 0)
        return 0;

    if (data)
        *data = NULL;
    if (sz)
        *sz = 0;
    if (sval)
        *sval = -123456789;

    quicklistNode *node;
    if (where == QUICKLIST_HEAD && quicklist->head) {
        node = quicklist->head;
    } else if (where == QUICKLIST_TAIL && quicklist->tail) {
        node = quicklist->tail;
    } else {
        return 0;
    }
    //P: 0 takes the first element of ziplist- 1 take the last element of ziplist;
    p = ziplistIndex(node->zl, pos);
    if (ziplistGet(p, &vstr, &vlen, &vlong)) {
        if (vstr) {
            if (data)
                *data = saver(vstr, vlen);
            if (sz)
                *sz = vlen;
        } else {
            if (data)
                *data = NULL;
            if (sval)
                *sval = vlong;
        }
        //Remove the element from QuickList
        quicklistDelIndex(quicklist, node, &p);
        return 1;
    }
    return 0;
}

Take another look at quicklistdelindex

/* Delete one entry from list given the node for the entry and a pointer
 * to the entry in the node.
 *
 * Note: quicklistDelIndex() *requires* uncompressed nodes because you
 *       already had to get *p from an uncompressed node somewhere.
 *
 * Returns 1 if the entire node was deleted, 0 if node still exists.
 * Also updates in/out param 'p' with the next offset in the ziplist. 
 Delete an entry from quicklistnode:
 1. Delete entry from ziplist.
 2. Subtract 1 from the number of entries in quicklistnode
    If the number of entries in quicklistnode is 0, the current quicklistnode is deleted from QuickList.
    Otherwise, update the SZ field in quicklistnode.
 */
REDIS_STATIC int quicklistDelIndex(quicklist *quicklist, quicklistNode *node,
                                   unsigned char **p) {
    int gone = 0;

    node->zl = ziplistDelete(node->zl, p);
    node->count--;
    if (node->count == 0) {
        gone = 1;
        __quicklistDelNode(quicklist, node);
    } else {
        quicklistNodeUpdateSz(node);
    }
    quicklist->count--;
    /* If we deleted the node, the original node is no longer valid */
    return gone ? 1 : 0;
}

 

So far, the analysis of the main structure code of QuickList and the code of the two main methods push and pop is over. The next article analyzes the source code of the underlying storage ziplist of QuickList.

This article refers to Qian wenpin’s “redis deep Adventure: core principles and application practice”, thank you!

Recommended Today

Dandelion · Jerry technology weekly Vol.21 – technology weekly · React Hooks vs Vue 3 + Composition API

Dandelion · Jerry technology weekly Vol.21 Choose react or Vue, everyone will have their own answer in their heart. There are many reasons to pick the framework of heart water, but when we ask ourselves, can we really evaluate the difference between the two. Perhaps we should return to the original intention and look at […]