Read the source code with Dabin – redis 6 – objects and data types (Part 2)

Time:2021-1-25

Continue with our objects and data types.

In the last section, we learned about strings and lists, then hashes, collections, and ordered collections.

1 hash object

The optional codes of hash objects are ziplist and hashtable.

1.1 zip list encoded hash object

The hash object encoded by ziplist uses compressed list as the underlying implementation. Whenever there is a new key value pair to be added to the hash object, the program will first save itkeyThe compressed list node is pushed to the end of the table, and then savedvalueThe compressed list node is pushed to the end of the table. Therefore:

  • The two nodes that save the key value pair are always close together, the node that saves the key is in the front, and the node that saves the value is in the back;
  • The key value pairs added to the hash object will be copied in the header direction of the compressed list, and the key value pairs added later will be placed in the tail direction of the compressed list.

Execute the following hset command, and the server will create a list object as shown in Figure 9 as the value of the profile key:

127.0.0.1:6379> HSET profile name "Tom"
(integer) 1
127.0.0.1:6379> HSET profile age "25"
(integer) 1
127.0.0.1:6379> HSET profile career "Programer"
(integer) 1
127.0.0.1:6379> OBJECT ENCODING profile
"ziplist"

Read the source code with Dabin - redis 6 - objects and data types (Part 2)

The compression list used by the object is shown in Figure 10

Read the source code with Dabin - redis 6 - objects and data types (Part 2)

1.2 hashtable coding

Hashtable encoded hash objects use dictionary as the underlying implementation. Each key value pair in the hash object is saved with a dictionary key value pair

  • Each key in the dictionary is a string object, and the key of the key value pair is saved in the object;
  • Each value in the dictionary is a string object, which holds the value of the key value pair.

If the previous profile key uses hashtable encoded hash object, the hash object should be as shown in Figure 11

Read the source code with Dabin - redis 6 - objects and data types (Part 2)

1.3 code conversion

When the hash object meets the following two conditions at the same time, ziplist encoding will be used:

  1. In all key value pairs saved by hash object, the string length of key and value is less than 64 bytes;
  2. The number of key value pairs saved by hash object is less than 512.

The critical values in the above conditions correspond to redis.conf Configuration in file:hash-max-ziplist-valueandhash-max-ziplist-entries

In version 3.2, when adding a hash key value pair, a ziplist encoded hash object is always created first, and then the transformation check is performed.
There are two situations about when to perform code conversion

  1. When updating or adding a key value pair, ifThe number of bytes of the valuegreater thanhash-max-ziplist-valueTo convert ziplist coding to hashtable coding;
  2. When adding a key value pair, if theKey value vs. quantitygreater thanhash-max-ziplist-entriesFrom ziplist coding to hashtable coding.

It should be noted that in the case of the above conversion, there will be no conversion from hashtable to ziplist, even if the conditions are met.

On the hash code conversion function, you can refer to t_ Hash.c/hashtypeconvert, the source code is as follows:

#O is the original object and enc is the target code.
void hashTypeConvert(robj *o, int enc) {
    if (o->encoding == OBJ_ ENCODING_ Zip list) {// the original encoding is obj_ ENCODING_ Zip list
        hashTypeConvertZiplist(o, enc);
    } else if (o->encoding == OBJ_ENCODING_HT) {
        serverPanic("Not implemented");
    } else {
        serverPanic("Unknown hash encoding");
    }
}

2 collection objects

The optional codes of set objects are: intset and hashtable.

2.1 set object of intset encoding

The set object encoded by intset uses integer set as the underlying implementation, and all the elements contained in the set object are stored in the integer set.

Execute the following Sadd command to create an intset encoded set object as shown in Figure 12:

127.0.0.1:6379> SADD numbers 1 3 5
(integer) 3
127.0.0.1:6379> OBJECT ENCODING numbers
"intset"

Read the source code with Dabin - redis 6 - objects and data types (Part 2)

2.2 hashtable encoded set objects

Hashtable encodes a set object using a dictionary as the underlying implementation. Each key of the dictionary is a string object, and each string object contains a set element, while the dictionary values are all set to null.

Execute the following Sadd command to create a hashtable encoded collection object as shown in Figure 13:

127.0.0.1:6379> SADD fruits "apple" "banana" "cherry"
(integer) 3
127.0.0.1:6379> OBJECT ENCODING fruits
"hashtable"

Read the source code with Dabin - redis 6 - objects and data types (Part 2)

2.3 code conversion

When the set object satisfies the following two conditions at the same time, the object uses intset encoding

  1. All elements stored in the collection object can be expressed as integer values by long double;
  2. The number of elements saved by the collection object is no more than 512.

The critical values in the above conditions correspond to redis.conf Configuration in file:set-max-intset-entries

For set objects, when the first key value pair is added, the value in the key value pair will be checked. If it is an integer value that meets the conditions, an intset encoded set object will be created. Otherwise, a hashtable encoded set object will be created.

There are two situations about when to perform code conversion

  1. When updating or adding a key value pair, ifvalueIt can’t be represented by long double. It will be converted from intset coding to hashtable coding;
  2. When adding a key value pair, if theKey value vs. quantitygreater thanset-max-intset-entriesTo convert from intset encoding to hashtable encoding.

Similarly, in the case of conversion mentioned above, there will be no conversion from hashtable to intset, even if the conditions are met.

On the hash code conversion function, you can refer to t_ Set.c/settypeconvert, the source code is as follows:

#Setobj is the original object and enc is the target code.
hvoid setTypeConvert(robj *setobj, int enc) {
    setTypeIterator *si;
    serverAssertWithInfo(NULL,setobj,setobj->type == OBJ_SET && setobj->encoding == OBJ_ENCODING_INTSET);
    if (enc == OBJ_ ENCODING_ HT) {// can only be converted to obj_ ENCODING_ HT coding
        int64_t intele;
        dict *d = dictCreate(&setDictType,NULL);
        robj *element;
        /* Presize the dict to avoid rehashing */
        dictExpand(d,intsetLen(setobj->ptr));
        /* To add the elements we extract integers and create redis objects */
        si = setTypeInitIterator(setobj);
        while (setTypeNext(si,&element,&intele) != -1) {
            element = createStringObjectFromLongLong(intele);
            serverAssertWithInfo(NULL,element,
                                dictAdd(d,element,NULL) == DICT_OK);
        }
        setTypeReleaseIterator(si);
        setobj->encoding = OBJ_ENCODING_HT;
        zfree(setobj->ptr);
        setobj->ptr = d;
    } else {
        serverPanic("Unsupported set conversion");
    }
}

3. Ordered collection object

The optional codes for ordered collection objects are ziplist and skiplist.

3.1 ziplist encoded ordered collection objects

The set object encoded by intset uses compressed list as the underlying implementation. Each collection element is saved with two compressed list nodes next to each other. The first node holds the member of the element, and the second node holds the score of the element.

The set elements in the compressed list are sorted according to the score from small to large. The elements with smaller score are placed in the direction of the header, while the elements with larger score are placed near the end of the table.

Execute the following Sadd command to create a ziplist encoded collection object as shown in Figure 14:

127.0.0.1:6379> ZADD price 8.5 apple 5.0 banana 6.0 cherry
(integer) 3
127.0.0.1:6379> OBJECT ENCODING price
"ziplist"

Read the source code with Dabin - redis 6 - objects and data types (Part 2)

The underlying structure ziplist is shown in Figure 15
Read the source code with Dabin - redis 6 - objects and data types (Part 2)

3.2 skiplist encoded set objects

The collection object encoded by skiplist uses Zset as the underlying implementation. A Zset structure contains both a dictionary and a jump table. The structure source code is as follows:

# server.h
typedef struct zset {
    dict *dict;
    zskiplist *zsl;
} zset;

ZSL jump table in Zset structure saves all set elements from small to large according to score, and each jump table node saves one set element.

The object attribute of the jump table node holds the members of the element, while the score attribute of the jump table node holds the branches of the element. **Program through the jump table, the ordered set of range type operation. For example, zrank, zrange and other commands are implemented based on jump table API.

In addition, the dict dictionary in the Zset structure creates a mapping from member to score for the ordered set. Each key value pair in the dictionary holds a set element: the key in the dictionary holds the members of the element, while the value in the dictionary holds the score of the element. Through this dictionary, the program uses o (1) complexity to find the score of a given member.

In an ordered set, each element’s member is a string object, and each element’s score is a floating-point number of double type. It is worth mentioning that although the Zset structure uses both jump table and dictionary to save the elements of the ordered set, these two data structures will share the members and scores of the same elements through pointers, so it will not produce any duplicate members and scores, and will not waste extra memory.

If the price key creates a skip list instead of a ziplist encoded ordered set object, the ordered set object will be as shown in Figure 16, and the Zset result used by the object will be as shown in Figure 17
Read the source code with Dabin - redis 6 - objects and data types (Part 2)

Read the source code with Dabin - redis 6 - objects and data types (Part 2)

In Figure 17, members and scores of each element are shown repeatedly for convenience. In fact, they are members and scores of shared elements.

3.3 code conversion

When an ordered set object satisfies the following two conditions at the same time, the object is encoded with ziplist:

  1. The number of elements stored in an ordered collection object is no more than 128.
  2. All element members stored in an ordered collection are less than 64 bytes in length.

The critical values in the above conditions correspond to redis.conf Configuration in file:zset-max-ziplist-entriesandzset-max-ziplist-value

For a set object, when a key value pair is added, the set element and the value in the key value pair will be checked. If the condition is met, a ziplist encoded set object will be created. Otherwise, a skiplist encoded set object will be created. The corresponding source code is as follows:

# t_zset.c/zaddGenericCommand
...
zobj = lookupKeyWrite(c->db,key);
if (zobj == NULL) {
    if (xx) goto reply_to_client; /* No key + XX option: nothing to do. */
    if (server.zset_max_ziplist_entries == 0 ||
        server.zset_max_ziplist_value < sdslen(c->argv[scoreidx+1]->ptr))
    {
        #The number of object elements is 0, or
        zobj = createZsetObject();
    } else {
        zobj = createZsetZiplistObject();
    }
    dbAdd(c->db,key,zobj);
} else {
    if (zobj->type != OBJ_ZSET) {
        addReply(c,shared.wrongtypeerr);
        goto cleanup;
    }
}

summary

  1. Hash objects haveziplistandhashtablecode.
  2. Collection objects haveintsetandhashtablecode.
  3. Ordered set objects haveziplistandskiplistcode.