Redis string object

The encoding of a string object can be int, raw, or embstr

If a string object holds an integer value, and the integer value can be usedlongType, the string object will save the integer value in theptrAttribute (willvoid*convert tolong）And set the encoding of the string object toint.

int

For example, if we perform the followingSETCommand, the server will create aintEncoded string object asnumberKey value:

redis> SET number 10086
OK

redis> OBJECT ENCODING number
"int"

Redis string object

It is worth noting that:
longThe type represents the long integer of 8 bytes in C language
codeintIn redisREDIS_ENCODING_INTcode.

It is said in the redis redisobject data structure that,ptrIs a pointer to the data structure that actually holds the value. This data structure consists oftypeProperties andencodingAttribute determined

My system is Ubuntu 16.04 64 bit, using redis version 3.0.7, which can be saved-9223372036854775808~9223372036854775807In other words, the values in this range are encoded asint.

raw

If the string object holds a string value and the length of the string value is greater than 39 bytes, the string object will use a simple dynamic string (SDS) to hold the string value and set the encoding of the object toraw 。

For example, if we execute the following command, the server will create arawEncoded string object asstoryThe value of the key

redis> SET story "Long, long, long ago there lived a king ..."
OK

redis> STRLEN story
(integer) 43

redis> OBJECT ENCODING story
"raw"

Redis string object

embstr

If the string object holds a string value and the length of the string value is less than or equal to 39 bytes, the string object will use theembstrEncoding to save the string value

embstrEncoding is an optimized encoding method specially used to save short strings. This encoding is the same as raw encodingredisObjectStructure andsdshdrStructure to represent a string object, butrawThe code calls the memory allocation function twice to create theredisObjectStructure andsdshdrStructure

andembstrEncoding allocates a continuous space by calling the memory allocation function once, and the space containsredisObjectandsdshdrTwo structures

Redis string object

useembstrEncoding string objects to hold short string values has the following advantages:

embstrThe number of memory allocations required to create a string object fromrawThe encoding is reduced from two to one
releaseembstrThe encoded string object only needs to call the memory release function once and releaserawThe encoded string object needs to call the memory release function twice
becauseembstrAll the data of the encoded string object is stored in a continuous memory, so the encoded string object is compared withrawEncoded string objects can make better use of the advantages of caching

So why use 39 bytes or lessembstrWhat about the coding?

firstembstrIs a contiguous area of memory, consisting ofredisObjectandsdshdrform.

amongredisObjectIt takes 16 bytes. When the string length in buf is 39,sdshdrThe size of is8+39+1=48Which byte is'\0'. it adds up to 64

You can find out in this wayredisObjectIs 16 bytes

typedef struct redisObject {
    unsigned type:4;
    unsigned encoding:4;
    unsigned lru:24; /* lru time (relative to server.lruclock) */
    int refcount;
    void *ptr;
} robj;

sizeof(redisObject);

The following code tells you why it is 48 bytes

struct sdshdr {
    Unsigned int len; // string length
    Unsigned int free; // the number of bytes left available
    char buf[];
};

unsigned int4 bytes, 2 bytes, 8 bytes
bufThat’s the content. The example is 39 bytes
final+1refer to'\0', the string can use a'\0'At the end of the char array

It is worth noting that:
In redis 5.0.5, 39 becomes 44 bytes
This is because of theunsigned intCan represent a large range, but for very short SDS, a lot of space is wasted (twounsigned int8 bytes)
And the originalsdshdrChanged tosdshdr8, sdshdr16, sdshdr32, sdshdr64, insideunsigned intBecomeuint8_t, uint16_tThis will optimize the memory usage of small s DS

sds

There are two data types related to the implementation of SDS, one is SDS:

//Alias for string type
typedef char *sds;

The other is sdshdr

//Structure of holding SDS
struct sdshdr {
    //The number of string spaces that have been used in buf
    int len;
    //The number of reserved string space in buf
    int free;
    //Where the string is actually stored
    char buf[];
};

sdsOnly character array typechar*AndsdshdrIt is used for holding and keepingsdsInformation

How to append a string?

//Expand the reserved space of SDS to ensure that after calling this function,
//Addlen + 1 bytes (for null) after the SDS string is writable
sds sdsMakeRoomFor(sds s, size_t addlen) {
    struct sdshdr *sh, *newsh;
    size_t free = sdsavail(s);
    size_t len, newlen;

    //The reserved space can meet the needs of this splicing
    if (free >= addlen) return s;

    len = sdslen(s);
    sh = (void*) (s-(sizeof(struct sdshdr)));

    //Set the string length of the new SDS
    //This length is larger than the actual length required to complete the splicing
    //Optimize the next splicing operation by reserving space
    newlen = (len+addlen);
    if (newlen < SDS_MAX_PREALLOC)
        newlen *= 2;
    else
        newlen += SDS_MAX_PREALLOC;

    //Reallocate sdshdr
    newsh = zrealloc(sh, sizeof(struct sdshdr)+newlen+1);
    if (newsh == NULL) return NULL;

    newsh->free = newlen - len;

    //Only the string part is returned
    return newsh->buf;
}

fromnewlenWe can see that if thenewlenless thanSDS_MAX_PREALLOC, sonewlenThe actual value of will be twice the length required

IfnewlenThe value of is greater thanSDS_MAX_PREALLOC, sonewlenThe actual value of will be addedSDS_MAX_PREALLOC(current version 3.0.7 of SDS_ MAX_ The default value of prealloc is 1024 * 1024)

This memory allocation strategy shows that when the SDS value is expanded, extra space is always reserved. By spending more memory, the number of reallocate is reduced, and the processing speed of the next expansion operation is optimized

An example of optimizing extension operations is the append commandAPPENDWhen the command is executed, it will call sdsmakeroomfor to reserve more space. When next execute append, if you want to splice the string lengthaddlenNo more thansdshdr.free(last timeAPPENDThen you can skip the memory reallocation operation and directly perform string concatenation

Instead, if you don’t use this strategy, do it every timeAPPENDMust reallocate the memory

Note that no extra space is reserved when the SDS value is first created (see the definition of sdsnewlen given earlier), only when thesdsMakeRoomForAt least once, there will be reserved space in SDS, and there is corresponding compact space function in SDS modulesdsRemoveFreeSpace.

Therefore, this expansion strategy of redis to SDS value will not waste much memory, but it will get good optimization effect for some redis modes which need to perform string splicing many times (because frequent memory reallocation is a relatively expensive work)

SDS vs C language string

Redis string object

Redis string object