SDS of redis source code

Time:2022-1-7

1: Introduction to SDS

We execute commands in redis

set key name

Key and name are string types, and strings are often used in redis. How does redis save strings? Let’s look down
As we all know, redis is written in C. in C, char is used to save the string, and 0 is used as the end of the string. However, redis is not saved in this way. Redis uses a structure called SDS to save the string. The structure is as follows (before redis3.2)

 struct sdshdr{
   int len;
   int free;
   char buf[];
}

So the question is, why does redis use the structure of SDS instead of the string of C language? Let’s see their differences

1: Calculate the difference in string length

For C, the way to calculate the length of the string is to traverse and stop when it meets 0, so the complexity pair is O (n), while SDS directly saves the length of the string, and the complexity is O (1)

2: Ensure binary security

Because SDS is not a flag ending with 0, it naturally ensures the security of binary

3: Memory management policy (pre allocated memory and lazy space release policy)

Redis is a high-speed cache database that requires frequent string operations. If the memory allocation is wrong, it will lead to serious consequences. Even if the memory allocation is OK, frequent memory allocation is very time-consuming, so these should be avoided

Inert space release strategy

In SDS, firstly, the inert space release strategy is used to optimize the string shortening operation of SDS.
When you want to shorten the string saved in SDS, the program does not immediately use the memory allocation to recover the shortened extra bytes, but uses the free member of the header to record these bytes and wait for future use.
The source code is as follows

Void sdsclear (SDS) {// resets the buf space of the SDS. Laziness is released
    struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
    sh->free += sh->len;    // Header free member + length of used space len = new free
    sh->len = 0;            // Used space becomes 0
    sh->buf[0] = '
Void sdsclear (SDS) {// resets the buf space of the SDS. Laziness is released
struct sdshdr *sh = (void*) (s-(sizeof(struct sdshdr)));
sh->free += sh->len;    // Header free member + length of used space len = new free
sh->len = 0;            // Used space becomes 0
sh->buf[0] = '\0';         // Empty string
}
'; // Empty string }
Pre allocated memory

The capacity expansion strategy is to double the capacity expansion space before the length of the string is less than 1m, that is, 100% redundant space is reserved. When the length exceeds 1m, in order to avoid waste caused by excessive redundant space after doubling, only 1m redundant space will be allocated for each expansion.

4: Compatible with C language function library (0 will be automatically added after the string)

SDS structure after version 3.2

The previous len, free and char structures look good, but there are some problems

 struct sdshdr{
   int len;
   int free;
   char buf[];
}

Len and free are both int types, both of which are 4byte, that is, 32bit. They can represent a range of about 4.2 billion, which greatly wastes space. Therefore, SDS has been changed after 3.2. The changes are as follows

typedef char *sds;

/* Note: sdshdr5 is never used, we just access the flags byte directly.
 * However is here to document the layout of type 5 SDS strings. */
struct __attribute__ ((__packed__)) sdshdr5 {
    unsigned char flags; /* 3 lsb of type, and 5 msb of string length */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr8 {
    uint8_t len; /* used */
    uint8_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
struct __attribute__ ((__packed__)) sdshdr16 {
    uint16_t len; /* used */
    uint16_t alloc; /* excluding the header and null terminator */
    unsigned char flags; /* 3 lsb of type, 5 unused bits */
    char buf[];
};
.........

Sdshdr5 represents the length of data with 5 bits, sdshdr8 represents the length of data with 8 bits, and so on
The memory allocation of sdshdr5 is shown in the figure below
SDS of redis source code
When the length of data to be stored exceeds 31, it needs to be represented by sdshdr8
The memory allocation of sdshdr8 is shown in the figure below
SDS of redis source code

The rest sdshdr16 and above are judged by analogy. The source code is as follows

static inline char sdsReqType(size_t string_size) {
    if (string_size < 1<<5) //2^5-1
        return SDS_TYPE_5;
    if (string_size < 1<<8) //2^8-1
        return SDS_TYPE_8;
    if (string_size < 1<<16) //2^16-1
        return SDS_TYPE_16;
#if (LONG_MAX == LLONG_MAX)
    if (string_size < 1ll<<32) //2^32-1
        return SDS_TYPE_32;
    return SDS_TYPE_64;
#else
    return SDS_TYPE_32;
#endif
}

My technical official account is highly recommended for every week.
Wechat can pay attention by scanning the QR code below:
SDS of redis source code