Redis 6.0 multithreading performance test results and analysis

Time:2021-4-21
 
Single threaded redis has always been known for its simplicity and efficiency, but it also has Achilles heel: blocking! Before the completion of the most bottleneck network read-write (redis big key, including some other heavyweight operations such as sort / Sunion / zunionstore / sdiff, centralized expired key cleaning, memory overflow maxmemory policy, etc.), all other requests will be blocked, seriously affecting its efficiency. Therefore, the voice of multi-threaded redis is getting higher and higher. Due to the low latency of memory based operations, CPU resources will not be the bottleneck of memory based operations even in single thread mode. The most likely bottleneck is network IO operation. After redis 6.0 started to support multithreading, the so-called multithreading is just multithreading at the socket level, and the core memory read-write mode is still single threaded.
After understanding the nature of multithreading, there will be a series of problems. How much improvement will multithreading have over single thread? How many threads are appropriate? See some big God test (at present, only Meitu has tested the unstable version in the whole network, and all the reprints are from this test), the result is also very ideal, but it’s not too enjoyable to just look at it, so I decided to have a try. This paper will make a superficial test and verification on the multi threading of redis. At the same time, we need to think about another problem: how to choose the multi-threaded version of redis and redis cluster?

Multi threaded redis

The “multithreading” feature of redis 6.0 makes many headlines climaxMeitu technical teamThe core thread is execute command or single thread. Multithreading refers to the multithreading of network IO (socket) reading and writing.
As shown in the figure below, multiple threads can be used to read and write the data in the network socket, so the multi thread of redis is also called IO thread, and the related parameter is “IO threads”. Another parameter is Io threads do reads, which involves another detail: multi thread IO is mainly used in the process of returning result set after request completion, that is, socket write operation. As for reading socket, the multiplexing technology adopted by single thread itself will not produce bottleneck. Therefore, redis gives an IO threads do reads Parameter to determine whether to enable multithreading when reading socket. In fact, whether IO threads do reads is enabled or not has little impact on the performance. Finally, a verification will be made.

 

Test environment and strategy

Local configuration: CentOS 7, 16C + 32GB memory


Redis version: 6.0.6
Under the configuration of 1 thread, 2 threads, 4 threads, 6 threads, 8 threads and 10 threads, 200 concurrent connections are used to make 100 million requests( ./ bin / redis – benchmark – d 128   – c 200 – n 1000000 – t set – q) At the same time, to avoid the impact of network delay, redis – Benchmark tests the get and set performance of redis locally.

 

Rollover

Before the whole test started, it took two rollovers to continue

Rollover site 1
The default version of GCC on CentOS 7 is 4. * and redis 6.0 can’t be compiled. Therefore, you need to upgrade GCC because the machine doesn’t support Yum installation,See hereUse source code package installation, GCC compile time that acid cool, native 16C + In the 32GB memory environment, due to the lack of some dependent packages, it failed several times, and the final compilation took about an hour and 10 minutes
Rollover site 2
I didn’t read the instructions in the configuration file carefully. After I / O threads is set and the redis service is restarted, I use redis benchmark to connect directly. The result is similar to that of single thread, which is very surprising.
Finally, I found this passage in the original configuration file:
If you want to test the Redis speedup using redis-benchmark, make sure you also run the benchmark itself in threaded mode, using the –threads option to match the number of Redis threads, otherwise you’ll not be able to notice the improvements.
It means it has to be in redis – Benchmark settings — The threads parameter should be set in match redis, — The threads parameter is redis 6 . A new parameter added after 0. Only add — The thread parameter can reflect the efficiency of multi-threaded redis.

Description of thread IO

After a second rollover, I decided to take a good look at redis . Comment information about thread IO in conf
################################ THREADED I/O #################################

# Redis is mostly single threaded, however there are certain threaded
# operations such as UNLINK, slow I/O accesses and other things that are
# performed on side threads.
#
# Now it is also possible to handle Redis clients socket reads and writes
# in different I/O threads. Since especially writing is so slow, normally
# Redis users use pipelining in order to speed up the Redis performances per
# core, and spawn multiple instances in order to scale more. Using I/O
# threads it is possible to easily speedup two times Redis without resorting
# to pipelining nor sharding of the instance.
#
# By default threading is disabled, we suggest enabling it only in machines
# that have at least 4 or more cores, leaving at least one spare core.
# Using more than 8 threads is unlikely to help much. We also recommend using
# threaded I/O only if you actually have performance problems, with Redis
# instances being able to use a quite big percentage of CPU time, otherwise
# there is no point in using this feature.
#
# So for instance if you have a four cores boxes, try to use 2 or 3 I/O
# threads, if you have a 8 cores, try to use 6 threads. In order to
# enable I/O threads use the following configuration directive:
#
# io-threads 4
#
# Setting io-threads to 1 will just use the main thread as usual.
# When I/O threads are enabled, we only use threads for writes, that is
# to thread the write(2) syscall and transfer the client buffers to the
# socket. However it is also possible to enable threading of reads and
# protocol parsing using the following configuration directive, by setting
# it to yes:
#
# io-threads-do-reads no
#
# Usually threading reads doesn't help much.
#
# NOTE 1: This configuration directive cannot be changed at runtime via
# CONFIG SET. Aso this feature currently does not work when SSL is
# enabled.
#
# NOTE 2: If you want to test the Redis speedup using redis-benchmark, make
# sure you also run the benchmark itself in threaded mode, using the
# --threads option to match the number of Redis threads, otherwise you'll not
# be able to notice the improvements.
The general meaning is as follows:
In most cases, redis runs in a single thread mode. However, some thread operations, such as disconnection, time-consuming I / O operations (such as bgsave, expired key cleaning) and other tasks are performed in the side thread (the sub thread of the main thread fork).
Now it’s also available in different I / The read and write of redis client socket are processed in O thread. Due to the slow speed of writing (socket writing), redis users usually use pipelining to improve the performance of redis on a single core, and use multiple instances to expand the capacity. Use I / O thread can easily improve the performance of redis in socket reading and writing, without the need to help the pipeline or instance partition.
In redis 6.0, multithreading is disabled by default. It is recommended to enable multithreading in a machine with at least four or more cores and leave at least one spare core. Using more than eight threads is unlikely to help much.
Because redis instances can make full use of CPU resources, multi-threaded I / O can only improve the efficiency if you do have performance problems, otherwise it is unnecessary to use this feature.
If you have a 4-core server, try 2 or 3 I / O threads. If it’s 8-core, try 6 threads. To enable multithreaded I / O, use the following configuration parameter: IO threads 4
Setting IO threads to 1 will enable a single main thread like traditional redis. When I / O threads are enabled, only write operations are supported. That is to say, IO threads call syscall and transmit client buffer to socket. However, you can also enable read-write threads and use the following configuration instructions for protocol resolution by setting it to “yes”: IO threads do reads No
Generally speaking, threading reads thread is not helpful for performance improvement
Note 1: this configuration instruction can’t be changed by configuration set at run time. It can only be restarted after modifying the configuration file. This feature is currently invalid when SSL is enabled.
Note 2: if you want to use redis – Benchmark to test the performance of redis, be sure to run redis in threaded mode – Benchmark, using — Threads option to match the number of redis threads, otherwise the improvement of test results cannot be observed.
 

Test results and analysis

The following is a horizontal comparison of the test results of requests per second in different threads. The QPS results of 100W get / set requests are performed in different threads

It can be seen that:
1, 1 thread, that is, the traditional single thread mode, the QPS of get operation can reach about 9W
For 2 or 2 threads, the QPS of get operation can reach about 18W, which is 100% higher than that of single thread
3, 4 threads, compared with 2 threads, there will be about 30% improvement, but it has not doubled from 1 thread to 2 threads
For set operation, QPS from 4 threads to 6 threads and 8 threads have no significant difference, which are between 23W and 24W
Compared with 4 threads and 6 threads, there is about 10% improvement under 8 threads
6 or 10 threads. Compared with the most efficient 8 threads, the performance starts to decline, which is equivalent to the efficiency of 4 or 6 threads
Therefore, in the local environment, if I / O threads 4 is set to 2 or 4 OK, the maximum number is no more than 8. If it is exceeded, the performance will be degraded, and the number of CPUs should not be exceeded. As the comment in the configuration file says, at least one CPU should be set aside.
 
The following are the requests per second of get and set in 10 test results under different threads   Average value comparison:

 

On the IO threads do reads parameter

I mentioned above – threads – do – The read parameter is used to determine whether the socket read operation is multithreaded or not. Redis’s socket read operation uses multiplexing technology, which will not become a bottleneck in itself. Therefore, this parameter has little impact on the test under multithreading. stillRefer to the picture hereIn my opinion, the IO threads do reads is the socket read thread. Whether it is enabled or not has little influence.
Redis . The comment about this parameter in conf is : When I / O threads are enabled , we only use threads for writes , that is to thread the write ( two ) syscall and transfer the client buffers to the socket . However it is also possible to enable threading of reads and protocol parsing using the following configuration directive , by setting it to yes Usually threading reads doesn ‘ t help much .
The following is a comparative test on whether this parameter is enabled or not

Refer to the following screenshot. When two threads are turned on, turn on and disable IO threads do reads respectively. On the whole, there is little difference in performance impact. Of course, professional God can be analyzed from the perspective of source code.
io – Threads is 2 and IO is started – threads – do – reads

I / O threads is 2 and I / O threads do reads is disabled

 

Selection of multi thread version of redis and redis cluster

There are two modes of redis cluster: sentinel and cluster. Let’s not mention sentinel for the moment. Let’s think about the selection of multi-threaded version of redis and redis cluster.
Redis cluster solves the problems of scalability and single node and single thread blocking. If redis supports multi thread (currently, it is not recommended to have more than 8 threads for multi thread redis), these two problems have been solved without considering the bottleneck of single node network card. A single node can support multi thread and make full use of multi-core CPU, and a single node can support 25W QPS, what more bicycles?
At the same time, the pain points of redis cluster should be considered
1, multiple / pipeline operation is not supported (third-party plug-ins are not necessarily stable).
2. Each master node in the cluster needs to hang a slave node, regardless of whether the node is an independent physical node or a node in a single machine multi instance, which eventually increases the maintenance cost.
3. Only one database can be used
4. A series of performance problems such as slot migration caused by cluster expansion and shrinkage, as well as cluster management problems
These so-called pain points no longer exist, so here we are faced with the problem of re selection: whether to use the multi-threaded version of redis or redis cluster? This is a question that needs to be considered.

 

doubt

About redis benchmark testing. / bin / redis benchmark – D 128 – C 200 – N 1000000 – t set – Q — threads 2, I don’t quite understand the two parameters – C and — threads involved
-C: Specifies the number of concurrent connections
–Threads is the number of threads in redis benchmark?
Tell me about itOld money(never met, thank you) after verification, it is said that this parameter is epoll of redis benchmark client. The principle of multiplexer on server side is very clear to me. The client side also has epoll, so I still don’t understand the relationship between the two.

 

Redis benchmark test site

The following is a partial screenshot of the redis benchmark test process

There are too many pictures to paste one by one.

Recommended Today

Envoy announced alpha version of native support for windows

Author: sunjay Bhatia Since 2016, porting envoy to the windows platform has been an important part of the projectOne of the goalsToday, we are excited to announce the alpha version of envoy’s windows native support. The contributor community has been working hard to bring the rich features of envoy to windows, which is another step […]