Redis learning 6 (causes of redis blocking and its troubleshooting direction)


1、 Slow query

Because redis is a single thread, a large number of slow queries may cause the redis server to block. You can get the slow log through slowlog get n to view the details.

2、 Bigkey large object

Problems caused by big bigkey objects include:

  • Memory space is uneven (balanced). For example, in redis cluster, bigkey will cause uneven memory space utilization of nodes.
  • Timeout blocking: due to the single thread feature of redis, the operation of bigkey is time-consuming, which means that the possibility of blocking redis is increased.
  • Network congestion: each time a bigkey is obtained, the network traffic is relatively large. If a bigkey is 1MB and the number of visits is 1000, then 1000mb of traffic will be generated per second. For ordinary gigabit network card (128MB / s in bytes), it is a disaster.

The existence of bigkey is not completely fatal. If the bigkey exists but is rarely accessed, only the problem of uneven memory space exists. Compared with the other two problems, it is not so important and urgent. However, if the bigkey exists, it will not be accessed It is a hot key (frequently accessed), and its harm is unimaginable. Therefore, we must pay close attention to the existence of bigkey in the actual development and operation and maintenance.

Large objects can be found through redis cli – H {IP} – P {port} bigkeys.

3、 Swap

Because the data of redis is put in memory, the amount of data stored depends on the amount of memory.

If the memory utilization rate of a redis instance exceeds the maximum available memory (used)_ Memory > maximum available memory), then the operating system starts to exchange the memory and swap space, and writes the old or unused contents in the memory to the hard disk (this space on the hard disk is called swap partition), so as to free up new physical memory for new pages.

Reading and writing on the hard disk is about five orders of magnitude slower than that on the memory – memory is 0.1 μ s unit, and hard disk is 10 ms. If memory exchange occurs on the redis process, redis and applications that rely on the data on redis will be seriously affected.

View used_ The memory index can know the memory in use by redis, and identify the checking method of redis memory exchange:

  1. Identify redis process number

redis-cli info server | grep process_id

  1. Query memory exchange information according to process number

cat /proc/{process_id}/smaps | grep Swap

If the exchange volume is 0kb or individual 4KB, it is normal.

Prevent memory swapping:

  1. Ensure that the machine has enough available memory;
  2. Ensure that all redis samples set the maximum available memory to prevent uncontrolled growth of redis memory in extreme cases;
  3. Reduce the priority of system using swap, such as echo 10 > / proc / sys / VM / swap.

4、 Fork subprocess

When RDB is generated and AOF is rewritten, a child process will be forked to complete the persistence. If the fork operation takes too much time, it will cause blocking. The blocking reason is that the operation will copy the spatial memory table of the parent process, that is, the fork operation time-consuming has a great relationship with the amount of memory (data set).

Fork operation is a heavyweight operation, which will copy the spatial memory table of the parent process (theoretically, it needs to copy the same memory as the parent process, but Linux has a copy on write mechanism. The physical memory pages contributed by the parent and child processes are the same, which will be much smaller. 10g only needs 20MB).

The time consumption of fork should be within 20ms / GB; the maximum memory available for each instance should be controlled within 10GB (copy spatial memory table); the execution frequency of fork operation should be reduced and the trigger time of AOF rewriting should be appropriately relaxed.

Use the info stats command to get lastest_ fork_ USEC indicator, which indicates the latest fork operation time of redis.

5、 Aof brush disk blocked

When AOF is turned on, the file will be flushed once a second. When the hard disk pressure is too high, fsync needs to wait for the writing to complete.

View the AOF in redis log or info persistence statistics_ delayed_ Fsync index.

6、 Possible blocking caused by redis input buffer

Input buffer: redis allocates input buffer for each client. It will save the client’s sent commands temporarily, and then take them out for execution. Qbuf is the total capacity (0 means no query buffer is allocated), qbuf free is the remaining capacity (0 is no remaining space); the size cannot exceed 1g, when the size exceeds 1g, the client will be automatically shut down, and the input buffer is not restricted by maxmemory.

When a large number of keys enter the input buffer and cannot be consumed, it can cause redis blocking; through the client list command to locate the blocked client; through the info clients command blocked_ The clients parameter shows the currently blocked commands.

7、 Possible blocking caused by redis output buffer

Client output buffer: it is a read buffer implemented on the redis server side. After receiving the request from the client, the redis server writes the obtained results to the client buffer instead of sending it directly to the client. Thus, it can continue to process other requests from the client, so that the redis server will not block the processing of other requests due to network reasons.

The configuration parameter of redis client buffer is “client output buffer limit”. The default value is:> CONFIG GET "*output*"
1) "client-output-buffer-limit"
2) "normal 0 0 0 slave 0 0 0 pubsub 0 0 0"
  • Class: client type, normal, slave, PubSub
    • Normal: a normal client
    • Slave: the replication client from the library
    • Pub / sub: the client of publish and subscribe
  • Hard limit: hard limit of buffer size.
  • Soft limit: the soft limit of buffer size.
  • Soft seconds: the buffer size has reached (exceeded) the duration of the soft limit value.

The client output buffer limit parameter limits the size of the allocated buffer and prevents excessive memory allocation. The default value of the parameter is 0, which means there is no restriction.

There are two kinds of trigger protection mechanism for redis server

  1. If the size of the client buffer reaches the soft limit and lasts for soft seconds, it will immediately disconnect from the client.
  2. When the size of the client buffer reaches the hard limit, the server will immediately disconnect from the client.

8、 Network problems

1. Connection rejection

  • Network flash off: generally in the case of network cutover or bandwidth depletion;
  • Redis connection denial: when the number of connections is greater than maxclients, new connections are rejected. You can pay attention to rejected info stats_ Connections index;
  • Connection overflow:
    • Process limit: the maximum number of files that can be opened by a process — ulimit – N, usually 1024. This value needs to be increased for a large number of connected redis;
    • Backlog queue overflow: the system uses the backlog queue to save TCP connections on specific ports. The default value of redis is 511, and the system’s backlog is 128. Cron can be used to perform netstat – s | grep overflowed statistics regularly;

2. Network delay

Measuring network latency between machines

redis-cli -h {ip} -p {port} –latency
Redis cli – H {IP} – P {port} – latency history default 15 seconds to complete one line of statistics, – I controls the sampling time
Redis-cli – H {IP} – P {port} – latency dist statistical chart, sampling every 1 second