Redis optimization experience summary (required)

Time:2019-10-18

Memory management optimization

Redis hash is a HashMap within value. If the number of members of the map is relatively small, the map will be stored in a compact format similar to one-dimensional linear, which saves a lot of pointer memory overhead. This parameter control corresponds to the following two items in redis.conf configuration file:

hash-max-zipmap-entries 64 hash-max-zipmap-value 512       

When there are no more than a few members in the value map, it will be stored in linear compact format. By default, it is 64. That is to say, if there are less than 64 members in the value map, linear compact storage will be used. If the value is exceeded, it will be automatically converted into a real HashMap.

hash-max-zipmap-value The meaning is that when the length of each member value in the value map does not exceed a few bytes, linear compact storage will be used to save space.

If any one of the above two conditions exceeds the set value, it will be converted into a real HashMap, which will no longer save memory. So is the larger the value set, the better? Of course, the answer is No. the advantage of HashMap is that the time complexity of search and operation is O (1), and the time complexity of O (n) is to give up the one-dimensional storage of hash.

If the number of members is small, the impact will be small. Otherwise, the performance will be seriously affected. Therefore, it is the most fundamental trade-off between time cost and space cost to balance the setting of this value.

list-max-ziplist-value 64 list-max-ziplist-entries 512       

If the value size of the list data type node is less than how many bytes, the compact storage format will be used, and if the value of the list data type node is less than how many bytes, the compact storage format of pointer removal will be used.

Memory pre allocation:

The internal implementation of redis does not optimize the memory allocation too much (compared with Memcache). To a certain extent, there will be memory fragmentation, but in most cases, this will not become the performance bottleneck of redis. However, if most of the data stored in redis is numerical, a shared integer method is used in redis to save the cost of memory allocation, that is, When the system starts, a number of value objects from 1 to n are allocated in a pool. If the stored data is exactly the data within the range, the object is directly taken out of the pool and shared by reference counting. This can save memory and improve performance to a certain extent when the system stores a large number of values. The parameter value n You need to modify a line of macro definition redis? Shared? Interfaces in the source code. The default value is 10000, which can be modified according to your own needs. You can recompile after modification.

Persistence mechanism:

Snapshot mode:

This persistence method is actually a timer event in redis. Every fixed time, check whether the number and time of current data changes meet the configured conditions for triggering persistence. If they meet the requirements, a subprocess will be created by using the fork call of the operating system. By default, the subprocess will share the same address space with the parent process. At this time, you can use the subprocess to The main process can still provide services. When there is a write, the operating system will copy on write according to the memory page to ensure that the parent-child processes will not affect each other.

The main disadvantage of this persistence is that the scheduled snapshot only represents the memory image for a period of time, so the system will lose all data between the last snapshot and the reboot when the system is restarted.

Based on statement append mode (AOF):

Aof mode is actually similar to MySQL’s statement based binlog mode, that is, each command that will change redis memory data will be appended to a log file, that is, the log file is redis’s persistent data.

The main disadvantage of AOF method is that appending log files may lead to too large volume. When the system restarts to recover data, if it is AOF method, the data will be loaded very slowly. It may take several hours for dozens of G data to be loaded. Of course, this time-consuming is not because the disk file reading speed is slow, but because all commands read must be executed once in memory. In addition, because each command has to write log, the read-write performance of redis will also be reduced by using AOF.

It can be considered to save data to different redis instances. The memory size of each instance is about 2G. To avoid putting eggs in one basket can not only reduce the impact of cache failure on the system, but also accelerate the speed of data recovery, but also bring some complexity to the system design.

Redis persistent crash problem:

People who have experience in online operation and maintenance of redis will find that redis uses more physical memory, but when it does not exceed the total physical memory capacity, it will cause instability or even crash. Some people think that it is the persistent fork system call based on snapshot mode that causes the memory occupation to multiply. This view is not accurate, because the copy on write mechanism of fork call is the base. In the operating system page unit, only the dirty pages that have been written will be copied, but generally your system will not have all pages written in a short period of time, resulting in copying. So what causes redis to crash?

The answer is that redis uses buffer io for persistence. Buffer IO refers to the page cache of physical memory used by redis for writing and reading persistent files, while most database systems use direct IO to bypass the page cache of this layer and maintain a data cache by themselves. When redis’s persistent file is too large (especially snapshot file), it should be processed. When reading and writing, the data in the disk file will be loaded into the physical memory as a layer of cache of the operating system for the file, and the data in this layer of cache and the data managed in redis memory are actually stored repeatedly. Although the kernel will do the page cache removal work when the physical memory is tight, the kernel is likely to think that a certain page cache is more important, and let your process start to swap. At this point, your system will start to appear unstable or crash. Our experience is that when your redis physical memory usage exceeds 3 / 5 of the total memory capacity, it will start to be dangerous.

Conclusion:

1、Select appropriate data types according to business needs, and set corresponding compact storage parameters for different application scenarios.

2、When business scenarios do not need data persistence, turning off all persistence methods can get the best performance and maximum memory usage.

3、If you need to use persistence, choose between snapshot mode and statement appending mode according to whether you can tolerate the loss of data during restart. Do not use virtual memory or diskstore mode.

4、Don’t let your redis machine use more than 3 / 5 of the total physical memory.

The maxmemory option in redis.conf tells redis to reject subsequent write requests after using how much physical memory. This parameter can well protect your redis from swap caused by using too much physical memory, which will seriously affect performance and even crash.

VM enabled is no in redis.conf file

The above redis optimization experience summary (must be read) is all the content shared by the editor. I hope it can give you a reference, and I hope you can support developpaer more.

Recommended Today

JQuery’s promise

Promise is an asynchronous programming specification, transliteration to understand –“Prometheus” Prometheus is a hero in Greek mythology, whose name means “prophet”. Promise was originally named “futures”. It can be seen that promise does not mean literally “oath”, “promise”, etc., but “prophet” and “future”. Promise is to solve the most headache in asynchronous programmingCallback nesting region […]