Loser! Redis wasted so much memory!

Time:2021-8-27

As an in memory database, the size of memory space is very important for redis. More memory means more data will be stored. But I don’t know if you have ever encountered such a situation. Obviously, the space is very large, but the use of memory is not very ideal.

Why did this happen? In this issue, let’s take a look at this “strange” event.

Sit down and get ready to start!
Loser! Redis wasted so much memory!
Note: mind map

View memory usage

First of all, we need to obtain relevant information to know the usage of redis memory.

Viewing memory related information in redis is very simple. You only need to enter “info memory” on the command line to see various related data. Here I list some important parameters:

  • used_memory: the amount of memory that has been used.
  • used_memory_rss: the size of redis physical memory.
  • mem_fragmentation_ratio: memory fragmentation rate.

Here is a term of memory fragmentation rate that needs attention. It can be used to indicate the current memory usage.

Specific calculation method:
Loser! Redis wasted so much memory!
For the memory fragmentation rate, it is generally maintained at  1~1.5Is the most reasonable.

What is memory fragmentation

After understanding the memory fragmentation rate, what is memory fragmentation?

Here’s the definition: because a continuous free space is smaller than the space to be applied for, this space is unavailable. For the memory as a whole, it is a memory fragment.

for instance:

Suppose there is a continuous free memory space of 100MB, and you will request a memory of 30MB from it every time. Then, after you apply three times, there is only 10MB left in this memory. The fourth application will fail. If no other space is released and the space requested each time is larger than 10MB, the remaining space is a memory fragment for the whole memory.
Loser! Redis wasted so much memory!

Causes of memory fragmentation

In redis, the most common methods are to write, modify and delete data. These operations will produce a certain degree of memory fragmentation after execution.

Write data

Redis allocates memory according to a fixed size. In order to reduce the allocation times, redis will allocate a corresponding size of space according to the closest fixed value of the requested memory.

What do you mean? If redis allocates memory according to 8 bytes, 16 bytes, 32 bytes, 48 bytes, etc. When you want to store an 18 byte data, redis will allocate 32 bytes (because 32 is the closest fixed value to 18). If the memory space required for the re written data is within 14 bytes, redis does not need to allocate any more.

It’s like you have different boxes. In order to hold things, you need to find a box with the closest volume. But after you put it in, you find that there is still room for some small things, so you don’t have to look for the box.

However, this way of allocating space will bring a certain degree of memory fragmentation. We can regard the divided space of fixed size as boxes of different volumes, and there will be surplus space in each box to varying degrees. These remaining spaces are memory fragments.

Modify data

When a key value pair is modified, it may become larger or smaller, which will occupy additional space or release unused space.
Loser! Redis wasted so much memory!
As shown in the figure, at present, a, B and C occupy 3, 2 and 4 bytes respectively. When a is modified from 3 bytes to 2 bytes, one byte of space will be empty, and one byte fragment will appear.
Loser! Redis wasted so much memory!
What if I change data a from 3 bytes to 4 bytes? At this time, in order to maintain the spatial continuity of data a, the operating system will copy B to another space. At this time, another 1-byte fragment will appear.

Delete data

It’s easy to understand how to modify data and delete data. In the above example, if data B is deleted, 2 bytes of space will be released. In this way, a 2-byte fragment is generated for the whole memory space.
Loser! Redis wasted so much memory!

How to resolve memory fragmentation

You may wonder, what harm will memory fragmentation do?

We still use the box above to represent it. If you want to load these boxes and transport them away, and there is free space (memory fragments) in each box, will the efficiency and cost performance of running once be very low. Similarly, in redis, due to the existence of a large number of fragments, the actual utilization will become low.

So is there any way to solve memory fragmentation?

Push down and start over

The first way is very simple, just push it down and start over. That is to restart redis directly, and the world will be clean as soon as the memory is powered off. But this violent and easy way has many hidden dangers.

If you do this in the production environment, you have to burn incense in advance. I hope there will be no problem. If you haven’t done persistence, don’t burn it. It’s useless to burn it. If there is persistence, the recovery time depends on the size of your persistent file. Services cannot be provided at this stage. Is it bad?

Space displacement

So is there a less exciting way.

Yes, the higher version of redis provides a way to clean up memory fragments. In a word, it is space replacement.

What kind of replacement method? Our goal is to eliminate memory fragmentation, so don’t we just rearrange the used memory data together? Let the discontinuous space become continuous, and continue to allocate the remaining space.

Draw a picture to understand:
Loser! Redis wasted so much memory!
However, it’s easy to talk about it. There is performance loss between theory and practice.

In the process of multiple data copies, a single threaded redis can only wait and cannot respond to the client’s request. At this time, you can only stare. The performance is too affected.

Cool, what should I do?! Don’t worry. There are mitigation strategies. You go on.

Redis has special parameter settings to automatically clean up memory fragments:activedefrag yes

This command starts the cleaning function, which is not enough. Redis needs other conditions to clean up.

The following parameters can be cleaned after any condition is met:

  • active-defrag-ignore-bytes 100mb: when the fragment reaches 100MB, start cleaning.  
  • active-defrag-threshold-lower 10: when the debris exceeds 10%, start cleaning.
  • active-defrag-threshold-upper 100: if the memory fragment exceeds 100%, clean it up as much as possible.

In the process of processing, in order to avoid the impact on normal requests and ensure performance. Redis also provides a parameter to monitor the CPU utilization ratio, so that the normal cleaning can be guaranteed only when the following conditions are met:

  • active-defrag-cycle-min 5: the proportion of CPU time occupied by cleaning memory fragments shall not be lower than this value to ensure that the cleaning can be carried out normally.
  • active-defrag-cycle-max 75: the percentage of CPU time spent cleaning up memory fragments is not higher than this value. Once exceeded, stop cleaning, so as to avoid a large number of memory copies blocking redis during cleaning, resulting in other request delays.

summary

View memory usage

  • Execute info memory on the command line to view redis memory related information. According to the memory fragmentation rate, it can be cleaned up at a certain time.

Causes of memory fragmentation

  • When writing data, redis allocates memory according to a fixed size in order to reduce the number of allocations. Additional memory space will be released or occupied when modifying data, and space will be released when deleting data. This will produce varying degrees of memory fragmentation.

How to resolve memory fragmentation

  • It is handled by restarting redis. If there is no persistence, it may lead to accidents. In the case of persistence, the recovery speed needs to depend on the size of the file.
  • Through space replacement, that is, reorganize the used memory data together.

Personal Java novice learning exchange Garden

Recommended Today

Swift advanced (XV) extension

The extension in swift is somewhat similar to the category in OC Extension can beenumeration、structural morphology、class、agreementAdd new features□ you can add methods, calculation attributes, subscripts, (convenient) initializers, nested types, protocols, etc What extensions can’t do:□ original functions cannot be overwritten□ you cannot add storage attributes or add attribute observers to existing attributes□ cannot add parent […]