Detailed explanation of how to customize redis command with Lua

Time:2021-10-17

preface

As a very successful database, redis provides a wealth of data types and commands. Using these, we can easily and efficiently complete many cache operations, but there are always some special problems or needs to be solved. At this time, we may need to customize our own redis data structure and commands.

Redis command problem

Thread safety issues

We all know that redis is single threaded, but how can it have thread safety problems?

The thread safety problem we normally understand refers to the data resource burst caused by multiple threads operating in the single process multithreading model and sharing memory in the process. The thread safety problem of redis does not come from within the redis server.

As a data server, redis is equivalent to the shared memory of multiple clients, and multiple clients are equivalent to multiple threads in the same process. If there is no good data synchronization strategy between multiple clients, similar thread safety problems will occur.

Typical scenarios are:

  • Redis stores the status of a user: user5277 = idle;
  • Client connection a reads the user status and obtains the user’s idle status status status = get (“user5277”);
  • The client connection B also reads the user status;
  • Client connection a arranges a task for the user and sets the user status in redis to busy set (“user5277”, “busy”);
  • Client connection B also sets the user to the busy state.
  • However, at this time, the user is assigned two tasks at the same time.

The reason for this problem is that although redis is single threaded and can ensure the serialization of commands, due to its high execution efficiency, the commands of multiple clients do not do a good job in request synchronization, which will also cause the order of commands to be disordered.

Of course, this problem is also easy to solve. Just lock the user state, so that only one client can operate the user state at the same time. However, when adding locks, we need to consider lock granularity, deadlock and other issues. Undoubtedly, it adds the complexity of the program and is not conducive to maintenance.

Efficiency issues

Redis is an extremely efficient in memory data server. Its command execution speed is extremely fast. I have seen a pressure test result of Alibaba cloud redis before. The execution efficiency can reach 10W writing QPS and 60W reading QPS. Then, where does its efficiency problem come from?

The answer is the network. All web users know that efficiency optimization should start from the network. The server is to optimize the code and database. It is not as good as an optimization of network connection, and the most effective way of network optimization is to reduce the number of requests. We should know that it takes about 100ns to perform a memory access, while it takes about 500000 ns to go back and forth between different computer rooms. We can imagine the gap.

Redis is extremely efficient in a single machine, but the industrial deployment will not put the server and redis on the same machine. If there is an efficiency bottleneck, it is the network.

A typical scenario is that we read a piece of data from redis, and then use this data as a key to read another piece of data. In this way, there are two network round trips.

The reason for this problem is that redis’s common commands do not have the ability of server-side computing and cannot perform composite command operations on the server. Although redis also provides the pipeline feature, it requires that there is no dependency between the requests and responses of multiple commands. If you want to simplify multiple interdependent commands, you can only pull the data back to the client and request redis after being processed by the client.

To sum up, we need to “Customize” some commands to use redis more efficiently and conveniently.

Execution of embedded Lua

Fortunately, redis has built-in Lua execution environment to support the execution of lua scripts. By executing Lua scripts, we can compound multiple commands into one Lua script, and use Lua scripts to realize the ordering of redis commands and redis server-side computing mentioned above.

Lua

Lua is a concise, lightweight and extensible scripting language with the following features:

  • Lightweight: the source package has only the core library, and the compiled volume is very small.
  • High efficiency: written by ANSI C, it is fast to start and run.
  • Embedded: can be embedded into variousprograming languageOr run in the system to improve the flexibility of static language. For example, openresty embeds Lua into nginx for execution.

And there is no need to worry about grammar. Lua’s grammar is very simple and can be used every minute.

Perform steps

After version 2.6, redis will create Lua environment, load Lua library, define redis global table, store redis.pcall and other redis commands to prepare for the execution of lua script.

The execution steps of a typical Lua script are as follows:

  1. Check whether the script has been executed. If not, generate a Lua function using the SHA1 checksum of the script;
  2. Tick for function binding timeout and error handling;
  3. Create a pseudo client and execute the redis command in Lua through this pseudo client;
  4. Process the return value of the pseudo client and finally return it to the client;

The interaction sequence is shown in the figure

Detailed explanation of how to customize redis command with Lua

Although the Lua script uses a pseudo client, redis handles it like an ordinary client, and also performs RDB AOF master-slave replication and other operations on the executed redis commands.

use

The Lua script can be used through the eval and evalsha commands of redis.

Eval is applicable to the single execution of lua script. Before executing the script, SHA1 checksum will be generated from the script content. Query whether the function has been defined in the function table. If it is not defined, redis will cache the checksum of the script in the global table as the function name after successful execution. If you execute this command again later, no new function will be created.

To use the evalsha command, you must first use the script load command to load the function into redis. Redis will return the SHA1 checksum of this function, and then you can directly use this checksum to execute the command.

The following is an example of using the above command:

?
1
2
3
4
5
6
7
8
127.0.0.1:6379> EVAL "return 'hello'" 0 0
"hello"
 
127.0.0.1:6379> SCRIPT LOAD "return redis.pcall('GET', ARGV[1])"
"20b602dcc1bb4ba8fca6b74ab364c05c58161a0a"
 
127.0.0.1:6379> EVALSHA 20b602dcc1bb4ba8fca6b74ab364c05c58161a0a 0 test
"zbs"

The prototype of the eval command isEVAL script numkeys key [key ...] arg [arg ...] , can be used inside the Lua functionKEYS[N]andARGV[N] To reference keys and parameters, it should be noted that the parameter serial numbers of keys and argv start from 1.

It should also be noted that in Lua script, when redis returns null, the result is false instead of nil;

Lua script instance

Here are some examples of lua script to introduce the syntax for reference only.

The value of field B of HashSet a in redis is C. take out the value with the key C in redis.

?
1
2
3
4
//Use: Eval script 2 a B
 
local tmpKey = redis.call('HGET', KEYS[1], KEYS[2]);
return redis.call('GET', tmpKey);

Lpop multiple values at a time until the value is n or the list is empty (pipeline can also be easily implemented);

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
//Use: Eval script 2 list count
local list = {};
local item = false;
local num = tonumber(KEYS[2]);
while (num > 0)
do
  item = redis.call('LPOP', KEYS[1]);
  if item == false then
    break;
  end;
  table.insert(list, item);
  num = num - 1;
end;
return list;

Obtain the details in the HashSet corresponding to the n elements with the most scores in Zset;

?
1
2
3
4
5
6
7
local elements = redis.call('ZRANK', KEYS[1], 0, KEY[2]);
local detail = {};
for index,ele in elements do
  local info = redis.call('HGETALL', ele);
  table.insert(detail, info);
end;
return detail;

This is the basic usage syntax. More applications depend on each specific scenario.

Some thinking

In addition to implementation, there are also some things to think about:

Usage scenario

First, let’s summarize the usage scenarios of lua in redis:

  • The Lua script can be used to implement atomic operations to avoid data conflicts caused by different clients accessing the redis server.
  • When the results of previous and subsequent requests are dependent, you can use Lua script to integrate multiple requests into one request.

Attention

When using Lua script, we also need to pay attention to:

  • To ensure security, do not use global variables in the Lua script to avoid polluting the Lua environment. Although all errors are reported when using global variables and the Lua script stops executing, the local keyword is added when defining variables.
  • Pay attention to the time complexity of lua script. Redis’s single thread will also block the execution of lua script.
  • When using Lua script to implement atomic operations, it should be noted that if Lua script reports an error, the previous commands cannot be rolled back.
  • When multiple redis requests are issued at one time, but there is no dependency before and after the request, pipeline is more convenient than Lua script.

Summary

Recently, there have been great changes in work. From business to technology stack, they are completely different from the original. It’s really uncomfortable that all codes and businesses are out of their control. In work, they all “start a search engine and rely on grammar checking”. They have to stay up late every day to get familiar with new things. They are a little tired. Sure enough, changing jobs is to find a crime. However, the sense of fullness after walking out of the comfort zone also reminds me that I am making continuous progress, which is also quite a sense of achievement.

There’s no precipitation when I just come into contact with new things, and I don’t want to write some hydrology like “take you to master Java in three days”. My spare time is taken to supplement the technology stack needed by my work, and I don’t have time to study something interesting. I need materials to write articles. In order not to smash signboards, there may be less recently..

summary

The above is the whole content of this article. I hope the content of this article has a certain reference value for everyone’s study or work. If you have any questions, you can leave a message. Thank you for your support for developepper.