Explain how redis implements ack for queue messages

Time:2019-10-9

Preface

Because the queues provided by the company are too painful and limited to use other queues, in order to ensure data security, a queue with ack function is needed.

In the native redis, the function of queue is realized by L/R PUSH/POP. Of course, this can not meet the demand (without ACK function), so we need to make a small adjustment to the list of redis.

The general idea is to put the pop data in the backup place when POP, delete the backup information after ACK request (confirmation message is consumed); check whether there is no ack in the backup queue before pop, if there is, PUSH in the list and then POP out of the list.

The following script is implemented in lua, and only needs to be loaded into redis before execution.

The message itself needs to contain the ID attribute

Push is OK. Native is OK. (Take LPUSH for example)

Pop time script

local not_empty = function(x)
 return (type(x) == "table") and (not x.err) and (#x ~= 0)
end

LocalqName = ARGV [1] -- Queue name
Local currentTime = ARGV [2] -- Current time, which needs to be passed in from outside, cannot use redis own time, if using its own time may lead to inconsistencies in redis own backup when replaying requests
Local ConsiderAs FailMaxTimeSpan = ARGV [3] -- Timeout settings, when a message has not been acked for more than a certain period of time, the message needs to be re-entered

local zsetName= qName ..'BACKUP'
local hashName= qName ..'CONTEXT'

local tmp = redis.call('ZRANGEBYSCORE',zsetName , '-INF', tonumber(currentTime) - tonumber(considerAsFailMaxTimeSpan), 'LIMIT', 0, 1)
if (not_empty(tmp)) then
 Redis. call ('ZREM', zsetName, TMP [1]) -- The unique ID of the message is presented here
 redis.call('LPUSH', qName, redis.call('HGET', hashName, tmp[1]))
end
tmp = redis.call('RPOP', qName)
if (tmp) then
 local msg = cjson.decode(tmp)
 local id = msg['id']
 redis.call('ZADD', zsetName, tonumber(currentTime), id)
 redis.call('HSET',hashName , id, tmp)
end
return tmp

The ack time is relatively simple, just delete the specified ID from the set and hash.


 local key = ARGV[1]
 local qName=ARGV[2]
 redis.call('ZREM', qName..'BACKUP', key)
 redis.call('HDEL', qName..'CONTEXT', key)

The load scripts need to be displayed before they are used in the program, and then the Sha values of the two scripts can be directly invoked to execute.

summary

Above is the whole content of this article. I hope the content of this article can bring some help to everyone’s study or work. If you have any questions, you can leave a message and exchange it. Thank you for your support to developpaer.