Detailed explanation of redis stream type commands

Time:2021-10-14

This article was first published on personal blog, and the article link is: https://blog.d77.xyz/archives/d1007463.html

preface

Recently, the stream data type in redis is used as message queue in the project, which is more convenient than the message queue implemented by other redis. Since it is the first time to use, record the knowledge points for future use.

Stream type

There are many implementations of redis based message queues, but most of them have their own characteristics and problems. Redis itself is just a cache. It is estimated that the official can’t see it anymore. This is why a data type is added in redis 5.0 to implement typical message queues.

Stream type has almost all the functions required for a message queue, including but not limited to:

  • Serialization generation of message ID
  • Message traversal
  • Processing unacknowledged messages
  • Blocking and non blocking reads of messages
  • Packet consumption of messages
  • Message queue monitoring

wait….

The following describes how to use the stream data type.

Use of stream type

Xadd command

The syntax format is:

XADD key ID field value [field value …]

  • Key, used to specify the name of the stream
  • ID, used to specify the ID value. The most commonly used is*
  • Field value [field value…], key value type data

Xadd is used to add messages to the specified key. If the key does not exist, it will be created automatically. The added message iskey-valueType, you can add multiple messages at a time.

Specify the ID, the most commonly used is *, which means that redis automatically generates the ID, and the automatically generated ID is1526919030474-55Format, which is composed of millisecond timestamp and sequence number. The sequence number is used to distinguish messages generated in the same millisecond to ensure that the ID is always incremented. If the system clock is slow due to some other reasons, resulting in the generated timestamp being less than the value recorded in redis, the maximum value recorded in the system will be taken to continue to increase, so as to ensure the increasing status of ID.

127.0.0.1:6379> xadd message 1 key1 value1 key2 value2
"1-0"
127.0.0.1:6379> xadd message 1-2 key1 value1 key2 value2
"1-2"

In general, the ID is automatically specified by redis, but in fact, the ID can be customized. In order to ensure the self increment status of the ID, the manually specified ID must be greater than the ID existing in the system, but it is generally not done.

127.0.0.1:6379> xadd message * key1 value1 key2 value2
"1604475735664-0"

The above command adds key1 value and key2 Value2 messages to the message key. The return value is the ID of the current message, which is automatically generated by redis. At this time, there is a message in the message queue that can be read.

127.0.0.1:6379> xadd message maxlen 10 * key3 value3
"1604476672762-0"

The maxlen parameter is used to limit the maximum number of messages in the key. However, it is inefficient to precisely limit the number of messages in the key, which can be used~The symbol roughly limits the number of messages in the key. Redis will delete redundant messages only when the entire macro node can be deleted. The actual number may be dozens more than the limit. This is normal, but it will not be less than the limit.

Xread command

The syntax format is:

XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key …] id [id …]

  • [count], used to get the number of messages
  • [block milliseconds] is used to set the blocking mode and blocking timeout. It is non blocking by default
  • ID [ID…], which is used to set the starting ID of reading, equivalent to where id > $ID. in blocking mode, $can be used to obtain the latest message ID, which is meaningless in non blocking mode.
  • Key, specify the name of the stream

The xread command is used to read messages from one or more keys and only return messages whose ID value is greater than the ID passed in the parameter. This command has blocking and non blocking usage. If there is no message in the key, it returns null.

127.0.0.1:6379> xread streams message 0
1) 1) "message"
   2) 1) 1) "1-0"
         2) 1) "key1"
            2) "value1"
            3) "key2"
            4) "value2"
      2) 1) "1-2"
         2) 1) "key1"
            2) "value1"
            3) "key2"
            4) "value2"
      3) 1) "1604476672762-0"
         2) 1) "key3"
            2) "value3"

The above command outputs all messages in non blocking mode. Because there is no message with ID smaller than 0, it outputs all messages.

In blocking mode, you can use the $symbol to get the latest message. Returns NULL if there is no new message within the specified timeout.

127.0.0.1:6379> xread count 10 block 10000 streams message $
(nil)
(10.02s)
127.0.0.1:6379> xread count 10 block 10000 streams message $
1) 1) "message"
   2) 1) 1) "1604478070071-0"
         2) 1) "keyblock2"
            2) "value"
(5.00s)

After entering the command, you can observe that the command has no output. At this time, open a new redis CLI and enterxadd message * keyblock2 valueAdd a new message to the key. You can see that the above command returns the value and blocking time just added.

Xlen command

Syntax format:

XLEN key

Returns the number of messages in the key. If the key does not exist, 0 will be returned. Even if the number of messages in the key is 0, the key will not be automatically deleted because there may still be a consumer group associated with the key.

127.0.0.1:6379> xlen message
(integer) 5

Returns the number of all messages in the message.

Xrange command

The syntax is as follows:

XRANGE key start end [COUNT count]

  • Key, specify the name of the stream
  • Start, start ID
  • End, termination ID
  • [count count], number of reads

This command returns a message that matches the given ID range. The range of IDS is specified by the start and end parameters.

127.0.0.1:6379> xrange message - + 
1) 1) "1-0"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"
2) 1) "1-2"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"
3) 1) "1604476672762-0"
   2) 1) "key3"
      2) "value3"
4) 1) "1604478059261-0"
   2) 1) "keyblock"
      2) "value"
5) 1) "1604478070071-0"
   2) 1) "keyblock2"
      2) "value"

This command consists of two special IDs. Use – to represent the minimum ID value and + to represent the maximum ID value. You can query all messages.

127.0.0.1:6379> xrange message 1 1
1) 1) "1-0"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"
2) 1) "1-2"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"

Even if the ID is incomplete, just entering the ID value will output all messages with the same ID and different serial numbers.

127.0.0.1:6379> xrange message - + count 3
1) 1) "1-0"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"
2) 1) "1-2"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"
3) 1) "1604476672762-0"
   2) 1) "key3"
      2) "value3"

Use the count parameter to limit the number of output messages.

Through a simple loop, you can use a small amount of memory to iterate over all the values in a key. You only need to take the largest ID in the result of the last iteration as the starting ID of the next iteration.

Xrevrange command

Syntax description:

XREVRANGE key end start [COUNT count]

The syntax of the xrevrange command and the xrange command are exactly the same, with one difference. Xrevrange is traversed in reverse and will not be repeated.

127.0.0.1:6379> xrevrange message + -
1) 1) "1604478070071-0"
   2) 1) "keyblock2"
      2) "value"
2) 1) "1604478059261-0"
   2) 1) "keyblock"
      2) "value"
3) 1) "1604476672762-0"
   2) 1) "key3"
      2) "value3"
4) 1) "1-2"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"
5) 1) "1-0"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"

Xtrim command

Syntax description:

XTRIM key MAXLEN [~] count

  • Key, specify the name of the stream
  • Maxlen, specifies the pruning policy. Currently, only this one is implemented
  • [~], approximate trim
  • Count, quantity after trimming

The xtrim command starts discarding messages with small ID values.

127.0.0.1:6379> xread streams message 0
1) 1) "message"
   2) 1) 1) "1-0"
         2) 1) "key1"
            2) "value1"
            3) "key2"
            4) "value2"
      2) 1) "1-2"
         2) 1) "key1"
            2) "value1"
            3) "key2"
            4) "value2"
      3) 1) "1604476672762-0"
         2) 1) "key3"
            2) "value3"
      4) 1) "1604478059261-0"
         2) 1) "keyblock"
            2) "value"
      5) 1) "1604478070071-0"
         2) 1) "keyblock2"
            2) "value"
127.0.0.1:6379> xtrim message maxlen 4
(integer) 1

The return value of the xtrim command is the number of trimmed IDs.

127.0.0.1:6379> xrange message - +
1) 1) "1-2"
   2) 1) "key1"
      2) "value1"
      3) "key2"
      4) "value2"
2) 1) "1604476672762-0"
   2) 1) "key3"
      2) "value3"
3) 1) "1604478059261-0"
   2) 1) "keyblock"
      2) "value"
4) 1) "1604478070071-0"
   2) 1) "keyblock2"
      2) "value"

Looking again, you can see that the number of messages in the key has been trimmed out, leaving only four.

127.0.0.1:6379> xtrim message maxlen ~ 2
(integer) 0

If the ~ parameter is used, trimming may not occur. This parameter tells redis to perform pruning only when the entire macro node can be deleted, which is more efficient and can ensure that the number of messages is not less than the required number.

Xdel command

Syntax description:

XDEL key ID [ID …]

  • Key, specify the name of the stream
  • ID [ID…], ID value to be deleted

The xdel command is used to delete messages with a specified ID from the key. When the ID does not exist, the number returned may be inconsistent with the number deleted. When executing the xdel command, redis will not delete the corresponding message in memory, but will only mark it as deleted. After all nodes are deleted, the whole node is destroyed and the memory is recycled.

127.0.0.1:6379> xdel message 1-2
(integer) 1
127.0.0.1:6379> xrange message - +
1) 1) "1604476672762-0"
   2) 1) "key3"
      2) "value3"
2) 1) "1604478059261-0"
   2) 1) "keyblock"
      2) "value"
3) 1) "1604478070071-0"
   2) 1) "keyblock2"
      2) "value"

Message with ID 1-2 was deleted.

Xgroup command

Syntax description:

XGROUP [CREATE key groupname id-or-$] [SETID key groupname id-or-$] [DESTROY key groupname] [CREATECONSUMER key groupname consumername] [DELCONSUMER key groupname consumername]

  • [create key groupname ID or – $], create a group in the specified key and specify the starting point of the group to read messages. If 0 is specified, the group can read all historical messages of the specified key. If $isspecified, the group can read new messages of the specified key and cannot read historical messages. You can also specify any start ID.
  • [setid key groupname ID or – $], reset the starting point of message reading for existing groups. For example, setting the starting point to 0 can re read all historical messages
  • [destroy key groupname], destroy a group in the specified key
  • [create consumer key groupname consumername], create a consumer in the specified key and the specified group. When a command mentions a new consumer name, a new consumer is automatically created.
  • [delconsumer key groupname consumername], destroy a consumer in the specified key and the specified group.

Xgroup is a command group that can execute different commands through different keywords.

127.0.0.1:6379> xgroup create message read_group $
OK
127.0.0.1:6379> xreadgroup group read_group read streams message >
(nil)
127.0.0.1:6379> xadd message * readkey readvalue
"1604494179721-0"
127.0.0.1:6379> xreadgroup group read_group read streams message >
1) 1) "message"
   2) 1) 1) "1604494179721-0"
         2) 1) "readkey"
            2) "readvalue"

Use the create command to create a read_ Group grouping. The starting point of the specified ID is the last ID. if it is read directly, it is null (xreadgroup command is described below).

Use the xadd command to add a new message and read it again. It is found that the newly added message can be read normally.

127.0.0.1:6379> xgroup setid message read_group 0
OK
127.0.0.1:6379> xreadgroup group read_group read streams message >
1) 1) "message"
   2) 1) 1) "1604476672762-0"
         2) 1) "key3"
            2) "value3"
      2) 1) "1604478059261-0"
         2) 1) "keyblock"
            2) "value"
      3) 1) "1604478070071-0"
         2) 1) "keyblock2"
            2) "value"
      4) 1) "1604494179721-0"
         2) 1) "readkey"
            2) "readvalue"

Use the setid command to reset the starting point of reading ID, and all historical messages can be read.

127.0.0.1:6379> xinfo groups message
1) 1) "name"
   2) "read_group"
   3) "consumers"
   4) (integer) 1
   5) "pending"
   6) (integer) 4
   7) "last-delivered-id"
   8) "1604494179721-0"

Use the Xinfo command to query the newly created group information. You can see the group name, the number of consumers, and the ID value of the last added message (mentioned below the Xinfo command).

127.0.0.1:6379> xinfo consumers message read_group
1) 1) "name"
   2) "read"
   3) "pending"
   4) (integer) 4
   5) "idle"
   6) (integer) 476449

Use the Xinfo command to view the information of the newly added consumer. You can see the name of the consumer and the number of messages in the pending state (said under the pending state).

127.0.0.1:6379> xgroup delconsumer message read_group read
(integer) 4
127.0.0.1:6379> xinfo consumers message read_group
(empty array)

Use the delconsumer command to delete the consumers in the group, and use the Xinfo command to view the consumers in the group. An empty array is returned, indicating that the deletion is successful. The return value of the delconsumer command is the number of messages in pending status owned by the current consumer.

127.0.0.1:6379> xgroup destroy message read_group
(integer) 1
127.0.0.1:6379> xinfo groups message
(empty array)

Use the destroy command to delete the group in message, and use the Xinfo command to view it. An empty array is returned, indicating that the deletion is successful. The return value of the destroy command is the number of groups successfully deleted.Note: even messages with active consumers and pending status will still be deleted. You need to ensure that this command is executed when necessary.

Xinfo command

Syntax description:

XINFO [CONSUMERS key groupname] [GROUPS key] [STREAM key] [HELP]

  • [consumers key groupname], query the consumer information in the specified key and the specified group.
  • [groups key] to query the grouping information in the specified key.
  • [stream key] to query all the information in the specified key.
127.0.0.1:6379> xinfo consumers message read_group
1) 1) "name"
   2) "read"
   3) "pending"
   4) (integer) 4
   5) "idle"
   6) (integer) 206796

Read of read message_ All consumer information in the group.

127.0.0.1:6379> xinfo groups message
1) 1) "name"
   2) "read_group"
   3) "consumers"
   4) (integer) 1
   5) "pending"
   6) (integer) 4
   7) "last-delivered-id"
   8) "1604494179721-0"

Read all packet information in message.

127.0.0.1:6379> xinfo stream message
 1) "length"
 2) (integer) 4
 3) "radix-tree-keys"
 4) (integer) 1
 5) "radix-tree-nodes"
 6) (integer) 2
 7) "last-generated-id"
 8) "1604494179721-0"
 9) "groups"
10) (integer) 1
11) "first-entry"
12) 1) "1604476672762-0"
    2) 1) "key3"
       2) "value3"
13) "last-entry"
14) 1) "1604494179721-0"
    2) 1) "readkey"
       2) "readvalue"

Read all messages in message.

Extending command

Syntax description:

XPENDING key group [start end count] [consumer]

  • Key, specified key
  • Group, the specified group
  • [start end count], start ID and end ID, and quantity
  • Consumer, consumer name
127.0.0.1:6379> xpending message readgroup 
1) (integer) 2
2) "1604496633846-0"
3) "1604496640734-0"
4) 1) 1) "read"
      2) "2"

The expanding command can view the number of unconfirmed messages in the corresponding group, the name of the corresponding consumer, and the start and end IDs.

127.0.0.1:6379> xpending message readgroup - + 10 read
1) 1) "1604496633846-0"
   2) "read"
   3) (integer) 513557
   4) (integer) 4
2) 1) "1604496640734-0"
   2) "read"
   3) (integer) 482927
   4) (integer) 1

Use the extending command to view the details of a message in an unacknowledged state.

Xreadgroup command

Syntax description:

XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key …] ID [ID …]

  • Group, fixed
  • Group, group name
  • Consumer, consumer name
  • [count count], the number of messages obtained each time
  • [block milliseconds], blocking mode and timeout
  • [Noack], which does not require confirmation messages, is applicable to less important messages that can be lost
  • Streams, fixed
  • Key [key…], the specified key
  • ID [ID…], the specified message ID, > specifies to read all non consumed messages, and other values specify suspended messages

The xreadgroup command can read and confirm messages by combining with consumer groups and consumers. The operation of reading messages is refined on the basis of xread.

Syntactically, the xreadgroup and xread commands are almost the same. The xreadgroup command has one mandatory parameter:GROUP groupname consumername

When multiple consumers consume the same message queue at the same time, they will consume the same message repeatedly, and each message will be consumed by each consumer. However, if you want multiple consumers to consume the same message queue together, you need to use the consumer group.

For example, the push system must not be pushed repeatedly, that is, each message can only be consumed once. At this time, multiple consumers can use the same push queue to reduce the pressure on each consumer system.

127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> xadd message * key1 value1
"1604496633846-0"
127.0.0.1:6379> xadd message * key2 value2
"1604496640734-0"
127.0.0.1:6379> xgroup create message readgroup 0
OK
127.0.0.1:6379> xadd message * key3 value3
"1604496696501-0"
127.0.0.1:6379> xadd message * key4 value4
"1604496704823-0"
127.0.0.1:6379> xreadgroup group readgroup read count 1 streams message >
1) 1) "message"
   2) 1) 1) "1604496633846-0"
         2) 1) "key1"
            2) "value1"
127.0.0.1:6379> xreadgroup group readgroup read count 1 streams message >
1) 1) "message"
   2) 1) 1) "1604496640734-0"
         2) 1) "key2"
            2) "value2"

From the beginning, empty the database, add messages to message again, create groups, and use the readgroup command to read the latest messages.

127.0.0.1:6379> xinfo consumers message readgroup
1) 1) "name"
   2) "read"
   3) "pending"
   4) (integer) 2
   5) "idle"
   6) (integer) 53146
127.0.0.1:6379> xpending message readgroup - + 10 read
1) 1) "1604496633846-0"
   2) "read"
   3) (integer) 513557
   4) (integer) 4
2) 1) "1604496640734-0"
   2) "read"
   3) (integer) 482927
   4) (integer) 1

After reading two messages, you can see that there are two messages in pending status by using Xinfo command. You can view the specific information of messages in unconfirmed status by using expanding command.

Xack command

Syntax description:

XACK key group ID [ID …]

  • Key, specified key
  • Group, specified group
  • D [ID…], the ID of the message to be confirmed

The xack command deletes pending messages from the pending queue, that is, messages that have not been acknowledged before. When the xreadgroup command is used to read the message, the message is stored in the pel at the same time, waiting for confirmation. Calling the xack command can delete the pending message from the pel and free the memory to ensure that the message is not lost.

127.0.0.1:6379> xack message readgroup 1604496633846-0
(integer) 1
127.0.0.1:6379> xpending message readgroup 
1) (integer) 1
2) "1604496640734-0"
3) "1604496640734-0"
4) 1) 1) "read"
      2) "1"
127.0.0.1:6379> xinfo consumers message readgroup
1) 1) "name"
   2) "read"
   3) "pending"
   4) (integer) 1
   5) "idle"
   6) (integer) 418489

Use the xack command to confirm a message. Use the extending command again to check the number of unconfirmed messages. There is only one unconfirmed message left. Use the Xinfo command to check that the number of pending messages is also 1. The confirmation message is successful.

Xclaim command

Syntax description:

XCLAIM key group consumer min-idle-time ID [ID …] [IDLE ms] [TIME ms-unix-time] [RETRYCOUNT count] [FORCE] [JUSTID]

  • Key, specified key
  • Group, the specified group
  • Consumer, the specified consumer
  • Min idle time, specify the minimum number of idle messages, and specify how long idle messages will be selected
  • ID [ID…], the ID of the message
  • [idle MS], set the idle time of the message. If it is not provided, it defaults to 0
  • [time MS UNIX time], same as idle, UNIX timestamp
  • Retrycount sets the number of retries. Usually xclaim does not change this value. It is usually used in the extending command to find some messages that have not been processed for a long time.
  • Force, create a pending message in the pel, even if the specified ID has not been assigned to the pel of the client.
  • Justid, only the claimed message ID array is returned, and the actual message is not returned.

The xclaim command is used to change the ownership of an unconfirmed message. If a consumer hangs up without processing after reading the message, the message will always be in the pending queue and occupy memory. At this time, you need to use the xclaim command to change the owner of the message and let other consumers consume the message.

127.0.0.1:6379> xreadgroup group readgroup read2 count 1 streams message >
1) 1) "message"
   2) 1) 1) "1604496696501-0"
         2) 1) "key3"
            2) "value3"
127.0.0.1:6379> xack message readgroup 1604496696501-0
(integer) 1
127.0.0.1:6379> xpending message readgroup - + 10 read
1) 1) "1604496640734-0"
   2) "read"
   3) (integer) 1258517
   4) (integer) 2
127.0.0.1:6379> xpending message readgroup - + 10 read2
(empty array)

Create a new consumer, read a message, and then confirm the message. Check the number of unconfirmed messages in the two consumers. There is one in read and no unconfirmed message in read2.

127.0.0.1:6379> xclaim message readgroup read2 0 1604496640734-0
1) 1) "1604496640734-0"
   2) 1) "key2"
      2) "value2"
127.0.0.1:6379> xpending message readgroup - + 10 read2
1) 1) "1604496640734-0"
   2) "read2"
   3) (integer) 3724
   4) (integer) 4
127.0.0.1:6379> xpending message readgroup - + 10 read
(empty array)

Use the xclaim command to transfer the ownership of the message to the consumer read2. You can see that there is an unconfirmed message in the pending queue of consumer read2, and there is no message in the pending queue of consumer read.

summary

It’s a good experience to use redis as a message queue for the first time. It’s the first time to summarize this command. If there are omissions and errors in the above contents, please correct them.

Reference link

https://redis.io/commands#stream

Recommended Today

The selector returned by ngrx store createselector performs one-step debugging of fetching logic

Test source code: import { Component } from ‘@angular/core’; import { createSelector } from ‘@ngrx/store’; export interface State { counter1: number; counter2: number; } export const selectCounter1 = (state: State) => state.counter1; export const selectCounter2 = (state: State) => state.counter2; export const selectTotal = createSelector( selectCounter1, selectCounter2, (counter1, counter2) => counter1 + counter2 ); // […]