Redis (8) — publish / subscribe and stream

Time:2020-9-4

Redis (8) -- publish / subscribe and stream

1、 Publish / subscribe function in redis

Publish / subscribe systemIt is a common function in web system. In a nutshellThe publisher publishes the message and the subscriber receives the messageThis is a bit like our newspaper / magazine(using the previous picture)

Redis (8) -- publish / subscribe and stream

  • The picture is quoted from “message queue”- https://www.wmyskxz.com/2019/07/16/xiao-xi-dui-lie-kan-guo-lai/

From usFront (related reading below)From the perspective of learning knowledge, although we can use onelistList structure combinationlpushandrpopTo achieve the function of message queue, but it seems difficult to achieveMessage multicastFunctions of:

Redis (8) -- publish / subscribe and stream

To support message multicast,RedisIt can no longer rely on the five basic data structures. It uses a single module to support message multicastPubSubThat is to sayPublisherSubscriber (publisher / subscriber model)

Introduction to PubSub

We start fromThe picture aboveBased on thelistStructure of message queue is a kind ofPublisherAndConsumerPoint to point strong correlation,RedisIn order to eliminate such strong association, another concept is introducedchannel (channel)

Redis (8) -- publish / subscribe and stream

WhenPublishertochannelWhen publishing a message in, thechannelOfConsumerCan receive messages at the same time. But here it isproblemYes, it is necessary for consumers to subscribe to a channelSpecify channel name explicitlyThat means, if we want toSubscribe to multipleChannel, then you have toExplicitly focus on multiplename.

In order to simplify the tedious operation of subscription,RedisProvidedMode subscriptionFunction ofPattern SubscribeThat’s OKFocus on multiple channels at onceEven if the producer adds a new channel of the same mode, the consumer can receive the message immediately:

Redis (8) -- publish / subscribe and stream

For example, in the figure above,AllAt the bottom of the pictureConsumerCan receive information

Publishertowmyskxz.chatthischannelA message was sent in, not just on this channelConsumer 1andConsumer 2Can receive messages, two of the pictureschannelDuhe modewmyskxz.*Match, soRedisThe message is also sent to the subscriptionwmyskxz.*This modelConsumer 3And followed another channel in this modewmyskxz.logUnder theConsumer 4andConsumer 5

On the other hand, if the channel receiving the message iswmyskxz.chat, soConsumer 3There will be news.

Quick experience

stayRedisMedium,PubSubThe use of the module is very simple, and the common commands are as follows:

#Subscription channel:
Subscribe channel [channel...] ා subscribe to information for a given channel or channels
Psubscribe pattern [pattern...] ා subscribe to one or more channels that match a given pattern
#Release channel:
Publish channel message? Sends a message to the specified channel
#Unsubscribe channel:
Unsubscribe [channel [channel...]]? Unsubscribe from the specified channel
Punsubscribe [pattern [pattern...]]? Unsubscribe all channels of a given pattern

We can quickly experience it locallyPubSub

Redis (8) -- publish / subscribe and stream

The specific steps are as follows:

  1. Open the local redis service and create two new console windows;
  2. Enter in one of the windowsSUBSCRIBE wmyskxz.chatfollowwmyskxz.chatChannel, let this window becomeconsumer
  3. Enter in another windowPUBLISH wmyskxz.chat 'message'Send a message to this channel and you’ll see it at this timeAnother window appears in real timeTest message sent.

Implementation principle

As you can see, we can almost simply use one of these two commandsPublish / subscribe systemBut how to achieve it?

Each redis server process maintains an identity server stateOfredis.h/redisServerStructure, in whichThere are channels with subscriptionas well assubscription model Information for:

struct redisServer {
    // ...
    dict *pubsub_ Channels; // subscribe to channels
    list *pubsub_ Patterns; // subscription mode
    // ...
};

Subscription channel principle

When the client subscribes to a channel, redis will go topubsub_channelsA new piece of data is added to the dictionary, which is actually this onedictThe dictionary maintains a linked list, as shown in the following figurepubsub_channelsIn the example,client 1client 2Just subscribechannel 1Other channels are also subscribed to by other clients:

Redis (8) -- publish / subscribe and stream

Subscribe command

SUBSCRIBEThe behavior of the command can be represented by the following pseudo code:

def SUBSCRIBE(client, channels):
    #Traverse all input channels
    for channel in channels:
        #Add the client to the end of the linked list
        redisServer.pubsub_channels[channel].append(client)

adoptpubsub_channelsDictionary, the program only needs to check whether a channel is the key of the dictionary, it can know whether the channel is being subscribed by the client; as long as the value of a key is taken out, the information of all clients who subscribe to the channel can be obtained.

Publish command

understandSUBSCRIBE, soPUBLISHThe implementation of the command is also very simple. You only need to locate the specific client through the above dictionary, and then send the message to them(pseudo code implementation is as follows)

def PUBLISH(channel, message):
    #Traverse all clients that subscribe to channel
    for client in server.pubsub_channels[channel]:
        #Send information to them
        send_message(client, message)

Unsubscribe command

useUNSUBSCRIBEThe command can unsubscribe from the specified channel. This command performs the reverse operation of the subscription: it can unsubscribe from thepubsub_channelsIn the given channel (key) of the dictionary, delete the information about the current client, so that the information about the unsubscribed channel will not be sent to this client again.

Subscription mode principle

Redis (8) -- publish / subscribe and stream

As we mentioned above, when sending a message towmyskxz.chatIn this channel, redis will send it not only to the current channel, but also to all channels that match the current mode,pubsub_patternsIt also maintains oneredis.h/pubsubPatternStructure:

typedef struct pubsubPattern {
    Redisclient * client; // client in subscription mode
    Robj * pattern; // subscription mode
} pubsubPattern;

Whenever calledPSUBSCRIBEWhen the command subscribes to a pattern, the program creates apubsubPatternStructure and add the structure to theredisServer.pubsub_patternsIn the linked list.

Let’s take a look at onepusub_patternsExamples of linked lists:

Redis (8) -- publish / subscribe and stream

At this time, the clientclient 3implementPSUBSCRIBE wmyskxz.java.*, sopubsub_patternsThe linked list will be updated as follows:

Redis (8) -- publish / subscribe and stream

By traversing the entirepubsub_patternsLinked list, the program can check all the patterns being subscribed to, as well as the clients subscribing to these patterns.

Publish command

The pseudo code given above does notComplete description PUBLISHThe act of ordering becausePUBLISHExcept formessageSend toall subscriptions channelClient forIn addition, it willchannelandpubsub_patternsMediumpatternFor comparison, ifchannelIf it matches a pattern, it will alsomessageSend toClients subscribing to that mode

Complete descriptionPUBLISHThe pseudo code for the function is set as follows:

def PUBLISH(channel, message):
    #Traverse all clients that subscribe to channel
    for client in server.pubsub_channels[channel]:
        #Send information to them
        send_message(client, message)
    #Take out all modes, as well as the client of subscription mode
    for pattern, client in server.pubsub_patterns:
        #If the channel matches the pattern
        if match(channel, pattern):
            #Then send the information to the client who subscribes to the pattern
            send_message(client, message)

Punsubscribe command

usePUNSUBSCRIBEThe command can unsubscribe the specified mode. This command performs the reverse operation of subscription mode: the order will be deletedredisServer.pubsub_patternsIn a linked list, all associated with the unsubscribed modepubsubPatternStructure so that the client will no longer receive messages from channels that match the pattern.

Disadvantages of PubSub

althoughRedisYesPubSubMode to achieveMulticast message queueBut in the field of actual message queuing, almostWe couldn’t find a particularly suitable sceneBecause its disadvantages are obvious

  • There is no ack mechanism and data continuity is not guaranteedThe producer of PubSub sends a message, and redis will directly find the corresponding consumer and pass it on. If there is no consumer, the message is discarded. If there are three consumers at first, one of them suddenly hangs up, and after a while, when it reconnects, then the information during the reconnection period will be completely lost to this consumer.
  • Non persistent message:If redis stops and restarts, PubSub’s messages will not be persistent. After all, when redis goes down, it means that a consumer does not have one, and all messages will be discarded directly.

Based on the above shortcomings, the author of redis even started a unique project for multicast message queuing, but it seems that the project is not mature at present. But then in June 2018,Redis 5.0AddedStreamData structure, this function brings redisPersistent message queueSince then, the function of PubSub as a message queue has disappeared

Redis (8) -- publish / subscribe and stream

2、 More powerful stream | persistent publish / subscribe system

Redis StreamConceptually, it’s like aAdditional content onlyOfMessage linked list, string all the messages one by one, and each message has a unique ID and content. This is very simple. What makes it complicated is another concept borrowed from KafkaConsumer group (same idea, different realization)

Redis (8) -- publish / subscribe and stream

The image above shows a typical exampleStreamStructure. Each stream has a unique name, which is redis’skey, in our first usexaddAutomatically created when the command appends a message. Let’s explain some concepts in the figure

  • Consumer GroupConsumer group, which can be simply regarded as a data structure to record the flow state. Consumers can choose to useXREADOrder to proceedIndependent consumptionIt is also possible for multiple consumers to join a consumer group at the same timeIntra group consumption。 Consumers in the same consumer group share all stream information,Only one consumer will consume the same messageIn this way, it can be applied in distributed application scenarios to ensure the uniqueness of messages.
  • last_delivered_id: used to indicate that the consumer group is spending on the streamConsumption positionCursor information for. Each consumer group has a streamUnique name, the consumer group will not be created automatically. You need to use theXGROUP CREATEInstruction to explicitly create, and you need to specify which message ID to start consumption from to initializelast_delivered_idThis variable.
  • pending_ids: each consumer has a state variable, which is used to representalreadyClient sideobtain, butNo ack yetNews of. The purpose of recording is toEnsure that the client consumes the message at least onceAnd will not be lost in the middle of network transmission without processing the message. If the client does not have ACK, there will be more and more message IDS in this variable. Once a message is backed, it will start to decrease accordingly. This variable is also officially known as redisPEL (Pending Entries List)

Message ID and message content

Message ID

Message ID ifXADDIf the command returns to automatic creation, its format will look like this:timestampInMillis-sequence (MS timestamp serial number)For example1527846880585-5, which indicates that the current message is a millisecond timestamp1527846880585Is the fifth message generated in that millisecond.

The format of these IDS looks a little strange,Why use time as part of your ID?On the one hand, we want toSatisfy ID auto incrementThe attributes of, on the other hand, are also forSupport range lookupThe function of. Since the ID is related to the time when the message is generated, there is basically no additional loss in searching according to the time range.

Of course, the message ID can also be customized by the client, but the form must beInteger integerAnd the ID of the message added later must be greater than the previous message ID.

Message content

Message content is a common key value pair, such as the key value pair of hash structure.

Examples of addition, deletion and modification

The order of adding, deleting, modifying and checking is very simple. The details are as follows:

  1. xadd: append message
  2. xdel: delete message. The deletion here only sets the flag bit and does not affect the total length of the message
  3. xrange: get the message list and automatically filter the deleted messages
  4. xlen: message length
  5. del: delete stream

Examples of use:

#The * sign indicates that the server automatically generates the ID, followed by a pile of keys / values
1:6379 > xadd codehole * name Laoqian age 30
Message generated by ID 㮹
127.0.0.1:6379> xadd codehole * name xiaoyu age 29
1527849629172-0
127.0.0.1:6379> xadd codehole * name xiaoqian age 1
1527849637634-0
127.0.0.1:6379> xlen codehole
(integer) 3
0.0.1:6379 > xrange codehole - + ා - represents the minimum value, + represents the maximum value
1) 1) 1527849609889-0
   2) 1) "name"
      2) "laoqian"
      3) "age"
      4) "30"
2) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
3) 1) 1527849637634-0
   2) 1) "name"
      2) "xiaoqian"
      3) "age"
      4) "1"
0.0.1:6379 > xrange codehole 1527849629172-0 + ා specifies the list of minimum message IDs
1) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
2) 1) 1527849637634-0
   2) 1) "name"
      2) "xiaoqian"
      3) "age"
      4) "1"
127.0.0.1:6379 > xrange codehole - 1527849629172-0 ා specifies the list of maximum message IDs
1) 1) 1527849609889-0
   2) 1) "name"
      2) "laoqian"
      3) "age"
      4) "30"
2) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
127.0.0.1:6379> xdel codehole 1527849609889-0
(integer) 1
0.0.1:6379 > xlen codehole ා length is not affected
(integer) 3
The deleted message is gone
1) 1) 1527849629172-0
   2) 1) "name"
      2) "xiaoyu"
      3) "age"
      4) "29"
2) 1) 1527849637634-0
   2) 1) "name"
      2) "xiaoqian"
      3) "age"
      4) "1"
127.0.0.1:6379 > del codehole ා delete the whole stream
(integer) 1

Examples of independent consumption

We can implement stream message without defining consumption groupsIndependent consumptionWhen the stream has no new messages, it can even block the wait. Redis designed a separate consumption instructionxreadThe stream can be used as a normal message queue (list). usexreadWe can ignore it completelyConsumer groupJust like a stream is a common list

#Read two messages from the stream header
127.0.0.1:6379> xread count 2 streams codehole 0-0
1) 1) "codehole"
   2) 1) 1) 1527851486781-0
         2) 1) "name"
            2) "laoqian"
            3) "age"
            4) "30"
      2) 1) 1527851493405-0
         2) 1) "name"
            2) "yurui"
            3) "age"
            4) "29"
#Read a message from the end of the stream, and there is no doubt that no message will be returned here
127.0.0.1:6379> xread count 1 streams codehole $
(nil)
#From the tail block waiting for a new message to arrive, the following instructions will block until the new message arrives
127.0.0.1:6379> xread block 0 count 1 streams codehole $
#We open a new window and fill the stream with messages
127.0.0.1:6379> xadd codehole * name youming age 60
1527852774092-0
#Then switch to the previous window, we can see that the blocking is lifted and the new message content is returned
#It also shows a waiting time, where we wait for 93s
127.0.0.1:6379> xread block 0 count 1 streams codehole $
1) 1) "codehole"
   2) 1) 1) 1527852774092-0
         2) 1) "name"
            2) "youming"
            3) "age"
            4) "60"
(93.11s)

If the client wants to usexreadconductSequential consumptionYou have toRemember current consumptionWhere is the message ID returned. Next continue callxreadIf the ID of the last message returned last time is passed in as a parameter, you can continue to consume subsequent messages.

block 0Always blocked until the message arrives,block 1000Indicates blocking1s, if1sIf there’s no message coming in, go backnil

127.0.0.1:6379> xread block 1000 count 1 streams codehole $
(nil)
(1.07s)

Create a consumer example

Stream throughxgroup createThe instruction creates a consumer group. It needs to pass the start message ID parameter to initializelast_delivered_idVariable:

Xcroup create codehole CG1 0-0 ᦇ means to start from scratch
OK
#$indicates that consumption starts from the tail, only new messages are accepted, and all current stream messages are ignored
127.0.0.1:6379> xgroup create codehole cg2 $
OK
127.0.0.1:6379 > Xinfo codehole ා get stream information
 1) length
 2) (integer) 3 ා 3 messages in total
 3) radix-tree-keys
 4) (integer) 1
 5) radix-tree-nodes
 6) (integer) 2
 7) groups
 8) (integer) 2 ා two consumption groups
 9) First entry ා first message
10) 1) 1527851486781-0
    2) 1) "name"
       2) "laoqian"
       3) "age"
       4) "30"
11) Last entry ා last message
12) 1) 1527851498956-0
    2) 1) "name"
       2) "xiaoqian"
       3) "age"
       4) "1"
127.0.0.1:6379 > Xinfo groups codehole ා get the consumption group information of stream
1) 1) name
   2) "cg1"
   3) consumers
   4) (integer) 0 ා there are no consumers in this consumption group
   5) pending
   6) (integer) 0 ා the consumer group has no messages being processed
2) 1) name
   2) "cg2"
   3) Consumers ා there are no consumers in this consumption group
   4) (integer) 0
   5) pending
   6) (integer) 0 ා the consumer group has no messages being processed

Examples of intra group consumption

Stream providesxreadgroupThe instruction can be used for intra group consumption of consumption group, which needs to be providedConsumer group name, consumer name and start message ID。 It is the same asxreadSimilarly, you can block waiting for new messages. After reading the new message, the corresponding message ID will enter the consumer’sPEL (message being processed)In the structure, the client is used after processingxackinstructionsNotification serverAfter the message has been processed, the message ID will change fromPELThe following is an example:

#The > sign indicates the last from the current consumption group_ delivered_ Read after ID
#Every time a consumer reads a message, last_ delivered_ The ID variable goes forward
127.0.0.1:6379> xreadgroup GROUP cg1 c1 count 1 streams codehole >
1) 1) "codehole"
   2) 1) 1) 1527851486781-0
         2) 1) "name"
            2) "laoqian"
            3) "age"
            4) "30"
127.0.0.1:6379> xreadgroup GROUP cg1 c1 count 1 streams codehole >
1) 1) "codehole"
   2) 1) 1) 1527851493405-0
         2) 1) "name"
            2) "yurui"
            3) "age"
            4) "29"
127.0.0.1:6379> xreadgroup GROUP cg1 c1 count 2 streams codehole >
1) 1) "codehole"
   2) 1) 1) 1527851498956-0
         2) 1) "name"
            2) "xiaoqian"
            3) "age"
            4) "1"
      2) 1) 1527852774092-0
         2) 1) "name"
            2) "youming"
            3) "age"
            4) "60"
#If you continue reading, there will be no new messages
127.0.0.1:6379> xreadgroup GROUP cg1 c1 count 1 streams codehole >
(nil)
#Then stop waiting
127.0.0.1:6379> xreadgroup GROUP cg1 c1 block 0 count 1 streams codehole >
#Open another window and fill in the message
127.0.0.1:6379> xadd codehole * name lanying age 61
1527854062442-0
#Back to the previous window, we found that the blocking was relieved and a new message was received
127.0.0.1:6379> xreadgroup GROUP cg1 c1 block 0 count 1 streams codehole >
1) 1) "codehole"
   2) 1) 1) 1527854062442-0
         2) 1) "name"
            2) "lanying"
            3) "age"
            4) "61"
(36.54s)
0.0.1:6379 > Xinfo groups codehole ා observe the consumption group information
1) 1) name
   2) "cg1"
   3) consumers
   4) (integer) 1: a consumer
   5) pending
   6) (integer) 5 ා is there any ack for 5 pieces of information being processed
2) 1) name
   2) "cg2"
   3) consumers
   4) (integer) 0 ා consumer group CG2 has no change, because we have been manipulating CG1
   5) pending
   6) (integer) 0
#If there are multiple consumers in the same consumption group, we can observe the status of each consumer through the Xinfo consumers command
At present, there is one consumer
1) 1) name
   2) "c1"
   3) pending
   4) (integer) 5 ා 5 messages to be processed
   5) idle
   6) (integer) 418715 ා how long has MS not read the message
#Next, let's ack a message
127.0.0.1:6379> xack codehole cg1 1527851486781-0
(integer) 1
127.0.0.1:6379> xinfo consumers codehole cg1
1) 1) name
   2) "c1"
   3) pending
   4) (integer) 4 becomes 5
   5) idle
   6) (integer) 668504
#All of the following ack messages
127.0.0.1:6379> xack codehole cg1 1527851493405-0 1527851498956-0 1527852774092-0 1527854062442-0
(integer) 4
127.0.0.1:6379> xinfo consumers codehole cg1
1) 1) name
   2) "c1"
   3) pending
   4) (integer) 0 ා pel is empty
   5) idle
   6) (integer) 745505

QA 1: what if there are too many stream messages? |Upper limit of stream

It is easy to think that if there is too much information accumulated, the chain list of the stream will not be very long, and whether the content will explode is a problem.xdelThe command does not delete the message, it just makes a flag bit for the message.

Redis naturally takes this into account, so it provides a fixed length stream function. stayxaddThe instruction provides a fixed lengthmaxlenYou can kill the old message to ensure that it does not exceed the specified length at most. It is also very simple to use

> XADD mystream MAXLEN 2 * value 1
1526654998691-0
> XADD mystream MAXLEN 2 * value 2
1526654999635-0
> XADD mystream MAXLEN 2 * value 3
1526655000369-0
> XLEN mystream
(integer) 2
> XRANGE mystream - +
1) 1) 1526654999635-0
   2) 1) "value"
      2) "2"
2) 1) 1526655000369-0
   2) 1) "value"
      2) "3"

If usedMAXLENWhen the stream reaches the specified length, the old messages will be eliminated automatically, so the size of the stream is constant. Currently, there is no option for the stream to keep only a given number of entries, because in order to run consistently, such commands must block for a long time to weed out messages.(for example, during the peak period of adding data, you have to pause for a long time to weed out old messages and add new ones)

Additional useMAXLENIn order to save memory space, stream uses a special structure, and the adjustment of this structure needs extra cost. So we can use a kind of~Special commands for:

XADD mystream MAXLEN ~ 1000 * ... entry fields here ...

It will reasonably prune the nodes based on the current structure to ensure that at least there will be1000Data, possibly1010It could be1030

QA 2: how does pel avoid message loss?

When the client consumer reads the stream message and the redis server replies the message to the client, the client suddenly disconnects and the message is lost. However, the message ID sent out has been saved in the pel. After the client reconnects, the message ID list in pel can be received again. But at this timexreadgroupThe starting message ID of cannot be a parameter>, and must be any valid message ID. generally, the parameter is set to0-0To read all pel messages and fromlast_delivered_idNew news after that.

Redis Stream Vs Kafka

Redis is based on memory storage, which means that it will be faster than the disk based Kafka, which also means that we can use redisCannot store large amounts of data for a long time。 But if you want toMinimum delayFor real-time processing of messages, you can consider redis, but ifThe message is large and data should be reusedKafka should be considered first.

And in some ways,Redis StreamIt’s also better for small, inexpensive applications becauseKafkaRelatively speaking, it is more difficult to configure.

Related reading

  1. Redis (1) — five basic data structures- https://www.wmyskxz.com/2020/02/28/redis-1-5-chong-ji-ben-shu-ju-jie-gou/
  2. Redis (2) – jump table- https://www.wmyskxz.com/2020/02/29/redis-2-tiao-yue-biao/
  3. Redis (3) — deep research on distributed lock- https://www.wmyskxz.com/2020/03/01/redis-3/
  4. Reids (4) — magical hyperloglog to solve statistical problems- https://www.wmyskxz.com/2020/03/02/reids-4-shen-qi-de-hyperloglog-jie-jue-tong-ji-wen-ti/
  5. Redis (5) – 100 million level data filtering and bloom filter- https://www.wmyskxz.com/2020/03/11/redis-5-yi-ji-shu-ju-guo-lu-he-bu-long-guo-lu-qi/
  6. Redis (6) – geohash finds people nearby https://www.wmyskxz.com/2020/03/12/redis-6-geohash-cha-zhao-fu-jin-de-ren/
  7. Redis (7) – persistence- https://www.wmyskxz.com/2020/03/13/redis-7-chi-jiu-hua-yi-wen-liao-jie/

reference material

  1. Subscription and publishing — Design and implementation of redis- https://redisbook.readthedocs.io/en/latest/feature/pubsub.html
  2. Deep adventure of redis by Qian wenpin- https://book.douban.com/subject/30386804/
  3. Introduction to redis streams- https://redis.io/topics/streams-intro
  4. Kafka vs. Redis: Log Aggregation Capabilities and Performance – https://logz.io/blog/kafka-vs-redis/
  • This article has been included in my GitHub programmer growth series[more than Java], learn more than code. Welcome to star: https://github.com/wmyskxz/MoreThanJava
  • Official account number :wmyskxz,Personal independent domain name blog: wmyskxz.com , adhere to the original output, scan the code below, 2020, grow together with you!

Redis (8) -- publish / subscribe and stream

Thank you very much for your talentSee hereIf you think this article is well writtenThere’s something about “I don’t have three hearts.”And then,Please like, pay attention, share and leave a message!

It’s not easy to create. Your support and recognition is the biggest driving force for my creation. See you in the next article!

Recommended Today

The selector returned by ngrx store createselector performs one-step debugging of fetching logic

Test source code: import { Component } from ‘@angular/core’; import { createSelector } from ‘@ngrx/store’; export interface State { counter1: number; counter2: number; } export const selectCounter1 = (state: State) => state.counter1; export const selectCounter2 = (state: State) => state.counter2; export const selectTotal = createSelector( selectCounter1, selectCounter2, (counter1, counter2) => counter1 + counter2 ); // […]