Getting started with stream

Time：2022-5-29

1、 Stream introduction

Springcloud stream is developed on the basis of spring messaging and spring integration projects. We may use multiple MQ in the project, or we need to replace MQ at a certain stage of the project. If we use stream, we can just change the configuration without modifying the code for sending and receiving messages. Moreover, the messaging model defined by stream makes our code more concise and understandable.

For detailed introduction documents, please refer to:
Introduction to spring cloud stream system and principle (qq.com)

2、 Introductory case

First we start withstart.spring.ioDownload a brand-new project. The required dependencies include web and Lombok. After downloading, introduce the stream rocketmq dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>com.alibaba.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rocketmq</artifactId>
    <version>2021.1</version>
</dependency>
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <optional>true</optional>
</dependency>

Then we start creating producers:

@Slf4j
@Component
@EnableBinding(Source.class)
public class MyMessageProducer {

    @Autowired
    private Source source;

    public void send(String message){
        Log Info ("producer starts sending messages!");
        source.output().send(MessageBuilder.withPayload(message).build());
    }

}

We use the controller to send messages to the departure producer:

@RestController
public class ProducerController {

    @Autowired
    private MyMessageProducer producer;

    @GetMapping("/sendMessage")
    public String sendMessage(){
        producer.send("hello spring cloud stream!");
        return "OK";
    }

}

Then we create consumers:

@Slf4j
@Component
@EnableBinding(Sink.class)
public class MyMessageConsumer {

    @StreamListener(Sink.INPUT)
    public void receive(String message){
        Log Info ("the message content listened to is: {}", message);
    }
    
}

Finally, the most important content of our configuration file:

server:
  port: 8081

spring:
  application:
    name: stream-demo
  cloud:
    stream:
      bindings:
        output:
          destination: myRocketMQTopic
          group: test-stream
        input:
          destination: myRocketMQTopic
          group: test-stream
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876

On the premise of starting the MQ service and starting our project, we can successfully run a QuickStart case of stream rocketmq. Our browser accesses the link so that the producer sends a message, and then the consumer can listen to the message.

2021-07-16 09:35:26.805 info 16920 -- [nio-8081-exec-1] c.e.s.producer Mymessageproducer: the producer starts sending messages!
2021-07-16 09:35:49.737 info 16920 -- [messagethread_1] c.e.s.consumer Mymessageconsumer: the message content is: Hello spring cloud stream!
2021-07-16 09:36:14.371 info 16920 -- [nio-8081-exec-3] c.e.s.producer Mymessageproducer: the producer starts sending messages!
2021-07-16 09:36:14.379 info 16920 -- [messagethread_1] c.e.s.consumer Mymessageconsumer: the message content is: Hello spring cloud stream!

3、 Custom channel

In the above introductory case, we use the output and input channels of the stream, and we can customize our own channels by imitating them.

public interface MyOutputChannel {

    @Output("myOutput")
    MessageChannel myOutput();

}

public interface MyInputChannel {

    @Input("myInput")
    SubscribableChannel myInput();

}

Then we create a new producer and use a custom output channel.

@Slf4j
@Component
@EnableBinding(MyOutputChannel.class)
public class MyMessageProducer2 {

    @Autowired
    private MyOutputChannel myOutputChannel;

    public void send(String message){
        Log Info ("mymessageproducer2 producer starts sending messages!");
        myOutputChannel.myOutput().send(MessageBuilder.withPayload(message).build());
    }

}

Create a new consumer to listen to the customized input channel.

@Slf4j
@Component
@EnableBinding(MyInputChannel.class)
public class MyMessageConsumer2 {

    @StreamListener("myInput")
    public void receive(String message){
        Log Info ("myinput listens to the message content as: {}", message);
    }

}

Create a new controller to trigger sending messages:

@RestController
public class ProducerController2 {

    @Autowired
    private MyMessageProducer2 producer;

    @GetMapping("/sendMessage2")
    public String sendMessage(){
        producer.send("hello spring cloud stream using customize channel!");
        return "OK";
    }

}

Finally, we need to add our customized output and input channels in the configuration file:

server:
  port: 8081

spring:
  application:
    name: stream-demo
  cloud:
    stream:
      bindings:
        output:
          destination: myRocketMQTopic
          group: test-stream
        myOutput:
          destination: myRocketMQTopic2
          group: test-stream2
        input:
          destination: myRocketMQTopic
          group: test-stream
        myInput:
          destination: myRocketMQTopic2
          group: test-stream2
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876

Thus, the channels in our two examples can work at the same time.

2021-07-16 10:16:46.495 info 18580 -- [nio-8081-exec-3] c.e.s.producer Mymessageproducer2: mymessageproducer2 producer starts sending messages!
2021-07-16 10:16:52.220 info 18580 -- [nio-8081-exec-2] c.e.s.producer Mymessageproducer2: mymessageproducer2 producer starts sending messages!
2021-07-16 10:16:52.808 info 18580 -- [nio-8081-exec-4] c.e.s.producer Mymessageproducer2: mymessageproducer2 producer starts sending messages!
2021-07-16 10:16:55.610 info 18580 -- [nio-8081-exec-5] c.e.s.producer Mymessageproducer: the producer starts sending messages!
2021-07-16 10:16:55.729 info 18580 -- [messagethread_1] c.e.s.consumer Mymessageconsumer: the message content is: Hello spring cloud stream!
2021-07-16 10:16:56.306 info 18580 -- [messagethread_1] c.e.s.consumer Mymessageconsumer2: myinput listens to the message: Hello spring cloud stream using customize channel!
2021-07-16 10:16:56.306 info 18580 -- [messagethread_1] c.e.s.consumer Mymessageconsumer2: myinput listens to the message: Hello spring cloud stream using customize channel!
2021-07-16 10:16:58.249 info 18580 -- [nio-8081-exec-6] c.e.s.producer Mymessageproducer: the producer starts sending messages!
2021-07-16 10:16:58.256 info 18580 -- [messagethread_1] c.e.s.consumer Mymessageconsumer: the message content is: Hello spring cloud stream!
2021-07-16 10:16:59.310 info 18580 -- [messagethread_1] c.e.s.consumer Mymessageconsumer2: myinput listens to the message: Hello spring cloud stream using customize channel!

4、 Message filtering

There are two methods to filter messages. The first is to set the consumption tag of a cosumer so that the specified tag will be consumed, and the messages of unspecified tags will be filtered; The other is to filter according to the customized header content.

To distinguish the above examples, we re create a channel:

public interface NewChannel {

    String INPUT = "newInput";
    String OUTPUT = "newOutput";

    @Output(NewChannel.INPUT)
    MessageChannel output();

    @Input(NewChannel.INPUT)
    SubscribableChannel input();
}

Then there are producers and consumers:

@Slf4j
@Component
@EnableBinding(NewChannel.class)
public class MyMessageProducer3 {

    @Autowired
    private NewChannel newChannel;

    public void send(Person person){
        Log Info ("mymessageproducer3 producer starts sending messages!");
        Message<Person> message1 = MessageBuilder.withPayload(person)
                //Specify the tag of messages in rocketmq
                .setHeader(RocketMQHeaders.TAGS, "animal")
                //Custom headers are used for filtering by consumer listeners
                .setHeader("MyTag", "TAG1")
                .build();

        Message<Person> message2 = MessageBuilder.withPayload(person)
                .setHeader(RocketMQHeaders.TAGS, "person")
                .setHeader("MyTag", "TAG2")
                .build();

        Message<Person> message3 = MessageBuilder.withPayload(person)
                .setHeader(RocketMQHeaders.TAGS, "person")
                .setHeader("MyTag", "TAG3")
                .build();
        newChannel.output().send(message1);
        newChannel.output().send(message2);
        newChannel.output().send(message3);
    }

}
@Slf4j
@Component
@EnableBinding(NewChannel.class)
public class MyMessageConsumer3 {

    @StreamListener(value = NewChannel.INPUT, condition = "headers['MyTag']=='TAG2'")
    public void receive(Message<Person> message){
        Log Info ("mymessageconsumer3 listens that the message content is: {}", message);
        Log Info ("payload of mymessageconsumer3 message:{}", message.getpayload());
        Log Info ("mymessageconsumer3 message header:{}", message.getheaders() Get ("mytag");
    }

}

The entity class of the message is:

@Data
@NoArgsConstructor
@AllArgsConstructor
public class Person {

    private String name;
    private Integer age;
}

Create a new controller to send messages:

@RestController
public class ProducerController3 {

    @Autowired
    private MyMessageProducer3 producer;

    @GetMapping("/sendMessage3")
    public String sendMessage(){
        Person tom = new Person("tom", 21);
        producer.send(tom);
        return "OK";
    }

}

The configuration file is modified to:

server:
  port: 8081

spring:
  application:
    name: stream-demo
  cloud:
    stream:
      bindings:
        newInput:
          destination: newRocketMQTopic
          group: test-stream3
        newOutput:
          destination: newRocketMQTopic
          group: test-stream3
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true
        bindings:
          newInput:
            consumer:
              #Accept only messages with the specified Tags
              tags: person || plant

When the application is restarted, we can access the controller interface and send three messages at the same time.

In the configuration file, we specify that consumers only consume messages with tags of person and plant, and messages with tags of animal will be displayedCONSUMED_BUT_FILTERED；

In the consumer consumption configuration code, our annotation @streamlistener specifies that the condition is consumption onlyheaders['MyTag']=='TAG2'Message of; The message of tag3 is also considered, but the corresponding consumer is not found, so the console displays:

Cannot find a @StreamListener matching for message with id: 48260ac4-772b-54b8-2ae0-8b91635729a9

5、 Functional programming

Since springcloud has been upgraded to version 3.1 or above, the command-line programming annotations @enablebinding, @output, @input, etc. in the above example have been marked as abandoned. The abandonment instructions are as follows:

@deprecated as of 3.1 in favor of functional programming model

So how can we use functional programming to transform the above example.

5.1 timing drive model

We need to re create a project, the parent project stream function, and introduce the following dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>

<!-- https://mvnrepository.com/artifact/com.alibaba.cloud/spring-cloud-starter-stream-rocketmq -->
<dependency>
    <groupId>com.alibaba.cloud</groupId>
    <artifactId>spring-cloud-starter-stream-rocketmq</artifactId>
    <version>2021.1</version>
</dependency>

<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <optional>true</optional>
</dependency>

Then create a sub project consumer and create a producer class:

@Slf4j
@Component
public class FunctionConsumer {

    @Bean
    public Consumer<Date> mySink(){
        return (message) -> {
            Log Info ("received message is: {}", message);
        };
    }

}

Add profile:

server:
  port: 8082

spring:
  application:
    name: stream-demo-consumer
  cloud:
    stream:
      bindings:
        #Format: method name - type - sequence number
        #The method name represents the name of the method that consumes the message
        #Type in indicates message receiving, and out indicates message sending
        #Serial number is used to distinguish different consumers
        mySink-in-0:
          # topic
          destination: functionRocketMQTopic
          group: test-stream4
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true
    function:
      #Name of the method by which the consumer processes the message
      definition: mySink

At this point, the consumer is finished, can start running, and will start listening for messages on the functionrocketmqtopic topic.

Then we need to create a sub project producer and create a new producer class:

@Slf4j
@Component
public class FunctionProducer {

    @Bean
    public Supplier<Date> mySource(){
        return () -> {
            Log Info ("start of sending message!");
            return new Date();
        };
    }

}

Then add the following configuration files:

server:
  port: 8081

spring:
  application:
    name: stream-demo-producer
  cloud:
    stream:
      bindings:
        #Format: method name - type - sequence number
        #The method name represents the name of the method that consumes the message
        #Type in indicates message receiving, and out indicates message sending
        #Serial number is used to distinguish different consumers
        mySource-out-0:
          # topic
          destination: functionRocketMQTopic
          group: test-stream4
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true
    function:
      #Name of the method by which the producer generated the message
      definition: mySource

In this way, our producer is also configured and can run after startup.

5.2 streambridge model

This model is mainly used to meet the scenarios where business actions trigger message sending. We just need to create a new producer class:

@Slf4j
@Service
public class StreamBridgeProducer {

    @Autowired
    private StreamBridge streamBridge;

    public void sendDateMessage(){
        Message<Date> message = MessageBuilder.withPayload(new Date()).build();
        Log Info ("start sending message!");
        streamBridge.send("mySource-out-1", message);
    }

}

Then create a new controller to call the trigger message sending:

@RestController
public class SendController {

    @Autowired
    private StreamBridgeProducer streamBridgeProducer;

    @GetMapping("/sendMessage")
    public String sendMessage(){
        streamBridgeProducer.sendDateMessage();
        return "OK";
    }

}

Finally, the configuration file is changed as follows:

server:
  port: 8081

spring:
  application:
    name: stream-demo-producer
  cloud:
    stream:
      bindings:
        mySource-out-1:
          destination: functionRocketMQTopic
          group: test-stream4
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true

In this way, we can start the producer, and the message can be sent successfully when the interface is called.

5.3 multi channel configuration

We can configure multiple producers and consumers in a project at the same time, even both producers and consumers.

Only the configuration file is given here. For the rest, please refer to the example given above.

server:
  port: 8081

spring:
  application:
    name: stream-demo-producer
  cloud:
    stream:
      bindings:
        #When there are multiple producer channels, the timing message mode destination can be the same, but the streambridge mode destination cannot be the same
        mySource-out-0:
          destination: functionRocketMQTopic0
        mySource-out-1:
          destination: functionRocketMQTopic1
        mySource-out-2:
          destination: functionRocketMQTopic2
        mySource2-out-3:
          destination: functionRocketMQTopic3
        #Consumer channel  
        mySink-in-0:
          destination: functionRocketMQTopic0
          group: test-stream4
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true
    function:
      definition: mySource;mySource2;mySink

6、 Message partition

Springcloudstream message partitioning ensures that messages with the same characteristics are consumed by the same consumer in some scenarios. It can help some message queues that do not support message partitioning to achieve message partitioning.

The producer’s profile is changed to:

server:
  port: 8084

spring:
  application:
    name: stream-demo-producer
  cloud:
    stream:
      bindings:
        mySource-out-0:
          destination: functionRocketMQTopic
          producer:
            #Expression rules for specifying partition keys
            partition-key-expresion: payload
            #Specify the number of partitions
            partition-count: 2
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true
    function:
      definition: mySource

The profile of multiple consumers is changed to:

server:
  port: 8082

spring:
  application:
    name: stream-demo-consumer2
  cloud:
    stream:
      #Specify the current number of consumer partitions
      instance-count: 2
      #Specify the index of the current consumer
      instance-index: 0
      bindings:
        mySink-in-0:
          destination: functionRocketMQTopic
          group: test-stream4
          consumer:
            #Open partition
            partitioned: true
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true
    function:
      definition: mySink
server:
  port: 8083

spring:
  application:
    name: stream-demo-consumer2
  cloud:
    stream:
      #Specify the current number of consumer partitions
      instance-count: 2
      #Specify the index of the current consumer
      instance-index: 1
      bindings:
        mySink-in-0:
          destination: functionRocketMQTopic
          group: test-stream4
          consumer:
            #Open partition
            partitioned: true
      rocketmq:
        binder:
          name-server: 127.0.0.1:9876
          enable-msg-trace: true
    function:
      definition: mySink

To be verified by experiment.

7、 Multi message queue

We can use stream to configure multiple MQS of different types. Specific examples will be added

Springcloudstream self study document – zhihu.com

8、 Other features

In fact, we have already talked about the usage of stream. The rest is mainly about some configuration properties of rocketmq binder, such as:

  • consumer
    • Tags message filtering
    • Start of broadcast mode
    • Sequential consumption
  • producer
    • Start transaction
    • Send synchronously
    • Send timeout setting and message maximum setting
    • retry count

For other features, please refer to the rocketmq documentation on GitHub. This article will not list them one by one.

RocketMQ en · alibaba/spring-cloud-alibaba Wiki · GitHub

9、 Supplement

Sometimes, sending an MQ message shows that the message was not sent successfully. A very strange problem occurs. The solution found on the Internet is to close the VIP channel.

spring:
 cloud:
   stream:
      rocketmq:
          default:
            producer:
              #Close the VIP channel, or the message will fail to be sent
              vipChannelEnabled: false

After trying, it really works.

Tags: , , , ,

Recommended Today

How to use cache (Cache) for front-end performance optimization?

1. What is cache? Caching is a technique that saves a copy of a resource and uses that copy directly on the next request, reducing latency and network traffic and significantly improving website performance. 1. Common cache Private cache: can only be used for individual users, such as browser cache; Shared cache: can be used […]