An example of using rocketmq gracefully in spring boot

Time:2021-1-27

preface

MQ, a cross process communication mechanism, is used to deliver messages upstream and downstream. In traditional Internet architecture, MQ is usually used to decouple upstream and downstream.

For example: when system a communicates with system B, such as system a publishes a system announcement, system B can subscribe to the channel for system announcement synchronization. In the whole process, system a does not care whether system B will synchronize, but the system subscribing to the channel will handle it by itself.

What is rocketmq? #

Official note:

As more and more queues and virtual topics are used, ActiveMQ IO module encounters bottlenecks. We try our best to solve this problem by throttling, circuit breaker or degradation, but the effect is not good. So we started to focus on Kafka, the popular messaging solution. Unfortunately, Kafka can’t meet our requirements, especially in terms of low latency and high reliability.

It is clear that rkoketmq is a low latency, high reliability, scalable and easy-to-use message middleware.

It has the following characteristics:

  • Support publish / subscribe (Pub / sub) and peer-to-peer (P2P) message models
  • It can ensure strict message order, reliable FIFO and strict order delivery in a queue
  • Provide rich message pull mode, support pull and push mode
  • The single queue has the ability to stack millions of messages and hundreds of millions of messages
  • Support multiple message protocols, such as JMS, mqtt, etc
  • Distributed and highly available deployment architecture, which can satisfy at least one message passing semantics

Rocketmq environment installation#

Download address: https://rocketmq.apache.org/dowloading/releases/

Download binary or source code from the official to use. Source code compilation needs maven 3.2X, jdk8

Package in the root directory:


mvn -Prelease-all -DskipTests clean packager -U

There will be a folder version, zip and tar in the distribution / target / Apache rocketmq folder.

Use rocketmq-4.6.0. Zip:

  • Start name service mqnamesrv.cmd
  • Start data center mqbroker.cmd -n localhost:9876

Using rocketmq in springboot environment#

Getting started with springboot: https://www.jb51.net/article/177449.htm

Common use of springboot start:https :// www.jb51.net/article/177451.htm

The current environment version is:

  • SpringBoot 2.0.6.RELEASE
  • SpringCloud Finchley.RELEASE
  • SpringCldod Alibaba 0.2.1.RELEASE
  • RocketMQ 4.3.0

Import in project engineering:


<!-- MQ Begin -->
<dependency>
 <groupId>org.apache.rocketmq</groupId>
 <artifactId>rocketmq-client</artifactId>
 <version>${rocketmq.version}</version>
</dependency>
<!-- MQ End -->

As we already have a project here, we are not in the process of creating it. Mainly to see how to use rocketmq.

Create rocketmqproperties configuration property class. The contents of the class are as follows:

@ConfigurationProperties(prefix = "rocketmq")
public class RocketMQProperties {
 private boolean isEnable = false;
 private String namesrvAddr = "localhost:9876";
 private String groupName = "default";
 private int producerMaxMessageSize = 1024;
 private int producerSendMsgTimeout = 2000;
 private int producerRetryTimesWhenSendFailed = 2;
 private int consumerConsumeThreadMin = 5;
 private int consumerConsumeThreadMax = 30;
 private int consumerConsumeMessageBatchMaxSize = 1;
 //Omit get set
}

Now the producers and consumers in all our subsystems correspond to:

Is isenable enable MQ

Namesrvaddr cluster address

Groupname group name

It is set to be unified to facilitate system docking. If there are other requirements to be extended, we have given the default value in the class. You can also obtain the configuration in the configuration file or configuration center. The configuration is as follows:

#Sending the same type of message is set to the same group to ensure uniqueness. It does not need to be set by default. Rocketmq will use it [email protected] (PID stands for the name of the JVM) as the unique identifier
rocketmq.groupName=please_rename_unique_group_name
#Enable auto configuration
rocketmq.isEnable=true
#The nameserver address of MQ
rocketmq.namesrvAddr=127.0.0.1:9876
#The maximum message length is 1024 * 4 (4m) by default
rocketmq.producer.maxMessageSize=4096
#The timeout of sending message is 3000 by default
rocketmq.producer.sendMsgTimeout=3000
#Failed to send message, the number of retries is 2 by default
rocketmq.producer.retryTimesWhenSendFailed=2
#Number of consumer threads
rocketmq.consumer.consumeThreadMin=5
rocketmq.consumer.consumeThreadMax=32
#Set the number of consumption messages at a time. The default is 1
rocketmq.consumer.consumeMessageBatchMaxSize=1

Create consumer interface RocketConsumer.java The core steps of this interface are as follows

/**
 *Consumer interface
 * 
 * @author SimpleWu
 *
 */
public interface RocketConsumer {

/**
 *Initial consumer
 */
 public abstract void init();

 /**
 *Register for monitoring
 * 
 * @param messageListener
 */
 public void registerMessageListener(MessageListener messageListener);

}

Creating abstract consumer ab stractRocketConsumer.java :

/**
 *Basic information of consumers
 * 
 * @author SimpelWu
 */
public abstract class AbstractRocketConsumer implements RocketConsumer {

 protected String topics;
 protected String tags;
 protected MessageListener messageListener;
 protected String consumerTitel;
 protected MQPushConsumer mqPushConsumer;

 /**
 *Necessary information
 * 
 * @param topics
 * @param tags
 * @param consumerTitel
 */
 public void necessary(String topics, String tags, String consumerTitel) {
 this.topics = topics;
 this.tags = tags;
 this.consumerTitel = consumerTitel;
 }

 public abstract void init();

 @Override
 public void registerMessageListener(MessageListener messageListener) {
 this.messageListener = messageListener;
 }
 
}

In the class, we must specify the topics, tags and message listening logic

public abstract void init();This method is used to initialize the consumer and is implemented by subclasses.

Next, we write the auto configuration class RocketMQConfiguation.java , which initializes a default producer connection and loads all consumers.

@EnableConfigurationProperties({ RocketMQProperties.class }) using the profile

@Configuration is marked as a configuration class

@Conditionalonproperty (prefix = – rocketmq “, value = – isenable”, having value = – true “) can only be specified in the configuration rocketmq.isEnable =It doesn’t take effect until it’s true

The core contents are as follows:

/**
 *MQ configuration
 * 
 * @author SimpleWu
 */
@Configuration
@EnableConfigurationProperties({ RocketMQProperties.class })
@ConditionalOnProperty(prefix = "rocketmq", value = "isEnable", havingValue = "true")
public class RocketMQConfiguation {

 private RocketMQProperties properties;

 private ApplicationContext applicationContext;

 private Logger log = LoggerFactory.getLogger(RocketMQConfiguation.class);

 public RocketMQConfiguation(RocketMQProperties properties, ApplicationContext applicationContext) {
 this.properties = properties;
 this.applicationContext = applicationContext;
 }

 /**
 *Inject a default consumer
 * @return
 * @throws MQClientException
 */
 @Bean
 public DefaultMQProducer getRocketMQProducer() throws MQClientException {
 if (StringUtils.isEmpty(properties.getGroupName())) {
  throw new MQClientException(-1, "groupName is blank");
 }

 if (StringUtils.isEmpty(properties.getNamesrvAddr())) {
  throw new MQClientException(-1, "nameServerAddr is blank");
 }
 DefaultMQProducer producer;
 producer = new DefaultMQProducer(properties.getGroupName());

 producer.setNamesrvAddr(properties.getNamesrvAddr());
 // producer.setCreateTopicKey("AUTO_CREATE_TOPIC_KEY");

 //If you need different producers in the same JVM to send messages to different MQ clusters, you need to set different InstanceName
 // producer.setInstanceName(instanceName);
 producer.setMaxMessageSize(properties.getProducerMaxMessageSize());
 producer.setSendMsgTimeout(properties.getProducerSendMsgTimeout());
 //If the message fails to be sent, set the number of retries, which is 2 by default
 producer.setRetryTimesWhenSendFailed(properties.getProducerRetryTimesWhenSendFailed());

 try {
  producer.start();
  log.info("producer is start ! groupName:{},namesrvAddr:{}", properties.getGroupName(),
   properties.getNamesrvAddr());
 } catch (MQClientException e) {
  log.error(String.format("producer is error {}", e.getMessage(), e));
  throw e;
 }
 return producer;

 }

 /**
 *Load all consumers when springboot starts
 */
 @PostConstruct
 public void initConsumer() {
 Map<String, AbstractRocketConsumer> consumers = applicationContext.getBeansOfType(AbstractRocketConsumer.class);
 if (consumers == null || consumers.size() == 0) {
  log.info("init rocket consumer 0");
 }
 Iterator<String> beans = consumers.keySet().iterator();
 while (beans.hasNext()) {
  String beanName = (String) beans.next();
  AbstractRocketConsumer consumer = consumers.get(beanName);
  consumer.init();
  createConsumer(consumer);
  log.info("init success consumer title {} , toips {} , tags {}", consumer.consumerTitel, consumer.tags,
   consumer.topics);
 }
 }

 /**
 *Creating consumers through consumer confidence
 * 
 * @param consumerPojo
 */
 public void createConsumer(AbstractRocketConsumer arc) {
 DefaultMQPushConsumer consumer = new DefaultMQPushConsumer(this.properties.getGroupName());
 consumer.setNamesrvAddr(this.properties.getNamesrvAddr());
 consumer.setConsumeThreadMin(this.properties.getConsumerConsumeThreadMin());
 consumer.setConsumeThreadMax(this.properties.getConsumerConsumeThreadMax());
 consumer.registerMessageListener(arc.messageListenerConcurrently);
 /**
  *Set whether consumption starts from the head of the queue or the tail of the queue when the consumer starts for the first time. If it is not started for the first time, consumption will continue according to the last consumption position
  */
 // consumer.setConsumeFromWhere(ConsumeFromWhere.CONSUME_FROM_FIRST_OFFSET);
 /**
  *Set consumption model, cluster or broadcast, the default is cluster
  */
 // consumer.setMessageModel(MessageModel.CLUSTERING);

 /**
  *Set the number of consumption messages at a time. The default is 1
  */
 consumer.setConsumeMessageBatchMaxSize(this.properties.getConsumerConsumeMessageBatchMaxSize());
 try {
  consumer.subscribe(arc.topics, arc.tags);
  consumer.start();
  arc.mqPushConsumer=consumer;
 } catch (MQClientException e) {
  log.error("info consumer title {}", arc.consumerTitel, e);
 }

 }

}

Then create the directory and file meta-inf in Src / main / Resources folder/ spring.factories Add an automatic configuration class to start the startup configuration. We just need to import the dependency


org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.xcloud.config.rocketmq.RocketMQConfiguation

Next, import the dependency into the service, and then get all the necessary information through our abstract class to create the consumer. This step will be performed after all the consumers are initialized, and only the consumers who are spring beans will be managed.

Now let’s see how to create a consumer. The steps to create a consumer are very simple. We only need to inherit abstractrocketconsumer and then add spring’s @ component to complete the creation of a consumer. We can customize the theme and label of consumption in the class.

The project can be started according to the demand when the consumer fails to create.

Create a default consumer DefaultConsumerMQ.java

@Component
public class DefaultConsumerMQ extends AbstractRocketConsumer {
 /**
 *Initial consumer
 */
 @Override
 public void init() {
 //Set theme, label and consumer title
 super.necessary ("topic test", "*", "this is the title");
 //Consumer specific execution logic
 registerMessageListener(new MessageListenerConcurrently() {
  @Override
  public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
  msgs.forEach(msg -> {
   System.out.printf("consumer message boyd %s %n", new String(msg.getBody()));
  });
  //Mark that the message has been successfully consumed
  return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
  }
 });
 }
}

super.necessary (“topictest”, “*”, and “this is the title”); it must be set to listen to all tags under the topic of topictest on behalf of the consumer. The title field is defined by myself, so it is meaningless for this configuration.

We can inject spring bean here for any logical processing.

Create a message sending class to test

@Override
public String qmtest(@PathVariable("name")String name) throws MQClientException, RemotingException, MQBrokerException, InterruptedException, UnsupportedEncodingException {
 Message msg = new Message("TopicTest", "tags1", name.getBytes(RemotingHelper.DEFAULT_CHARSET));
 //Send a message to a broker
 SendResult sendResult = defaultMQProducer.send(msg);
 //Return whether the message was delivered successfully through sendresult
 System.out.printf("%s%n", sendResult);
 return null;
}

Let’s pass the HTTP request test

http://localhost:10001/demo/base/mq/hello consumer message boyd hello 
http://localhost : 10001 / demo / base / MQ / hehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehe

Well, the simple start design is completed, and there are a series of functions behind it: sequential message production, sequential consumption message, asynchronous message production, etc. officials can refer to the official to handle it by themselves.

  • ActiveMQ has not been verified by large-scale throughput scenarios, and the community is not very active.
  • Rabbitmq cluster dynamic expansion trouble, and with the current programming language is not difficult to customize.
  • Kafka supports the main MQ functions, which can not meet the requirements of the program, so it is not used, and it is not difficult to customize with the current program language.
  • Rocketmq has been baptized by women all over the world and is already very powerful; MQ has relatively complete functions and is distributed with good scalability; it supports complex MQ business scenarios. (complex business can be preferred)

summary

The above is the whole content of this article, I hope the content of this article has a certain reference learning value for your study or work, thank you for your support to developer.