Spring cloudbus uses Kafka to implement message bus

Time:2021-2-4

KafkaIs a distributed publish subscribe messaging system, initially developed by LinkedIn company, and later became a part of Apache foundationScalaandJavato write. Kafka is a fast, scalable, distributed, partitioned and replicable commit log service.

Before starting this article, you need to build a Kafka environment. If you are in CentOS environment, you can see my previous article:Installation of Kafka in centos7. Other platforms can be Baidu or Google.

In the previous environment, it needed to be modified server.properties File, open 9092 port monitoring:

listeners=PLAINTEXT://your.host.name:9092

Springboot simply integrates Kafka

Because spring cloud is based on spring boot, before using spring cloudbus to integrate, use spring boot to integrate and record.

Create project

Here you create a springboot project called Kafka hello and add the following dependencies:

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>

  <dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <optional>true</optional>
  </dependency>

  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
  </dependency>

  <dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>1.1.1.RELEASE</version>
  </dependency>

  <dependency>
    <groupId>com.google.code.gson</groupId>
    <artifactId>gson</artifactId>
    <version>2.8.2</version>
  </dependency>
</dependencies>

Message entity class

@Data
public class Message {
    private Long id;//id
    Private string MSG; // message
    Private date sendtime; // send time
}

Message producer

Create a message sending method in this class, using the KafkaTemplate.send () send message,wqhIt’s topic in Kafka.

@Component
@Slf4j
public class KafkaSender {

    @Autowired
    private KafkaTemplate<String,String> kafkaTemplate;

    private Gson gson = new GsonBuilder().create();

    public void send(Long i){
        Message message = new Message();
        message.setId(i);
        message.setMsg(UUID.randomUUID().toString());
        message.setSendTime(new Date());
        log.info ("= = = = = send message" + I + "> > > > {} < = = = = = = =", gson.toJson (message));
        kafkaTemplate.send("wqh",gson.toJson(message));
    }
}

Message receiving class,

In this class, create a consumer method, and use the @ kafkalistener annotation to listen to the specified topics. For example, listen to wanqh and wqh topics.

@Component
@Slf4j
public class KafkaConsumer {

    @KafkaListener(topics = {"wanqh","wqh"})
    public void consumer(ConsumerRecord<?,?> consumerRecord){
        //Determine whether it is null
        Optional<?> kafkaMessage = Optional.ofNullable(consumerRecord.value());
        log.info(">>>>>>>>>> record =" + kafkaMessage);
        if(kafkaMessage.isPresent()){
            //Get the value in the optional instance
            Object message = kafkaMessage.get();
            log.info ("> > > > receive message, message =" + message));
        }
    }
}

Modify startup class

@SpringBootApplication
public class KafkaApplication {

    @Autowired
    private KafkaSender kafkaSender;

    @PostConstruct
    public void init(){
      for (int i = 0; i < 10; i++) {
        //Call the message sending method in the message sending class
        kafkaSender.send((long) i);
      }
    }
    public static void main(String[] args) {
       SpringApplication.run(KafkaApplication.class, args);
    }
}

configuration file

spring.application.name=kafka-hello
server.port=8080
#============== kafka ===================
#Specify the Kafka proxy address, which can be multiple
spring.kafka.bootstrap-servers=192.168.18.136:9092

#=============== provider  =======================
spring.kafka.producer.retries=0
#Number of messages sent in bulk each time
spring.kafka.producer.batch-size=16384
spring.kafka.producer.buffer-memory=33554432

#Specify the encoding and decoding methods of message key and message body
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.apache.kafka.common.serialization.StringSerializer

#=============== consumer  =======================
#Specify the default consumer group ID
spring.kafka.consumer.group-id=test-consumer-group

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=true
spring.kafka.consumer.auto-commit-interval=100

#Specify the encoding and decoding methods of message key and message body
spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer
spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer

test

Start the project directly:

Spring cloudbus uses Kafka to implement message bus

Spring cloudbus integrates Kafka

In the previous section, we implemented the message bus by integrating spring cloudbus with rabbitmq, and tested the dynamic refresh configuration file. Rabbitmq is introduced byspring-cloud-starter-bus-amqpModule to implement the message bus. If we use Kafka to implement message bus, we can directly add thespring-cloud-starter-bus-amqpreplace withspring-cloud-starter-bus-kafka

Here I will copy the previous config client and change it to config client Kafka. Portal:Spingcloud bus integrates rabbitmq

  • Added dependency:
<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-config</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.retry</groupId>
            <artifactId>spring-retry</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-eureka</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-bus-kafka</artifactId>
        </dependency>
    </dependencies>
  • Add configuration information of Kafka
#Kafka server list, default localhost
spring.cloud.stream.kafka.binder.brokers=192.168.18.136:9092
#The default port of the Kafka server. When there is no port information configured in the brokers attribute, the default port is 9092
spring.cloud.stream.kafka.binder.defaultBrokerPort=9092
#The list of zookeeper nodes connected to Kafka server. The default is localhost
spring.cloud.stream.kafka.binder.zkNodes=192.168.18.136:2181
#The default port of zookeeper node. When the port information is not configured in the zknodes attribute, the default port is 2181
spring.cloud.stream.kafka.binder.defaultZkPort=2181

The test method is the same as the previous one, which is not introduced.


reference resources:

Project address:

Original [address: