Using Kafka in spring boot

Time:2021-7-22

Kafka is a high throughput distributed publish subscribe message system, which can replace the traditional message queue for decoupling data processing, caching unprocessed messages, etc. at the same time, it has higher throughput, supports partition, multiple copies, redundancy, so it is widely used in large-scale information data processing applications. Kafka supports Java and other language clients, and can be combined with Hadoop, storm, spark and other big data tools.

Environment installation

Build high throughput Kafka distributed publish subscribe message cluster

test case

GitHub code

I’ve put the code into GitHub to importspring-boot-kafkaproject

github https://github.com/souyunku/spring-boot-examples/tree/master/spring-boot-kafka

Add dependency

Add to projectkafka-clientsrely on

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>0.10.2.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

Enable Kafka

@Configuration
@EnableKafka
public class KafkaConfiguration {

}

Message producer

@Component
public class MsgProducer {

    private static final Logger log = LoggerFactory.getLogger(MsgProducer.class);

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    public void sendMessage(String topicName, String jsonData) {
        Log. Info ("push data to Kafka: [{}]], jsondata];
        try {
            kafkaTemplate.send(topicName, jsonData);
        } catch (Exception e) {
            Log. Error ("error in sending data)! {}}, topicName, jsonData);
            Log. Error ("error in sending data = = = = >", e);
        }

        //The listener of message sending, which is used to call back the returned information
        kafkaTemplate.setProducerListener(new ProducerListener<String, String>() {
            @Override
            public void onSuccess(String topic, Integer partition, String key, String value, RecordMetadata recordMetadata) {
            }

            @Override
            public void onError(String topic, Integer partition, String key, String value, Exception exception) {
            }

            @Override
            public boolean isInterestedInSuccess() {
                Log.info ("data sending completed");
                return false;
            }
        });
    }

}

News consumer

@Component
public class MsgConsumer {

    @KafkaListener(topics = {"topic-1","topic-2"})
    public void processMessage(String content) {

        System. Out. Println ("message consumed" + content) ";
    }
    
}

Parameter configuration

application.properties

#kafka
#Specify the Kafka proxy address, which can be multiple
spring.kafka.bootstrap-servers=YZ-PTEST-APP-HADOOP-02:9092,YZ-PTEST-APP-HADOOP-04:9092
#Specifies the number of threads in the listener container to increase concurrency
spring.kafka.listener.concurrency=3
#Number of messages sent in bulk each time
spring.kafka.producer.batch-size=1000
#Specify the default consumer group ID
spring.kafka.consumer.group-id=myGroup
#Specify the default topic ID
spring.kafka.template.default-topic=topic-1

Start service

@SpringBootApplication
@ComponentScan(value = {"io.ymq.kafka"})
public class Startup {

    public static void main(String[] args) {
        SpringApplication.run(Startup.class, args);
    }
}

unit testing

import io.ymq.kafka.MsgProducer;
import io.ymq.kafka.run.Startup;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;
/**
 *Description: testing Kafka
 *
 * @author yanpenglei
 * @create 2017-10-16 18:45
 **/
@RunWith(SpringRunner.class)
@SpringBootTest(classes = Startup.class)
public class BaseTest {

    @Autowired
    private MsgProducer msgProducer;

    @Test
    public void test() throws Exception {

        msgProducer.sendMessage("topic-1", "topic--------1");
        msgProducer.sendMessage("topic-2", "topic--------2");
    }
}

Message producer, response

2017-10-17 15:54:44.814 info 2960 -- [main] io.ymq.kafka.msgproducer: push data to Kafka: [Topic -------- 1]
2017-10-17 15:54:44.860 info 2960 -- [main] io.ymq.kafka.msgproducer: push data to Kafka: [Topic -------- 2]
2017-10-17 15:54:44.878 info 2960 -- [AD | producer-1] io.ymq.kafka.msgproducer: data transmission completed
2017-10-17 15:54:44.878 info 2960 -- [AD | producer-1] io.ymq.kafka.msgproducer: data transmission completed

Message consumer, response

Message consumed topic -------- 1
Message consumed topic -------- 2

I’ve put the code into GitHub to importspring-boot-kafkaproject

github https://github.com/souyunku/spring-boot-examples/tree/master/spring-boot-kafka

There are some pits

[2017-10-16 19:20:08.340] - 14884 serious [main] - org.springframework.kafka.support.loggingproducerlistener: exception throw when sending a message with key '='null' and payload '='topic -------- 2' to topic topic-2:

After debugging, it is found that Kafka connection uses the same host name, so modify hosts

C:\Windows\System32\drivers\etc\hosts

10.32.32.149 YZ-PTEST-APP-HADOOP-02
10.32.32.154 YZ-PTEST-APP-HADOOP-04

Contact

  • Author: Peng Lei
  • source:http://www.ymq.io
  • Email:[email protected]
  • Copyright belongs to the author, please indicate the source of reprint
  • Wechat: pay attention to official account, Search Cloud database, focus on developing technology research and knowledge sharing.

Using Kafka in spring boot