Spring cloud stream binder Kafka property configuration

Time:2021-8-13

order

This article briefly introduces some attribute configurations of spring cloud stream binder Kafka.

maven

       <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-stream-binder-kafka</artifactId>
            <version>1.0.3.RELEASE</version>
        </dependency>

Stream properties

spring-cloud-stream-1.0.3.RELEASE-sources.jar!/org/springframework/cloud/stream/config/ChannelBindingServiceProperties.java

spring:
  cloud:
     stream:
       Instanceindex: 0 ## supports environment variable instance_ INDEX
                        ## The instance index of the application: a number from 0 to instanceCount-1. Used for partitioning and with Kafka
       instanceCount: 1 ## The number of deployed instances of an application. Must be set for partitioning and if using Kafka.
                        ## used to partition data across different consumers.

Logically, topic can be regarded as a queue. Each consumer must specify its topic, which can be simply understood as indicating which queue to put the message in. In order to expand Kafka’s throughput horizontally, the topic is physically divided into one or more partitions. Each partition physically corresponds to a folder in which all messages and index files of the partition are stored. The naming rule of partition is topic name + ordered sequence number. The sequence number of the first partition starts from 0, and the maximum sequence number is the number of partitions minus 1.

Messages in the same partition can only be consumed by one consumer in the same group.

When the number of consumers is more than the number of partitions, redundant consumers are idle.

When the number of consumers is less than or equal to the number of partitions, multiple partitions will correspond to one consumer, and individual consumers will consume more than others.

Instancecount is mainly used by consumers. It is generally less than or equal to the number of partitions of topic. It is mainly used for consumers’ consumption partitions.

Bindings property

spring-cloud-stream-1.0.3.RELEASE-sources.jar!/org/springframework/cloud/stream/config/BindingProperties.java

spring:
  cloud:
     stream:
        bindings:
          output:
            destination: event-demo
            content-type: text/plain
            #Group: Test ##consumer attribute
            #producer:
            #consumer:

producer

spring-cloud-stream-1.0.3.RELEASE-sources.jar!/org/springframework/cloud/stream/binder/ProducerProperties.java

spring:
  cloud:
     stream:
        bindings:
          output:
            destination: event-demo
            content-type: text/plain
            producer:
              partitionCount: 1
              headerMode
              partitionKeyExtractorClass: org.springframework.cloud.stream.partitioning.CustomPartitionKeyExtractorClass
              partitionSelectorClass: org.springframework.cloud.stream.partitioning.CustomPartitionSelectorClass
              headerMode: raw
  • Kafka producer extended properties
    spring-cloud-stream-binder-kafka-1.0.3.RELEASE-sources.jar!/org/springframework/cloud/stream/binder/kafka/KafkaProducerProperties.java
spring:
  cloud:
     stream:
        bindings:
          output:
            destination: event-demo
            content-type: text/plain
            producer:
              bufferSize: 16384
              maxRequestSize: 1048576
              sync: true
              batchTimeout: 0

consumer

spring-cloud-stream-1.0.3.RELEASE-sources.jar!/org/springframework/cloud/stream/binder/ConsumerProperties.java

spring:
  cloud:
     stream:
        bindings:
          input:
            destination: event-demo
            content-type: text/plain
            consumer:
              concurrency: 1 ## The concurrency of the inbound consumer.
              partitioned: false ## Whether the consumer receives data from a partitioned producer.Default: false.
              headerMode: raw
  • Kafka consumer extended properties
    spring-cloud-stream-binder-kafka-1.0.3.RELEASE-sources.jar!/org/springframework/cloud/stream/binder/kafka/KafkaConsumerProperties.java
spring:
  cloud:
     stream:
        bindings:
          input:
            destination: event-demo
            content-type: text/plain
            consumer:
              autoCommitOffset: false
              resetOffsets: true
              startOffset: earliest
              enableDlq: false
              recoveryInterval: 5000

Native API

ConsumerConfig consumerConfig =  new kafka.consumer.ConsumerConfig(props);
        ConsumerConnector consumerConnector = kafka.consumer.Consumer.createJavaConsumerConnector(consumerConfig);
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(topic, consumerCount);
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumerConnector
                .createMessageStreams(topicCountMap);

Here, the header topiccountmap tells Kafka how many threads we will use to consume the topic in the consumer. The key of topiccountmap is topic name, and the value is the number of threads for this topic.

Summary

On the whole, spring cloud stream abstracts a part of itself, but one hard injury is that spring.cloud.stream.instanceindex is not friendly, which makes the service instances stateful. It is more troublesome to deploy based on docker, so it is better to directly use the native API. If there are not many partitions or each consumer has strong performance, at least two can be deployed, and the configuration is acceptable.

doc