This article briefly introduces some attribute configurations of spring cloud stream binder Kafka.
<dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-stream-binder-kafka</artifactId> <version>1.0.3.RELEASE</version> </dependency>
spring: cloud: stream: Instanceindex: 0 ## supports environment variable instance_ INDEX ## The instance index of the application: a number from 0 to instanceCount-1. Used for partitioning and with Kafka instanceCount: 1 ## The number of deployed instances of an application. Must be set for partitioning and if using Kafka. ## used to partition data across different consumers.
Logically, topic can be regarded as a queue. Each consumer must specify its topic, which can be simply understood as indicating which queue to put the message in. In order to expand Kafka’s throughput horizontally, the topic is physically divided into one or more partitions. Each partition physically corresponds to a folder in which all messages and index files of the partition are stored. The naming rule of partition is topic name + ordered sequence number. The sequence number of the first partition starts from 0, and the maximum sequence number is the number of partitions minus 1.
Messages in the same partition can only be consumed by one consumer in the same group.
When the number of consumers is more than the number of partitions, redundant consumers are idle.
When the number of consumers is less than or equal to the number of partitions, multiple partitions will correspond to one consumer, and individual consumers will consume more than others.
Instancecount is mainly used by consumers. It is generally less than or equal to the number of partitions of topic. It is mainly used for consumers’ consumption partitions.
spring: cloud: stream: bindings: output: destination: event-demo content-type: text/plain #Group: Test ##consumer attribute #producer: #consumer:
spring: cloud: stream: bindings: output: destination: event-demo content-type: text/plain producer: partitionCount: 1 headerMode partitionKeyExtractorClass: org.springframework.cloud.stream.partitioning.CustomPartitionKeyExtractorClass partitionSelectorClass: org.springframework.cloud.stream.partitioning.CustomPartitionSelectorClass headerMode: raw
- Kafka producer extended properties
spring: cloud: stream: bindings: output: destination: event-demo content-type: text/plain producer: bufferSize: 16384 maxRequestSize: 1048576 sync: true batchTimeout: 0
spring: cloud: stream: bindings: input: destination: event-demo content-type: text/plain consumer: concurrency: 1 ## The concurrency of the inbound consumer. partitioned: false ## Whether the consumer receives data from a partitioned producer.Default: false. headerMode: raw
- Kafka consumer extended properties
spring: cloud: stream: bindings: input: destination: event-demo content-type: text/plain consumer: autoCommitOffset: false resetOffsets: true startOffset: earliest enableDlq: false recoveryInterval: 5000
ConsumerConfig consumerConfig = new kafka.consumer.ConsumerConfig(props); ConsumerConnector consumerConnector = kafka.consumer.Consumer.createJavaConsumerConnector(consumerConfig); Map<String, Integer> topicCountMap = new HashMap<String, Integer>(); topicCountMap.put(topic, consumerCount); Map<String, List<KafkaStream<byte, byte>>> consumerMap = consumerConnector .createMessageStreams(topicCountMap);
Here, the header topiccountmap tells Kafka how many threads we will use to consume the topic in the consumer. The key of topiccountmap is topic name, and the value is the number of threads for this topic.
On the whole, spring cloud stream abstracts a part of itself, but one hard injury is that spring.cloud.stream.instanceindex is not friendly, which makes the service instances stateful. It is more troublesome to deploy based on docker, so it is better to directly use the native API. If there are not many partitions or each consumer has strong performance, at least two can be deployed, and the configuration is acceptable.