DWQA QuestionsCategory: ProgramKafka multiple consumers have only one consumption
mataye asked 9 months ago

Using goal“ github.com/Shopify/sarama “This library
I now have two consumers, two topics. The Kafka partition is set to 50, but in the actual operation process, only one consumer is consuming. Close the consumers that can currently consume, and the other can consume, but it is not OK to run at the same time
Looking at the documents on the Internet, it is said that there are not enough partitions, but now I have set up 50 partitions with only two consumers
Code 1

producerConfig := sarama.NewConfig()
    producerConfig.Producer.Partitioner = sarama.NewHashPartitioner
    producerConfig.Producer.Return.Successes = true
    producerConfig.Producer.Timeout = 5 * time.Second
    producer, err =  sarama.NewSyncProducer ([broker instance address], producterconfig)


    kafka_msg := &sarama.ProducerMessage{
        Topic:topic,
        Key:sarama.StringEncoder(key),
        Value: sarama.StringEncoder(value),
    }
    partition, offset, err := producer.SendMessage(kafka_msg)

In this way, the partition ID returned after successful delivery will always be 0. I wonder if it is because my partition setting value has not taken effect,
But because it’s the first time to use Kafka, I’m not sure,
So use the next way
Code 2

producerConfig := sarama.NewConfig()
    // producerConfig.Producer.Partitioner = sarama.NewHashPartitioner
    producerConfig.Producer.Return.Successes = true
    producerConfig.Producer.Timeout = 5 * time.Second
    producer, err =  sarama.NewSyncProducer ([broker instance address], producterconfig)


    kafka_msg := &sarama.ProducerMessage{
        Topic:topic,
        Key:sarama.StringEncoder(key),
        Value: sarama.StringEncoder(value),
        Partition: 7 ,
    }
    partition, offset, err := producer.SendMessage(kafka_msg)

After this setting, the delivery is successful, and the partition return is also 0.
What I want to confirm is that before the partition setting takes effect, if I use code 2 to write it, is the final partition delivered to 7?
Now I suspect it’s because the partition configuration doesn’t work
Please answer

4 Answers
nextGood answered 9 months ago

There are differences between unicast and broadcast in Kafka. For a message, consumers in the same consumption group have a competitive relationship, and only one consumer can consume, which is unicast. Similarly, for a message, consumers in different consumption groups can consume at the same time, which is multicast. If you want two consumers to consume messages at the same time, you can put the two consumers in different consumption groups. This needs to be set by the groupid attribute of the consumer side.

Simeone_xu answered 9 months ago

A partition can only be consumed by one consumer in the consumer group. I guess that’s because every message sent is sent to the same partition

mataye replied 9 months ago

Yes, I know this concept. Code return is also reality. It has been pushed to the same partition 0, so I use the following code, Kafka_ msg := & sarama.ProducerMessage {
Topic:topic,
Key:sarama.StringEncoder(key),
Value: sarama.StringEncoder(value),
Partition: 7 ,
}

Add partition: 7 to it. Does this mean that I specify to push to partition 7? I want to confirm this

Simeone_xu replied 9 months ago

Yes, I haven’t used the Kafka client of golang. You can see the relevant documents

Rain before the wind answered 9 months ago

Your two consumers should be in the same group. In Kafka, a message can be consumed by multiple groups, but only by one consumer in a group.

HammerMax answered 9 months ago

In Sarama package, when initializing producer, you need to specify the partitioner
Paste a source code:
图片描述
Unlike the default pationer in Java producer, it selects partitions in order (specified partition, key hash, roundrobin).
So if you want to specify a partition.
take

producerConfig.Producer.Partitioner = sarama.NewHashPartitioner

Change to

producerConfig.Producer.Partitioner = sarama.NewManualPartitioner