Kafka — how are partitions specified

Time:2021-12-1

When building the producerrecord, I said that in addition to topic and value, the producerrecord also has two options, including partition and key. After pulling the metadata, we start sending the message according to the partition.

Specify partition

Specifying the partition means that when building the producerrecord, the producer record whose topic is topic1 specifies the partition as 1. Assuming that there are three brokers, broker0, broker1 and broker2, this message will be sent to broker1.

Kafka -- how are partitions specified

No key

If no partition is specified, the partition calculation will be performed according to the partition. Here, we directly use the default partition, so it can be divided into the case of specifying a key and the case without a key.

If no key is specified, there are two cases:

  1. If the number of available partitions corresponding to topic1 is greater than 0, poll according to the number of available partitions corresponding to topic1.
  2. The number of available partitions corresponding to topic1 is equal to 0. At this time, polling is performed according to the number of partitions corresponding to topic1.

Polling means sending it to broker 0 this time, broker 1 next time, broker 2 next time, and then starting from broker 0.

Kafka -- how are partitions specified

Specify key

If the key is specified, Kafka will calculate the value of the key through murmur2 algorithm. For example, when the key is a, the calculated value is 32. This 32 will take the modulus of the number of brokers 3, and the remainder is 2. At this time, it will be sent to broker 2.

Note that this key is already a serialized key.

Kafka -- how are partitions specified