Kafka series — 4.2, consumer partition strategy

kafkaAllow configurationpartition.assignment.strategyTo change the partition strategy of consumer groups.
kafkaThe following partition policies are provided

• RangeAssignor
• RoundRobinAssignor
• StickyAssignor

The default isRangeAssignor

At the same time,kafkaIt also allows us to customize the partition policy by simply inheritingAbstractPartitionAssignorAbstract class is enough.

1、RangeAssignor

There is a consumption section on leavec1, c2, all subscriptiont0, t1, eachtopicThere are two zones under itt0p0, t0p1, t1p0, t1p1
The partition is as follows:

c1: t0p0, t1p0
c2: t0p1, t1p1

If eachtopicIf there are three partitions, the distribution will be uneven

c1: t0p0, t0p1,   t1p0, t1p1
c2: t0p2,         t1p2

The algorithm follows eachtopicUnder the number of partitions, the average score.

2、RoundRobinAssignor

The consumption groups are sorted by dictionary and then allocated by polling.
There is a consumption section on leavec1, c2。 All subscriptiont0, t1。 eachtopicThere are three zones below.
The division is as follows

c1: t0p0, t0p2, t1p1
c2: t0p1, t1p0, t1p2

There is a consumption section on leavec1, c2, c3。 among
c1subscribet0,
c2subscribet1,
c3subscribet0, t1, t2,
t0There is 1 partitiont0p0,
t1There are 2 partitionst1p0, t1p1,
t2There are three partitionst2p0, t2p1, t2p3
The division is as follows

c1: t0
c2: t1p0
c3: t1p1, t2p0, t2p1, t2p3

3、StickyAssignor

The allocation strategy follows the following two principles

1. The partition should be distributed as evenly as possible
2. The partition should be assigned as much as possible to the last one

The first condition has a higher priority than the second

There is a consumption section on leavec1, c2, c3。 All subscriptiont0, t1Each partition has 3 partitions
The division is as follows

c1: t0p0, t1p0
c2: t0p1, t1p1
c3: t0p2, t1p2

You can see thatRoundRobinAssignorThe algorithm is similar.

Let’s look at what happens when the partitions of different consumer groups are inconsistent
There is a consumption section on leavec1, c2, c3。 among
c1subscribet0,
c2subscribet1,
c3subscribet0, t1, t2,
t0There is 1 partitiont0p0,
t1There are 2 partitionst1p0, t1p1,
t2There are three partitionst2p0, t2p1, t2p3
The division is as follows

c1: t0p0
c2: t1p0, t1p1
c3: t2p0, t2p1, t2p3

Stick it hereRoundRobinAssignorThe partition of the algorithm is compared

c1: t0p0
c2: t1p0
c3: t1p1, t2p0, t2p1, t2p3

You can seeStickyAssignorThe allocation ratio of the algorithmRoundRobinAssignorBetter. namelyThe partition should be distributed as evenly as possible

Suppose C1 quits the subscription. What happens to partition allocation at this time?

c2: t0p0, t1p0, t1p1
c3: t2p0, t2p1, t2p3

4. Summary

RocketMQproposalThe consumer group subscribes to only one topicIn my actual development process, it is basically the same. If you subscribe to more than onetopic, the consumer group will not work properly. aboutkafkaFor example, a consumer group can subscribe to more than onetopicIt’s really flexible.