Principle analysis of Kafka Technology

Time:2021-4-8

Preface: a lot of students leave a message saying that they can’t understand the blog Kafka application example I wrote before. They don’t understand why the code is written so. I found the PPT about Kafka technology analysis I wrote before. Now I will sort out some key points and related principles for your reference.

1. Introduction of message queue

Principle analysis of Kafka Technology
Kafka definition: Kafka is a distributed message queue based on publish / subscribe mode, which is mainly used in the field of big data real-time processing.
Definition of message queue
Principle analysis of Kafka Technology
The difference between the two models
Principle analysis of Kafka Technology
Principle analysis of Kafka Technology
How does the publish subscribe model achieve load balancing?
Principle analysis of Kafka Technology

2. Model comparison of popular queues

RabbitMQ
Principle analysis of Kafka Technology
The production side sends a message to the queue through routing, and only one consumer can consume it.
Principle analysis of Kafka Technology
When rabbitmq needs to support multiple subscriptions, the message sent by the publisher is written to multiple queues simultaneously through routing, and the message is consumed by different subscription groups.
Kafka
Principle analysis of Kafka Technology

3. Kafka architecture

Principle analysis of Kafka Technology
Principle analysis of Kafka Technology
Note 1: messages in a partition of a topic can only be consumed by one consumer in a consumer group
Principle analysis of Kafka Technology
Note 2: however, different consumer groups can appear: messages in a partition can be consumed by multiple consumers in different consumer groups at the same time
Principle analysis of Kafka Technology
Note 3: when the number of consumers in a consumer group is greater than the number of partitions, it will cause a waste of consumer resources
Principle analysis of Kafka Technology
Note 4: when the number of consumers is less than the number of partitions: a consumer can consume messages in two consumer groups at the same time

4. Kafka production process analysis

1. Write mode — producer uses push mode to publish messages to broker
Principle analysis of Kafka Technology
Principle analysis of Kafka Technology

5. Kafka’s storage strategy

Principle analysis of Kafka Technology
Principle analysis of Kafka Technology
File storage mode
Principle analysis of Kafka Technology