A distributed, partitioned, multi replica and multi subscriber distributed logging system based on zookeeper coordination
The main application scenarios are: log collection system and message system
Publish subscribe model
The message is persisted into a topic. Consumers can subscribe to one or more topics. Consumers can consume all the data in the topic. The same data can be consumed by multiple consumers. The data will not be deleted immediately after consumption.
Broker (server node)
A Kafka server is a broker. A cluster is composed of multiple brokers, and each broker can accommodate multiple topics;
The publisher publishes the message to Kafka’s topic. After the broker receives the message sent by the producer, the broker appends the message to the segment file currently used to append data;
Consumers can read data from the broker. Consumers can consume data in multiple topics;
Each consumer belongs to a specific consumer group (you can specify a group name for each consumer. If you do not specify a group name, it belongs to the default group);
Topic (message category)
Each message published to the Kafka cluster has a category called topic; (similar to database table name)
The data in topic is divided into one or more partitions. Each topic has at least one partition;
Storing metadata; It means that all the data of the subject partition is saved in zookeeper, and the data saved by it is the authority, and the others should be aligned with it;
Member management; It refers to the registration, logout and attribute change of the broker node;
(b) elections; It refers to the election of cluster controller, and other management tasks include but are not limited to topic deletion, parameter configuration, etc;
Graphical interface management tool (Kafka manager)
1. Will messages in Kafka be lost and consumed repeatedly?
2. How does Kafka reflect message sequencing?
3. How are transactions implemented in Kafka?