Kafkar introduction

Time:2021-9-17

Kafka introduction

A distributed, partitioned, multi replica and multi subscriber distributed logging system based on zookeeper coordination
The main application scenarios are: log collection system and message system

Kafkar introduction

Core concept

Publish subscribe model

The message is persisted into a topic. Consumers can subscribe to one or more topics. Consumers can consume all the data in the topic. The same data can be consumed by multiple consumers. The data will not be deleted immediately after consumption.
Kafkar introduction

Broker (server node)

A Kafka server is a broker. A cluster is composed of multiple brokers, and each broker can accommodate multiple topics;

Producer (publisher)

The publisher publishes the message to Kafka’s topic. After the broker receives the message sent by the producer, the broker appends the message to the segment file currently used to append data;

Consumers

Consumers can read data from the broker. Consumers can consume data in multiple topics;

Consumer group

Each consumer belongs to a specific consumer group (you can specify a group name for each consumer. If you do not specify a group name, it belongs to the default group);

Topic (message category)

Each message published to the Kafka cluster has a category called topic; (similar to database table name)

Partition

The data in topic is divided into one or more partitions. Each topic has at least one partition;

Zookeeper

Storing metadata; It means that all the data of the subject partition is saved in zookeeper, and the data saved by it is the authority, and the others should be aligned with it;
Member management; It refers to the registration, logout and attribute change of the broker node;
(b) elections; It refers to the election of cluster controller, and other management tasks include but are not limited to topic deletion, parameter configuration, etc;

Code example

Kafkar introduction

Kafkar introduction

Graphical interface management tool (Kafka manager)

Kafkar introduction

Kafkar introduction

problem

1. Will messages in Kafka be lost and consumed repeatedly?

2. How does Kafka reflect message sequencing?

3. How are transactions implemented in Kafka?