Kafka — the general flow of sending messages from the producer client

Time:2021-12-4

Kafka also has the concept of producer and consumer. Producers send messages to Kafka, and consumers consume messages from Kafka. This is similar to the previous ActiveMQ and rabbitmq.
Kafka -- the general flow of sending messages from the producer client
Let’s start with the producer’s message and discuss the whole process of Kafka. The version here is0.10.1.0

Message object

The message object producerrecord contains topic, partition, key, value and timestamp, where topic and value are required when building producerrecord.

Topic represents the name of a set. When a message is sent to Kafka, Kafka needs to store the message. At this time, it needs the name of a set to store the messages of the same set together. This is similar to the MySQL table. The data of the same table is stored in the same table.

Value is the body of the message. For example, when we send a hello message to topic named topic1, this value is equal to hello.

Partition represents the area code. Kafka is a distributed system, that is, the data of a topic will be stored in multiple servers, so there are many partitions. Setting partition can specify which partition to send messages.

As the key of the message, the message can be partitioned according to the key.

Timestamp refers to the timestamp of the message, which can represent either the timestamp of the message or the time when the message is appended to the log file.

We assume that a message with a message body of Hello is sent to topic1.
Kafka -- the general flow of sending messages from the producer client

KafkaProducer

After the message is built, we need to send the message. Otherwise, why build it, right.

Kafkaproducer is responsible for sending messages. Each kafkaproducer has a ClientID. If it is not set, it will default toproducer-+Number, this number isAtomicIntegerType, so it is thread safe. If it is not set, the ClientID isproducer-1producer-2。 Since kafkaproducer is also thread safe, it is also possible for the client to build a kafkaproducer to send messages.

When sending messages, we need to know the address of the server (i.e. broker), so when instantiating kafkaproducer,bootstrap.serversIt also needs to be assigned. If there are multiple, separate them with commas in the format ofhost1:port1,host2:port2, here we assume that the server address ishadoop1:9092
Kafka -- the general flow of sending messages from the producer client

Interceptor

Interceptors are not necessary, such as doing some customized operations on messages, modifying topic and value, or customizing callback functions, but they are generally not used under normal circumstances.
Kafka -- the general flow of sending messages from the producer client

serialize

In network transmission, the broker receives the message in the form of byte array (byte []), so the key and value of the message need to be serialized in advance when sending.

When initializing kafkaproducer, setkey.serializerandvalue.serializeTo specify the serializers for key and value.
Kafka -- the general flow of sending messages from the producer client

Partition device

Kafka is a distributed system, and each topic corresponds to multiple brokers, so we need to allocate its partition to the message before sending it.
Kafka -- the general flow of sending messages from the producer client

Message accumulator

Kafka does not send messages one by one. It is stored in the cache of the message accumulator and then sent in batches. This can reduce the resource consumption of network transmission and improve performance.
Kafka -- the general flow of sending messages from the producer client

Sender thread

After the message is saved in memory, the sender thread will send the qualified messages in batches. In addition to sending messages, metadata loading is also handled through the sender thread.

Sneder thread sends and receives messages based on Java NiO selector. Send the message through the selector and receive the message through the selector.

Later, we will slowly expand the content of this chapter.
Kafka -- the general flow of sending messages from the producer client