Distributed message system: Kafka



As a message middleware, Kafka serves the asynchronous transmission of messages
The function is similar to JMS: the producer puts the message into the queue, and the consumer gets the data from the queue
But it’s totally different

Kafka flow chart

Distributed message system: Kafka

1. Producer sends messages by connecting to broker
2. Producer delivers messages to a topic
3. Messages sent to topic can be processed by different brokers
4. The consumer is connected to a broker to listen on the topic of the subscription
5. Brokers are managed by zookeeper and inform each other of their operation
6. If the broker to which producer or consumer is connected has no related topic, the message will be automatically routed to the related broker, and the next time producer or consumer will automatically remember the related broker

Distributed message system: Kafka

1. After sending messages, each message will be arranged to each partition in turn
2. Messages can be configured to determine how long they should be retained on the partition
3. Each consumer can start to consume messages from different queue locations, and can consume them repeatedly

Kafka installation


curl -L -O http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz

Modify configuration

vi kakfa/config/server.properties

Log directory

Allocate memory (less than 256 will cause memory overflow)
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" 

Register with zookeeper

Delete data (prevent disk explosion)

vi /config/server.properties

According to the storage time

According to the data size


bin/kafka-server-start.sh config/server.properties &



Create a topic

bin/kafka-topics.sh --create --zookeeper --replication-factor 1 --partitions 1 --topic test

View all topics

bin/kafka-topics.sh --list --zookeeper

Producer sends message

bin/kafka-console-producer.sh --broker-list --topic test

Consumer receives messages

bin/kafka-console-consumer.sh --zookeeper --topic test --from-beginning

Delete the specified topic

./bin/kafka-topics.sh  --delete --zookeeper  --topic test

Actual operation code

Using spring to operate Kafka

Import Maven dependency:


Message producer:

Configuration class:

public class KafkaProducerConfig {
    private String bootstrapServers = "IP:9092";

    public ProducerFactory<String, String> producerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        return new DefaultKafkaProducerFactory<>(configProps);

    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());

Send message class:

public class KafkaSender {
    private KafkaTemplate kafkaTemplate;

    public void send(){

Message receiver
Configuration class:

public class KafkaConsumerConfig {

    private String bootstrapServers="IP:9092";

    private String topic = "topic_1";

    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        configProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        configProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        configProps.put(ConsumerConfig.GROUP_ID_CONFIG, topic);
        return new DefaultKafkaConsumerFactory<>(configProps);

    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();

        return factory;


Message acceptance class:

public class KafkaReceive {
    @KafkaListener(topics = {"topic_1"})
    public void receive(String content){

The above is the basic installation and use of Kafka, thank you for reading