Distributed message system: Kafka

Time:2021-7-7

preface

As a message middleware, Kafka serves the asynchronous transmission of messages
The function is similar to JMS: the producer puts the message into the queue, and the consumer gets the data from the queue
But it’s totally different

Kafka flow chart

Distributed message system: Kafka

1. Producer sends messages by connecting to broker
2. Producer delivers messages to a topic
3. Messages sent to topic can be processed by different brokers
4. The consumer is connected to a broker to listen on the topic of the subscription
5. Brokers are managed by zookeeper and inform each other of their operation
6. If the broker to which producer or consumer is connected has no related topic, the message will be automatically routed to the related broker, and the next time producer or consumer will automatically remember the related broker

Distributed message system: Kafka

1. After sending messages, each message will be arranged to each partition in turn
2. Messages can be configured to determine how long they should be retained on the partition
3. Each consumer can start to consume messages from different queue locations, and can consume them repeatedly

Kafka installation

Download:

curl -L -O http://mirror.bit.edu.cn/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz

Modify configuration

vi kakfa/config/server.properties

Log directory
log.dirs=/tmp/kafka/logs

Allocate memory (less than 256 will cause memory overflow)
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M" 

Register with zookeeper
zookeeper.connect=106.15.205.155:2181

Delete data (prevent disk explosion)

vi /config/server.properties

According to the storage time
log.retention.hours=168

According to the data size
log.segment.bytes=1073741824

start-up

bin/kafka-server-start.sh config/server.properties &

close

bin/kafka-server-stop.sh

Create a topic

bin/kafka-topics.sh --create --zookeeper 106.15.205.155:2181 --replication-factor 1 --partitions 1 --topic test

View all topics

bin/kafka-topics.sh --list --zookeeper 106.15.205.155:2181

Producer sends message

bin/kafka-console-producer.sh --broker-list 106.15.205.155:9092 --topic test

Consumer receives messages

bin/kafka-console-consumer.sh --zookeeper 106.15.205.155:2181 --topic test --from-beginning

Delete the specified topic

./bin/kafka-topics.sh  --delete --zookeeper 106.15.205.155:2181  --topic test

Actual operation code

Using spring to operate Kafka

Import Maven dependency:

        <!--kafka-->
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
            <version>1.3.0.RELEASE</version>
        </dependency>

Message producer:

Configuration class:


@Configuration
public class KafkaProducerConfig {
    private String bootstrapServers = "IP:9092";

    @Bean
    public ProducerFactory<String, String> producerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);

        return new DefaultKafkaProducerFactory<>(configProps);
    }

    @Bean
    public KafkaTemplate<String, String> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }
}

Send message class:

@Component
public class KafkaSender {
    @Autowired
    private KafkaTemplate kafkaTemplate;

    public void send(){
        kafkaTemplate.send("topic_1","luxiaotao");
    }
}

Message receiver
Configuration class:

@Configuration
public class KafkaConsumerConfig {

    private String bootstrapServers="IP:9092";


    private String topic = "topic_1";

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> configProps = new HashMap<>();
        configProps.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        configProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        configProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        configProps.put(ConsumerConfig.GROUP_ID_CONFIG, topic);
        return new DefaultKafkaConsumerFactory<>(configProps);
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());

        return factory;
    }

}

Message acceptance class:

@Component
public class KafkaReceive {
    @KafkaListener(topics = {"topic_1"})
    public void receive(String content){
        System.out.println("============================="+content+"============================");
    }
}

The above is the basic installation and use of Kafka, thank you for reading