The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

Time:2021-6-10

Kafka is a very popular message middleware. According to the official website, thousands of companies are using it. Recently, we have practiced a wave of Kafka, which is really good and powerful. Today, we will learn Kafka from three aspects: the installation of kafaka under Linux, the visualization tool of Kafka, and the combination of Kafka and springboot. I hope you can quickly start Kafka and master this popular message middleware after watching it!

Springboot real e-commerce project Mall (40K + star) address:https://github.com/macrozheng/mall

About Kafka

Kafka was created byLinkedInThe company developed an open source distributed message flow platform, written by Scala and Java. Its main function is to provide a unified, high throughput, low latency platform for processing real-time data, and its essence is based onPublish subscribe modeThe message engine system of.

Kafka has the following characteristics:

  • High throughput and low latency: Kafka can send and receive messages very fast, and the latency of cluster processing messages can be as low as 2ms.
  • High scalability: Kafka can flexibly expand and contract, and can expand to thousands of brokers and hundreds of thousands of partitions, processing trillions of messages every day.
  • Permanent storage: Kafka can store data safely in a distributed, persistent, fault-tolerant cluster.
  • High availability: Kafka can effectively expand the cluster in the availability zone. When a node goes down, the cluster can still work normally.

Kafka installation

We will adopt the installation mode under Linux, and the installation environment is CentOS 7.6. I don’t use docker to install and deploy here. I think it’s easier to install directly (mainly because the official doesn’t provide docker image)!

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • After downloading, unzip Kafka to the specified directory:
cd /mydata/kafka/
tar -xzf kafka_2.13-2.8.0.tgz
  • After decompression, enter the decompression Directory:
cd kafka_2.13-2.8.0
  • Although it is reported that Kafka is about to remove zookeeper, it has not been removed in the latest version of Kafka, so you still need to start zookeeper before starting Kafka;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • Start the zookeeper service, which will run in2181Port;
#Run the service in the background and output the log to the zookeeper-out.file file in the current folder
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper-out.file 2>&1 &
  • Because Kafka is currently deployed on Linux server, if you want to access the external network, you need to modify the configuration file of Kafkaconfig/server.properties, modify the listening address of Kafka, otherwise it will be unable to connect;
############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://192.168.5.78:9092
  • Finally, start the Kafka service, and the service will run in the9092Port.
#Run the service in the background and output the log to the kafka-out.file file in the current folder
nohup bin/kafka-server-start.sh config/server.properties > kafka-out.file 2>&1 &

Kafka command line operation

Next, we use the command line to operate Kafka, familiar with the use of Kafka.

  • First, create a file calledconsoleTopicThe topic of the project;
bin/kafka-topics.sh --create --topic consoleTopic --bootstrap-server 192.168.5.78:9092
  • Next, look at topic;
bin/kafka-topics.sh --describe --topic consoleTopic --bootstrap-server 192.168.5.78:9092
  • The following topic information will be displayed;
Topic: consoleTopic    TopicId: tJmxUQ8QRJGlhCSf2ojuGw    PartitionCount: 1    ReplicationFactor: 1    Configs: segment.bytes=1073741824
    Topic: consoleTopic    Partition: 0    Leader: 0    Replicas: 0    Isr: 0
  • Send a message to topic:
bin/kafka-console-producer.sh --topic consoleTopic --bootstrap-server 192.168.5.78:9092
  • Input information directly in the command line to send;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • Re open a window and get the message from topic by the following command:
bin/kafka-console-consumer.sh --topic consoleTopic --from-beginning --bootstrap-server 192.168.5.78:9092

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

Kafka visualization

Using the command line to manipulate Kafka is a bit of a hassle, so let’s try the visualization toolkafka-eagle

Install JDK

If you are using CentOS, the full version of JDK is not installed by default. You need to install it yourself!

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • After downloading, extract the JDK to the specified directory;
cd /mydata/java
tar -zxvf OpenJDK8U-jdk_x64_linux_xxx.tar.gz
mv OpenJDK8U-jdk_x64_linux_xxx.tar.gz jdk1.8
  • stay/etc/profileAdd environment variable to fileJAVA_HOME
vi /etc/profile
#Add in profile file
export JAVA_HOME=/mydata/java/jdk1.8
export PATH=$PATH:$JAVA_HOME/bin
#Make the modified profile file effective
. /etc/profile

installkafka-eagle

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • When the download is complete, thekafka-eagleUnzip to the specified directory;
cd /mydata/kafka/
tar -zxvf kafka-eagle-web-2.0.5-bin.tar.gz
  • stay/etc/profileAdd environment variable to fileKE_HOME
vi /etc/profile
#Add in profile file
export KE_HOME=/mydata/kafka/kafka-eagle-web-2.0.5
export PATH=$PATH:$KE_HOME/bin
#Make the modified profile file effective
. /etc/profile
  • Install MySQL and add databasekekafka-eagleIt will be used later;
  • Modify configuration file$KE_HOME/conf/system-config.properties, mainly modifying zookeeper configuration and database configuration, commenting out SQLite configuration and using MySQL instead;
######################################
# multi zookeeper & kafka cluster list
######################################
kafka.eagle.zk.cluster.alias=cluster1
cluster1.zk.list=localhost:2181

######################################
# kafka eagle webui port
######################################
kafka.eagle.webui.port=8048

######################################
# kafka sqlite jdbc driver address
######################################
# kafka.eagle.driver=org.sqlite.JDBC
# kafka.eagle.url=jdbc:sqlite:/hadoop/kafka-eagle/db/ke.db
# kafka.eagle.username=root
# kafka.eagle.password=www.kafka-eagle.org

######################################
# kafka mysql jdbc driver address
######################################
kafka.eagle.driver=com.mysql.cj.jdbc.Driver
kafka.eagle.url=jdbc:mysql://localhost:3306/ke?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull
kafka.eagle.username=root
kafka.eagle.password=root
  • Start with the following commandkafka-eagle
$KE_HOME/bin/ke.sh start
  • After the command is executed, the following information will be displayed, but it does not mean that the service has been started successfully and needs to wait for a while;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • Here are some more useful oneskafka-eagleCommand:
#Stop service
$KE_HOME/bin/ke.sh stop
#Restart service
$KE_HOME/bin/ke.sh restart
#View service running status
$KE_HOME/bin/ke.sh status
#View service status
$KE_HOME/bin/ke.sh stats
#Dynamic view of service output log
tail -f $KE_HOME/logs/ke_console.out
  • You can access it directly after successful startup, and enter the account and passwordadmin:123456, visit address:http://192.168.5.78:8048/

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • After successful login, you can access the dashboard, and the interface is still great!

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

Use of visualization tools

  • Previously, we used the command line to create a topic, which can be created directly through the interface;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • We can also go through it directlykafka-eagleTo send messages;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • We can consume the messages in topic through the command line;
bin/kafka-console-consumer.sh --topic testTopic --from-beginning --bootstrap-server 192.168.5.78:9092
  • The information obtained by the console is shown as follows:;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • There’s another interesting feature calledKSQL, you can query the messages in topic through SQL statements;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • Visualization tools naturally need monitoring, if you want to turn it onkafka-eagleFor the monitoring function of Kafka, you need to modify the startup script of Kafka to expose the port of JMX;
vi kafka-server-start.sh
#Expose JMX ports
if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
    export KAFKA_HEAP_OPTS="-server -Xms2G -Xmx2G -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"
    export JMX_PORT="9999"
fi
  • Take a look at the monitoring chart interface;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • There is also a big screen monitoring function;

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • There is also the command line function of zookeeper. In short, the function is very complete and powerful!

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

Springboot integrates Kafka

Operating Kafka in springboot is also very simple. For example, the message mode of Kafka is very simple. There is no queue, only topic.

  • First, in the applicationpom.xmlAdd the spring Kafka dependency in;
<!-- Spring integrates Kafka -- >
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>2.7.1</version>
</dependency>
  • Modify application profileapplication.yml, configure the Kafka service address and thegroup-id
server:
  port: 8088
spring:
  kafka:
    bootstrap-servers: '192.168.5.78:9092'
    consumer:
      group-id: "bootGroup"
  • Create a producer to send messages to Kafka’s topic;
/**
 *Kafka message producer
 * Created by macro on 2021/5/19.
 */
@Component
public class KafkaProducer {
    @Autowired
    private KafkaTemplate kafkaTemplate;

    public void send(String message){
        kafkaTemplate.send("bootTopic",message);
    }
}
  • Create a consumer to get messages from Kafka and consume them;
/**
 *Kafka news consumer
 * Created by macro on 2021/5/19.
 */
@Slf4j
@Component
public class KafkaConsumer {

    @KafkaListener(topics = "bootTopic")
    public void processMessage(String content) {
        log.info("consumer processMessage : {}",content);
    }

}
  • Create an interface to send messages and call the producer to send messages;
/**
 *Kafka function test
 * Created by macro on 2021/5/19.
 */
@API (tags = kafkacontroller, description = Kafka function test)
@Controller
@RequestMapping("/kafka")
public class KafkaController {

    @Autowired
    private KafkaProducer kafkaProducer;

    @Apioperation ("send message")
    @RequestMapping(value = "/sendMessage", method = RequestMethod.GET)
    @ResponseBody
    public CommonResult sendMessage(@RequestParam String message) {
        kafkaProducer.send(message);
        return CommonResult.success(null);
    }
}
  • The interface is directly invoked in Swagger for testing.

The Kafka graphic tool eagle, which is used for hanging and exploding the sky, must be recommended to you!

  • The project console will output the following information, indicating that the message has been received and consumed.
2021-05-19 16:59:21.016  INFO 2344 --- [ntainer#0-0-C-1] c.m.mall.tiny.component.KafkaConsumer    : consumer processMessage : Spring Boot message!

summary

Through a wave of practice in this paper, you can basically get started with Kafka. Installation, visualization tools and spring boot are all operations related to developers, and they are also the only way to learn Kafka.

reference material

Project source address

https://github.com/macrozheng…

This article is about GitHubhttps://github.com/macrozheng/mall-learningHas been included, welcome to star!