Talk about Kafka: building Kafka on Linux

Time:2021-9-14

Welcome to my WeChat official account, old Chou chat architecture, the principle of Java back-end technology stack, source code analysis, architecture and various Internet high concurrency, high-performance and high availability solutions.

1、 Environmental preparation

JDK download address link:jdk 1.8, extraction code: dv5h
Zookeeper download address link:zookeeper3.4.14 , extraction code: 3dch
Kafka download address link:kafka2.12, extraction code: 61bc

1.1 Java environment

1.1.1 uploadjdk-8u261-linux-x64.rpmConnect to the server and install

#Installation command
rpm -ivh jdk-8u261-linux-x64.rpm

Talk about Kafka: building Kafka on Linux

1.1.2 configuring environment variables

#Edit the configuration file from the JDK bin directory to the / etc / profile file, which is valid for all user shells
vim /etc/profile
#Take effect
source /etc/profile

Talk about Kafka: building Kafka on Linux

export JAVA_HOME=/usr/java/jdk1.8.0_261-amd64
export PATH=$PATH:$JAVA_HOME/bin

Talk about Kafka: building Kafka on Linux

#Verify
java -version

Talk about Kafka: building Kafka on Linux
At this point, the JDK installation is successful.

1.2 installation and configuration of zookeeper

1.2.1 uploadzookeeper-3.4.14.tar.gzTo the server, unzip to/opt

#Extract ZK to the specified directory
tar -zxf zookeeper-3.4.14.tar.gz -C /opt

Talk about Kafka: building Kafka on Linux
1.2.2 modificationZookeeperThe directory where the data is stored,dataDir

#Enter the conf configuration directory
cd /opt/zookeeper-3.4.14/conf
#Copy Zoo_ Sample.cfg is named zoo.cfg
cp zoo_sample.cfg zoo.cfg
#Edit the zoo.cfg file
vim zoo.cfg
dataDir=/var/riemann/zookeeper/data

Talk about Kafka: building Kafka on Linux
Talk about Kafka: building Kafka on Linux

1.2.3 editing/etc/profileTo make the configuration effective

Setting environment variablesZOO_LOG_DIR, specifyZookeeperWhere to save the log;

`ZOOKEEPER_ Prefix ` points to the decompression directory of 'zookeeper';

Put zookeeper’sbinAdd directory toPATHMedium:

Talk about Kafka: building Kafka on Linux

export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/var/riemann/zookeeper/log

Take effect after configuration:

source /etc/profile 

1.2.4 startupZookeeper, confirmZookeeperState of

zkServer.sh start 

Talk about Kafka: building Kafka on Linux

So far, zookeeper has been successfully installed.

1.3 installation and configuration of Kafka

1.3.1 uploadkafka_2.12-1.0.2.tgzTo the server and unzip

tar -zxf kafka_2.12-1.0.2.tgz -C /opt 

Talk about Kafka: building Kafka on Linux
1.3.2 configure environment variables and take effect

vim /etc/profile 
export KAFKA=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA/bin
source /etc/profile 

1.3.3 configuration/opt/kafka_2.12-1.0.2/configMediumserver.propertiesfile

vi  /opt/kafka_2.12-1.0.2/config/server.properties

Talk about Kafka: building Kafka on Linux

`Kafka ` connects to the address of 'zookeeper'. The locally started 'zookeeper' instance is used here
The connection address is ` localhost: 2181`
The following 'mykafka' is the root node path of 'Kafka' in zookeeper

Talk about Kafka: building Kafka on Linux
Configure Kafka storage persistent data directory
Talk about Kafka: building Kafka on Linux

log.dirs=/var/riemann/kafka/kafka-logs

Create the above persistent data directory

mkdir -p /var/riemann/kafka/kafka-logs

1.4 start Kafka

Enter the root directory of Kafka installation and execute the following command:

kafka-server-start.sh ../config/server.properties

Talk about Kafka: building Kafka on Linux
After successful startup, you can see the started status in the last line of console output: at this time, Kafka installation is successful.

Talk about Kafka: building Kafka on Linux

1.5 reopen a window to view the zookeeper node
Talk about Kafka: building Kafka on Linux

1.6 at this time, Kafka is started in the foreground mode. To stop, use Ctrl + C

If you want to start in the background, use the command:

kafka-server-start.sh -daemon config/server.properties 

To view the background process of Kafka:

ps -ef | grep kafka

Talk about Kafka: building Kafka on Linux

Stop Kafka running in the background:

kafka-server-stop.sh 

2、 Production and consumption

Check the status of zookeeper. Zookeeper starts successfully, and then start Kafka.

Talk about Kafka: building Kafka on Linux
2.1 kafka-topics.sh is used to manage topics

View help for commands

[[email protected] bin]# kafka-topics.sh
Create, delete, describe, or change a topic.
Option                                   Description                            
------                                   -----------                            
--alter                                  Alter the number of partitions,        
                                           replica assignment, and/or           
                                           configuration for the topic.         
--config <String: name=value>            A topic configuration override for the 
                                           topic being created or altered.The   
                                           following is a list of valid         
                                           configurations:                      
                                             cleanup.policy                        
                                             compression.type                      
                                             delete.retention.ms                   
                                             file.delete.delay.ms                  
                                             flush.messages                        
                                             flush.ms                              
                                             follower.replication.throttled.       
                                           replicas                             
                                             index.interval.bytes                  
                                             leader.replication.throttled.replicas 
                                             max.message.bytes                     
                                             message.format.version                
                                             message.timestamp.difference.max.ms   
                                             message.timestamp.type                
                                             min.cleanable.dirty.ratio             
                                             min.compaction.lag.ms                 
                                             min.insync.replicas                   
                                             preallocate                           
                                             retention.bytes                       
                                             retention.ms                          
                                             segment.bytes                         
                                             segment.index.bytes                   
                                             segment.jitter.ms                     
                                             segment.ms                            
                                             unclean.leader.election.enable        
                                         See the Kafka documentation for full   
                                           details on the topic configs.        
--create                                 Create a new topic.                    
--delete                                 Delete a topic                         
--delete-config <String: name>           A topic configuration override to be   
                                           removed for an existing topic (see   
                                           the list of configurations under the 
                                           --config option).                    
--describe                               List details for the given topics.     
--disable-rack-aware                     Disable rack aware replica assignment  
--force                                  Suppress console prompts               
--help                                   Print usage information.               
--if-exists                              if set when altering or deleting       
                                           topics, the action will only execute 
                                           if the topic exists                  
--if-not-exists                          if set when creating topics, the       
                                           action will only execute if the      
                                           topic does not already exist         
--list                                   List all available topics.             
--partitions <Integer: # of partitions>  The number of partitions for the topic 
                                           being created or altered (WARNING:   
                                           If partitions are increased for a    
                                           topic that has a key, the partition  
                                           logic or ordering of the messages    
                                           will be affected                     
--replica-assignment <String:            A list of manual partition-to-broker   
  broker_id_for_part1_replica1 :           assignments for the topic being      
  broker_id_for_part1_replica2 ,           created or altered.                  
  broker_id_for_part2_replica1 :                                                
  broker_id_for_part2_replica2 , ...>                                           
--replication-factor <Integer:           The replication factor for each        
  replication factor>                      partition in the topic being created.
--topic <String: topic>                  The topic to be create, alter or       
                                           describe. Can also accept a regular  
                                           expression except for --create option
--topics-with-overrides                  if set when describing topics, only    
                                           show topics that have overridden     
                                           configs                              
--unavailable-partitions                 if set when describing topics, only    
                                           show partitions whose leader is not  
                                           available                            
--under-replicated-partitions            if set when describing topics, only    
                                           show under replicated partitions     
--zookeeper <String: urls>               REQUIRED: The connection string for    
                                           the zookeeper connection in the form 
                                           host:port. Multiple URLS can be      
                                           given to allow fail-over.            
[[email protected] bin]# 
#List existing topics
[[email protected] ~]# kafka-topics.sh --list --zookeeper localhost:2181/myKafka
#Create a theme, which contains a partition, which is a leader partition, and it does not have a copy of the follower partition.
[[email protected] ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --create --topic topic_test --partitions 1 --replication-factor 1
#View partition information
[[email protected] ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --list
#View the details of the specified topic
[[email protected] ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --describe --topic topic_test 
#Delete the specified topic
[[email protected] ~]# kafka-topics.sh --zookeeper localhost:2181/myKafka --delete --topic topic_test

A topic appears in the column, and a topic is created. The topic contains a partition. The partition is a leader partition, and it does not have a copy of the follower partition.

Talk about Kafka: building Kafka on Linux

View the details of the specified topic
Talk about Kafka: building Kafka on Linux
Create a theme that contains multiple partitions

Multiple partitions: scale out
Multiple replicas: highly available

Talk about Kafka: building Kafka on Linux

2.2 kafka-console-consumer.sh is used to consume messages

#Open consumer
[[email protected] ~]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_test 
#Open consumer方式二,从头消费,不按照偏移量消费
[[email protected] ~]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_test --from-beginning

2.3 kafka-console-producer.sh for production messages

#Turn on producer
[[email protected] ~]# kafka-console-producer.sh --topic topic_test --broker-list  localhost:9020

2.4 specific operation

Open consumers and producers, produce and consume messages.

Talk about Kafka: building Kafka on Linux
Consumer, consume according to offset

Talk about Kafka: building Kafka on Linux
Consumers consume from scratch, not according to the offset

Talk about Kafka: building Kafka on Linux