Common script practice of Kafka in PHP

Time:2020-10-7

Before reading this tutorial, you’d better try to read it first: Kafka’s practice in PHP

Practice of own command

Kafka knowledge tried to practice:

  1. Create a topic
  2. Production news
  3. Consumer News
  4. Topic information
  5. Get consumer group
  6. Get the offset of the consumption group

Own order

#Kafka installation directory bin directory
#* represents the scripts we will use
connect-distributed.sh        kafka-log-dirs.sh                    kafka-streams-application-reset.sh
connect-standalone.sh         kafka-mirror-maker.sh                kafka-topics.sh*
kafka-acls.sh                 kafka-preferred-replica-election.sh  kafka-verifiable-consumer.sh
kafka-broker-api-versions.sh  kafka-producer-perf-test.sh          kafka-verifiable-producer.sh
kafka-configs.sh              kafka-reassign-partitions.sh         trogdor.sh
kafka-console-consumer.sh*    kafka-replay-log-producer.sh         windows
kafka-console-producer.sh*    kafka-replica-verification.sh        zookeeper-security-migration.sh
kafka-consumer-groups.sh*     kafka-run-class.sh                   zookeeper-server-start.sh
kafka-consumer-perf-test.sh   kafka-server-start.sh                zookeeper-server-stop.sh
kafka-delegation-tokens.sh    kafka-server-stop.sh                 zookeeper-shell.sh*
kafka-delete-records.sh       kafka-simple-consumer-shell.sh

Create a topic (Kafka- topics.sh )

#Create one partition and one replica. This is the topic of test. In fact, it can be understood as at least the number of copies in the broker, must > = 1
# --zookeeper  localhost:2181  (Kafka's default port: 2181)
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

#Create TEST02 topic of 2 partitions and 1 copy
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic test02

#List all topics
bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
test
test02
#Pay attention here__ consumer_ Offsets are created by Kafka by default and are used to store Kafka consumption records. We do not need to pay attention to them for the time being


#List information about specific topics
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
Topic:test      PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: test     Partition: 0    Leader: 0       Replicas: 0     Isr: 0


bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test02
Topic:test02    PartitionCount:2        ReplicationFactor:1     Configs:
        Topic: test02   Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: test02   Partition: 1    Leader: 0       Replicas: 0     Isr: 0

#From the above display, we can see that the first sentence is to display the overall information, and the indent below shows the partition information

Consumer (Kafka console)- consumer.sh )

#Start a consumption group consumption. Here we need to open a shell terminal, because it will wait for the output
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
#Waiting for output

Kafka console- consumer.sh )

#Here we need to open a shell terminal because it will wait for input
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
#Wait for us to enter the message and wait for > to appear
> msg01
> msg02
> msg03

#Pay attention to the above consumer terminal, automatically output our message
msg01
msg02
msg03

View consumer groups

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
Note: This will not show information about old Zookeeper-based consumers.
console-consumer-25379
console-consumer-73410
console-consumer-27127
console-consumer-61887
console-consumer-61324

#Here we start a new consumer and output again (because previous consumers do not know what the latest consumption group ID is)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
Note: This will not show information about old Zookeeper-based consumers.
console-consumer-25379
console-consumer-73410
console-consumer-27127
Console-consumer-39416 ා this is our new consumer group ID. let's practice according to this consumption group
console-consumer-61887
console-consumer-61324

#View specific information of consumption group
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group console-consumer-39416
Note: This will not show information about old Zookeeper-based consumers.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
test            0          9               9               0               consumer-1-94afec29-5042-4108-8619-ba94812f10a8 /127.0.0.1      consumer-1

#View offline console-consumer-25379 consumer group
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group console-consumer-25379
Note: This will not show information about old Zookeeper-based consumers.
Consumer group 'console-consumer-25379' has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
test            0          5               9               4               -               -               -

#View offline console-consumer-27127 consumer groups
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group console-consumer-27127
Note: This will not show information about old Zookeeper-based consumers.
Consumer group 'console-consumer-27127' has no active members.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
test            0          6               9               3               -               -               -

#Here we find that we generate one consumer group and one consumer each time, which is not convenient for us to do multiple consumer tests for one consumer group

Start a consumer group and multiple consumers

cp config/consumer.properties config/consumer_g1.properties
vim config/consumer_g1.properties
#Modify consumption group name
group.id=test-consumer-group => group.id=test-g1

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer_g1.properties
msg01
msg02
msg03

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --consumer.config config/consumer_g1.properties
#No output (because we only have one partition for the test topic, a partition can only be consumed by a consumer under the same consumption group, so this one is idle)

#View consumer groups
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
Note: This will not show information about old Zookeeper-based consumers.
console-consumer-25379
console-consumer-73410
console-consumer-27127
console-consumer-39416
test-g1
console-consumer-61887
console-consumer-61324

#View test-g1 consumer group
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group test-g1
Note: This will not show information about old Zookeeper-based consumers.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
test            0          9               9               0               consumer-1-43922b0c-34e0-47fe-b597-984c9e6a2884 /127.0.0.1      consumer-1

#Next, let's open the two consumption groups of TEST02 to see the situation

#Let's start two test-g1 consumers for TEST02
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test02 --from-beginning --consumer.config config/consumer_g1.properties

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test02 --from-beginning --consumer.config config/consumer_g1.properties

#View test-g1 consumer group
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group test-g1
Note: This will not show information about old Zookeeper-based consumers.

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
test02          0          0               0               0               consumer-1-90ccc960-557a-46ab-a799-58c35ee670d8 /127.0.0.1      consumer-1
test02          1          0               0               0               consumer-1-c6d79321-8212-4594-8a11-353f684c54fc /127.0.0.1      consumer-1
test            0          9               9               0               consumer-1-7a2706f7-a206-4c29-ae1f-3726ad21af96 /127.0.0.1      consumer-1

#Here, you can see a message generated in the topic test, and then generate several more messages in TEST02. You will find that the two consumption groups of TEST02 are almost loaded consumption messages

Some information commands of consumption group

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members --verbose
Note: This will not show information about old Zookeeper-based consumers.

CONSUMER-ID                                     HOST            CLIENT-ID       #PARTITIONS     ASSIGNMENT
consumer-1-b8eeb8bc-6aa6-439b-84d8-0fcbed4a2899 /127.0.0.1      consumer-1      1               test02(1)
consumer-1-7109f789-a5cf-4862-94d0-976146dbc769 /127.0.0.1      consumer-1      1               test(0)
consumer-1-90ccc960-557a-46ab-a799-58c35ee670d8 /127.0.0.1      consumer-1      1               test02(0)


bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
Note: This will not show information about old Zookeeper-based consumers.

COORDINATOR (ID)          ASSIGNMENT-STRATEGY       STATE                #MEMBERS
localhost:9092 (0)        range                     Stable               3


bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group console-consumer-25379

zookeeper(zookeeper-shell.sh)

connect

bin/zookeeper-shell.sh 127.0.0.1:2181
Connecting to 127.0.0.1:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

Common commands

ls /
[cluster, controller_epoch, controller, brokers, zookeeper, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]