Using docker to quickly establish a Kafka instance

Time:2020-10-22

0 Preface

Kafka is very popular in recent years. Although it is positioned as a distributed streaming platform, we still use it as an MQ most of the time: P.

It happens that Kafka is also being studied recently. It is necessary to prepare the development environment for research and development test.

Docker is also a very popular virtual application container platform. It can manage the mainstream open-source frameworks in the form of containers. By combining the basic containers, it can build more complex application services, and greatly reduce the cost of building the environment. For example, in the machine learning project, we quickly integrated the main open source frameworks of machine learning, such as CONDA + tenserflow + keras, without affecting the host environment. If you don’t like it, it’s very fast to delete the image and rebuild it with a new version.

Adapt to this idea, so the construction of this environment is also the implementation goal of docker + Kafka. In the process, record this article as a reference and review in the future.

1 install docker and docker compose

The above official documents also describe clearly, and I will not repeat them here.

For docker installation, please refer to

For docker compose installation, please refer to

2 find docker image

With the docker platform, whenever you need to build an open-source framework, you can go to the docker hub to find out if there is an official docker image. After searching for hairstyles, Kafka didn’t provide any official images. All we could find were some community maintained image packages. For example: wurstmeister / Kafka, etc. <p>
Although the image is not officially maintained, there are still 25m + docker pull and nearly 700 likes, which is the highest score of the same type of image on docker hub. Therefore, this image is preferred as the basis of the service.

3 define docker compose

Here we introduce the concept of docker compose. For docker compose, we can understand it as follows: docker compose = n * docker container = a complex service.

Since Kafka’s cluster based high availability feature is based on zookeeper (called zk), building available Kafka images depends on ZK foundation. Therefore, docker compose is needed to combine different containers into a complete service. <p>To build a service composed of multiple containers, an independent docker is required- compose.yml Documents.

version: '3'
<! -- compatible version of docker compose, which can be configured according to the actual large version number of docker -- >
services:
<! -- define ZK layer service -- >
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
<! -- define Kafka layer -- >
  kafka:
    image: wurstmeister/kafka
    <! -- priority should be given to the basic image rather than the local dockerfile reconstruction method. The actual situation can be selected from the build configuration item. Therefore, the following line is commented out -- >
    <!--build: .-->
    depends_on: [ zookeeper ]
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 47.106.143.76
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Follow the docker above- compose.yml File, ZK and Kafka can be integrated into a service package. The basic idea here is that if there is a basic image, the basic image should be used first, otherwise the dockerfile reconstruction method should be used.

docker- compose.yml For further customization of containers, please refer to the official documentation of docker compose.

4 package service

Switch the path to and docker- compose.yml Package the defined services in the directory with consistent files

[[email protected] kafka]# docker-compose build
zookeeper uses an image, skipping
kafka uses an image, skipping

After that, you can use the docker compose up – D command to start

[[email protected] kafka]# docker-compose up -d
Starting kafka_kafka_1     ... done
Starting kafka_zookeeper_1 ... done
[[email protected] kafka]#

You can stop the service by using a similar command.
It should be noted that the services operated by docker compose should be based on the docker in the current directory- compose.yml If this definition is not found, docker compose cannot know which service to operate.

5 test service

After the docker service is up, we can test the service. Here, we can use the CLI tool provided by docker for basic test, and then we can use API call method to test the service.

First, we need to check the containers that have been started

[[email protected] kafka]# docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
2c3e1aacdd56        wurstmeister/kafka       "start-kafka.sh"         2 hours ago         Up 2 minutes        0.0.0.0:9092->9092/tcp                               kafka_kafka_1
20cd04496a9c        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   2 days ago          Up 2 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka_zookeeper_1
[[email protected] kafka]#

It can be seen from the above that Kafka’s container ID is 2c3e1aacdd56. We can enter the container by using the following command line:

docker exec -it {container id} /bin/bash
[[email protected] kafka]# docker exec -it 2c3e1aacdd56 /bin/bash
bash-4.4#
bash-4.4# ls -al
total 72
drwxr-xr-x    1 root     root          4096 Jul 13 02:00 .
drwxr-xr-x    1 root     root          4096 Jul 13 02:00 ..
-rwxr-xr-x    1 root     root             0 Jul 13 02:00 .dockerenv
drwxr-xr-x    1 root     root          4096 Apr 15 16:03 bin
drwxr-xr-x    5 root     root           340 Jul 13 03:29 dev
drwxr-xr-x    1 root     root          4096 Jul 13 02:00 etc
drwxr-xr-x    2 root     root          4096 Jan  9  2018 home
drwxr-xr-x    5 root     root          4096 Jul 13 02:00 kafka
drwxr-xr-x    1 root     root          4096 Apr 15 16:03 lib
drwxr-xr-x    2 root     root          4096 Apr 15 16:03 lib64
drwxr-xr-x    5 root     root          4096 Jan  9  2018 media
drwxr-xr-x    2 root     root          4096 Jan  9  2018 mnt
drwxr-xr-x    1 root     root          4096 Apr 15 16:03 opt
dr-xr-xr-x  101 root     root             0 Jul 13 03:29 proc
drwx------    2 root     root          4096 Jan  9  2018 root
drwxr-xr-x    1 root     root          4096 Jul 13 02:00 run
drwxr-xr-x    1 root     root          4096 Apr 15 16:03 sbin
drwxr-xr-x    2 root     root          4096 Jan  9  2018 srv
dr-xr-xr-x   13 root     root             0 Jul 13 03:29 sys
drwxrwxrwt    1 root     root          4096 Jul 13 03:29 tmp
drwxr-xr-x    1 root     root          4096 Apr 15 16:03 usr
drwxr-xr-x    1 root     root          4096 Apr 15 16:03 var
bash-4.4#

As above, you can see the contents of the Kafka container.

Then execute the Kafka cli test tool

<! -- configure environment variables -- >
bash-4.4# export ZK=zookeeper:2181
bash-4.4# echo $ZK
zookeeper:2181

<! -- run the CLI series tools provided by Kafka -- >
bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic topic --partitions 4 --zookeeper $ZK --replication-factor 1
Created topic "topic".

At this point, a complete ZK + Kafka container service of docker is opened. Although there is still some distance from the production node, a complete application development can be established on it. Have a chance to talk.

reference

Kafka documentation
Docker documentation
Docker compose documentation
Wurstmeister / Kafka docker image description document

Recommended Today

Layout of angular material (2): layout container

Layout container Layout and container Using thelayoutDirective to specify the layout direction for its child elements: arrange horizontally(layout=”row”)Or vertically(layout=”column”)。 Note that if thelayoutInstruction has no value, thenrowIs the default layout direction. row: items arranged horizontally.max-height = 100%andmax-widthIs the width of the item in the container. column: items arranged vertically.max-width = 100%andmax-heightIs the height of the […]