Kafka monitoring of k8s (Prometheus + grafana)


Welcome to my GitHub


Content: classification and summary of all original articles and supporting source code, involving Java, docker, kubernetes, Devops, etc;

For Kafka deployed on k8s,Prometheus+GrafanaIt is a common monitoring scheme. Today, we will monitor Kafka of k8s environment through Prometheus + grafana;


Today, we focus on Kafka monitoring, so k8s, helm, Kafka, Prometheus, grafana and other services are ready. Here are some links, which can be used for reference when you make relevant deployment:

  1. Build k8s:Kube spray 2.11 installing kubernetes1.15
  2. Build Helm:Deploying and experiencing helm (version 2.16.1)
  3. Building Prometheus and grafana:Rapid deployment of Prometheus and grafana in kubernetes 1.15
  4. Deployment of Kafka:Rapid deployment of Kafka in k8s environment

Version information

  1. Kubernetes:1.15
  2. Kubernetes host: CentOS Linux release 7.7.1908
  3. NFS service: IP address, folder / Volume1 / NFS storageclass test
  4. Helm:2.16.1
  5. Kafka:2.0.1
  6. Zookeeper:3.5.5
  7. Prometheus:2.0.0
  8. Grafana:5.0.0

After the preparation, we can start the actual combat;

Confirm the Kafka exporter parameter

The Kafka exporter to be deployed needs to obtain data from Kafka, so prepare Kafka exporter parameters;

  1. Check the Kafka service with the type of clusterip, as shown in the red box below:
  2. The service name in the red box above iskafka, port is9092Therefore, the Kafka information configured in Kafka exporter later iskafka:9092

Practical operation

  1. Add helm warehouse (the warehouse has Kafka exporter we need)helm repo add gkarthiks https://gkarthiks.github.io/helm-charts
  2. Download Kafka exporter:helm fetch gkarthiks/prometheus-kafka-exporter
  3. Unzip the downloaded chart file:tar -zxvf prometheus-kafka-exporter-0.1.0.tgz
  4. Enter the extracted Directory:cd prometheus-kafka-exporter
  5. modify values.yaml File, as shown in the red box below,kafka:9092This is the address to access Kafka in the same namespace:
  6. stay values.yaml File directory execution:helm install –name-template kafka-exporter -f values.yaml . –namespace kafka-test(Kafka’s namespace is Kafka test, and the namespace here should be consistent with Kafka)
  7. To see if the service and pod are ready:
  8. At this point, we have deployedkafka-exporterIt can provide monitoring data to Prometheus, but Prometheus does not know that it is coming here to collect data, so next, Prometheus should be configured;
  9. confirmkafka-exporterThe access address of the service is used by Prometheus to collect data. The name and port are shown in the red box below. Therefore, the access address across the namespace can be spliced:kafka-exporter-prometheus-kafka-exporter.kafka-test.svc.cluster.local:9308
  10. If you are a referenceRapid deployment of Prometheus and grafana in kubernetes 1.15Deployment Prometheus, please open theconfigmap.yamlThe file contains the configuration of Prometheus. If it is deployed through other ways, please find it according to your own deployment prometheus.yml The location of the;
  11. configmap.yamlIn this way, Prometheus can collect the data of Kafka exporter
  12. Make the configuration work:kubectl apply -f configmap.yaml
  13. At this time, the Prometheus container still uses the old configuration. In order to make the configuration effective, delete the Prometheus pod, so that the new pod created automatically by k8s uses the new configuration. Find out the Prometheus Pod:kubectl get pods -n kube-system
  14. Delete old Pod:kubectl delete pod prometheus-68545d4fd8-f5q2w -n kube-system
  15. Wait for the new pod to be created automatically;
  16. Next, log in to grafana, configure the monitoring page, as shown in the figure below, and do the template import operation:
  17. Enter 7589 on the import template page:
  18. When selecting a data source, select Prometheus, as shown in the following figure:
  19. At this point, if your Kafka has received and received messages, you can immediately see the data:
    So far, Kafka monitoring in k8s environment has been deployed. I hope this paper can give you some reference;

Welcome to the official account: programmer Xin Chen

Wechat search “programmer Xinchen”, I am Xinchen, looking forward to traveling with you in Java World