Build Prometheus system resources and Kafka monitoring

Time:2021-12-5

<meta charset=”utf-8″>

What is Prometheus?

Prometheus is an open source monitoring and alarm system and time series database (TSDB) developed by Soundcloud. Prometheus is developed in go language and is an open source version of Google borgmon monitoring system.
In 2016, Google launched the cloud native Computing Foundation under the Linux foundation to include Prometheus into its second largest open source project.
Prometheus is currently active in the open source community.
Prometheus and heapster (heapster is a sub project of k8s to obtain cluster performance data) have more complete and comprehensive functions than each other. Prometheus also has enough performance to support tens of thousands of clusters.

Characteristics of Prometheus

  • Multidimensional data model.
  • Flexible query language.
  • Independent of distributed storage, a single server node is autonomous.
  • The timing data is collected by HTTP based pull.
  • The time sequence data can be pushed through the intermediate gateway.
  • Discover the target service object through service discovery or static configuration.
  • Support a variety of charts and interface displays, such as grafana.

Official website address:https://prometheus.io/

Architecture diagram

Build Prometheus system resources and Kafka monitoring
Build Prometheus system resources and Kafka monitoring

Basic principles

The basic principle of Prometheus is to periodically capture the status of the monitored components through the HTTP protocol. Any component can access the monitoring as long as it provides the corresponding HTTP interface. No SDK or other integration process is required. This is very suitable for virtual environment monitoring systems, such as VM, docker, kubernetes, etc. The HTTP interface that outputs the monitored component information is called an exporter. At present, most of the components commonly used by Internet companies have exporters that can be used directly, such as varnish, haproxy, nginx, MySQL and Linux system information (including disk, memory, CPU, network, etc.).

Service process

  • Prometheus daemon is responsible for fetching metrics data on the target regularly. Each fetching target needs to expose an HTTP service interface to fetch it regularly. Prometheus supports specifying the capture target through configuration file, text file, zookeeper, consul, DNS SRV lookup, etc. Prometheus uses the pull method for monitoring, that is, the server can push data directly through the target pull data or indirectly through the intermediate gateway.
  • Prometheus stores all the captured data locally, cleans and arranges the data through certain rules, and stores the obtained results in a new time series.
  • Prometheus visually displays the collected data through promql and other APIs. Prometheus supports chart visualization in many ways, such as grafana, its own promdash and its own template engine. Prometheus also provides the query method of HTTP API to customize the required output.
  • Pushgateway supports the client to actively push metrics to pushgateway, while Prometheus just grabs data from the gateway regularly.
  • Alertmanager is a component independent of Prometheus. It can support Prometheus query statements and provide very flexible alarm methods.

Introduction to this tutorial

  • 1. Demonstrate the installation of Prometheus server
  • 2. Demonstrate node exporter and JMX_ prometheus_ Java agent is installed and used to monitor Linux system resources and Kafka indicators respectively
  • 3. Demonstrate the use of grafana

1、 Prometheus installation

  • Download prometheus-2.18.1.linux-amd64.tar.gz from the official website and unzip it
[[email protected] prometheus]# ll
total 148584
drwxr-xr-x  2 3434 3434     4096 May  8  2020 console_libraries
drwxr-xr-x  2 3434 3434     4096 May  8  2020 consoles
drwxr-xr-x 16 root root     4096 Nov 10 13:00 data
drwxr-xr-x  2 root root     4096 Nov  4 16:28 lib
-rw-r--r--  1 3434 3434    11357 May  8  2020 LICENSE
-rw-r--r--  1 3434 3434     3184 May  8  2020 NOTICE
-rwxr-xr-x  1 3434 3434 87173843 May  8  2020 prometheus
-rw-r--r--  1 3434 3434     1209 Nov  4 16:41 prometheus.yml
-rwxr-xr-x  1 3434 3434 49973547 May  8  2020 promtool
-rwxr-xr-x  1 3434 3434 14957614 May  8  2020 tsdb
  • Modify the configuration file prometheus.yml

The configuration is as follows

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']
  • Run Prometheus
./prometheus

visithttp://10.4.4.16:9090/graphAs shown in the figure below, and indicate successful installation

Build Prometheus system resources and Kafka monitoring

image.png

2、 Exporter installation and use

  1. node_ Exporter installation and operation
  • Download node_ Exporter-1.0.1.linux-amd64.tar.gz and unzip it
[[email protected] node_exporter]# ll
total 19220
-rw-r--r-- 1 3434 3434    11357 Jun 16 21:19 LICENSE
-rwxr-xr-x 1 3434 3434 19657731 Jun 16 20:44 node_exporter
-rw------- 1 root root       47 Sep 30 03:02 nohup.out
-rw-r--r-- 1 3434 3434      463 Jun 16 21:19 NOTICE
  • function
./node_exporter
  1. kafka_ Exporter installation and operation

Mode 1: non intrusive Kafka monitoring (recommended)

  • Download Kafka_ Exporter-1.2.0.linux-amd64.tar.gz and unzip it
[[email protected] kafka_exporter]# ll
total 13276
-rwxr-xr-x 1 2000 2000 13578776 Jul  7  2018 kafka_exporter
-rw-rw-r-- 1 2000 2000    11357 Jul  7  2018 LICENSE
  • function
./kafka_exporter --kafka.server=127.0.0.1:9092

Mode 2: intrusive Kafka monitoring exporter installation (it is complex and can be skipped)

jmx_ prometheus_ Javaagent installation and operation

  • Download binary JMX_ prometheus_ Javaagent-0.14.0.jar and configuration file kafka-0-8-2.yml this tutorial places the binary files in the / opt / Prometheus / lib directory
[[email protected] prometheus]# ll
total 148584
drwxr-xr-x  2 3434 3434     4096 May  8  2020 console_libraries
drwxr-xr-x  2 3434 3434     4096 May  8  2020 consoles
drwxr-xr-x 17 root root     4096 Nov 10 15:00 data
drwxr-xr-x  2 root root     4096 Nov  4 16:28 lib
-rw-r--r--  1 3434 3434    11357 May  8  2020 LICENSE
-rw-r--r--  1 3434 3434     3184 May  8  2020 NOTICE
-rwxr-xr-x  1 3434 3434 87173843 May  8  2020 prometheus
-rw-r--r--  1 3434 3434     1209 Nov  4 16:41 prometheus.yml
-rwxr-xr-x  1 3434 3434 49973547 May  8  2020 promtool
-rwxr-xr-x  1 3434 3434 14957614 May  8  2020 tsdb
[[email protected] prometheus]# ll lib
total 412
-rw-r--r-- 1 root root 413862 Nov  4 16:25 jmx_prometheus_javaagent-0.14.0.jar
-rw-r--r-- 1 root root   2820 Nov  4 16:28 kafka-0-8-2.yml
  • Modify the script that starts Kafka and add the following
export KAFKA_OPTS="$KAFKA_OPTS -javaagent:/opt/prometheus/lib/jmx_prometheus_javaagent-0.14.0.jar=7071:/opt/prometheus/lib/kafka-0-8-2.yml"
  • Start Kafka
[[email protected] prometheus]# ps axu|grep kafka
root     16400  0.0  0.0 103320   896 pts/2    S+   15:08   0:00 grep kafka
root     26565  3.9 34.4 13512408 5680568 pts/3 Sl  Nov04 341:41 /opt/local/jdk/bin/java -Xmx6G -Xms2G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/logs/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/logs/kafka -Dlog4j.configuration=file:bin/../config/log4j.properties -cp :/opt/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b05.jar:/opt/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/local/kafka/bin/../libs/connect-api-0.10.2.1.jar:/opt/local/kafka/bin/../libs/connect-file-0.10.2.1.jar:/opt/local/kafka/bin/../libs/connect-json-0.10.2.1.jar:/opt/local/kafka/bin/../libs/connect-runtime-0.10.2.1.jar:/opt/local/kafka/bin/../libs/connect-transforms-0.10.2.1.jar:/opt/local/kafka/bin/../libs/guava-18.0.jar:/opt/local/kafka/bin/../libs/hk2-api-2.5.0-b05.jar:/opt/local/kafka/bin/../libs/hk2-locator-2.5.0-b05.jar:/opt/local/kafka/bin/../libs/hk2-utils-2.5.0-b05.jar:/opt/local/kafka/bin/../libs/jackson-annotations-2.8.0.jar:/opt/local/kafka/bin/../libs/jackson-annotations-2.8.5.jar:/opt/local/kafka/bin/../libs/jackson-core-2.8.5.jar:/opt/local/kafka/bin/../libs/jackson-databind-2.8.5.jar:/opt/local/kafka/bin/../libs/jackson-jaxrs-base-2.8.5.jar:/opt/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.8.5.jar:/opt/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.8.5.jar:/opt/local/kafka/bin/../libs/javassist-3.20.0-GA.jar:/opt/local/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/local/kafka/bin/../libs/javax.inject-1.jar:/opt/local/kafka/bin/../libs/javax.inject-2.5.0-b05.jar:/opt/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/local/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/local/kafka/bin/../libs/jersey-client-2.24.jar:/opt/local/kafka/bin/../libs/jersey-common-2.24.jar:/opt/local/kafka/bin/../libs/jersey-container-servlet-2.24.jar:/opt/local/kafka/bin/../libs/jersey-container-servlet-core-2.24.jar:/opt/local/kafka/bin/../libs/jersey-guava-2.24.jar:/opt/local/kafka/bin/../libs/jersey-media-jaxb-2.24.jar:/opt/local/kafka/bin/../libs/jersey-server-2.24.jar:/opt/local/kafka/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jetty-http-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jetty-io-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jetty-security-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jetty-server-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jetty-util-9.2.15.v20160210.jar:/opt/local/kafka/bin/../libs/jopt-simple-5.0.3.jar:/opt/local/kafka/bin/../libs/kafka_2.11-0.10.2.1.jar:/opt/local/kafka/bin/../libs/kafka_2.11-0.10.2.1-sources.jar:/opt/local/kafka/bin/../libs/kafka_2.11-0.10.2.1-test-sources.jar:/opt/local/kafka/bin/../libs/kafka-clients-0.10.2.1.jar:/opt/local/kafka/bin/../libs/kafka-log4j-appender-0.10.2.1.jar:/opt/local/kafka/bin/../libs/kafka-streams-0.10.2.1.jar:/opt/local/kafka/bin/../libs/kafka-streams-examples-0.10.2.1.jar:/opt/local/kafka/bin/../libs/kafka-tools-0.10.2.1.jar:/opt/local/kafka/bin/../libs/log4j-1.2.17.jar:/opt/local/kafka/bin/../libs/lz4-1.3.0.jar:/opt/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/local/kafka/bin/../libs/reflections-0.9.10.jar:/opt/local/kafka/bin/../libs/rocksdbjni-5.0.1.jar:/opt/local/kafka/bin/../libs/scala-library-2.11.8.jar:/opt/local/kafka/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/opt/local/kafka/bin/../libs/slf4j-api-1.7.21.jar:/opt/local/kafka/bin/../libs/slf4j-log4j12-1.7.21.jar:/opt/local/kafka/bin/../libs/snappy-java-1.1.2.6.jar:/opt/local/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/local/kafka/bin/../libs/zkclient-0.10.jar:/opt/local/kafka/bin/../libs/zookeeper-3.4.9.jar -javaagent:/opt/prometheus/lib/jmx_prometheus_javaagent-0.14.0.jar=7071:/opt/prometheus/lib/kafka-0-8-2.yml kafka.Kafka /opt/local/kafka/config/server.properties
  1. Modify prometheus.yml and add Linux resource monitoring and Kafka monitoring
- job_name: linux
    static_configs:
      - targets: ['10.4.4.16:9100']
        labels:
          instance: node

  - job_ Name: 'Kafka exporter' # mode 1, non intrusive (recommended)
    static_configs:
    - targets: ['10.4.4.16:9308']

  - job_ Name: 'Kafka' # mode 2, intrusive
    static_configs:
    - targets: ['10.4.4.16:7071']
  1. Restart the Prometheus program

  2. visithttp://10.4.4.16:9090/targets
    It can be seen that Linux and Kafka monitoring are added, as shown in the red box below:

    Build Prometheus system resources and Kafka monitoring

    image.png

3、 Installation and use of grafana

  • Download and unzip grafana-7.2.0.linux-amd64.tar.gz
[[email protected] grafana]# ll
total 48
drwxr-xr-x  2 root root  4096 Sep 23 20:19 bin
drwxr-xr-x  3 root root  4096 Sep 23 20:19 conf
drwxr-xr-x  5 root root  4096 Nov 10 15:16 data
-rw-r--r--  1 root root 11343 Sep 23 20:16 LICENSE
-rw-r--r--  1 root root   108 Sep 23 20:16 NOTICE.md
drwxr-xr-x  4 root root  4096 Sep 23 20:19 plugins-bundled
drwxr-xr-x 12 root root  4096 Sep 23 20:19 public
-rw-r--r--  1 root root  2799 Sep 23 20:16 README.md
drwxr-xr-x  2 root root  4096 Sep 23 20:19 scripts
-rw-r--r--  1 root root     5 Sep 23 20:19 VERSION
  • Run grafana
[[email protected] grafana]# ps axu|grep graf
root     18047  0.0  0.0 103320   900 pts/2    S+   15:20   0:00 grep graf
root     24002  0.0  0.2 1695092 38780 ?       Sl   Sep30  45:16 ./grafana-server

visithttp://10.4.4.16:3000/loginThe following message appears, indicating that the installation is successful. The default login user name / password is admin / Admin

Build Prometheus system resources and Kafka monitoring

image.png
  • Add the data source as Prometheus. Note that the URL is the server address plus 9090 port

    Build Prometheus system resources and Kafka monitoring

    image.png
  • Download and import the dashboard JSON file from the official website

    Build Prometheus system resources and Kafka monitoring

    image.png
  • View the node exporter as follows:

    Build Prometheus system resources and Kafka monitoring

    image.png
  • View the Kafka exporter as follows:

Method 1 Result:

Build Prometheus system resources and Kafka monitoring

image.png

Result of mode 2:

Build Prometheus system resources and Kafka monitoring

image.png