Build Prometheus + grafana cloud platform monitoring system

Time:2021-9-27


1.1 Prometheus and grafana overview
1.1.1 Prometheus overview
Prometheus (Prometheus) is an open-source combination of monitoring & Alarm & time series database, which was initially developed by Soundcloud company. In the most common docker, mesos and kubernetes container management systems, Prometheus is usually used for monitoring. Prometheus [pr əˈ mi θ Ju: s] Prometheus
The basic principle of Prometheus is to periodically capture the status of monitored components through HTTP protocol. The advantage of this is that any component can access the monitoring system as long as it provides HTTP interface, without any SDK or other integration process. This is very suitable for virtualized environments, such as VM or docker.
The HTTP interface that outputs the monitored component information is called an exporter. At present, most of the components commonly used by Internet companies can be directly used by exporters, such as varnish, haproxy, nginx, MySQL and Linux system information (including disk, memory, CPU, network, etc.). See the specific support sources:https://github.com/prometheus。 exporter ([ek ˈ sp ɔ: t ə (r) [exporter]
Compare the operation interface of mainstream monitoring tools:
Nagios monitoring interface:
Build Prometheus + grafana cloud platform monitoring system

ZABBIX monitoring interface

Grafana + Prometheus monitoring interface:

Through the above interface, we can see that the display interface of grafana + Prometheus is more beautiful
1.1.2 Prometheus architecture diagram
The overall architecture of Prometheus and some of its ecosystem components:

retrieval [r ɪˈ tri ː Vl] data retrieval
The operation process of Prometheus components is as follows:
1. Prometheus server: Prometheus sever is the core part of Prometheus component, which is responsible for obtaining, storing and querying monitoring data. Prometheus server can manage monitoring targets through static configuration, or use service discovery to dynamically manage monitoring targets and obtain data from these monitoring targets. Secondly, Prometheus sever needs to store the collected data. Prometheus server itself is a real-time database, which stores the collected monitoring data in the local disk in the way of time series. Prometheus server provides customized promql externally to realize data query and analysis. In addition, the federated clustering capability of Prometheus server enables it to obtain data from other Prometheus server instances.
2. Exporters: the exporter exposes the endpoint of monitoring data collection to Prometheus server in the form of HTTP service. Prometheus server can obtain the monitoring data to be collected by accessing the endpoint provided by the exporter. Exporters can be divided into two categories:
(1) . direct collection: this kind of exporter has built-in support for Prometheus monitoring, such as cdadvisor, kubernetes, etcd, gokit, etc., and has built-in endpoints for exposing monitoring data to Prometheus.
(2) Indirect acquisition: the original monitoring target does not directly support Prometheus, so the monitoring and acquisition program of the monitoring target needs to be written through the client library provided by Prometheus. For example, MySQL exporter, JMX exporter, consul exporter, etc.
3. Alertmanager: the Prometheus server supports the creation of alarm rules based on Prom QL. If the rules defined by Prom QL are met, an alarm will be generated. Common receiving methods are: e-mail, webhook, etc.
4. Pushgateway: the Prometheus data collection is based on the Prometheus server pulling data from the exporter. Therefore, when the network environment does not allow the Prometheus server to communicate with the exporter, the pushgateway can be used for transit.
Prometheus workflow:
1. Prometheus server regularly pulls metrics from configured jobs and exporters, or receives metrics sent from pushgateway, or pulls metrics from other Prometheus servers.
metrics [ ˈ metr ɪ Ks] metrics
2. Prometheus server stores the collected metrics locally and runs the defined alerts.rules to record new time series or push alerts to alert manager.
3. The alertmanager processes the received alarms and sends alarms according to the configuration file.
4. Collect data visually in the graphical interface.
Experimental topology:

Workflow of establishing Prometheus + grafana cloud platform monitoring system:
1. Install the exporter to collect data
2. Beautify the collected data
3. View in web interface
Understand: 1 don’t understand: 2
1.2 installing and configuring Prometheus monitoring service
1. Install go locale
Method 1: use tar package to install go locale
Upload to go1.13.3.linux-amd64.tar.gz to Linux server
Or download online:
[[email protected] ~]# wget -c https://storage.googleapis.co…
[[email protected] ~]# tar -zxvf go1.13.3.linux-amd64.tar.gz -C /usr/local
[[email protected] ~]# echo “export PATH=$PATH:/usr/local/go/bin” > /etc/profile.d/go.sh
[[email protected] ~]# source /etc/profile.d/go.sh
[[email protected] ~]# go

Method 2: install the go locale using Yum or up2date
[[email protected] ~]# yum install -y epel-release
[[email protected] ~]# yum install go -y
[[email protected] ~]# go version
Extension:
Go (also known as golang) is a programming language developed by Google, which is statically strongly typed, compiled, and hairstyle, and has garbage collection function.
Download and install Prometheus(https://prometheus.io/download/
[[email protected] ~]#wget https://github.com/prometheus…
[[email protected] ~]# tar zxvf prometheus-2.27.1.linux-amd64.tar.gz -C /usr/local/
[[email protected] ~]# cd /usr/local/prometheus-2.27.1.linux-amd64/
Modify the Prometheus configuration file to configure and monitor the Linux host and MySQL database. The configuration file is YML syntax. Pay attention to indent alignment
[ [email protected] Prometheus-2.7.1.linux-amd64]# VIM prometheus.yml # insert the following at the end of the document

  • job_name: system-status
    static_configs:

    • targets: [‘192.168.1.64:9100’]
      labels:
      instance: xuegod64
  • job_name: mysql-status
    static_configs:

    • targets: [‘192.168.1.64:9104’]
      labels:
      instance: xuegod64-mysql

When inserting, the job before the file and_ Keep name aligned, as shown in the following figure:

notes:

  • job_ Name: system status # every job_ Name is a monitoring item, which represents monitoring the operating system status
    static_configs:

    • Targets: [‘192.168.1.64:9100 ‘] # monitored host IP address and port
      labels:
      Instance: xuegod64 # instance name, which represents the corresponding host in grafana.
  • job_ Name: MySQL status # monitors the status of MySQL database
    static_configs:

    • Targets: [‘192.168.1.64:9104 ‘] # target host IP address and port
      labels:
      Instance: xuegod64 MySQL # instance name, which represents the corresponding host in grafana.

[[email protected]]# ./prometheus –config.file=prometheus.yml
Build Prometheus + grafana cloud platform monitoring system

The following message indicates successful startup.
Or run directly in the background:
[[email protected]]# nohup ./prometheus –config.file=prometheus.yml &
[ [email protected] ~]#Systemctl stop firewalld & & systemctl disable firewalld # turn off the firewall
Browser inputhttp://192.168.1.63:9090

Note: because we haven’t started configuring xuegod64, we can’t get the information of xuegod64 yet
1.3 start the exporter service for collecting Linux system and MySQL service status
Prometheus common exporters and functions:
(1)、node_ The exporter is used to monitor the performance and running status of the operating system
(2)、mysqld_ Exporter is used to monitor MySQL services
(3)、snmp_ Exporter is used to monitor network devices
More exporters can be downloaded from the official website:https://prometheus.io/downloa…

1. Configure node_ The exporter monitors the running status of xuegod64
[ [email protected] ~]#Systemctl stop firewalld & & systemctl disable firewalld # turn off the firewall
Upload node_ Exporter-1.1.2.linux-amd64.tar.gz to Linux.
[[email protected] ~]# tar zxvf /root/node_exporter-1.1.2.linux-amd64.tar.gz -C /usr/local/
[[email protected] ~]# nohup /usr/local/node_exporter-1.1.2.linux-amd64/node_exporter &
2. Configure mysqld_ The exporter is used to monitor MySQL services on xuegod64
(1) , installing database services
[[email protected] ~]# yum install mariadb-server mariadb -y
[[email protected] ~]# systemctl start mariadb
[ [email protected] ~]#MySQL # direct login to MySQL
mysql> GRANT REPLICATION CLIENT,PROCESS ON . TO ‘mysql_monitor’@’localhost’ identified by ‘123456’;
mysql> GRANT SELECT ON . TO ‘mysql_monitor’@’localhost’;
MariaDB [(none)]> exit;
Note: MySQL user permission Description:
(1) . replication client permission # with this permission, you can query the status of master server and slave server.
(2) . process permission # through this permission, users can execute show processlist and kill commands. By default, each user can execute the show processlist command, but can only query the user’s process. For example:
mysql> show processlist;
In addition, the management permission process cannot specify a database, and on must be followed by.
(3) . select permission # must have select permission before you can use select * from table to view data
(2) . install mysqld_ exporter

Upload mysqld_ Exporter to Linux system
[[email protected] ~]# tar xf mysqld_exporter-0.13.0-rc.0.linux-amd64.tar.gz -C /usr/local/
Create a hidden configuration file. My.cnf for mysqld_ The exporter connects to MYSQL to collect data.
[[email protected] ~]# vim /usr/local/mysqld_exporter-0.13.0-rc.0.linux-amd64/.my.cnf
[client]
user=mysql_monitor
password= 123456
[[email protected] ~]# nohup /usr/local/mysqld_exporter-0.13.0-rc.0.linux-amd64/mysqld_exporter –config.my-cnf=”/usr/local/mysqld_exporter-0.13.0-rc.0.linux-amd64/.my.cnf” &
In the web interface, check the monitoring information again. The Linux and MySQL nodes have been successfully added and have been up
Browser inputhttp://192.168.1.63:9090

View network ports:
[[email protected] ~]# netstat -antup | grep 91
tcp6 0 0 :::9100 :::* LISTEN 2158/node_exporter
tcp6 0 0 :::9104 :::* LISTEN 41308/mysqld_export
Summarize the usage of Prometheus:
1. To monitor the operating system, you only need one step:
Install node on the monitored system_ The exporter and start it, and it’s OK
2. To monitor and operate the MySQL database, you only need two steps:
(1) Create a MySQL user
(2) . install mysqld_ Exporter and start
1.4 actual combat – beautify the display effect of monitoring data with grafana visualization tool
1.4.1 configuring grafana
1. Upload grafana to Linux system / root /
[[email protected] ~]# yum install -y ./grafana-7.5.7-1.x86_64.rpm
Or download online:
[[email protected] ~]# wget https://mirrors.tuna.tsinghua…
2. Install graphic pie chart plug-in to facilitate the display of pie chart
Method 1: offline installation
Download:https://grafana.com/api/plugi…
Then upload grafana-piechart-panel-1.6.1.zip to Linux system
[ [email protected] ~]#MKDIR / var / lib / grafana / plugins # creates the path to store the plug-in
[[email protected] ~]# unzip grafana-piechart-panel-1.6.1.zip
[[email protected] ~]# mv grafana-piechart-panel /var/lib/grafana/plugins/grafana-piechart-panel
[[email protected] ~]# systemctl restart grafana-server
Method 2: online installation
[[email protected] ~]# grafana-cli plugins install grafana-piechart-panel
[ [email protected] ~]#Systemctl restart grafana server # after installing the plug-in, you need to restart the service
3. Access:http://192.168.1.63:3000/loginUser name: admin password admin

The first login will prompt to change the password. Let’s change it to 123456

Configure data sources

URL address:http://192.168.1.63:9090

At the bottom of the page, click Save & test

Scrape [skre ɪ p] Scraping; interval [ ˈɪ nt ə Vl] interval; Scratch interval
The following message pops up, indicating success

1.4.2 import the web dashboard plug-in monitoring the running status of Linux system and MySQL service into grafana
1. Find a dashboard plug-in that beautifies Linux system monitoring data

Build Prometheus + grafana cloud platform monitoring system

Open the official website:https://grafana.com/grafana/d…
Find this on the page:
https://grafana.com/grafana/d…
Enter the keyword node to query and view the display Kanban you want:

I use this ID: 8919
Open this link:https://grafana.com/grafana/d…

You can also download the JSON file directly, and then import the offline file directly later:
Build Prometheus + grafana cloud platform monitoring system

The ID number is 8919. Note: if the Kanban Imported Online cannot see the data, you can change an ID number and try again.
Start import:

Click load,

Refresh browser interface:

Note: you can also import the previously downloaded node exporter for Prometheus dashboard cn v20201010_ Rev24.json offline file upload.

2. Find out the ID number of MySQL’s Web dashboard plug-in
Open this link to directly search for the dashboard corresponding to the course
https://grafana.com/grafana/d…
Then select the interface developed by nasskach

Open link:https://grafana.com/grafana/d…ID of replication 7362
Build Prometheus + grafana cloud platform monitoring system

Go to the main interface of grafana and import the dashboard:

Build Prometheus + grafana cloud platform monitoring system

Build Prometheus + grafana cloud platform monitoring system

Drop down to select Prometheus

1.5 actual combat – grafana configuration mail alarm
1.5.1 configure SMTP to send mail
Configure 163 mailbox SMTP service

Click to open the SMTP service. After clicking to open, we will be prompted to download the mobile app. We click to continue to open without downloading the app.

Send the mobile phone verification code according to the prompt to open it successfully

EMXCERHXVHVDXKJF
After successful opening, the option of authorization password management will be added. If you have opened it before, click Add authorization password directly.

Build Prometheus + grafana cloud platform monitoring system

Grafana configuring SMTP accounts
[[email protected] ~]# vim /etc/grafana/grafana.ini
Line 522 modify SMTP configuration

Note: before modifying the configuration; The annotation symbol needs to be cancelled. Change the corresponding line to the following:
[smtp]
Enabled = true # enable SMTP
Host = smtp.163.com: 465 #163 SMTP service address
user = [email protected] #163 mailbox user name

If the password contains # or ; you have to wrap it with triple quotes. Ex “””#password;”””

Password = emxcerhxvhvdxkjf # authorization password
; cert_file =
; key_file =
; skip_verify = false
from_ address = [email protected] #The email from address is the same as the login user.
;from_name = Grafana
; ehlo_identity =
After modifying the email configuration, modify the domain information. The email content contains the access address of grafana. The default address is localhost.
Change:
41 ;domain = localhost
Is:
41 domain = 192.168.1.63
1.5.2 configure picture rendering function
Configure plug-in address
Change:
813 ;server_url =
Is:
813 server_url = http://192.168.1.63:8081/render
Configure callback_ URL, which is the address of grafana.
Change:
815 ;callback_url =
815 callback_url = http://192.168.1.63:3000/
Modify the default language. The default configuration does not support Chinese when rendering pictures.
Change:
840 ;rendering_language =
Is:
840 rendering_language = zh
Install the picture rendering plug-in (tested, only the plug-in built in docker mode supports Chinese rendering)
Install docker environment
[[email protected] ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
Configure Yum source for domestic docker (alicloud)
[[email protected] ~]# yum-config-manager –add-repo http://mirrors.aliyun.com/doc…
Install docker CE
[[email protected] ~]# yum makecache fast
[[email protected] ~]# yum install docker-ce docker-ce-cli containerd.io -y
[[email protected] ~]# systemctl start docker && systemctl enable docker.service
[[email protected] ~]#tee /etc/docker/daemon.json << ‘EOF’
{
“registry-mirrors”:[“https://rsbud4vc.mirror.aliyuncs.com”,”https://registry.docker-cn.com”,”https://docker.mirrors.ustc.edu.cn”,”https://dockerhub.azk8s.cn”,”http://hub-mirror.c.163.com”,”http://qtid6917.mirror.aliyuncs.com”,”https://rncxm540.mirror.aliyuncs.com”,”https://e9yneuy4.mirror.aliyuncs.com”]
}
EOF
[[email protected] ~]# systemctl daemon-reload
[[email protected] ~]# systemctl restart docker
Run the docker version of the picture rendering plug-in
Upload grafana-image-renderer.tar.gz in the course materials
[[email protected] ~]# docker load -i grafana-image-renderer.tar.gz
[[email protected] ~]# docker run -d -p 8081:8081 –restart=always index.docker.io/grafana/grafana-image-renderer:latest
Restart grafana for the configuration to take effect
[[email protected] ~]# systemctl restart grafana-server
Check whether the mail service is configured successfully and add an alarm channel.

Take any name, select email and fill in the recipient address to send the test email. Include image needs to be checked manually, so that the monitoring chart information will be attached when sending mail. In the new version, it is not checked by default because plug-ins need to be installed.

Click save after the test is successful.
Build Prometheus + grafana cloud platform monitoring system

Received email from grafana
Build Prometheus + grafana cloud platform monitoring system

The content is test information

1.5.3 user defined monitoring items and alarm rules
Configure the alarm and return to the instrument panel for operation.

Build Prometheus + grafana cloud platform monitoring system

Add panel
Build Prometheus + grafana cloud platform monitoring system

Modify a new title
Panel title: xuegod64 CPU utilization

Modify monitor item configuration. Delete monitoring item. Just keep the user configuration. Because we can only test user monitoring items by manual testing. Delete AFDC reservation B
metrics [ ˈ metr ɪ Ks] metrics
Metrics: (1 – avg(rate(node_cpu_seconds_total{instance=~”xuegod64″,mode=”idle”}[30s])) by (instance)) * 100
Legend: CPU utilization
Note: idle indicates the total load, system indicates the system utilization rate, and user indicates the user utilization rate.
Leave only this monitor parameter
Build Prometheus + grafana cloud platform monitoring system

Add alarm
Build Prometheus + grafana cloud platform monitoring system

Configure alarm rules.
Xuegod64 CPU usage alarm
Random output point alarm description information: xuegod64 CPU > 50% (test)

Note: the query time is the average data of 5 minutes by default. If you want to trigger the alarm quickly, the query time can be shortened to 1m. 0.5=50%
Save the configuration and click the button

Click Save to save
You can see this sign here:

Apply the following:

1.5.4 pressure test trigger alarm
Install the pressure test tool
[[email protected] ~]# yum install -y epel-release
[[email protected] ~]# yum install -y stress
Pressure test the CPU. Note: xuegod64 is allocated with 8 core CPUs in the teaching environment. If more than 50% alarms are configured for our monitoring indicators, more than half of the CPU resources need to be exhausted. Therefore, the – C parameter can be greater than half of the number of CPU cores in the virtual machine. Students can adjust it according to their own learning environment.
[[email protected] ~]# stress -c 6
Build Prometheus + grafana cloud platform monitoring system

Create a new terminal window and use the top command to view the system load
[[email protected] ~]# top

Build Prometheus + grafana cloud platform monitoring system

Six processes that occupy 100% of CPU have been created.
Viewing the status information, you can see that the interval between pending and sending mail is 1 minute.
Build Prometheus + grafana cloud platform monitoring system

View mailbox
Build Prometheus + grafana cloud platform monitoring system

Monitoring indicator data can be seen in the email

CTRL + C close the pressure test tool
Build Prometheus + grafana cloud platform monitoring system

Wait for the average load indicator to come down and receive the recovery message.

Summary:
1.1 Prometheus and grafana overview
1.2 installing and configuring Prometheus monitoring service
1.3 start the exporter service for collecting Linux system and MySQL service status
1.4 actual combat – beautify the display effect of monitoring data with grafana visualization tool
1.5 actual combat – grafana configuration mail alarm

Pay attention to the ‘little god’ not to get lost, and continuously update the technical dry goods for everyone every day
Build Prometheus + grafana cloud platform monitoring system