Docker advanced – container monitoring

Time:2019-12-2

Summary

The previous article introduced the use of docker compose composite application and the use of scale to quickly expand the container.

Since the services started by docker compose are all on the same host, when multiple container applications are running on one host, the operation of the container, such as CPU utilization, memory utilization, network status, disk space and a series of time-series data information that changes with time, needs to be understood, so monitoring is necessary.

Selection of container monitoring scheme

There are various monitoring schemes for containers, including docker stats command, scout, data dog, sysdig cloud, sensu monitoring framework, cadviser, etc.

Through the docker stats command, you can easily see the CPU, memory, network traffic and other data of all containers on the current host. But the disadvantage of docker stats command is that it only counts all containers of the current host, and the monitoring data obtained is real-time, without local storage and alarm function.

docker stats

While scout, sysdig cloud and data dog all provide relatively complete services, but they are all managed services and charged. The sensu monitoring framework is highly integrated and free, but the deployment is too complex. In a comprehensive consideration, we choose cadviser as the container monitoring tool.

Cadvisor is produced by Google. It has the advantages of open source products, complete monitoring indicators, convenient deployment, and official docker image. The disadvantage is that the integration is not high, and only 2 minutes of data is saved locally by default. However, with the data stored in influxdb and the graph displayed in grafana, the container monitoring system can be built conveniently. The effect of data collection and graph display is good, which has little impact on the system performance.

CAdvisor

Cadviser is a container resource monitoring tool, including container memory, CPU, network IO, disk IO and other monitoring, and provides a web page to view the real-time running status of the container. Cadviser stores 2 minutes of data by default, and only for a single physical machine. However, cadviser provides many data integration interfaces, which support the integration of influxdb, redis, Kafka, elasticsearch, etc. monitoring data can be sent to these databases for storage with corresponding configuration.

Cadviser has two main functions:

  • Display the monitoring data of host and container.
  • Display historical change data.

InfluxDB

Influxdb is an open source distributed timing, event and indicator database written in go language, without external dependency.

As mentioned earlier, cadviser only stores the data of the last two minutes on the local machine by default. In order to store data persistently and collect display monitoring data uniformly, data needs to be stored in the influxdb. Influxdb is a time-series database, which is specially used to store time-series related data. It is very suitable to store cadviser’s data. In addition, cadviser itself has provided the integration method of influxdb, and it is enough to specify the configuration when starting the container.

Main functions of influxdb:

  • Based on time series, time-dependent correlation functions (such as maximum, minimum, sum, etc.) are supported;
  • Measurability: you can calculate a large amount of data in real time;
  • Event based: it supports arbitrary event data;

Main features of influxdb:

  • No structure (no mode);
  • Can be any number of columns;
  • Expandable;
  • It supports a series of functions such as min, Max, sum, count, mean, and median, which is convenient for statistics;
  • Native HTTP support, built-in HTTP API;
  • Powerful SQL like syntax;
  • With management interface, easy to use

Granfana

Grafana is an open-source data monitoring and analysis visualization platform, which supports multiple data source configurations (including influxdb, mysql, elasticsearch, opentsdb, graphite, etc.) and rich plug-in and template functions, chart permission control and alarm.

Main features of grafan:

  • Flexible and rich graphic options
  • Can mix multiple styles
  • Support day and night mode
  • Multiple data sources

CAdvisor+InfluxDB+Granfana

Cadviser: responsible for collecting container data over time
Influxdb: responsible for storing timing data
Grafana: responsible for analyzing and displaying timing data

Installation and deployment

Deploy the influxdb service

docker run -d --name influxdb -p 8086:8086 \
-v /data/influxdb:/var/lib/influxdb \
--hostname=influxdb \
influxdb

Explain:

  • –Name: shipping container assignment Name: influxdb
  • -P: mapping port. Port 8083 is the management port of influxdb, and port 8086 is the data port of influxdb
  • -V: mount data – D: run this container in the background
  • Influxdb: it runs through this container. It will be pulled down in the official warehouse of docker by default

1. The browser accesses the management end of influxdb, http://ip:8083

2. Create the test database and root user, which is used for later configuration of granfa

docker exec it influxdb influx

CREATE DATABASE "test"
CREATE USER "root" WITH PASSWORD 'root' WITH ALL PRIVILEGES

Deploy cadviser service

Google’s cadviser can be used to collect the timing information of docker containers, including resource usage and performance data during container operation.

docker run --volume=/:rootfs:ro --volume=/var/run:/var/run:ro\
--volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro\
--volume=/dev/disk/:/dev/disk:ro --volume=/cgroup:/cgroup:ro \
--publish=8080:8080 --detach=true --privileged=true --name=cadvisor \
google/cadvisor:latest\
-storage_driver=influxdb-storage_driver_host=influxdb:8086\
-storage_driver_db=test-storage_driver_user=root\
-storage_driver_password=root

Explain:

  • -D: run this container in the background
  • –Name: start container assignment name cadviser
  • -P: Map port 8080
  • –Mount: bind the related directories of the host to the container. These directories are the directory files and monitoring contents that cadviser needs to collect
  • -Storage driver: you need to specify the storage driver, database host and database name of cadviser

Pass the IP: 8080 port access test

Deploy granana services

Grafana is an open-source time series data analysis tool, and the interface is professional and simple.

docker run -d -p 3000:3000 \
-v /data/grafana:/var/lib/grafana \
--link=influxdb:influxdb\
--name grafana  grafana/grafana

Note: if you use – V to mount the data, the container fails to start. The error is MKDIR: cannot create directory ‘/ var / lib / grafana / plugins’: permission denied. At this time, you can use docker run — entrypoint “Id” grafana / grafana to get uid, GID, groups (472 by default);

Then use chown – R 472:472 / data / grafana / to modify the directory permission to start successfully.

Access the web service of grafana through the IP: 3000 port. The first access needs to modify the admin user password. The default user name / password is admin / Admin

Docker compose rapid deployment

version: '3.1'

volumes:
  grafana_data: {}

services:
 influxdb:
  image: tutum/influxdb:0.9
  #image: tutum/influxdb
  #image: influxdb
  restart: always
  #user: 
  environment:
    - PRE_CREATE_DB=cadvisor
  ports:
    - "8083:8083"
    - "8086:8086"
  expose:
    - "8090"
    - "8099"
  volumes:
    - ./data/influxdb:/data

 cadvisor:
  #image: google/cadvisor:v0.29.0
  image: google/cadvisor
  links:
    - influxdb:influxsrv
  command: -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=influxsrv:8086
  restart: always
  ports:
    - "8080:8080"
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

 grafana:
  #image: grafana/grafana:2.6.0
  user: "104"
  image: grafana/grafana
  user: "104"
  #user: "472"
  restart: always
  links:
    - influxdb:influxsrv
  ports:
    - "3000:3000"
  volumes:
    - grafana_data:/var/lib/grafana
  environment:
    - HTTP_USER=admin
    - HTTP_PASS=admin
    - INFLUXDB_HOST=influxsrv
    - INFLUXDB_PORT=8086
    - INFLUXDB_NAME=cadvisor
    - INFLUXDB_USER=root
    - INFLUXDB_PASS=root

Start docker compose file

docker-compose up

Check whether the three service containers are started

Docker advanced - container monitoring

Browse grafana services, http://ip:3000

Docker advanced - container monitoring

Log in to grafana service

Docker advanced - container monitoring

Browse the influxdb service, http: / / IP: 8083/

Docker advanced - container monitoring

Configure the grafana tool

1. Configure data source

Docker advanced - container monitoring

Choose an influxdb data source

Docker advanced - container monitoring

The specific configuration is as follows

Docker advanced - container monitoring
Docker advanced - container monitoring

After configuration, as shown below

Docker advanced - container monitoring

2. Configuration panel

Docker advanced - container monitoring

Select panel type

Docker advanced - container monitoring

Select graphics

Docker advanced - container monitoring

Configure query criteria

Docker advanced - container monitoring

After configuration, the effect is as follows

Docker advanced - container monitoring

Grafana supports query conditions, which can be found in influxdb

Docker advanced - container monitoring

Browse C advisor service, http://ip:8080/

Docker advanced - container monitoring