Linux uses SS command combined with ZABBIX to monitor sockets

Time:2021-12-8

preface

Here we use ZABBIX to monitor it. We use SS command instead of netstat command, because SS is much faster. If you don’t believe it, you can test it. The more sockets a machine has, the more obvious the comparison is. Moreover, the SS command can display more contents. In fact, I am not particularly familiar with these two commands. You can see through man SS:

1、 SS command

The SS command is used to display the socket status. It can display the statistics of packet sockets, TCP sockets, UDP sockets, DCCP sockets, raw sockets, UNIX domain sockets, etc. it displays more TCP and state information than other tools. It is a very practical, fast and effective new tool to track IP connections and sockets. The SS command can provide the following information:

  • All TCP sockets
  • All UDP sockets
  • All SSH / ftp / TTP / HTTPS persistent connections
  • All local processes connected to xserver
  • Use state (for example: connected, synchronized, syn-recv, syn-sent, time-wait), address and port filtering
  • All state fin-wait-1 tcpsocket connections and more

Many popular Linux distributions support SS and many monitoring tools use SS command. Being familiar with this tool will help you better find and solve system performance problems. I strongly recommend using SS command instead of some netstat commands, such as netsat – ant / LNT, etc

Direct SS command

Explain the above wave:

Netid #socket type. In the above example, there are TCP and U_ Str (UNIX stream) and other sockets
State # socket status. The following are all States and descriptions of TCP socket, which are actually all States of TCP's three handshakes and four waves
Recv-q # in the estab state indicates how many bytes of data in the kernel have not been read by the upper application. If the value here is large, the application may be blocked
Send-q # in the estab state indicates how many bytes of data in the kernel's send queue have not received the ack of confirmation. If this value is large, it indicates that the receiving and processing at the receiving end need to be strengthened
Local address: port # local address and port
Peer address: port # remote address and port

Then let’s look at the states above. If people who are particularly familiar with the network should understand it, at least I’m not particularly familiar with the state of three handshakes and four waves:

Listen # server listens on socket and waits for client connection
Syn-sent # client has sent socket connection request message, waiting for the connection to be received by the server
After receiving the connection request message, the syn-received # server waits for the reply message from the client confirming the connection
An effective connection has been successfully established between the established # server and the client to transmit data to each other
The fin-wait-1 # server or client calls the close function to actively send a request message for terminating the connection to the other party, while waiting for the other party's reply message confirming the termination of the connection
Fin-wait-2 # actively closes the connection, receives the reply message from the other party confirming the termination of the connection, and waits for the request message of the other party's connection termination. At this time, the state is the semi closed state of the TCP connection, which can accept data, but cannot send data
The close-wait # passive shutdown end sends a reply message confirming the termination of the connection to the active shutdown end after receiving the request message of the active shutdown end to terminate the connection. At the same time, the passive shutdown end waits for the local user to terminate the connection. At this time, the passive shutdown end is in the semi closed state of the TCP connection and can send data, but can not receive data
The closing # server and client send the connection termination request message (call the close function) to the other party at the same time, and both parties receive the connection termination request message sent by the other party before receiving the connection termination reply message sent by the other party. At this time, both parties enter the closing state. After entering the closing state, As long as you receive the reply message from the other party to terminate your connection, you will enter the time-wait state. Therefore, the duration of the closing state will be very short, which is generally difficult to capture
After sending all data, the last-ack # passive shutdown end sends a request message for terminating the connection to the active shutdown end and waits for the active shutdown end to send a reply message for terminating the connection
After the time-wait # active shutdown end receives the request message of the passive shutdown end to terminate the connection, it sends the reply message of terminating the connection to the passive shutdown end and waits for enough time to ensure that the passive shutdown end receives the reply message of terminating the connection sent by the active shutdown section
Closed # has no connection at all. The socket connection has been terminated

So how do these status SS commands correspond? (the following is the status information displayed by SS command)


[TCP_ESTABLISHED] = "ESTAB",
[TCP_SYN_SENT] = "SYN-SENT",
[TCP_SYN_RECV] = "SYN-RECV",
[TCP_FIN_WAIT1] = "FIN-WAIT-1",
[TCP_FIN_WAIT2] = "FIN-WAIT-2",
[TCP_TIME_WAIT] = "TIME-WAIT",
[TCP_CLOSE] = "UNCONN",
[TCP_CLOSE_WAIT] = "CLOSE-WAIT",
[TCP_LAST_ACK] = "LAST-ACK",
[TCP_LISTEN] =  "LISTEN",
[TCP_CLOSING] = "CLOSING",

Jiang can actually do the following monitoring here. Continue to look at the use of SS command.

Usage: ss [ OPTIONS ]

       ss [ OPTIONS ] [ FILTER ]

-h. -- help information
-5. -- version program version information
-n. -- numeric does not resolve the service name
-r. -- resolve resolve hostname
-a. -- all displays all sockets
-l. -- listening sockets showing listening status
-o. -- options display timer information
-e. -- extended displays detailed sockets information
-m. -- memory displays the memory usage of sockets
-p. -- processes displays the processes that use sockets
-i. -- info displays TCP internal information
-s. -- Summary displays an overview of socket usage
-4, - - IPv4 displays only IPv4 sockets
-6, - - IPv6 displays only IPv6 sockets
-0, - - packet displays the packet socket
-t. -- TCP displays only TCP sockets
-u. -- UDP displays only UCP sockets
-d. -- DCCP displays only DCCP sockets
-w. -- raw displays only raw sockets
-x. -- UNIX displays only UNIX sockets
-f. -- family = family displays sockets of family type. Family is optional and supports UNIX, INET, inet6, link and netlink
-A, --query=QUERY, --socket=QUERY
QUERY := {all|inet|tcp|udp|raw|unix|packet|netlink}[,QUERY]
-D. -- diag = file dumps the original TCP sockets information to a file
-F. -- filter = file removes filter information from the file
FILTER := [ state TCP-STATE ] [ EXPRESSION ]

Focus on the following monitoring

2、 ZABBIX monitors the overall socket status of the machine

Before doing this monitoring, you can be familiar with the awk command

The monitoring system used is ZABBIX. We will combine the template of ZABBIX (the template is selected here for later expansion) and the way of user-defined script for monitoring.

2.1. Script writing

I won’t say much. Let’s start with the script:

vim tcp_status.sh
#################Script content#################
#!/bin/bash
if [ $# -ne 1 ];then
    echo "Follow the script name with an argument "
fi

case $1 in

    LISTEN)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/LISTEN/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    ESTAB)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/ESTAB/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;


    CLOSE-WAIT)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/CLOSE-WAIT/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    TIME-WAIT)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/TIME-WAIT/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    SYN-SENT)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/SYN-SENT/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    SYN-RECV)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/SYN-RECV/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    FIN-WAIT-1)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/FIN-WAIT-1/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    FIN-WAIT-2)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/FIN-WAIT-2/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    UNCONN)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/UNCONN/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    LAST-ACK)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/LAST-ACK/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;

    CLOSING)
        result=`ss -ant | awk 'NR>1 {a[$1]++} END {for (b in a) print b,a[b]}' | awk '/CLOSING/{print $2}'`
        if [ "$result" == "" ];then
               echo 0
        else
           echo $result
        fi
        ;;
 esac

2.2. Configure the configuration file of ZABBIX agent

vim  zabbix_agent.conf
##############Add the following#################
Unsafeuserparameters = 1 # this parameter needs to be configured by the user-defined script
UserParameter=tcp.status[*],sh /home/zabbix/tcp_ Status.sh $1 # here is used to specify the script just written, followed by a parameter

Remember to restart ZABBIX agent after configuration

2.3. Configure ZABBIX template

Add item, trigger and graph to it

Add a new template, and then add item to it, as shown in the following figure

The tcp.status in the key in the figure above refers to the configuration userparameter = tcp.status [*], SH / home / ZABBIX / TCP in step 2_ status.sh \$1

Then, the content in brackets is the parameter for \ $1 to transfer parameters. The specific parameter is the unconn in [unconn]. These values correspond to each situation in the case in the monitoring script in the first step. It is basically completed here. No, let’s draw a diagram and add a graph to the template, as shown in the following figure:

The most important step is to add the host configured with the monitoring script (step 1) to the template. So far, the monitoring is completed. Let’s see the result diagram

3、 ZABBIX monitors the number of requests from each machine

The purpose of this monitoring is to see which machines access the target machine more frequently.

This monitoring adopts self discovery monitoring, which is a little more difficult than the one above. Why do you choose self discovery monitoring? Because the item is not certain, here you choose the original IP address and the destination IP address as the item. We monitor the destination IP address. This is unchanged, so the IP value of the original address will change, Therefore, the method of automatically generating items is used for monitoring, and automatically adding and deleting items is actually very useful. As long as you learn it, it is super simple

There are also three steps: writing scripts and configuring ZABBIX_ Agent.conf file to configure discovery

3.1 script writing

Two scripts are needed here, one for self discovery (JSON format needs to be output) and the other for item

vim tcp_monitory.sh
##################tcp_monitor.sh##################
#!/bin/bash
#Get the data and output it to the data.txt file. The format is: original address IP: Count: destination address IP
#In addition, the data with count less than 200 is filtered out. There is no socket status here. The eyebrows and beard are grasped. Individuals can improve according to specific needs
ip_addr=`ip addr | grep -w inet | grep -v  "127.0.0.1" | awk '{print $2}'| awk -F "/" '{print $1}'`
ss -ant | awk '{ print $5}'|grep -Ev '127.0.0.1' | cut -d ':' -f4 | awk -v ip_addr=$ip_addr 'NR>1 {++s[$1]} END {for(k in s)if(s[k]>=200){print k,s[k],ip_addr}}' | grep -E  "^([0-9]{1,3}\.){3}[0-9]" > /home/zabbix/data.txt

#Execute Python script to output JSON format,
python /home/zabbix/get_json.py

#####################################
#Here is get_ Content of json.py
##############get_json.py################
#!/usr/bin/env python
#coding=utf-8
import json

def create_json(path):
    json_list = []
    with open(path) as f:
        for line in f.readlines():
            dict = {}
            split = line.split(" ")
            dict["{#DES_IP}"] = split[0]
            //Dict ["{#link_count}"] = split [1] // this is optional
            dict["{#SOU_IP}"] = split[2][:-1]
            json_list.append(dict)
    sum = {}
    sum["data"] = json_list
    sum = json.dumps(sum)
    print sum


if __name__ == '__main__':
    path = "/home/zabbix/data.txt"
    create_json(path)

##############Split line: the above is a self discovery script###############
##############Split line: the following is the script related to item###############
vim  tcp_item.sh
##################tcp_item.sh####################
#!/bin/bash
export LANG="en_US.UTF-8"
path=/home/zabbix/data.txt
count=`cat $path | grep $1 | grep $2 | awk '{print $2}'`
[ 1"$count" -eq 1 ] && echo 0 || echo $count

When both scripts are done, you can ZABBIX_ Agent.conf is configured

3.2 configuration ZABBIX_ Agent.conf file

Add the following content to the configuration file:

Unsafeuserparameters = 1 # if it has been configured, it does not need to be configured

UserParameter=discovery.tcp_ monitor[*],sh /home/zabbix/tcp_ Monitor.sh # self discovery
UserParameter=alert.tcp_ count[*],sh /home/zabbix/tcp_ Item.sh $1 and $2 #item, where $1 and $2 are transfer parameters in item to distinguish items

3.3. Configure discovery, item, trigger and graph

Here, you still choose to configure on the ZABBIX template. Now add a discovery

Then configure item, trigger and graph on discovery

Configure item:

Des above_ IP,SOU_ The IP is derived from the python script in the self discovery script and is used for the output format. alter.tcp_ Count is userparameter = alert.tcp_ count[*],sh /home/zabbix/tcp_ Item. Sh \ $1 \ $2, followed by \ $1, \ $2 and des_ IP,SOU_ Generate a unique item corresponding to the IP.

After configuring item, you can configure trigger:

Next, continue to configure the graph

Finally, add the template to the machine and see the results

The above is a detailed explanation of how Linux uses SS command combined with ZABBIX to monitor sockets. For more information about Linux SS command ZABBIX socket monitoring, please pay attention to other related articles of developeppaer!

Recommended Today

Heavyweight Tencent cloud open source industry’s first etcd one-stop governance platform kstone

​ Kstone open source At the kubecon China Conference held by CNCF cloud native foundation on December 9, 2021,Tencent cloud container tke team released the open source project of kstone etcd governance platform. KstoneIt was initiated by the TKE team of Tencent cloud containerCloud native one-stop etcd governance project based on kubernetes。 The project originates […]