10 basic commands commonly used in Linux performance testing

Today, I’d like to introduce 10 basic commands commonly used to detect performance.

1. uptime

$ uptime
23:51:26 up 21:31, 1 user, load average: 30.02, 26.43, 19.02

The command can roughly see the overall load of the computer. The number after load average represents the average load of the computer in 1min, 5min and 15min respectively.

2. dmesg | tail

$ dmesg | tail  
[1880957.563150]perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0  
[...]  
[1880957.563400] Out of memory: Kill process18694 (perl) score246or sacrifice child  
[1880957.563408] Killed process18694 (perl) total-vm:1972392kB, anon-rss:1953348kB,file-rss:0kB  
[2320864.954447] TCP: Possible SYN floodingon port7001. Dropping request. Check SNMP counters.

Print the contents in the ring buffer of the kernel, which can be used to view some errors;

In the above example, the process 18694 is killed due to memory overrun and the TCP request is discarded. Dmesg can quickly determine whether there are problems that lead to abnormal system performance.

3. vmstat 1

$ vmstat1  
procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 rb swpd free buff cache siso    bibo   incs ussy idwast  
34 0 0 200889792 73708 591828 0 0 0 5 610961300  
32 0 0 200889920 73708 591860 0 0 0 592 132844282981100  
32 0 0 200890112 73708 591860 0 0 0 0 95012154991000  
32 0 0 200889568 73712 591856 0 0 0 48 119002459990000  
32 0 0 200890208 73712 591860 0 0 0 0 158984840981100

Print statistical information of process, memory, switch partition, IO and CPU;

The format of vmstat is as follows

vmstat [options] [delay [count]]

The first output of vmstat represents the average value from Startup to vmstat running. The remaining output is the average value within the specified time interval. In the above example, the delay value is set to 1. Except for the first output, the remaining output is counted once a second. If count is not set, it will be printed circularly all the time.

$ vmstat 10 3procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- 
r b   swpd free buff cache si so    bi bo   in cs us sy id wa st 
1  0      0 2527112 1086888 13720228    0    0     1    14    2    1  1  1 99  0  0 
0  0      0 2527156 1086888 13719856    0    0     0   104 3003 4901  0  0 99  0  0 
0  0      0 2526412 1086888 13719904    0    0     0    10 3345 4870  0  1 99  0  0

In the above example, delay is set to 10 and count is set to 3, which means that the average value in 10 seconds is printed for each line, and only three times are printed.

Columns to check

• r: Represents the number of processes that are running or waiting for CPU scheduling. Because this column does not contain I / O statistics, it can be used to detect whether the CPU is saturated. If the number in column R is greater than the number of CPU cores, the CPU is saturated.
• Free: current remaining memory;
• Si, so: the number of swap in and swap out of the swap partition. If the number of swap in and swap out is greater than 0, it means that the memory is insufficient;
• Us, sy, ID, Wa: statistical information of CPU, representing user time, system time (kernel), idle and wait I / O respectively. I / O processing time is included in system time, so if system time exceeds 20%, I / O may have bottleneck or exception;

4. mpstat -P ALL 1

$ mpstat -P ALL
Linux 3.10.0-229.el7.x86_64 (localhost.localdomain) 05/30/2018 _x86_64_ (16 CPU)
04:03:55 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
04:03:55 PM all 3.67 0.00 0.61 0.71 0.00 0.00 0.00 0.00 0.00 95.02
04:03:55 PM 0 3.52 0.00 0.57 0.76 0.00 0.00 0.00 0.00 0.00 95.15
04:03:55 PM 1 3.83 0.00 0.61 0.71 0.00 0.00 0.00 0.00 0.00 94.85
04:03:55 PM 2 3.80 0.00 0.61 0.60 0.00 0.00 0.00 0.00 0.00 94.99
04:03:55 PM 3 3.68 0.00 0.58 0.60 0.00 0.00 0.00 0.00 0.00 95.13
04:03:55 PM 4 3.54 0.00 0.57 0.60 0.00 0.00 0.00 0.00 0.00 95.30
[...]

This command is used to print statistics of each CPU once per second, and can be used to check whether the CPU scheduling is uniform.

5. pidstat 1

$ pidstat 1
Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015    _x86_64_ (32 CPU)

07:41:02 PM UID PID %usr %system  %guest %CPU CPU Command
07:41:03 PM 0         9    0.00    0.94    0.00    0.94     1  rcuos/0
07:41:03 PM 0      4214    5.66    5.66    0.00   11.32    15  mesos-slave
07:41:03 PM 0      4354    0.94    0.94    0.00    1.89     8  java
07:41:03 PM 0      6521 1596.23    1.89    0.00 1598.11    27  java
07:41:03 PM 0      6564 1571.70    7.55    0.00 1579.25    28  java
07:41:03 PM 60004     60154    0.94    4.72    0.00    5.66     9  pidstat

07:41:03 PM UID PID %usr %system  %guest %CPU CPU Command
07:41:04 PM 0      4214    6.00    2.00    0.00    8.00    15  mesos-slave
07:41:04 PM 0      6521 1590.00    1.00    0.00 1591.00    27  java
07:41:04 PM 0      6564 1573.00   10.00    0.00 1583.00    28  java
07:41:04 PM 108      6718    1.00    0.00    0.00    1.00     0  snmp-pass
07:41:04 PM 60004     60154    1.00    4.00    0.00    5.00     9  pidstat
^C

This command is used to print the CPU usage of each process, similar to the content displayed in the top command. The advantage of pidstat is that it can scroll through the running process, rather than clear the screen as top does.

In the above example, the CPU utilization of the two Java processes in the% CPU is 1590% and 1573% respectively, which means that the Java processes occupy 16 CPUs.

6. iostat -xz 1

Similar to vmstat, the first output is the sampling data from the system startup to statistics;

$ iostat -xz 1
Linux 3.13.0-49-generic (titanclusters-xxxxx) 07/14/2015  _x86_64_ (32 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle          
73.96    0.00    3.73    0.03    0.06   22.21

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
xvda 0.00     0.23    0.21    0.18     4.52     2.08    34.37     0.00    9.98   13.80    5.42   2.44   0.09
xvdb 0.01     0.00    1.02    8.94   127.97   598.53   145.79     0.00    0.43    1.78    0.28   0.25   0.25
xvdc 0.01     0.00    1.02    8.86   127.79   595.94   146.50     0.00    0.45    1.82    0.30   0.27   0.26
dm-0        0.00     0.00    0.69    2.32    10.47    31.69    28.01     0.01    3.23    0.71    3.98   0.13   0.04
dm-1        0.00     0.00    0.00    0.94     0.01     3.78     8.00     0.33  345.84    0.04  346.81   0.01   0.00
dm-2        0.00     0.00    0.09    0.07     1.35     0.36    22.50     0.00    2.55    0.23    5.62   1.78   0.03
[...]^C

Check column

• R / s, w / s, RKB / s, WKB / s, indicating the number of reads, writes, read Kbytes, and write Kbytes sent to the I / O device per second.
• This indicates that the average I / O waiting time of the application is greater than the expected value.
• Avgqu SZ is the average time that requests are sent to the I / O device. If the value is greater than 1, the I / O device may be saturated;
• %Util, the utilization rate of the device per second; if the utilization rate exceeds 60%, it indicates that the device has abnormal performance;

7. free -m

$ free -m            
      total used free shared buffers     cached
Mem: 245998      24545     221453         83         59        541
-/+ buffers/cache: 23944     222053
Swap: 0          0          0

Columns checked:

• buffers: For the buffer cache, used for block device I/O.
• cached: For the page cache, used by file systems.
• If buffers and cached are close to 0, it indicates that I / O utilization is too high and the system has performance problems.
• Linux will use free memory as cache, if the application needs to allocate memory, the system can quickly reclaim the memory occupied by cache, so free memory contains the part occupied by cache.

8. sar -n DEV 1

SAR is the abbreviation of system activity reporter.

-n { keyword [,… ]| all}, used to report network statistics. Keywords can be one or more of the following: dev, edev, NFS, NFSD, sock, IP, EIP, ICMP, eicmp, TCP, etcp, UDP, sock6, IP6, eip6, icmp6, eicmp6 and udp6.

• -N dev 1, statistics the network usage every second;
• -N edev 1, counts the wrong network information every second;

$ sar -n DEV 1
Linux 3.10.0-229.el7.x86_64 (localhost.localdomain) 05/31/2018  _x86_64_ (16 CPU)

03:54:57 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:54:58 PM ens32 3286.00   7207.00    283.34  18333.90      0.00      0.00      
0.003:54:58 PM lo      0.00      0.00      0.00      0.00      0.00      0.00      
0.0003:54:58 PM vethe915e51 0.00      0.00      0.00      0.00      0.00      0.00     
0.0003:54:58 PM docker0 0.00      0.00      0.00      0.00      0.00      0.00     

0.0003:54:58 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
03:54:59 PM ens32 3304.00   7362.00    276.89  18898.51      0.00      0.00      
0.0003:54:59 PM lo      0.00      0.00      0.00      0.00      0.00      0.00      
0.0003:54:59 PM vethe915e51 0.00      0.00      0.00      0.00      0.00      0.00     
0.0003:54:59 PM docker0 0.00      0.00      0.00      0.00      0.00      0.00      0.00
^C

• Iface, network interface name;
• Rxpck / s, packets received per second;
• Txpck / s, the number of messages transmitted per second; (transmit packages)
• Rxkb / s, the number of kilobytes received per second;
• Txkb / s, the number of kilobytes sent per second;
• Rxcmp / s, the number of compressed packets received per second;
• Txcmp / s, the number of compressed packets sent per second;
• Rxmcst / s, the number of group packets received per second;

9. sar -n TCP,ETCP 1

This command can be used to roughly judge the network throughput, such as the number of network connections initiated and received;

• TCP, report the statistical information about tcpv4 network traffic;
• Etcp reports the statistical information about tcpv4 network errors;

$ sar -n TCP,ETCP 1
Linux 3.10.0-514.26.2.el7.x86_64 (aushop) 05/31/2018 _x86_64_ (2 CPU)

04:16:27 PM active/s passive/s iseg/s oseg/s04:16:44 PM 0.00 2.00 15.00 13.00
04:16:45 PM 0.00 3.00 126.00 203.00
04:16:46 PM 0.00 0.00 99.00 99.00
04:16:47 PM 0.00 0.00 18.00 9.00
04:16:48 PM 0.00 0.00 5.00 6.00
04:16:49 PM 0.00 0.00 1.00 1.00
04:16:50 PM 0.00 1.00 4.00 4.00
04:16:51 PM 0.00 3.00 171.00 243.00
^C

Columns detected:

• Active / s: number of locally initiated TCP connections per second (e.g., via connect()), the number of network connections initiated;
• Passive / s: number of remotely initiated TCP connections per second (e.g., via accept());
• Retransmissions / s: number of TCP retransmits per second;

10. top

The top command contains more index statistics, which is equivalent to a comprehensive command.

$ top
top - 00:15:40 up 21:56, 1 user, load average: 31.09, 29.87, 29.92
Tasks: 871 total, 1 running, 868 sleeping, 0 stopped, 2 zombie
%Cpu(s): 96.8 us, 0.4 sy, 0.0 ni, 2.7 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 25190241+total, 24921688 used, 22698073+free, 60448 buffers
KiB Swap: 0 total, 0 used, 0 free. 554208 cached Mem   

PID USER      PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 
20248 root 20   0  0.227t 0.012t 18748 S 3090  5.2  29812:58 java  
4213 root 20   0 2722544  64640  44232 S 23.5  0.0 233:35.37 mesos-slave 
66128 titancl+ 20   0   24344   2332   1172 R 1.0  0.0   0:00.07 top  
5235 root 20   0 38.227g 547004  49996 S 0.7  0.2   2:02.74 java  
4299 root 20   0 20.015g 2.682g 16836 S 0.3  1.1  33:14.42 java 
1 root 20   0   33620   2920   1496 S 0.0  0.0   0:03.82 init    
2 root 20   0       0      0      0 S 0.0  0.0   0:00.02 kthreadd 
3 root 20   0       0      0      0 S 0.0  0.0   0:05.35 ksoftirqd/0   
5 root 0 -20       0      0      0 S 0.0  0.0   0:00.00 kworker/0:0H  
6 root 20   0       0      0      0 S 0.0  0.0   0:06.94 kworker/u256:0   
8 root 20   0       0      0      0 S 0.0  0.0   2:38.05 rcu_sched

11. Summary

The following pictures show the main functions of each command, such as using vmstat to view the overall performance of the system, mpstat to view the performance of CPU, pidstat to view the status of process, iostat to view the status of IO, free to view the status of memory, SAR to view the status of network, etc.

Original text:https://medium.com/netflix-te…

If there are errors or other problems, welcome to comment and correct. If you have any help, please click like + forward to share.

Welcome to the official account of the brother of migrant workers:The way of migrant workers’ Technology