How to perform Linux performance analysis in 60 seconds

Time:2021-8-30

When you log in to a Linux server and need to do performance analysis because of a problem: what tests will you do in the first minute?
Linux命令
In Netflix, we have many EC2 Linux machines, and we also need many performance analysis tools to monitor and check their performance. It includes atlas, a monitoring tool for the cloud, and vector for case analysis as needed. Although these tools can help us solve most problems, we sometimes need to log in to the machine instance to run some standard Linux performance analysis tools.

First 60 seconds: summary

In this article, Netflix’s performance analysis engineer team will show you how to use the existing Linux standard tools for performance optimization detection in command line mode in the first 60 seconds. You only need to run the following 10 commands in 60 seconds to have a high level of understanding of system resource usage and running process. Look for error information and saturation indicators, and can be displayed as the length of the request queue or the waiting time. Because they are easy to understand, and then resource utilization. Saturation means that a resource has exceeded its own load capacity.

uptimedmesg | tailvmstat 1mpstat -P ALL 1pidstat 1iostat -xz 1free -msar -n DEV 1sar -n TCP,ETCP 1top

Some commands need to be installedsysstattool kit. The indicators displayed by these commands will help you complete some use (utilization, saturation, errors) methods: the methodology of locating performance bottlenecks. It includes checking utilization, saturation, and error indicators for all resources (such as CPU, memory, disk, etc.). Also pay attention to when you check and eliminate a resource problem, because exclusion can narrow the scope of analysis and guide any subsequent inspection.

The following sections will introduce these commands through an example in a production system. To learn more about these tools, you can also check their help manuals.

1. uptime

$ uptime23:51:26 up 21:31, 1 user, load average: 30.02, 26.43, 19.02

This is a quick way to show the average load of the system, which also indicates the number of processes waiting to run. In Linux systems, these numbers include the number of processes waiting for the CPU to run, as well as those blocked by non interruptible I / O (usually disk I / O). This gives a very direct display of resource load, which can better understand these data without the help of other tools. It is the only quick way to view the system load.

These three figures are the average of the constants in the past 1 minute, 5 minutes and 15 minutes in a decreasing manner. These three numbers show us how the system load changes over time. For example, if you are called to view a problem server and the value represented by 1 minute is much lower than that of 15 minutes, you may miss the time point of the problem because you log in to the machine too late.

In the above example, the average load shows that it is increasing. The value of 1 minute is 30, which is increased compared with the value of 19 for 15 minutes. This number is so large that something has happened: it may be the CPU demand;vmstatperhapsmpstatIt will help confirm what it is. These commands will be introduced in the third and fourth commands in this series.

2. dmesg | tail

$ dmesg | tail[1880957.563150] perl invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0[...][1880957.563400] Out of memory: Kill process 18694 (perl) score 246 or sacrifice child[1880957.563408] Killed process 18694 (perl) total-vm:1972392kB, anon-rss:1953348kB, file-rss:0kB[2320864.954447] TCP: Possible SYN flooding on port 7001. Dropping request.  Check SNMP counters.

The last 10 system message logs are displayed here. If there are no system messages, they will not be displayed. It mainly depends on the errors caused by performance problems. The above example contains the problem of killing the oom process and discarding TCP requests.

So remember to use this command. The dmesg command is worth using.

3. vmstat 1

$ vmstat 1procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu----- r  b swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st34  0    0 200889792  73708 591828    0    0     0     5    6   10 96  1  3  0  032  0    0 200889920  73708 591860    0    0     0   592 13284 4282 98  1  1  0  032  0    0 200890112  73708 591860    0    0     0     0 9501 2154 99  1  0  0  032  0    0 200889568  73712 591856    0    0     0    48 11900 2459 99  0  0  0  032  0    0 200890208  73712 591860    0    0     0     0 15898 4840 98  1  1  0  0^C

For a brief presentation of virtual memory statistics, vmstat is a common tool (first created for BSD decades ago). It prints a statistical summary of key service information on each line.

When vmstat runs with parameter 1, it prints statistics every 1 second. In this version of vmstat, the first line of output shows the average value since startup, not the statistics of the previous second. So now, you can skip the first line unless you want to see the meaning of the header field.

Meaning description of each column:

  1. r: The number of runnable processes waiting to run on the CPU. This metric provides data to determine CPU saturation because it does not include I / O waiting processes. It can be explained that when the value of “R” is larger than the number of CPUs, it is saturated.
  2. Free: free memory, in K. If this number is large, you still have enough free memory“ “Free – M” and the seventh command below can analyze the state of free memory in more detail.
  3. Si, so: the amount of data exchanged in and out. If these two values are non-zero, then there is no memory.
  4. Us, sy, ID, WA, ST: These are the decomposition of CPU time and the average value of all CPUs. They are user time, system time (kernel), idle time, waiting I / O time, and stolen time (mainly referring to other customers, or using Xen, which have their own independent operating domains).

The decomposition of CPU time can help determine whether the CPU is very busy (judged by the accumulation of user time and system time). Continuous I / O waiting indicates that the disk is the bottleneck. In this case, the CPU is relatively idle because tasks are blocked by waiting for disk I / O. You can think of waiting for I / O as another form of CPU idle, and this command gives clues to why they are idle.

System time is necessary for I / O processing. If the average system time consumption is higher than 20%, it is necessary to further explore and analyze: it may also be caused by the insufficient I / O efficiency of the kernel.

In the above example, CPU time is almost user level, indicating that this is an application level usage. If the CPU utilization exceeds 90% on average. This is not necessarily a problem; You can use the “R” column to check saturation.

4. mpstat -P ALL 1

$ mpstat -P ALL 1Linux 3.13.0-49-generic (titanclusters-xxxxx)  07/14/2015  _x86_64_ (32 CPU)07:38:49 PM  CPU   %usr  %nice   %sys %iowait   %irq  %soft  %steal  %guest  %gnice  %idle07:38:50 PM  all  98.47   0.00   0.75    0.00   0.00   0.00    0.00    0.00    0.00   0.7807:38:50 PM    0  96.04   0.00   2.97    0.00   0.00   0.00    0.00    0.00    0.00   0.9907:38:50 PM    1  97.00   0.00   1.00    0.00   0.00   0.00    0.00    0.00    0.00   2.0007:38:50 PM    2  98.00   0.00   1.00    0.00   0.00   0.00    0.00    0.00    0.00   1.0007:38:50 PM    3  96.97   0.00   0.00    0.00   0.00   0.00    0.00    0.00    0.00   3.03[...]

This command prints the time statistics of each CPU to see whether the overall CPU usage is balanced. If you have a CPU with significantly higher utilization, you can clearly see that this is a single threaded application.

5. pidstat 1

$ pidstat 1Linux 3.13.0-49-generic (titanclusters-xxxxx)  07/14/2015    _x86_64_    (32 CPU)07:41:02 PM   UID       PID    %usr %system  %guest    %CPU   CPU  Command07:41:03 PM     0         9    0.00    0.94    0.00    0.94     1  rcuos/007:41:03 PM     0      4214    5.66    5.66    0.00   11.32    15  mesos-slave07:41:03 PM     0      4354    0.94    0.94    0.00    1.89     8  java07:41:03 PM     0      6521 1596.23    1.89    0.00 1598.11    27  java07:41:03 PM     0      6564 1571.70    7.55    0.00 1579.25    28  java07:41:03 PM 60004     60154    0.94    4.72    0.00    5.66     9  pidstat07:41:03 PM   UID       PID    %usr %system  %guest    %CPU   CPU  Command07:41:04 PM     0      4214    6.00    2.00    0.00    8.00    15  mesos-slave07:41:04 PM     0      6521 1590.00    1.00    0.00 1591.00    27  java07:41:04 PM     0      6564 1573.00   10.00    0.00 1583.00    28  java07:41:04 PM   108      6718    1.00    0.00    0.00    1.00     0  snmp-pass07:41:04 PM 60004     60154    1.00    4.00    0.00    5.00     9  pidstat^C

The pidstat command is a bit like the statistics function for each CPU in the top command, but it prints information in a way of continuous scrolling and updating, rather than every screen cleaning. This is useful for observing patterns over time and recording the information you see (copy and paste) in your survey records.

The above example shows that two Java processes are consuming CPU.%CPUColumn is the utilization of all CPUs; 1591% means that the java process consumes almost 16 CPU cores.

6. iostat -xz 1

$ iostat -xz 1Linux 3.13.0-49-generic (titanclusters-xxxxx)  07/14/2015  _x86_64_ (32 CPU)avg-cpu:  %user   %nice %system %iowait  %steal   %idle          73.96    0.00    3.73    0.03    0.06   22.21Device:   rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %utilxvda        0.00     0.23    0.21    0.18     4.52     2.08    34.37     0.00    9.98   13.80    5.42   2.44   0.09xvdb        0.01     0.00    1.02    8.94   127.97   598.53   145.79     0.00    0.43    1.78    0.28   0.25   0.25xvdc        0.01     0.00    1.02    8.86   127.79   595.94   146.50     0.00    0.45    1.82    0.30   0.27   0.26dm-0        0.00     0.00    0.69    2.32    10.47    31.69    28.01     0.01    3.23    0.71    3.98   0.13   0.04dm-1        0.00     0.00    0.00    0.94     0.01     3.78     8.00     0.33  345.84    0.04  346.81   0.01   0.00dm-2        0.00     0.00    0.09    0.07     1.35     0.36    22.50     0.00    2.55    0.23    5.62   1.78   0.03[...]^C

This tool is useful for understanding block devices (such as disks) and shows the request load and performance data. See the explanation of the following fields for specific data:

  1. R / s, w / s, RKB / s, WKB / s: these represent the number of reads and writes per second and the number of bytes read and written on the device (unit: K bytes). These can see the load of the equipment. The performance problem can be simple because of the large number of file load requests.
  2. Await: the average time (in milliseconds) I / O waits. This is the waiting time of the application, including the time in the waiting queue and the time of the scheduled service. Excessive average waiting time indicates that the equipment is overloaded or there is a problem with the equipment.
  3. Avgqu-sz: average number of requests on the device. A value greater than 1 may indicate that the device is saturated (although devices can usually support parallel requests, especially virtual devices with multiple disks attached to the back).
  4. %Util: device utilization. Is the percentage of utilization rate, showing the working time of the equipment per second. If this value is greater than 60%, the performance will be very low (you can see it in await). Of course, it also depends on the characteristics of the equipment. If this value is close to 100%, it indicates that the equipment is saturated.

If the storage device is a logical disk device with multiple disks mounted behind it, a 100% utilization rate only means that some I / O is being processed at 100%. However, the back-end disks may be far from saturated and can handle more requests.

Keep in mind that low disk I / O performance is not necessarily an application problem. Many technologies are usually used to implement asynchronous I / O, so the application will not directly block and bear the delay (such as pre read and write buffer technology).

7. free -m

$ free -m             total       used       free     shared    buffers     cachedMem:        245998      24545     221453         83         59        541-/+ buffers/cache:      23944     222053Swap:            0          0          0

The two columns on the right show:

  1. Buffers: buffers used for block device I / O buffering.
  2. Cached: page cache for the file system.

We just want to check whether these cached values are close to 0. Non-zero may cause high disk I / O (confirmed by iostat command) and poor performance problems. The above example looks ok. There are still a lot of M bytes.

The line “- / + buffers / cache” provides clear statistics of used and free memory. Linux uses free memory as a cache, which can be quickly retrieved if the application needs it. Therefore, it should include the column of free memory. This is the statistics here. There is even a website dedicated to the problem of Linux memory consumption: Linux ate myram.

If ZFS file system is used on Linux, it may be more chaotic, because when we are developing some services, ZFS has its own file system cache, and the memory consumption will not be reducedfree -mThis order is reasonably reflected. It shows that the system memory is insufficient, but this part of ZFS cache can be used by applications.

8. sar -n DEV 1

$ sar -n DEV 1Linux 3.13.0-49-generic (titanclusters-xxxxx)  07/14/2015     _x86_64_    (32 CPU)12:16:48 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil12:16:49 AM      eth0  18763.00   5032.00  20686.42    478.30      0.00      0.00      0.00      0.0012:16:49 AM        lo     14.00     14.00      1.36      1.36      0.00      0.00      0.00      0.0012:16:49 AM   docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.0012:16:49 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil12:16:50 AM      eth0  19763.00   5101.00  21999.10    482.56      0.00      0.00      0.00      0.0012:16:50 AM        lo     20.00     20.00      3.25      3.25      0.00      0.00      0.00      0.0012:16:50 AM   docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00^C

Using this tool, you can detect the throughput of the network interface: rxkb / s and txkb / s. as a measure of the transceiver data load, it is also to detect whether the transceiver limit is reached. In the above example, eth0 receives data up to 22 m bytes / s, that is, 176 Mbit / S (the upper limit of the network card is 1 Gbit / s).

This version of the tool also has a statistical field:%ifutil, which is used to count the device utilization (full duplex bidirectional maximum). This utilization can also be measured by Brendan’s nicstat tool. In this example, 0.00 seems to have no statistics. Like nicstat, this value is difficult to calculate correctly.

9. sar -n TCP,ETCP 1

$ sar -n TCP,ETCP 1Linux 3.13.0-49-generic (titanclusters-xxxxx)  07/14/2015    _x86_64_    (32 CPU)12:17:19 AM  active/s passive/s    iseg/s    oseg/s12:17:20 AM      1.00      0.00  10233.00  18846.0012:17:19 AM  atmptf/s  estres/s retrans/s isegerr/s   orsts/s12:17:20 AM      0.00      0.00      0.00      0.00      0.0012:17:20 AM  active/s passive/s    iseg/s    oseg/s12:17:21 AM      1.00      0.00   8359.00   6039.0012:17:20 AM  atmptf/s  estres/s retrans/s isegerr/s   orsts/s12:17:21 AM      0.00      0.00      0.00      0.00      0.00^C

This is the statistics of TCP key indicators, which includes the following contents:

  1. Active / s: the number of locally initiated TCP connections per second (for example, connections initiated through connect()).
  2. Passive / s: the number of remotely initiated connections per second (for example, connections accepted through accept()).
  3. Retrans / s: number of TCP retransmissions per second.

This kind of active and passive statistics is often used as a rough estimate of the system load: the number of newly accepted connections (passive) and the number of downstream connections (active). You can think of the initiative as external and the passive as internal, but this is usually not very accurate (for example, when there is a local to local connection).

Retransmission is a signal that there is a problem with the network or server; It may be an unreliable network (e.g. public network), or it may be because the server is overloaded and begins to lose packets. As can be seen from the above example, a new TCP connection is created every second.

10. top

$ toptop - 00:15:40 up 21:56,  1 user,  load average: 31.09, 29.87, 29.92Tasks: 871 total,   1 running, 868 sleeping,   0 stopped,   2 zombie%Cpu(s): 96.8 us,  0.4 sy,  0.0 ni,  2.7 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 stKiB Mem:  25190241+total, 24921688 used, 22698073+free,    60448 buffersKiB Swap:        0 total,        0 used,        0 free.   554208 cached Mem   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND 20248 root      20   0  0.227t 0.012t  18748 S  3090  5.2  29812:58 java  4213 root      20   0 2722544  64640  44232 S  23.5  0.0 233:35.37 mesos-slave 66128 titancl+  20   0   24344   2332   1172 R   1.0  0.0   0:00.07 top  5235 root      20   0 38.227g 547004  49996 S   0.7  0.2   2:02.74 java  4299 root      20   0 20.015g 2.682g  16836 S   0.3  1.1  33:14.42 java     1 root      20   0   33620   2920   1496 S   0.0  0.0   0:03.82 init     2 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kthreadd     3 root      20   0       0      0      0 S   0.0  0.0   0:05.35 ksoftirqd/0     5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H     6 root      20   0       0      0      0 S   0.0  0.0   0:06.94 kworker/u256:0     8 root      20   0       0      0      0 S   0.0  0.0   2:38.05 rcu_sched

The top command contains many of the metrics we mentioned earlier. This command can be easily seen that the change of indicator indicates the change of load, which looks very different from the previous command.

A defect of top is also obvious. It is difficult to see the change trend. Other tools such as vmstat and pidstat will be very clear. They output statistics in a rolling manner. So if you don’t pause in time when you see the problematic information (ctrl-s is pause and ctrl-q is continue), then these useful information will be cleared.

Follow-on Analysis

There are also many commands and technologies that can be used to dig deep into system problems. Take a look at Brendan’s introduction to Linux performance tools in 2015, which describes more than 40 commands, including observability, benchmarking, tuning, static performance tuning, analysis and tracking.

The above isLiangxu tutorial networkHow to conduct Linux performance analysis in 60 seconds is shared for all friends.

This article is composed of blog one article multi posting platformOpenWriterelease!