Notes on Linux performance optimization

Time:2021-1-15

Share basisLinux performance optimizationNotes of course pictures.

preface

Experimental environment: Ubuntu 18.04, switch to the root authority operation

To view the detailed usage of the command, you can:man xxx
When it comes to paths, you can generallycat /x/xx/xxx
When it comes to tools, if they are not installed, you can:apt-get install xxx

CPU

CPU performance index

Notes on Linux performance optimization

Find tools according to indicators

performance index tool explain
Average load uptime
top
Uptime is the simplest;
Top provides more comprehensive indicators
Overall CPU utilization of the system vmstat
mpstat
top
sar
/proc/stat
Top, vmstat and mpstat can only be viewed dynamically,

/Proc / STAT / is the data source of other performance tools

Process CPU utilization top
pidstat
ps
htop
atop
Top and PS can sort processes by CPU utilization,
Pidstat only shows the processes that actually use CPU

Htop and atop are more intuitive in different colors

System context switch vmstat In addition to the number of context switches,
It also provides the number of running and non interruptible processes
Process context switch pidstat Note the addition of the – W option
Softinterrupt top
/proc/softirqs
mpstat
Top provides soft interrupt CPU utilization,
And / proc / softirqs and mpstat provide the running times of various soft interrupts on each CPU
Hard interrupt vmstat
/proc/interrupts
Vmstat provides the total number of interrupts,
And / proc / interrupts provides the cumulative number of interrupts running on each CPU
network dstat
sar
tcpdump
Dstat and SAR provide the total network reception and transmission,
Tcpdump dynamically captures the ongoing network communication
I/O dstat
sar
Dstat and SAR both provide the overall situation of I / O
Number of CPUs /proc/cpuinfo
lscpu
Lscpu is more intuitive
Event analysis perf
execsnoop
Perf can be used to analyze CPU cache and kernel call chain, and execsnoop can be used to monitor short-term processes

Check indicators according to tools

Performance tools Cup performance index
uptime Average load
top Average load, run queue, overall CPU utilization, and status and performance of each process
CPU utilization
htop Top enhanced version, with different colors to distinguish different types of process, more intuitive
atop Comprehensive monitoring of CPU, memory, disk, network and other resources
vmstat CPU utilization, context switching times, interrupt times of the whole system
It also includes the number of processes that are running and non interruptible
mpstat Utilization rate and number of soft interrupts per CPU
pidstat CPU utilization and interrupt context switching times of processes and threads
/proc/softirqs Soft interrupt type and cumulative number of interrupts per CPU
/proc/interrupts Hard interrupt type and cumulative number of interrupts per CPU
ps Status and CPU utilization of each process
pstree Father son relationship of process
dstat CPU utilization rate of the whole system
sar Overall CPU utilization of the system, including configurable historical data
strace System call of process
perf CPU performance event analysis, such as call chain analysis, CPU cache, CPU scheduling
execsnoop Monitor short-term progress

Analysis of CPU performance optimization

Notes on Linux performance optimization

Memory

Memory performance index

Notes on Linux performance optimization

Find tools according to indicators

Memory index Performance tools
System used, available, remaining memory free
vmstat
sar
/proc/meminfo
Process virtual memory, resident memory, shared memory ps
top
Process memory distribution pmap
Process swap out memory top
/proc/pid/status
Process page missing exception ps
top
System page change sar
Cache / buffer usage free
vmstat
sar
cachestat
Cache / buffer usage cachetop
Swap used space and remaining space free
sar
Swap in swap out vmstat
Memory leak detection memleak
valgrind
Specifies the cache size of the file pcstat

Check indicators according to tools

Performance tools Memory index
free
/proc/meninfo
The amount of memory used, available, remaining, and cache and buffer used by the system
top
ps
Process virtualization, resident, shared memory and page missing exception
vmstat System remaining memory, cache, buffer, swap in, swap out
sar System part of the memory page, memory utilization, cache and buffer usage
And swap
cachestat Hit ratio of system cache and buffer
cachetop Hit ratio of process cache and buffer
slabtop System slab cache usage
/proc/pid/status Process swap memory, etc
/proc/pid/smaps
pmap
Process address space and memory state
valgrind Process memory error checker, which is used to detect memory initialization, leakage
Cross border access and other memory errors
memleak Memory leak detection
pcstat View the cache of the specified file

Analysis of memory performance optimization

Notes on Linux performance optimization

I / O

I / O performance index

Notes on Linux performance optimization

Find tools according to indicators

performance index tool explain
File system space capacity, usage and remaining space df Please refer to
info coreutils ‘df invocation’
Index node capacity, usage and surplus df Use the – I option
Page cache and recyclable slab cache /proc/meminfo
sar、vmstat
Use SAR – R option
buffer /proc/meminfo
sar、vmstat
Use SAR – R option
Caching of directory entries, index nodes, and file systems /proc/slabinfo
slabtop
Slabtop is more intuitive
Disk I / O utilization, IOPs
Throughput, response time, average I / O size
And the waiting queue length
iostat
sar、dstat
Use the iostat – D – X or SAR – D options
Process I / O size and I / O latency pidstat
iotop
Use pidstat – D option
Block device I / O event tracking blktrace Example: blktrace – D / dev / SDA – O – (pipe) blkparse – I-
Process I / O system call tracing strace Tracking I / O of process through system call
Process block device I / O size tracking biosnoop
biotop
BCC package needs to be installed

Check indicators according to tools

Performance tools performance index
iostat Disk I / O utilization, IOPs, throughput, response time, average I / O size and waiting queue length
pidstat Process I / O size and I / O latency
sar Disk I / O utilization, IOPs, throughput, and response time
dstat Disk I / O, IPOs and throughput
iotop Sort processes by I / O size
slabtop Cache of directory entry, index node and file system
/proc/slabinfo Cache of directory entry, index node and file system
/proc/meminfo Page cache and recyclable slab cache
/proc/diskstats IOPs, throughput and latency of disk
/proc/pid/io Process IOPs, I / O size, and I / O latency
vmstat Summary of cache and buffer usage
blktrace Trace block device I / o Events
biosnoop Track the block device I / O size of the process
biotop Track the status of processes and sort them by I / O size
strace I / O system calls to trace processes
perf Tracing I / O events in the kernel
df Disk space and index node usage and remaining
mount The mounting path and parameters of the file system
du The amount of disk space occupied by the directory
tune2fs Display and set file system parameters
hdparam Display and set disk parameters

I / O performance optimization analysis

Notes on Linux performance optimization

Network

Find tools according to indicators

performance index tool explain
Throughput (BPS) sar
nethogs
iftop
You can view the network throughput of network interface, process and IP address respectively
PPS sar
/proc/net/dev
View PPS of network interface
Number of connections nestat
ss
View network connections
delay ping
hping3
Test network delay by ICMP, TCP, etc
Number of connection traces conntrack View and manage connection tracking status
route mtr
route
traceroute
View route and test link information
DNS dig
nslookup
Check DNS resolution
Firewall and NAT iptables Configure and manage firewall and NAT rules
Network card function ethtool View and configure the functions of the network interface
Grab a bag tcpdump
Wireshark
Analysis of network traffic by packet capture
Kernel stack trace bcc
systemtap
Tracking the behavior of kernel protocol stack dynamically

Check indicators according to tools

Performance tools major function
ifconfig
ip
Configure and view network interfaces
ss View network connections
sar
/proc/net/dev/sys/class/net/eth0/statistics
View the receiving and sending status of network interface
nethogs View the network receiving and sending status of the process
iftop View the network receiving and sending of IP
ethool View and configure network interfaces
conntrack View and manage connection tracking status
nslookup
dig
Check DNS resolution
mtr
route
traceroute
View route and test link information
ping
hping3
Test network delay
tcpdump Network packet capture tool
Wireshark Network packet capture and GUI analysis tool
iptables Configure and manage firewall and NAT rules
perf Analyzing the performance of kernel protocol stack
systemtap
bcc
Dynamically tracing the behavior of kernel protocol stack

Conclusion & Cases

“I feel shallow when I get it on paper, but I have to practice when I absolutely know it.”. Here, I would like to provide my practical course cases, hoping to help you.

Case: the CPU utilization rate of an application is 100%. What should I do?
Case: the CPU utilization rate of the system is very high, but why can’t we find a high CPU application?
Case: how to deal with a large number of non interruptible processes (d) and zombie processes (z) in the system?

This work adoptsCC agreementReprint must indicate the author and the link of this article