CPU soft interrupt & A Case Study of ing

Time:2020-9-15

1 case

1.1 environmental deployment

Environment 1: Web server

docker run -itd --name=nginx -p 80:80 nginx

Environment 2: syn attack

curl http://xxx.xxx.xxx.xxx/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

#The - S parameter indicates the syn (synchronization sequence number) of TCP protocol, and - P indicates that the destination port is 80
#Every 100 micro frames are sent every 100 seconds
#Note: if the phenomenon is not obvious in practice, you can try to turn down 100, such as 10 or even 1
$ hping3 -S -p 80 -i u1 xxx.xxx.xxx.xxx

1.2 analysis process

1. CPU utilization is not high, but the soft interrupt has reached 10%, all the non idle state is used in soft interrupt.

CPU soft interrupt & A Case Study of ing

2. Identify which soft interrupt caused the problem

watch -d cat /proc/softirqs

Timer, net_ Among RX (network reception), sched (kernel scheduling) and RCU (RCU lock), network reception changes fastest.

CPU soft interrupt & A Case Study of ing

3. Confirm the network problem, continue to observe the network receiving and contracting situation

#- N dev means to display the reports sent and received by the network, and output a group of data at an interval of 1 second
sar -n DEV 1

Report time | network card | rxpck / s txpck / s received and received frames | exkb / s txkb / s received and received kilobytes
CPU soft interrupt & A Case Study of ing

  • For netcard ET0, the frame received is very large 112924.00, but the amount of data is very small 5955.00.
  • Calculate it5955.00*1024Byte/112924.00=54ByteOn average, each packet has only 54 bytes, which is the small packet problem.

4. Packet capture confirmation

tcpdump -i eth0 -n tcp port 80

Http: flags [S], SEQ 270293337, win 512, length 0
Http: flags [S], SEQ 830767629, win 512, length 0

Flags [S] indicates syn packets. PPS over 1.2W confirms SYN Flood attack.

2 Concept

2.1 concept of soft interrupt

Linux interrupt will interrupt the current work of CPU, interrupt is generally designed to be short and concise. However, in order to solve the problem of long execution time of interrupt handler and interrupt loss, Linux interrupt is divided into two stages

  • The upper part is used to process interrupts quickly. It runs in interrupt forbidden mode and mainly deals with hardware related and time sensitive work
  • The lower part is used to delay the unfinished work of the upper part, which is usually run as an inner core thread

An example of receiving packets from a network card:

After the network card receives the data packet, it will inform the kernel that there is new data by means of hardware interrupt: for the upper part, since it is a fast processing, it is actually to read the data of the network card into the memory, and then update the status of the hardware register (indicating that the data has been read), and finally send a soft interrupt signal to inform the lower half to do further processing. When the lower part is awakened by the soft interrupt signal, it needs to find the network data from the memory, and then analyze and process the data layer by layer according to the network protocol stack until it is sent to the application program.

  • The upper part directly processes the hardware request, which is often called hard interrupt. It is characterized by fast execution
  • The second half is triggered by the kernel, which is often called soft interrupt, which is characterized by delayed execution

In fact, the top half interrupts the task the CPU is performing, and then immediately executes the interrupt handler. In the lower half, the kernel thread is executed in the way of kernel thread, and each CPU corresponds to a soft interrupt kernel thread named “ksoftirqd / CPU number”. For example, the name of the soft interrupt kernel thread corresponding to CPU 0 is ksoftirqd / 0. However, it should be noted that software interrupts not only include the lower part of the hardware device interrupt handler just mentioned, but also some kernel custom events, such as kernel scheduling and RCU lock (RCU is one of the most commonly used locks in Linux kernel), etc.

2.2 view soft interrupts and kernel threads

  • cat /proc/interruptsSoft interrupt
  • cat /proc/softirqsHard interrupt