Understanding CPU Interrupts in Linux

What Is an Interrupt?

What Are RPS and RFS?

Figure 1 Case with only RPS enabled (sourcing from the Internet)
Figure 2 Case with both RPS and RFS enabled (sourcing from the Internet)

Viewing the Number of NIC Queues

# ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums:
RX: 0
TX: 0
Other: 0
Combined: 4
Current hardware settings:
RX: 0
TX: 0
Other: 0
Combined: 4
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 141 0 0 0 IO-APIC-edge timer
1: 10 0 0 0 IO-APIC-edge i8042
4: 807 0 0 0 IO-APIC-edge serial
6: 3 0 0 0 IO-APIC-edge floppy
8: 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 IO-APIC-fasteoi acpi
10: 0 0 0 0 IO-APIC-fasteoi virtio3
11: 22 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1
12: 15 0 0 0 IO-APIC-edge i8042
14: 0 0 0 0 IO-APIC-edge ata_piix
15: 0 0 0 0 IO-APIC-edge ata_piix
24: 0 0 0 0 PCI-MSI-edge virtio1-config
25: 4522 0 0 4911 PCI-MSI-edge virtio1-req.0
26: 0 0 0 0 PCI-MSI-edge virtio2-config
27: 1913 0 0 0 PCI-MSI-edge virtio2-input.0
28: 3 834 0 0 PCI-MSI-edge virtio2-output.0
29: 2 0 1557 0 PCI-MSI-edge virtio2-input.1
30: 2 0 0 187 PCI-MSI-edge virtio2-output.1
31: 0 0 0 0 PCI-MSI-edge virtio0-config
32: 1960 0 0 0 PCI-MSI-edge virtio2-input.2
33: 2 798 0 0 PCI-MSI-edge virtio2-output.2
34: 30 0 0 0 PCI-MSI-edge virtio0-virtqueues
35: 3 0 272 0 PCI-MSI-edge virtio2-input.3
36: 2 0 0 106 PCI-MSI-edge virtio2-output.3
input0 indicates the network interrupt handled by the first CPU (CPU 0).
If there are multiple Alibaba Cloud ECS network interrupts, input.1, input.2, and input.3 are available.
......
PIW: 0 0 0 0 Posted-interrupt wakeup event
# for i in $(egrep "\-input."  /proc/interrupts |awk -F ":" '{print $1}');do cat /proc/irq/$i/smp_affinity_list;done
5
7
1
3
# for i in $(egrep "\-input."  /proc/interrupts |awk -F ":" '{print $1}');do cat /proc/irq/$i/smp_affinity_list;done |tr -s '\n' ','
5,7,1,3,
#sar -P 5,7,1,3 1 Refresh the usage of CPUs 5, 7, 1, and 3 every second.
# sar -P ALL 1 Refresh the usage of all CPU cores every second when a few CPU cores are being monitored, so that we can determine whether slow processing is caused by insufficient queues.
Linux 3.10.0-957.5.1.el7.x86_64 (iZwz98aynkjcxvtra0f375Z) 05/26/2020 _x86_64_ (4 CPU)
05:10:06 PM CPU %user %nice %system %iowait %steal %idle
05:10:07 PM all 5.63 0.00 3.58 1.02 0.00 89.77
05:10:07 PM 0 6.12 0.00 3.06 1.02 0.00 89.80
05:10:07 PM 1 5.10 0.00 5.10 0.00 0.00 89.80
05:10:07 PM 2 5.10 0.00 3.06 2.04 0.00 89.80
05:10:07 PM 3 5.10 0.00 4.08 1.02 0.00 89.80
05:10:07 PM CPU %user %nice %system %iowait %steal %idle
05:10:08 PM all 8.78 0.00 15.01 0.69 0.00 75.52
05:10:08 PM 0 10.00 0.00 16.36 0.91 0.00 72.73
05:10:08 PM 1 4.81 0.00 13.46 1.92 0.00 79.81
05:10:08 PM 2 10.91 0.00 15.45 0.91 0.00 72.73
05:10:08 PM 3 9.09 0.00 14.55 0.00 0.00 76.36
sar -P 1,3,5,7 1 |tail -n+3|awk '$NF<10 {print $0}'
sar -P ALL 1 |tail -n+3|awk '$NF<10 {print $0}'
# grep -i "input" /proc/interrupts
27: 1932 0 0 0 PCI-MSI-edge virtio2-input.0
29: 2 0 1627 0 PCI-MSI-edge virtio2-input.1
32: 1974 0 0 0 PCI-MSI-edge virtio2-input.2
35: 3 0 284 0 PCI-MSI-edge virtio2-input.3
# for i in $(egrep "\-input." /proc/interrupts |awk -F ":" '{print $1}');do cat /proc/irq/$i/smp_affinity_list;done
1
3
1
3
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
# service irqbalance status
Redirecting to /bin/systemctl status irqbalance.service
● irqbalance.service - irqbalance daemon
Loaded: loaded (/usr/lib/systemd/system/irqbalance.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2020-05-27 14:39:28 CST; 2s ago
Process: 1832 ExecStart=/usr/sbin/irqbalance --foreground $IRQBALANCE_ARGS (code=exited, status=0/SUCCESS)
Main PID: 1832 (code=exited, status=0/SUCCESS)
May 27 14:11:40 iZbp1ee4vpiy3w4b8y2m8qZ systemd[1]: Started irqbalance daemon.
May 27 14:39:28 iZbp1ee4vpiy3w4b8y2m8qZ systemd[1]: Stopping irqbalance daemon...
May 27 14:39:28 iZbp1ee4vpiy3w4b8y2m8qZ systemd[1]: Stopped irqbalance daemon.
# for i in $(egrep "\-input."  /proc/interrupts |awk -F ":" '{print $1}');do cat /proc/irq/$i/smp_affinity_list;done
1
3
1
3
# echo 0 >> /proc/irq/27/smp_affinity_list
# cat /proc/irq/27/smp_affinity_list
0
Each bit in the binary value represents a CPU on the server. A simple demo is shown below:
CPU ID Binary Hexadecimal
CPU 0 0001 1
CPU 1 0010 2
CPU 2 0100 4
CPU 3 1000 8
echo ff > /sys/class/net/eth0/queues/rx-0/rps_cpusThis method is similar to the method used to configure the interrupt affinity. A mask is used, and is set to a value indicating that all CPUs can be used to handle interrupts. For example:
4core, f
8core, ff
16core, ffff
32core, ffffffff
Queue 0 is configured on CPU 0 by default.
# cat /sys/class/net/eth0/queues/rx-0/rps_cpus
0
# echo f >>/sys/class/net/eth0/queues/rx-0/rps_cpus
# cat /sys/class/net/eth0/queues/rx-0/rps_cpus
f
# sysctl -a |grep net.core.rps_sock_flow_entries
net.core.rps_sock_flow_entries = 0
# sysctl -w net.core.rps_sock_flow_entries=1024
net.core.rps_sock_flow_entries = 1024
#  cat /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
0
# echo 256 >> /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
# cat /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
256

Case Study: Adjustment in Stress Testing

Case Study: CPU Interrupt Processing Anomaly of a Customer

Original Source:

--

--

--

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

7 Things Every Translation Management Software Should Have

Automated Testing in Devops using Kubernetes

Roadmap for Storj Integration

What ONIE Means to Bare metal switching?

Open source plugin for automatic programming metrics

Cloud Based Test Automation

Store and Serve — Keeping Data Safe on the Cloud

Career lessons I learned in 8 years as a Software Developer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alibaba Cloud

Alibaba Cloud

Follow me to keep abreast with the latest technology news, industry insights, and developer trends. Alibaba Cloud website:https://www.alibabacloud.com

More from Medium

An Introduction To Kubernetes

Github Actions Vs Jenkins: Which Should You Use?

Learning Docker & Kubernetes: The Jorney

Golang Crash Course (The language of DevOps and Cloud)