You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/learning-paths/servers-and-cloud-computing/irq-tuning-guide/_index.md
+8-8Lines changed: 8 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,30 +1,30 @@
1
1
---
2
-
title: Learn about the impact of NIC IRQs and patterns on cloud
2
+
title: Learn about the impact of network interrupts on cloud workloads
3
3
4
4
draft: true
5
5
cascade:
6
6
draft: true
7
7
8
8
minutes_to_complete: 20
9
9
10
-
who_is_this_for: This is anyone interested in understanding how IRQ patterns can enhance networking workload performance on cloud.
11
-
10
+
who_is_this_for: This is a specialized topic for developers and performance engineers who are interested in understanding how network interrupt patterns can impact performance on cloud servers.
12
11
13
12
learning_objectives:
14
-
- Analyze the current IRQ layout on the machine.
15
-
- Test different options and patterns to improve performance.
13
+
- Analyze the current interrupt request (IRQ) layout on an Arm Linux system
14
+
- Experiment with different interrupt options and patterns to improve performance
16
15
17
16
prerequisites:
18
-
- An Arm computer running Linux installed.
19
-
- Some familiarity with running Linux command line commands.
title: Understand and Analyze network IRQ configuration
3
3
weight: 2
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
-
First you should run the following command to identify all IRQs on the system. Identify the NIC IRQs and adjust the system by experimenting and seeing how performance improves.
9
+
## Why IRQ management matters for performance
10
10
11
-
```
11
+
In modern cloud environments, network performance is critical to overall system efficiency. Network interface cards (NICs) generate interrupt requests (IRQs) to notify the CPU when data packets arrive or need to be sent. These interrupts temporarily pause normal processing, allowing the system to handle network traffic.
12
+
13
+
By default, Linux distributes these network interrupts across available CPU cores. However, this distribution is not always optimal for performance:
14
+
15
+
- High interrupt rates: In busy servers, network cards can generate thousands of interrupts per second
16
+
- CPU cache locality: Processing related network operations on the same CPU core improves cache efficiency
17
+
- Resource contention: When network IRQs compete with application workloads for the same CPU resources, both can suffer
18
+
- Power efficiency: IRQ management can help reduce unnecessary CPU wake-ups, improving energy efficiency
19
+
20
+
Understanding and optimizing IRQ assignment allows you to balance network processing loads, reduce latency, and maximize throughput for your specific workloads.
21
+
22
+
## Identifying IRQs on your system
23
+
24
+
To get started, run this command to display all IRQs on your system and their CPU assignments:
Now, you may notice that the NIC IRQs are assigned to a duplicate CPU by default.
51
+
## How to identify network IRQs
52
+
53
+
Network-related IRQs can be identified by looking at the "Device" column in the output.
39
54
40
-
like this example:
55
+
You can identify network interfaces using the command:
56
+
57
+
```bash
58
+
ip link show
41
59
```
60
+
61
+
Here are some common patterns to look for:
62
+
63
+
Common interface naming patterns include `eth0` for traditional ethernet, `enP3p3s0f0` and `ens5-Tx-Rx-0` for the Linux predictable naming scheme, or `wlan0` for wireless.
64
+
65
+
The predictable naming scheme breaks down into:
66
+
67
+
- en = ethernet
68
+
- P3 = PCI domain 3
69
+
- p3 = PCI bus 3
70
+
- s0 = PCI slot 0
71
+
- f0 = function 0
72
+
73
+
This naming convention helps ensure network interfaces have consistent names across reboots by encoding their physical
74
+
location in the system.
75
+
76
+
## Improve performance
77
+
78
+
Once you've identified the network IRQs, you can adjust their CPU assignments to try to improve performance.
79
+
80
+
Identify the NIC (Network Interface Card) IRQs and adjust the system by experimenting and seeing if performance improves.
81
+
82
+
You may notice that some NIC IRQs are assigned to the same CPU cores by default, creating duplicate assignments.
This can potential hurt performance. Suggestions and patterns to experiment with will be on the next step.
51
95
52
-
### reset
96
+
##Understanding IRQ performance impact
53
97
54
-
If performance reduces, you can return the IRQs back to default using the following commands.
98
+
When network IRQs are assigned to the same CPU cores (as shown in the example above where IRQ 101 and 104 both use CPU 12), this can potentially hurt performance as multiple interrupts compete for the same CPU core's attention, while other cores remain underutilized.
55
99
56
-
```
100
+
By optimizing IRQ distribution, you can achieve more balanced processing and improved throughput. This optimization is especially important for high-traffic servers where network performance is critical.
101
+
102
+
Suggested experiments are covered in the next section.
103
+
104
+
### How can I reset my IRQs if I make performance worse?
105
+
106
+
If your experiments reduce performance, you can return the IRQs back to default using the following commands:
107
+
108
+
```bash
57
109
sudo systemctl unmask irqbalance
58
110
sudo systemctl enable --now irqbalance
59
111
```
60
112
61
-
or you can run the following
113
+
If needed, install `irqbalance` on your system. For Debian based systems run:
62
114
63
-
```
64
-
DEF=$(cat /proc/irq/default_smp_affinity)
65
-
for f in /proc/irq/*/smp_affinity; do
66
-
echo "$DEF" | sudo tee "$f" >/dev/null || true
67
-
done
115
+
```bash
116
+
sudo apt install irqbalance
68
117
```
69
118
70
119
### Saving these changes
71
120
72
-
Any changes you make to IRQs will be reset at reboot. You will need to change your systems settings to make your changes permanent.
121
+
Any changes you make to IRQs will be reset at reboot. You will need to change your system's settings to make your changes permanent.
While a single pattern does not work for all workloads. Our testing found that under heavy network workloads, different patterns performed better based on sizing.
9
+
## Optimal IRQ Management Strategies
10
10
11
-
### upto and under 16 vCPUs
12
-
For best performance, reduce NIC IRQs to either one or two cores. Otherwise random or default performed second best.
11
+
Testing across multiple cloud platforms reveals that IRQ management effectiveness varies significantly based on system size and workload characteristics. No single pattern works optimally for all scenarios, but clear patterns emerged during performance testing under heavy network loads.
13
12
14
-
*If the number of NIC IRQS are more then the number of vCPUs, concentrating them over less cores improved performance significantly.
13
+
## Recommendations by system size
15
14
16
-
### over 16 vCPUs
17
-
No pattern showed significant improvement over default as long as all NIC IRQs were not on duplicate cores.
15
+
### Systems with 16 vCPUs or less
16
+
17
+
For smaller systems with 16 or less vCPUs, concentrated IRQ assignment may provide measurable performance improvements.
18
+
19
+
- Assign all network IRQs to just one or two CPU cores
20
+
- This approach showed the most significant performance gains
21
+
- Most effective when the number of NIC IRQs exceeds the number of vCPUs
22
+
- Use the `smp_affinity` range assignment pattern from the previous section with a very limited core range, for example `0-1`
23
+
24
+
Performance improves significantly when network IRQs are concentrated rather than dispersed across all available cores on smaller systems.
25
+
26
+
### Systems with more than 16 vCPUs
27
+
28
+
For larger systems with more than 16 vCPUs, the findings are different:
29
+
30
+
- Default IRQ distribution generally performs well
31
+
- The primary concern is avoiding duplicate core assignments for network IRQs
32
+
- Use the scripts from the previous section to check and correct any overlapping IRQ assignments
33
+
- The paired core pattern can help ensure optimal distribution on these larger systems
34
+
35
+
On larger systems, the overhead of interrupt handling is proportionally smaller compared to the available processing power. The main performance bottleneck occurs when multiple high-frequency network interrupts compete for the same core.
36
+
37
+
## Implementation Considerations
38
+
39
+
When implementing these IRQ management strategies, there are some important points to keep in mind.
40
+
41
+
Pay attention to the workload type. CPU-bound applications may benefit from different IRQ patterns than I/O-bound applications.
42
+
43
+
Always benchmark your specific workload with different IRQ patterns.
44
+
45
+
Monitor IRQ counts in real-time using `watch -n1 'grep . /proc/interrupts'` to observe IRQ distribution in real-time.
46
+
47
+
Also consider NUMA effects on multi-socket systems. Keep IRQs on cores close to the PCIe devices generating them to minimize cross-node memory access.
48
+
49
+
Make sure to set up IRQ affinity settings in `/etc/rc.local` or a systemd service file to ensure they persist across reboots.
50
+
51
+
Remember that as workloads and hardware evolve, revisiting and adjusting IRQ management strategies may be necessary to maintain optimal performance.
title: IRQ management patterns for performance optimization
3
3
weight: 3
4
4
5
5
### FIXED, DO NOT MODIFY
6
6
layout: learningpathall
7
7
---
8
8
9
-
The following patterns were ran on multiple cloud and on a variety of sizes. A recommended IRQ pattern will be suggested at the end. Based on your workload, a different pattern may result in higher performance.
9
+
## Optimizing network performance with IRQ management
10
+
11
+
Different IRQ management patterns can significantly impact network performance across multiple cloud platforms and virtual machine sizes. This Learning Path presents various IRQ distribution strategies, along with scripts to implement them on your systems.
12
+
13
+
Network interrupt requests (IRQs) can be distributed across CPU cores in various ways, each with potential benefits depending on your workload characteristics and system configuration. By strategically assigning network IRQs to specific cores, you can improve cache locality, reduce contention, and potentially boost overall system performance.
14
+
15
+
The following patterns have been tested on various systems and can be implemented using the provided scripts. An optimal pattern is suggested at the conclusion of this Learning Path, but your specific workload may benefit from a different approach.
10
16
11
17
### Patterns
18
+
12
19
1. Default: IRQ pattern provided at boot.
13
20
2. Random: All IRQs are assigned a core and do not overlap with network IRQs.
14
-
3. Housekeeping: All IRQs outside of network IRQs are assign to specific core(s).
15
-
4. NIC IRQs are set to single or multiple ranges of cores and including pairs. EX. 1, 1-2, 0-3, 0-7, [0-1, 2-3..], etc.
16
-
21
+
3. Housekeeping: All IRQs outside of network IRQs are assigned to specific core(s).
22
+
4. NIC IRQs are assigned to single or multiple ranges of cores, including pairs.
17
23
18
24
### Scripts to change IRQ
19
25
26
+
The scripts below demonstrate how to implement different IRQ management patterns on your system. Each script targets a specific distribution strategy:
27
+
28
+
Before running these scripts, identify your network interface name using `ip link show` and determine your system's CPU topology with `lscpu`. Always test these changes in a non-production environment first, as improper IRQ assignment can impact system stability.
29
+
20
30
To change the NIC IRQs or IRQs in general you can use the following scripts.
21
31
22
-
Housekeeping pattern example, you will need to add more to account for other IRQs on your system
32
+
### Housekeeping
23
33
24
-
```
25
-
HOUSEKEEP=#core range here
34
+
The housekeeping pattern isolates non-network IRQs to dedicated cores.
35
+
36
+
You need to add more to account for other IRQs on your system.
echo 0-15 | sudo tee /proc/irq/$irq/smp_affinity_list > /dev/null
59
84
done
60
-
```
85
+
```
86
+
87
+
Each pattern offers different performance characteristics depending on your workload. The housekeeping pattern reduces system noise, paired cores optimize cache usage, and range assignment provides dedicated network processing capacity. Test these patterns in your environment to determine which provides the best performance for your specific use case.
88
+
89
+
Continue to the next section for additional guidance.
0 commit comments