Merge pull request ArmDeveloperEcosystem#2271 from kielfriedt/irq

IRQ tuning guide

Author: pareenaverma

content/learning-paths/servers-and-cloud-computing/irq-tuning-guide/_index.md

@@ -0,0 +1,45 @@
+---
+title: Learn about the impact of NIC IRQs and patterns on cloud
+
+draft: true
+cascade:
+    draft: true
+    
+minutes_to_complete: 20
+
+who_is_this_for: This is anyone interested in understanding how IRQ patterns can enhance networking workload performance on cloud.
+
+
+learning_objectives:
+   - Analyze the current IRQ layout on the machine.
+   - Test different options and patterns to improve performance.
+
+prerequisites:
+    - An Arm computer running Linux installed.
+    - Some familiarity with running Linux command line commands.
+
+author: Kiel Friedt
+
+### Tags
+skilllevels: Introductory
+subjects: Performance and Architecture
+armips:
+    - AArch64
+tools_software_languages:
+
+operatingsystems:
+    - Linux
+
+
+further_reading:
+    - resource:
+        title: Perf for Linux on Arm (LinuxPerf)
+        link: https://learn.arm.com/install-guides/perf/
+        type: website
+
+### FIXED, DO NOT MODIFY
+# ================================================================================
+weight: 1                       # _index.md always has weight of 1 to order correctly
+layout: "learningpathall"       # All files under learning paths have this same wrapper
+learning_path_main_page: "yes"  # This should be surfaced when looking for related content. Only set for _index.md of learning path content.
+---

content/learning-paths/servers-and-cloud-computing/irq-tuning-guide/_next-steps.md

@@ -0,0 +1,8 @@
+---
+# ================================================================================
+#       FIXED, DO NOT MODIFY THIS FILE
+# ================================================================================
+weight: 21                  # Set to always be larger than the content in this path to be at the end of the navigation.
+title: "Next Steps"         # Always the same, html page title.
+layout: "learningpathall"   # All files under learning paths have this same wrapper for Hugo processing.
+---

content/learning-paths/servers-and-cloud-computing/irq-tuning-guide/checking.md

@@ -0,0 +1,72 @@
+---
+title: checking IRQs
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+First you should run the following command to identify all IRQs on the system. Identify the NIC IRQs and adjust the system by experirmenting and seeing how performance improves.
+
+```
+grep '' /proc/irq/*/smp_affinity_list | while IFS=: read path cpus; do
+    irq=$(basename $(dirname $path))
+    device=$(grep -E "^ *$irq:" /proc/interrupts | awk '{print $NF}')
+    printf "IRQ %s -> CPUs %s -> Device %s\n" "$irq" "$cpus" "$device"
+done
+```
+
+
+{{% notice Note %}}
+output should look similar to this:
+```
+IRQ 104 -> CPUs 12   -> Device ens34-Tx-Rx-5
+IRQ 105 -> CPUs 5    -> Device ens34-Tx-Rx-6
+IRQ 106 -> CPUs 10   -> Device ens34-Tx-Rx-7
+IRQ 11  -> CPUs 0-15 -> Device
+IRQ 14  -> CPUs 0-15 -> Device ttyS0
+IRQ 17  -> CPUs 0-15 -> Device ACPI:Ged
+IRQ 19  -> CPUs 0-15 -> Device ACPI:Ged
+IRQ 2   -> CPUs 0-15 -> Device
+IRQ 20  -> CPUs 0-15 -> Device ACPI:Ged
+IRQ 21  -> CPUs 0-15 -> Device ACPI:Ged
+...
+IRQ 26  -> CPUs 0-15 -> Device ACPI:Ged
+```
+{{% /notice %}}
+
+Now, you may notice that the NIC IRQs are assigned to a duplicate CPU by default.
+
+like this example:
+```
+IRQ 100 -> CPUs 2 -> Device ens34-Tx-Rx-1
+IRQ 101 -> CPUs 12 -> Device ens34-Tx-Rx-2
+IRQ 102 -> CPUs 14 -> Device ens34-Tx-Rx-3
+IRQ 103 -> CPUs 9 -> Device ens34-Tx-Rx-4
+IRQ 104 -> CPUs 12 -> Device ens34-Tx-Rx-5
+IRQ 105 -> CPUs 5 -> Device ens34-Tx-Rx-6
+IRQ 106 -> CPUs 10 -> Device ens34-Tx-Rx-7
+```
+This can potential hurt performance. Suggestions and patterns to expertiment with will be on the next step.
+
+### reset
+
+If performance reduces, you can return the IRQs back to default using the following commands.
+
+```
+sudo systemctl unmask irqbalance
+sudo systemctl enable --now irqbalance
+```
+
+or you can run the following
+
+```
+DEF=$(cat /proc/irq/default_smp_affinity)
+for f in /proc/irq/*/smp_affinity; do
+  echo "$DEF" | sudo tee "$f" >/dev/null || true
+done
+```
+
+### Saving these changes
+
+Any changes you make to IRQs will be reset at reboot. You will need to change your systems settings to make your changes permenant.

content/learning-paths/servers-and-cloud-computing/irq-tuning-guide/conclusion.md

@@ -0,0 +1,17 @@
+---
+title: conclusion
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+While a single pattern does not work for all workloads. Our testing found that under heavy network workloads, different patterns performed better based on sizing.
+
+### upto and under 16 vCPUs
+For best performance, reduce NIC IRQs to either one or two cores. Otherwise random or default performed second best.
+
+*If the number of NIC IRQS are more then the number of vCPUs, concentrating them over less cores improved performance significantly.
+
+### over 16 vCPUs
+No pattern showed significant improvement over default as long as all NIC IRQs were not on duplicate cores.

content/learning-paths/servers-and-cloud-computing/irq-tuning-guide/patterns.md

@@ -0,0 +1,60 @@
+---
+title: patterns
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+The following patterns were ran on multiple cloud and on a variety of sizes. A recommended IRQ pattern will be suggested at the end. Based on your workload, a different pattern may result in higher performance.
+
+### Patterns
+1. Default: IRQ pattern provided at boot.
+2. Random: All IRQs are assigned a core and do not overlap with network IRQs.
+3. Housekeeping: All IRQs outside of network IRQs are assign to specific core(s).
+4. NIC IRQs are set to single or multiple ranges of cores and including pairs. EX. 1, 1-2, 0-3, 0-7, [0-1, 2-3..], etc.
+
+
+### Scripts to change IRQ
+
+To change the NIC IRQs or IRQs in general you can use the following scripts.
+
+Housekeeping pattern example, you will need to add more to account for other IRQs on your system
+
+```
+HOUSEKEEP=#core range here
+
+# ACPI:Ged
+for irq in $(awk '/ACPI:Ged/ {sub(":","",$1); print $1}' /proc/interrupts); do
+    echo $HOUSEKEEP | sudo tee /proc/irq/$irq/smp_affinity_list >/dev/null
+done
+```
+
+This is for pairs on a 16 vCPU machine, you will need the interface name.
+
+```
+IFACE=#interface name
+
+PAIRS=("0,1" "2,3" "4,5" "6,7" "8,9" "10,11" "12,13" "14,15")
+
+# Match IRQs for the NIC
+mapfile -t irqs < <(grep "$IFACE-Tx-Rx" /proc/interrupts | awk '{gsub(":","",$1); print $1}')
+
+i=0
+for irq in "${irqs[@]}"; do
+    pair=${PAIRS[$((i % ${#PAIRS[@]}))]}
+    echo "$pair" | sudo tee /proc/irq/$irq/smp_affinity_list >/dev/null
+    echo "Set IRQ $irq -> CPUs $pair"
+    ((i++))
+done
+```
+
+This will assign a specific core(s) to NIC IRQs only
+
+```
+IFACE=#interface name
+
+for irq in $(awk '/$IFACE/ {sub(":","",$1); print $1}' /proc/interrupts); do
+    echo 0-15 | sudo tee /proc/irq/$irq/smp_affinity_list > /dev/null
+done
+```