Is Your Linux Version Hiding Interrupt CPU Usage From You? | Tanel Poder Consulting

[giscus[bot]]

Is Your Linux Version Hiding Interrupt CPU Usage From You? | Tanel Poder Consulting

TL;DR: Some Linux distros and even different kernel versions within a distro have disabled IRQ time accounting. In such case, your monitoring tools will report zero time spent in hardware interrupt handlers (shown as %hi, %irq, hiq, etc). It’s easy to check how your kernel is behaving by looking at /proc/stat and you can still measure interrupt CPU usage using perf and a little creativity.
Index Index Problem Explanation Kernel configuration options Measuring IRQ time with perf Different distros and kernel versions Summary Here’s a reminder that your Linux kernel may be not breaking down hardware interrupt CPU usage separately in its /proc/stat CPU time reporting, depending on the kernel’s compile-time settings. - Linux, Oracle, SQL performance tuning and troubleshooting - consulting & training.

https://tanelpoder.com/posts/linux-hiding-interrupt-cpu-usage/

[iDings]

Now, with IRQ time accounting enabled:
However, process/thread level metrics (like top process-list section) will still lump any IRQ time together with %usr and %sys metrics as Linux doesn’t keep track of “unexpected” IRQ service time at process/thread level.

https://elixir.bootlin.com/linux/v5.4.277/source/kernel/sched/cputime.c#L357

irqtime_account_process_tick
   account_other_time
      steal_account_process_time

[iDings]

irqtime_account_process_tick
   account_other_time
      steal_account_process_time
      irqtime_tick_accounted

when enable CONFIG_IRQ_TIME_ACCOUNTING irqtime_tick_accounted will have the delta irqtime, seems in irqtime_account_process_tick will decrease the other(steal,hardirq,softirq) from cputime cputime -= other.

So from the code, it seems that IRQ time accounting enabled, kernel already spilt out irq from %usr and %sys

[tanelpoder]

Thanks, that's interesting. I wonder if this is something relatively new (can check old versions) or I was unaware of it. I don't have time to test this right now, but it would be interesting to run a process 100% on a CPU, while doing lots of slow interrupts (on a crappy VirtualBox VM for example) using other processes and then see if "getrusage()" says that my process (that never slept) was <100% on CPU over last few seconds.

[iDings]

Thanks for the great blog and the guideline.
I will try to test it: a test process 100% on a CPU and maybe just bind the handware irq(such as ethernet) to the same cpu, then monitor with top or getrusage() to see the result

[tanelpoder]

Yes, that was what I had in mind. Maybe tune the MTU to smallest possible and disable any interrupt coalescing with ethtool :-) Easier to achieve high interrupt overhead on some desktop VM probably, but should be doable on bare metal too, especially if allowing just one CPU to process ethernet interrupts.

I guess the other option would be to use perf record -g on the program at the same time - as perf "cycles" sampling uses NMIs, could just use a really high sampling rate on the CPU you are running your test program on at the same time (added benefit would be that you can see all the perf samples from the same experiment too :-)

[tanelpoder]

Or perhaps two tests are needed, one with perf's NMIs and one with ethernet/hardware interrupts...