Adaptive tickless mode (aka full dynticks)

[raykzhao]

Hi everyone,

Since now CONFIG_NO_HZ_FULL is recommended in CacULE, here is the discussion thread for the technical details and feedback for using the adaptive tickless mode with CacULE.

Notes:

1. The adaptive tickless mode must be manually enabled in nohz_full= kernel parameter, and CPU0 should be excluded from the list. For example, if you have a 4-core CPU (w/o hyperthreading), you may use nohz_full=1-3 to enable adaptive tickless mode on all cores except CPU0.
2. In addition, as mentioned here, if your CPU supports hyperthreading, both threads from the same physical core must enable/disable adaptive tickless mode at the same time. For example, if you have a 4-core CPU with 8 threads, you may use nohz_full=1-3,5-7 to enable adaptive tickess mode on all physical cores and their sibling threads except CPU0.

[JohnyPeaN]

Hi,
thanks for explanation and usefull links. I have found that good tool for identifiing siblings is lstopo.

Here is topology of Ryzen 3900X with SMT:

siblings are 0-12, 1-13, 2-14...etc. So nohz clause should look like nohz_full=1-11,13-23

Intel i7-7700K cpu with HT:

siblings are 0-4,1-5,2-6,3-7 so nohz clause should look like nohz_full=1-3,5-7

isolcpus= seems to "blacklist" specified cores from scheduling to them. Userspace won't run on them until explicitly specified by some tool like taskset. This is usefull for benchmarks or virtualisation, otherwise it has the effect of disabling cores for normal desktop usage.

[hamadmarri]

siblings are 0-12, 1-13, 2-14...etc. So nohz clause should look like nohz_full=1-11,13-23

Hi @JohnyPeaN

Can we always assume a sibling cpu is:

s = cpu# + (nr_cpus / 2)

so for cpu2 in your Ryzen 3900X example:
s = 2 + ( 24 / 2 ) = 14

?

On my machine, lstopo shows

means 0-2, 1-3 are siblings?
nohz_full=1,3 ?

Thank you so much for the illustrations

[JohnyPeaN]

According to your output siblings are 0-1 and 2-3, which is a different numbering scheme and your proposed calculation wouldn't work. In this case kernel command should be nohz_full=2-3. What machine is it?

[hamadmarri]

According to your output siblings are 0-1 and 2-3, which is a different numbering scheme and your proposed calculation wouldn't work. In this case kernel command should be nohz_full=2-3. What machine is it?

Hi @JohnyPeaN

Shouldn't be nohz_full=1,3 ? since 2 and 3 are siblings?

It's thinkpad x1 carbon

Thank you

[JohnyPeaN]

The processor seems to be:
Intel Core i7-3667U processor (2 cores, 2.0GHz, 4MB cache)
I couldn't find details explaining difference in its topology. It might be that kernel is just counting the cores differently.

What I have found out so far is this:
nohz_full should be 2,3 in this case exactly because they are siblings and therefore influence each other. You dont get benefits of nohz_full on core 3 if core 2 gets woken up on scheduling ticks. For example if both siblings have ticks shut off (which nohz_idle does), they can go in deeper C states for longer. In reality HT or SMT siblings are just one physical core with many components shared between the two logical cores.
The whole point is to "unselect" the correct sibling together with CPU0 from nohz clause, because the ticks on CPU0 influence its sibling anyway.
But now that I am thinking about it, there might be a catch. nohz_full gives us reduced ticks when there is only 1 task on given cpu. Reduced ticks on idle cpu are achieved with nohz_idle already. So excluding the correct sibling with CPU0 might not have great benefit when CPU0 is active. But it is still recommended. Might be interesting to study it deeper.

[hamadmarri]

The processor seems to be:
Intel Core i7-3667U processor (2 cores, 2.0GHz, 4MB cache)
I couldn't find details explaining difference in its topology. It might be that kernel is just counting the cores differently.

What I have found out so far is this:
nohz_full should be 2,3 in this case exactly because they are siblings and therefore influence each other. You dont get benefits of nohz_full on core 3 if core 2 gets woken up on scheduling ticks. For example if both siblings have ticks shut off (which nohz_idle does), they can go in deeper C states for longer. In reality HT or SMT siblings are just one physical core with many components shared between the two logical cores.
The whole point is to "unselect" the correct sibling together with CPU0 from nohz clause, because the ticks on CPU0 influence its sibling anyway.
But now that I am thinking about it, there might be a catch. nohz_full gives us reduced ticks when there is only 1 task on given cpu. Reduced ticks on idle cpu are achieved with nohz_idle already. So excluding the correct sibling with CPU0 might not have great benefit when CPU0 is active. But it is still recommended. Might be interesting to study it deeper.

Thank you so much @JohnyPeaN . It is crystal clear now.

[FerdiScholten]

On Haswell mobile quadcores, this is the topology:

So here siblings are 0-1 2-3 4-5 6-7
Kernel command nohz_full=2-7 should be good.
This topology also applies to Broadwell

The topology for Intel processors changed with Skylake to the one where hamadmarri's calculation works.

[raykzhao]

Hi @FerdiScholten @hamadmarri

I think this might be because the housekeeping CPU needs to handle all the scheduling-clock interrupts, as mentioned in the kernel document:

If there might potentially be some adaptive-ticks CPUs, there will be at least one CPU keeping the scheduling-clock interrupt going, even if all CPUs are otherwise idle.

So it is possible that the housekeeping CPU will never enter the idle state and it will probably keep running at high frequency.

[FerdiScholten]

The housekeeping core's are reporting frequency correctly and do enter idle state when there are no tasks running.

[ptr1337]

I dont really know about that topic much, but could think that it could be a patch from xanmods kernel or it could be also the pstate driver which is not working correctly or the cstates.

[hamadmarri]

Just something I noticed, for the cores running tickless, the CPU frequency is always reported as the maximum frequency supported by the processor. Only the cores that do use ticks report the actual running frequency.
I really do not believe those cores are running at max frequency constantly, even when the system is idle. Would be nice to be able to check those frequencies, is this a known thing?

Hi @FerdiScholten

Please note that the current RDB is not updating cpu frequency correctly. Please see here:
#33 (comment)

However, if you are experiencing the same issue with CFS, then the issue could be different.
I got this from one of my friend on telegram send me this:

commit 99f925947ab0fd5c17b74460d8b32f1aa1c86e3a
Merge: 191aaf6cc4a7 68d7a190682a
Author: Linus Torvalds <[email protected]>
Date:   Sat Jun 12 11:41:28 2021 -0700

    Merge tag 'sched-urgent-2021-06-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
    
    Pull scheduler fixes from Ingo Molnar:
     "Misc fixes:
    
       - Fix performance regression caused by lack of intended batching of
         RCU callbacks by over-eager NOHZ-full code.
    
       - Fix cgroups related corruption of load_avg and load_sum metrics.
    
       - Three fixes to fix blocked load, util_sum/runnable_sum and util_est
         tracking bugs"
    
    * tag 'sched-urgent-2021-06-12' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
      sched/fair: Fix util_est UTIL_AVG_UNCHANGED handling
      sched/pelt: Ensure that *_sum is always synced with *_avg
      tick/nohz: Only check for RCU deferred wakeup on user/guest entry when needed
      sched/fair: Make sure to update tg contrib for blocked load
      sched/fair: Keep load_avg and load_sum synced

It seems the mainstream kernel team are working on fixing some issues related to nohz full.

Thank you

[FerdiScholten]

Today I finally got to test this on CFS, and the issue is exactly the same there, tickless cores only report max frequency.