[AMD][Zen2] Unimplemented Edit Min CPU Frequency

[nirvanaosc]

First of all, congratulations for the amazing job.

I am trying to adjust Min and Max frequencies on a Ryzen 3900X. Adjusting Max Frequency works flawlessly, but when I try to change Min Frequency, it says "Feature is not implemented". I tried both main and develop branches.

Kernel
5.6.0-050600-lowlatency

Kernel params
cpufreq.off=1 modprobe.blacklist=acpi_cpufreq,pcc_cpufreq idle=halt tsc=unstable notsc nowatchdog

ko params
Register_ClockSource=1 Register_Governor=1 Register_CPU_Freq=1 Register_CPU_Idle=1 Idle_Route=1 Experimental=1 PC6_Enable=0 Clock_Ratio_Unlock=3

Alternatively, is it possible to change C1 frequency?

Thank you very much in advance.

[cyring]

Hello,

Thank you for using CoreFreq

So far, I've not find a mean to rewrite the Min Frequency Ratio.
That undocumented MSR appears to be read-only

CoreFreq/corefreqk.c

Line 5280 in b263200

RDMSR(PstateDef, MSR_AMD_PC6_F17H_STATUS);

• You can try to read MSR
  

  CoreFreq/amdmsr.h

  Line 137 in b263200

  #define MSR_AMD_PC6_F17H_STATUS 0xc0010292

modprobe msr
rdmsr -aX 0xc0010292
104000012
...
104000012

• With the output value and the following P-State bits layout
  

  CoreFreq/amdmsr.h

  Line 466 in b263200

  CpuFid : 8-0, /* Core Frequency ID. RW: FFh-10h <Value>*25 */

... we found a CPU FID of 0x12 (18) and a VID of zero

thus we use formula FID / 4 = 18 / 4 = 4.5 rounded in CoreFreq to a Frequency Ratio of 4

• Let's try to write a ratio of 6 aka a FID of 0x18 (24)

wrmsr -a 0xc0010292 0x104000018
rdmsr -aX 0xc0010292
104000012
...
104000012

... The Ratio is unchanged.

• That's why in CoreFreq there is no implementation behind Clock_Ratio_Unlock=3

[nirvanaosc]

Thank you very much for your fast response.

That's a bit unfortunate.

I tried using wrmsr myself and got the same results (I wasn't expecting otherwise). Do you know if it is possible to edit C-States frequency, and if so, how I can use wrmsr to alter them? even just C1 frequency would do it for me, as I'm using idle=halt.

I had a look at the code to try to figure out C-States address and layout so I could give it a try without bothering you, but it is not as straightforward as I need it to be given my very limited knowledge on this topic.

[cyring]

Thank you very much for your fast response.

That's a bit unfortunate.

I tried using wrmsr myself and got the same results (I wasn't expecting otherwise). Do you know if it is possible to edit C-States frequency, and if so, how I can use wrmsr to alter them? even just C1 frequency would do it for me, as I'm using idle=halt.

I have not found any such register which deals about C-State and an associated frequency.

• The AMD PPR is the specification to look at for Zen MSR.
  Go to the Developer Guides, Manuals & ISA Documents, you will find one PPR per Model (most are similar).
  Unlike the Intel SDM specification, no C-State counters (MSR or CSR) are published in the AMD documentation.
  You can however retrieve their percent (only) values from the SMU tables.

I had a look at the code to try to figure out C-States address and layout so I could give it a try without bothering you, but it is not as straightforward as I need it to be given my very limited knowledge on this topic.

• As a workaround I have programmed atomic software counters which, based on the TSC, are measured each time the kernel is entering its idle loop:
  

  CoreFreq/corefreqk.c

  Line 13474 in 717e444

  #define Atomic_Write_VPMC( _Core, cycles, _lvl) \

  
  And this is how I'm providing C1, based on those VPMC:
  

  CoreFreq/corefreqk.c

  Line 9258 in 717e444

  Atomic_Read_VPMC(LOCKLESS, Core->Counter[T].C1, Core->VPMC.C1); \

• You may have noticed that when CoreFreq is fully registered to the cpu-idle Kernel, a new idle limit level can be set.

• The Idle_Route=<N> parameter of corefreqk.ko will direct the Kernel to one of the low level idle triggers: 1:I/O; 2:HALT; 3:MWAIT
  

  CoreFreq/corefreqk.c

  Line 13833 in 717e444

  switch (Idle_Route) {

• Used by Kernel, the per idle state Latency and Residency can be customized here:
  

  CoreFreq/corefreqk.h

  Line 5015 in 717e444

  static IDLE_STATE Zen_IdleState[] = {

In short, those VPMC provide an estimated frequency per idle state but they are off course less accurate than hardware counters

[nirvanaosc]

Thank you very much again for your fast and detailed response.

As a workaround I have programmed atomic software counters which, based on the TSC

I see. That is very clever. Despite those virtual performance counters not being 100% accurate, they are accurate enough to see that C1 frequency is too low for my needs (around 3.6GHz).

You may have noticed that when CoreFreq is fully registered to the cpu-idle Kernel, a new idle limit level can be set.

Yes, I have been playing with idle limits (C1/poll) to see what suits me best.

Long story short, I am running a latency-sensible workload. Unfortunately, the CPU's default boosting behaviour doesn't do a good job at maintaining high frequencies, probably because the workload is too light. The main thread's core's frequency is constantly jumping between 3.8GHz and 4.4Ghz. Each thread has a single physical core assigned to it, in combination with isolcpus to make sure nothing else is running on those cores, but most of the threads are idling most of the time waiting for work to be done. Fixing the whole CPU to 4.4GHz with manual OC in the BIOS has a big impact on the main thread's performance. I also have the feeling that the CPU doesn't boost its frequency fast enough (if at all) when there is work to be done on the other threads (due to it being short and light the majority of the time).

Ideally, I'd like each core to be at its maximum possible frequency at all times without polling during idle, so other cores have more power and temperature headroom to boost higher. Basically having the benefits of idle-polling without its drawbacks.

I can't think of anything else to achieve those results, but if you have any ideas please let me know. I'll be more than happy to explore other avenues.

Thank you very much again for your time.

[h1z1]

Have you tried something like zenstates.py? ex https://github.com/r4m0n/ZenStates-Linux/blob/master/zenstates.py I run into the same issues especially with early threadrippers.

[nirvanaosc]

I gave it a quick try, but unfortunately it doesn't offer any benefits over CoreFreq. Thank you for mentioning though.

[h1z1]

Wasn't aware corefreq could edit pstates though?

[nirvanaosc]

I don't really need to. Editing Max frequency has the same effect

[h1z1]

Maybe I misunderstood your intent. I thought the issue was clock variably with the cpu staying too low thus causing latency spikes? You can force the cpu to hold on one clock by disabling all but one P-state (or whatever fits your use). How high depends on your hardware.

I can absolutely attest to Zen hardware doing weird things, one of which is instability with hpet and tsc.