Merge pull request #51612 from amolnar-rh/TELCODOCS-644

TELCODOCS-644/645: Per-Core Runtime Tuning of power states - CRI-O and PAO/NTO

Author: mburke5678

modules/cnf-provisioning-real-time-and-low-latency-workloads.adoc

@@ -231,8 +231,6 @@ status:
   runtimeClass: performance-manual
 ----
 
-* The pod must have the `cpu-load-balancing.crio.io: true` annotation.
-
 The Node Tuning Operator is responsible for the creation of the high-performance runtime handler config snippet under relevant nodes and for creation of the high-performance runtime class under the cluster. It will have the same content as default runtime handler except it enables the CPU load balancing configuration functionality.
 
 To disable the CPU load balancing for the pod, the `Pod` specification must include the following fields:
@@ -325,4 +323,121 @@ spec:
 [source,terminal]
 ----
 $ oc apply -f my-performance-profile.yaml
-----
+----
+
+[id="node-tuning-operator-pod-power-saving-config_{context}"]
+== Optional: Power saving configurations
+
+You can enable power savings for a node that has low priority workloads that are colocated with high priority workloads without impacting the latency or throughput of the high priority workloads. Power saving is possible without modifications to the workloads themselves.
+
+[IMPORTANT]
+====
+The feature is supported on Intel Ice Lake and later generations of Intel CPUs. The capabilities of the processor might impact the latency and throughput of the high priority workloads.
+====
+
+When you configure a node with a power saving configuration, you must configure high priority workloads with performance configuration at the pod level, which means that the configuration applies to all the cores used by the pod.
+
+By disabling P-states and C-states at the pod level, you can configure high priority workloads for best performance and lowest latency.
+
+.Power saving configurations
+[cols="1,2", options="header"]
+|====
+|Annotation
+|Description
+
+a|[source,yaml]
+----
+annotations:
+  cpu-c-states.crio.io: "enable"
+  cpu-freq-governor.crio.io: "<governor>"
+----
+|Provides the best performance for a pod by disabling C-states and specifying the governor type for CPU scaling. The `performance` governor is recommended for high priority workloads.
+|====
+
+
+.Prerequisites
+
+* You enabled C-states and OS-controlled P-states in the BIOS
+
+.Procedure
+
+. Generate a `PerformanceProfile` with `per-pod-power-management` set to `true`:
++
+[source,terminal]
+----
+$ podman run --entrypoint performance-profile-creator -v \
+/must-gather:/must-gather:z registry.redhat.io/openshift4/performance-addon-rhel8-operator:v4.11 \
+--mcp-name=worker-cnf --reserved-cpu-count=20 --rt-kernel=true \
+--split-reserved-cpus-across-numa=false --topology-manager-policy=single-numa-node \
+--must-gather-dir-path /must-gather -power-consumption-mode=low-latency \ <1>
+--per-pod-power-management=true > my-performance-profile.yaml
+----
+<1> The `power-consumption-mode` must be `default` or `low-latency` when the `per-pod-power-management` is set to `true`.
+
++
+.Example `PerformanceProfile` with `perPodPowerManagement`
+
+[source,yaml]
+----
+apiVersion: performance.openshift.io/v2
+kind: PerformanceProfile
+metadata:
+     name: performance
+spec:
+    [.....]
+    workloadHints:
+        realTime: true
+        highPowerConsumption: false
+        perPodPowerManagement: true
+----
+
+. Set the default `cpufreq` governor as an additional kernel argument in the `PerformanceProfile` custom resource (CR):
++
+[source,yaml]
+----
+apiVersion: performance.openshift.io/v2
+kind: PerformanceProfile
+metadata:
+     name: performance
+spec:
+    ...
+    additionalKernelArgs:
+    - cpufreq.default_governor=schedutil <1>
+----
+<1> Using the `schedutil` governor is recommended, however, you can use other governors such as the `ondemand` or `powersave` governors.
+
+. Set the maximum CPU frequency in the `TunedPerformancePatch` CR:
++
+[source,yaml]
+----
+spec:
+  profile:
+  - data: |
+      [sysfs]
+      /sys/devices/system/cpu/intel_pstate/max_perf_pct = <x> <1>
+----
+<1> The `max_perf_pct` controls the maximum frequency the `cpufreq` driver is allowed to set as a percentage of the maximum supported cpu frequency. This value applies to all CPUs. You can check the maximum supported frequency in `/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_max_freq`.
+
+
+. Add the desired annotations to your pods. The annotations override the `default` settings.
++
+.Example power saving annotation
+[source,yaml]
+----
+apiVersion: v1
+kind: Pod
+metadata:
+  ...
+  annotations:
+    ...
+    cpu-c-states.crio.io: "enable"
+    cpu-freq-governor.crio.io: "<governor>"
+    ...
+  ...
+spec:
+  ...
+  runtimeClassName: performance-<profile_name>
+  ...
+----
+
+. Restart the pods.

modules/cnf-understanding-workload-hints.adoc

@@ -49,4 +49,15 @@ realTime: true
     | Far edge clusters, latency critical workloads
     | Optimized for absolute minimal latency and maximum determinism at the cost of increased power consumption.
 
-|===
+    | Per-pod power management
+    a|[source,terminal]
+----
+workloadHints:
+realTime: true
+highPowerConsumption: false
+perPodPowerManagement: true
+----
+    | Critical and non-critical workloads
+    | Allows for power management per pod.
+
+|===

modules/ztp-du-bios-config-reference.adoc

@@ -65,8 +65,8 @@ The following table is a general recommendation for vDU cluster host BIOS config
 |Disable Energy Efficient Turbo to prevent the processor from using an energy-efficiency based policy.
 
 |Hardware P-States
-|Disabled
-|Disable `P-states` (performance states) to optimize the operating system and CPU for performance over power consumption.
+|Enabled or Disabled
+|Enable OS-controlled P-States to allow power saving configurations. Disable `P-states` (performance states) to optimize the operating system and CPU for performance over power consumption.
 
 |Package C-State
 |C0/C1 state
@@ -89,3 +89,8 @@ The following table is a general recommendation for vDU cluster host BIOS config
 ====
 Enable global SR-IOV and VT-d settings in the BIOS for the host. These settings are relevant to bare-metal environments.
 ====
+
+[NOTE]
+====
+Enable both `C-states` and OS-controlled `P-States` to allow per pod power management.
+====

scalability_and_performance/cnf-low-latency-tuning.adoc

@@ -17,6 +17,11 @@ include::modules/cnf-about_hyperthreading_for_low_latency_and_real_time_applicat
 
 include::modules/cnf-provisioning-real-time-and-low-latency-workloads.adoc[leveloffset=+1]
 
+[role="_additional-resources"]
+.Additional resources
+
+* For more information about recommended BIOS configuration, see xref:../scalability_and_performance/ztp-vdu-validating-cluster-tuning.adoc#ztp-du-bios-config-reference_vdu-config-ref[Recommended BIOS configuration for vDU cluster hosts].
+
 include::modules/cnf-managing-device-interrupt-processing-for-guaranteed-pod-isolated-cpus.adoc[leveloffset=+2]
 
 include::modules/cnf-use-device-interrupt-processing-for-isolated-cpus.adoc[leveloffset=+2]