Add CPUPSIPressureByUtilization ActualUtilizationProfile

CPU PSI pressure as mesured by the Kernel is not able to
discriminate self imposed pressure due to CPU throttling
on CPU limited containers from pressure caused
by resource contention with other neighbors.
See: kubernetes/enhancements#5062
In oder to filter out evident false positives,
we can introduce a new ActualUtilizationProfile
that is consuming CPU PSI pressure value only when the
actual average CPU utilization on the node is above a certain
threshold (70%), otherwise 0.
This is not a systematic solution but a mitigation.

Fixes: https://issues.redhat.com/browse/OCPBUGS-52341

Signed-off-by: Simone Tiraboschi <[email protected]>

Author: tiraboschi

README.md

@@ -181,9 +181,10 @@ the `profileCustomizations` field:
 ## Prometheus query profiles
 The operator provides the following profiles:
 - `PrometheusCPUUsage`: `instance:node_cpu:rate:sum` (metric available in OpenShift by default)
-- `PrometheusCPUPSIPressure`: `rate(node_pressure_cpu_waiting_seconds_total[1m])` (`node_pressure_cpu_waiting_seconds_total` is a custom metric that needs to be provided)
-- `PrometheusMemoryPSIPressure`: `rate(node_pressure_memory_waiting_seconds_total[1m])` (`node_pressure_memory_waiting_seconds_total` is a custom metric that needs to be provided)
-- `PrometheusIOPSIPressure`: `rate(node_pressure_io_waiting_seconds_total[1m])` (`node_pressure_memory_waiting_seconds_total` is a custom metric that needs to be provided)
+- `PrometheusCPUPSIPressure`: `rate(node_pressure_cpu_waiting_seconds_total[1m])` (`node_pressure_cpu_waiting_seconds_total` is reported in OpenShift only for nodes configured with psi=1 kernel argument)
+- `PrometheusCPUPSIPressureByUtilization`: `avg by (instance) ( rate(node_pressure_cpu_waiting_seconds_total[1m])) and (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1m]))) > 0.7 or avg by (instance) ( rate(node_pressure_cpu_waiting_seconds_total[1m])) * 0` (`node_pressure_cpu_waiting_seconds_total` is reported in OpenShift only for nodes configured with psi=1 kernel argument; the query is filtering out PSI pressure on nodes with average CPU utilization < 0.7 to filter out false positives pressure spikes due to self imposed CPU throttling)
+- `PrometheusMemoryPSIPressure`: `rate(node_pressure_memory_waiting_seconds_total[1m])` (`node_pressure_memory_waiting_seconds_total` is reported in OpenShift only for nodes configured with psi=1 kernel argument)
+- `PrometheusIOPSIPressure`: `rate(node_pressure_io_waiting_seconds_total[1m])` (`node_pressure_memory_waiting_seconds_total` is reported in OpenShift only for nodes configured with psi=1 kernel argument)
 
 ```yaml
 apiVersion: operator.openshift.io/v1

pkg/apis/descheduler/v1/types_descheduler.go

@@ -129,6 +129,8 @@ const (
 	PrometheusCPUUsageProfile ActualUtilizationProfile = "PrometheusCPUUsage"
 	// PrometheusCPUPSIPressureProfile sets rate(node_pressure_cpu_waiting_seconds_total[1m]) query
 	PrometheusCPUPSIPressureProfile ActualUtilizationProfile = "PrometheusCPUPSIPressure"
+	// PrometheusCPUPSIPressureUtilizationProfile sets a query based on a combination of PSI CPU pressure and average CPU utilization
+	PrometheusCPUPSIPressureByUtilizationProfile ActualUtilizationProfile = "PrometheusCPUPSIPressureByUtilization"
 	// PrometheusMemoryPSIPressureProfile sets rate(node_pressure_memory_waiting_seconds_total[1m]) query
 	PrometheusMemoryPSIPressureProfile ActualUtilizationProfile = "PrometheusMemoryPSIPressure"
 	// PrometheusIOPSIPressureProfile sets rate(node_pressure_io_waiting_seconds_total[1m]) query

pkg/operator/target_config_reconciler.go

@@ -672,6 +672,8 @@ func lifecycleAndUtilizationProfile(profileCustomizations *deschedulerv1.Profile
 				query = "instance:node_cpu:rate:sum"
 			case deschedulerv1.PrometheusCPUPSIPressureProfile:
 				query = "rate(node_pressure_cpu_waiting_seconds_total[1m])"
+			case deschedulerv1.PrometheusCPUPSIPressureByUtilizationProfile:
+				query = "avg by (instance) ( rate(node_pressure_cpu_waiting_seconds_total[1m])) and (1 - avg by (instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[1m]))) > 0.7 or avg by (instance) ( rate(node_pressure_cpu_waiting_seconds_total[1m])) * 0"
 			case deschedulerv1.PrometheusMemoryPSIPressureProfile:
 				query = "rate(node_pressure_memory_waiting_seconds_total[1m])"
 			case deschedulerv1.PrometheusIOPSIPressureProfile: