Introducing soft-tainter controller (#464)

Introducing a new controller to dynamically
set/remove a soft taint (effect: PreferNoSchedule)
to/from nodes that the descheduler identified
as overutilized according to the nodeutilization plugin.

Now the nodeutilization plugin in the descheduler
can consume utilization data from actual kubernetes metrics
and not only static reservations (requests).
On the other side, the default kube-scheduler
is only going to ensure that Pod's specific
resource requests can be satisfied.

The soft taint will simply gave an hint to the scheduler
to try avoiding nodes that looks overutilized at
descheduler eyes.
Since it's just a soft taint (do it just if possible)
there is no need to restrict it only to a certain
amount of nodes in the cluster or be strict on error handling.

This could potentially be directly implemented
in the descheduler but it has been judged as out of scope there.
So an external controller seems a more manageable solution.
Since we want to limit the rights to amend nodes to a
dedicated serviceaccount, the new controller will be executed
in a new pod deployed by the operator.
The operator will also deploy the ClusterRole
and the RoleBindings forthis additional operand.

With a fine-grained ValidatingAdmissionPolicy (GA in k8s 1.30)
we can enforce that the softTainter service account
is only allowed to apply/remove soft-taints
(enforcing PreferNoSchedule effect)
and with a predefined key-prefix.
The operator will also deploy the ValidatingAdmissionPolicy.

Finally, a smarter node classification requires a more complex logic
that is implemented on Prometheous side with recording rules.
So the operator will now also deply the PrometheusRule
checking that the containing namespace is correctly labelled
to exposed them.

Signed-off-by: Simone Tiraboschi <[email protected]>

Author: tiraboschi

Dockerfile

@@ -1,10 +1,11 @@
-FROM brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_1.22 as builder
+FROM brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_1.23 as builder
 WORKDIR /go/src/github.com/openshift/cluster-kube-descheduler-operator
 COPY . .
 RUN make build --warn-undefined-variables
 
 FROM registry.redhat.io/rhel9-4-els/rhel-minimal:9.4
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-descheduler-operator/cluster-kube-descheduler-operator /usr/bin/
+COPY --from=builder /go/src/github.com/openshift/cluster-kube-descheduler-operator/soft-tainter /usr/bin/
 RUN mkdir /licenses
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-descheduler-operator/LICENSE /licenses/.
 

Dockerfile.rhel7

@@ -5,6 +5,7 @@ RUN make build --warn-undefined-variables
 
 FROM registry.ci.openshift.org/ocp/builder:rhel-9-base-openshift-4.19
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-descheduler-operator/cluster-kube-descheduler-operator /usr/bin/
+COPY --from=builder /go/src/github.com/openshift/cluster-kube-descheduler-operator/soft-tainter /usr/bin/
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-descheduler-operator/manifests /manifests
 COPY --from=builder /go/src/github.com/openshift/cluster-kube-descheduler-operator/metadata /metadata
 LABEL io.k8s.display-name="OpenShift Descheduler Operator" \

README.md

@@ -168,10 +168,17 @@ as they are commonly encountered during VM eviction and migration.
 Equivalent to enabling both `EvictPodsWithPVC` and `EvictPodsWithLocalStorage` profiles in parallel.
 In the future, additional configurations may be introduced through the operator based on user feedback.
 
+This profile sets a nodeSelector (`kubevirt.io/schedulable=true`)
+in the Descheduler policy to limit its action to nodes that are considered schedulable for `KubeVirt`.
+That nodeSelector is a top level configuration option affecting all the active descheduler profiles:
+this profile is not expected to be combined with other profiles
+unless all profiles are expected to operate over the same set of nodes.
+
 The profile exposes the following customization:
 - `devLowNodeUtilizationThresholds`: Sets experimental thresholds for the LowNodeUtilization strategy.
 - `devActualUtilizationProfile`: Enable load-aware descheduling.
 - `devDeviationThresholds`: Have the thresholds be based on the average utilization.
+- `devEnableSoftTainter`: Have the operator deploying the soft-tainter component to dynamically set/remove soft taints according to the same criteria used for load aware descheduling. Scheduling and descheduling decisions are in general fully independent. In case of load-aware descheduling we can potentially have a relevant asymmetry between the descheduling and successive scheduling decisions. The soft taints set by the descheduler soft-tainter act as a hint for the scheduler to mitigate this asymmetry and foster a quicker convergence.
 
 ### EvictPodsWithPVC
 By default, the operator prevents pods with PVCs from being evicted. Enabling this
@@ -199,7 +206,8 @@ the `profileCustomizations` field:
 |`devEnableEvictionsInBackground`|`bool`| Enables descheduler's EvictionsInBackground alpha feature. The EvictionsInBackground alpha feature is a subject to change. Currently provided as an experimental feature.|
 | `devHighNodeUtilizationThresholds` | `string` | Sets thresholds for the [HighNodeUtilization](https://github.com/kubernetes-sigs/descheduler#highnodeutilization) strategy of the `CompactAndScale` profile in the following ratios: `Minimal` for 10%, `Modest` for 20%, `Moderate` for 30%. Currently provided as an experimental feature.|
 |`devActualUtilizationProfile`|`string`| Sets a profile that gets translated into a predefined prometheus query |
-| `devDeviationThresholds` | `string` | Have the thresholds be based on the average utilization. Thresholds signify the distance from the average node utilization in the following setting: `Low`: 10%:10%, `Medium`: 20%:20%, `High`: 30%:30% |
+| `devDeviationThresholds` | `string` | Have the thresholds be based on the average utilization. Thresholds signify the distance from the average node utilization. An AsymmetricDeviationThreshold will force all nodes below the average to be considered as underutilized to help rebalancing overutilized outliers. Supported settings: `Low`: 10%:10%, `Medium`: 20%:20%, `High`: 30%:30%, `AsymmetricLow`: 0%:10%, `AsymmetricMedium`: 0%:20%, `AsymmetricHigh`: 0%:30% |
+| `devEnableSoftTainter` | `bool` | Have the operator deploying the soft-tainter component to dynamic set/remove soft taints as a hint for the scheduler based on the same criteria used for the descheduling |
 
 ## Prometheus query profiles
 The operator provides the following profiles:

bindata/assets/kube-descheduler/deployment.yaml

@@ -37,7 +37,7 @@ spec:
             readOnlyRootFilesystem: true
             capabilities:
               drop: ["ALL"]
-          image: ${IMAGE}
+          image: ${OPERAND_IMAGE}
           resources:
             requests:
               cpu: "100m"

bindata/assets/kube-descheduler/prometheusrule.yaml

@@ -0,0 +1,39 @@
+---
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: descheduler-rules
+  namespace: openshift-kube-descheduler-operator
+spec:
+  groups:
+    - name: recordingRules.rules
+      rules:
+        - record: descheduler:nodeutilization:cpu:avg1m
+          expr: avg by (instance) (1 - rate(node_cpu_seconds_total{mode='idle'}[1m]))
+
+        - record: descheduler:averageworkersutilization:cpu:avg1m
+          expr: avg(descheduler:nodeutilization:cpu:avg1m * on(instance) group_left(node) label_replace(kube_node_role{role="worker"}, 'instance', "$1", 'node', '(.+)'))
+
+        - record: descheduler:nodepressure:cpu:avg1m
+          # return the cpu pressure if the cpu usage is over 70% otherwise
+          # return cpu pressure as zero to (partially) filter out false
+          # positives pressure spikes due to CPU limited pods.
+          # See: https://github.com/kubernetes/enhancements/issues/5062
+          expr: |-
+            avg by (instance) (
+              rate(node_pressure_cpu_waiting_seconds_total[1m])
+            ) and (
+              1 - avg by (instance) (
+                rate(node_cpu_seconds_total{mode='idle'}[1m])
+              )
+            ) > 0.7
+            or
+            avg by (instance) (
+              rate(node_pressure_cpu_waiting_seconds_total[1m])
+            ) * 0    
+
+        - record: descheduler:combined_utilization_and_pressure:avg1m
+          expr: |-
+            (descheduler:nodeutilization:cpu:avg1m and on() descheduler:averageworkersutilization:cpu:avg1m < 0.8)
+            or
+            (descheduler:nodepressure:cpu:avg1m)

bindata/assets/kube-descheduler/softtainterclusterrole.yaml

@@ -0,0 +1,52 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: openshift-descheduler-softtainter
+rules:
+- apiGroups:
+  - "events.k8s.io"
+  resources:
+  - "events"
+  verbs:
+  - "get"
+  - "watch"
+  - "list"
+  - "create"
+  - "update"
+  - "patch"
+  - "delete"
+- apiGroups:
+  - ""
+  resources:
+  - "nodes"
+  verbs:
+  - "get"
+  - "watch"
+  - "list"
+  - "patch"
+  - "update"
+- apiGroups:
+  - "coordination.k8s.io"
+  resources:
+  - "leases"
+  verbs:
+  - "create"
+- apiGroups:
+  - "coordination.k8s.io"
+  resources:
+  - "leases"
+  resourceNames:
+  - "soft-tainter-lock"
+  verbs:
+  - "get"
+  - "patch"
+  - "update"
+  - "delete"
+- apiGroups:
+  - "operator.openshift.io"
+  resources:
+  - "kubedeschedulers"
+  verbs:
+  - "get"
+  - "watch"
+  - "list"

bindata/assets/kube-descheduler/softtainterclusterrolebinding.yaml

@@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: openshift-descheduler-softtainter
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: openshift-descheduler-softtainter
+subjects:
+  - kind: ServiceAccount
+    name: openshift-descheduler-softtainter
+    namespace: openshift-kube-descheduler-operator

bindata/assets/kube-descheduler/softtainterclusterrolebindingprometheus.yaml

@@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: openshift-descheduler-softtainter-monitoring
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-monitoring-view
+subjects:
+  - kind: ServiceAccount
+    name: openshift-descheduler-softtainter
+    namespace: openshift-kube-descheduler-operator

bindata/assets/kube-descheduler/softtainterdeployment.yaml

@@ -0,0 +1,69 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: "softtainer"
+  namespace: "openshift-kube-descheduler-operator"
+  labels:
+    app: "softtainer"
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: "softtainer"
+  template:
+    metadata:
+      annotations:
+        kubectl.kubernetes.io/default-container: openshift-softtainer
+      labels:
+        app: "softtainer"
+    spec:
+      securityContext:
+        runAsNonRoot: true
+        seccompProfile:
+          type: RuntimeDefault
+      volumes:
+        - name: "policy-volume"
+          configMap:
+            name: "descheduler"
+        - name: certs-dir
+          secret:
+            secretName: kube-descheduler-serving-cert
+      priorityClassName: "system-cluster-critical"
+      restartPolicy: "Always"
+      containers:
+        - name: "openshift-softtainer"
+          securityContext:
+            allowPrivilegeEscalation: false
+            readOnlyRootFilesystem: true
+            capabilities:
+              drop: ["ALL"]
+          image: ${SOFTTAINTER_IMAGE}
+          livenessProbe:
+            failureThreshold: 1
+            httpGet:
+              path: /livez
+              port: 6060
+              scheme: HTTP
+            initialDelaySeconds: 30
+            periodSeconds: 5
+          readinessProbe:
+            failureThreshold: 1
+            httpGet:
+              path: /readyz
+              port: 6060
+              scheme: HTTP
+            initialDelaySeconds: 5
+            periodSeconds: 5
+          resources:
+            requests:
+              cpu: "100m"
+              memory: "500Mi"
+          command: ["/usr/bin/soft-tainter"]
+          args:
+            - --policy-config-file=/policy-dir/policy.yaml
+          volumeMounts:
+            - mountPath: "/policy-dir"
+              name: "policy-volume"
+            - mountPath: "/certs-dir"
+              name: certs-dir
+      serviceAccountName: "openshift-descheduler-softtainter"

bindata/assets/kube-descheduler/softtainterserviceaccount.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: openshift-descheduler-softtainter
+  namespace: openshift-kube-descheduler-operator