[Enhancement]: Kube(CPU|Memory)QuotaOvercommit alerts use max number of nodes instead of current ones

[afarbos]

What's the general idea for the enhancement?

Currently both alerts (KubeCPUQuotaOvercommit and KubeMemortQuotaOvercommit) compute the total available capacity using the current amount of nodes in the cluster.
However, this alert can become really noisy if the current amount of nodes is not the maximum amount of nodes available.
Today, a lot of the time k8s clusters are deployed with cluster autoscaler or come with it builtin (AKS, GKE...).
We should have a way to influence those alerts to retrieve the maximum amount of nodes instead of looking at the current one.

See this metric that could be helpful for example: see https://github.com/kubernetes/autoscaler/blob/213a8595ea2bddf433dd56e50c31ca868ef1da80/cluster-autoscaler/metrics/metrics.go#L157-L163

Please provide any helpful snippets.

No response

What parts of the codebase does the enhancement target?

Alerts

Anything else relevant to the enhancement that would help with the triage process?

No response

I agree to the following terms:

• I agree to follow this project's Code of Conduct.
• I have filled out all the required information above to the best of my ability.
• I have searched the issues of this repository and believe that this is not a duplicate.
• I have confirmed this proposal applies to the default branch of the repository, as of the latest commit at the time of submission.

[Fresa]

The same things goes for KubeCPUOvercommit and KubeMemoryOvercommit.

We're hosting on AKS with the cluster auto scaler provisioned as a managed add-on, which means no metrics available that can be scraped by BYO Prometheus. We solved this by extracting the VMSS name from the node labels on the metrics that these alerts use.

metricRelabelings:
  - targetLabel: node_group
    sourceLabels:
      - node
    # Node name example in a VMSS: aks-foo-12345678-vmss00000v
    regex: ^aks-(.*)-([0-9]*)-vmss(.*)$
    replacement: ${1}
    action: replace

We then rewrote the alerts to group by the node_group label, and added a condition to only consider overcommitment if the node pool is using maximum number of nodes. KubeCPUOvercommit as an example becomes:

sum by (node_group, cluster) (kube_node_info{job="kube-state-metrics"}) >= 
(
label_replace(label_replace(vector(5), "node_group", "foo", "", ""), "cluster", "foo.bar.example.com", "", "") or
label_replace(label_replace(vector(10), "node_group", "bar", "", ""), "cluster", "foo.bar.example.com", "", "")
) 

and 

sum by (node_group, cluster) (namespace_cpu:node_group:kube_pod_container_resource_requests:sum) - (sum by (node_group, cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) - max by (node_group, cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"})) > 0 

and 

(sum by (node_group, cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"}) - max by (node_group, cluster) (kube_node_status_allocatable{job="kube-state-metrics",resource="cpu"})) > 0`

Note that this alert rule refers to a recording rule which also needs to consider the node_group label:

- record: namespace_cpu:node_group:kube_pod_container_resource_requests:sum
  expr: |-
    sum by (namespace, node_group, cluster) (
        sum by (namespace, pod, node_group, cluster) (
            max by (namespace, pod, container, node_group, cluster) (
              kube_pod_container_resource_requests{resource="cpu",job="kube-state-metrics"}
            ) * on (namespace, pod, cluster) group_left() max by (namespace, pod, cluster) (
              kube_pod_status_phase{phase=~"Pending|Running"} == 1
            )
        )
    )

If we had metrics from the auto scaler, or were using Karpenter, we'd probably have tried to source the max node values from there instead of constructing instant vectors statically.