[Bug]: `found duplicate series` in `KubeletTooManyPods` alert

[jonasbadstuebner]

What happened?

I got a prometheus rule evaluation failure alert for the kubelet alert:

found duplicate series for the match group {namespace="monitoring", pod="alertmanager-kube-prometheus-stack-alertmanager-0"} on the right hand-side of the operation: [{namespace="monitoring", node="node-01.mydomain.de", pod="alertmanager-kube-prometheus-stack-alertmanager-0"}, {namespace="monitoring", node="node-02.mydomain.de", pod="alertmanager-kube-prometheus-stack-alertmanager-0"}];many-to-many matching not allowed: matching labels must be unique on one side

I would expect the alert to work with the default labels. I think the grouping has to be rechecked.

https://github.com/kubernetes-monitoring/kubernetes-mixin/pull/1011/files#r1905954328 states there is an existing test for the change, but my problem seems to be not covered by it.
It seems to expect the pod-id to change, but it doesn't for a StatefulSet.

A solution here would be to use the uid of the pod, I provided a snippet that works for me.

Please provide any helpful snippets.

# The promrule as applied in the cluster
- alert: KubeletTooManyPods
  annotations:
    description: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage
      }} of its Pod capacity on cluster {{ $labels.cluster }}.
    runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubelettoomanypods
    summary: Kubelet is running at capacity.
  expr: |-
    count by (cluster, node) (
      (kube_pod_status_phase{job="kube-state-metrics", phase="Running"} == 1)
      * on (cluster, namespace, pod) group_left (node)
      group by (cluster, namespace, pod, node) (
        kube_pod_info{job="kube-state-metrics"}
      )
    )
    /
    max by (cluster, node) (
      kube_node_status_capacity{job="kube-state-metrics", resource="pods"} != 1
    ) > 0.95
  for: 15m
  labels:
    severity: warning
# suggested change
count by (cluster, node) (
  (kube_pod_status_phase{job="kube-state-metrics", phase="Running"} == 1)
  * on (cluster, namespace, pod, uid) group_left (node)
  group by (cluster, namespace, pod, uid, node) (
    kube_pod_info{job="kube-state-metrics"}
  )
)
/
max by (cluster, node) (
  kube_node_status_capacity{job="kube-state-metrics", resource="pods"} != 1
)

What parts of the codebase are affected?

Alerts

I agree to the following terms:

• I agree to follow this project's Code of Conduct.
• I have filled out all the required information above to the best of my ability.
• I have searched the issues of this repository and believe that this is not a duplicate.
• I have confirmed this bug exists in the default branch of the repository, as of the latest commit at the time of submission.

[skl]

Thanks for the report, @jonasbadstuebner! I'll take a look and add a test case for this.

[jonasbadstuebner]

It seems to expect the pod-id to change, but it doesn't for a StatefulSet.

I meant pod name, of course. Not the uid.

[skl]

@jonasbadstuebner I came up with a more efficient query here:

• fix: KubeletTooManyPods duplicate series error #1017

Feedback welcome!

[jonasbadstuebner]

I'm sorry I didn't answer over the weekend.
The label_replace in your proposed solution is incorrect. It's replacing the node name with the IP of the node.
kubelet_running_pods already has a node label that has the node name as value, so you can drop the label_replace.

[jonasbadstuebner]

max by (cluster, node) (
    kubelet_running_pods{job="kubelet"} > 1
)
/
max by (node) (
    kube_node_status_capacity{job="kube-state-metrics", resource="pods"} != 1
) > 0.95

This query already works. No need for the label_replace.

[skl]

@jonasbadstuebner apologies if I merged too quickly, I was just clearing out all my PRs for the week 😅

Thanks for the feedback! I guess that depends on your scrape configuration, I've just had a look at a bunch of clusters and I see mixed usage of instance with the node label not always present.

How about if I get the node name from kubelet_node_name instead? This should be a more reliable way to get the node name, regardless of scrape configuration. What do you think?

max by (cluster, instance) (kubelet_running_pods > 1)
* on (cluster, instance) group_left(node)
max by (cluster, instance, node) (kubelet_node_name)

[skl]

@jonasbadstuebner I've created a follow-up PR to address:

• fix(KubeletTooManyPods): handle inconsistent instance labels #1021

[jonasbadstuebner]

This query would work for me/us.
Looks good to me. Thank you for fixing it this swiftly!