kube_pod_container_status_restarts_total doesn't return 0 by default

[trip-pguedes]

What happened: kube_pod_container_status_restarts_total metric only returns a value when restart happens, returning null while pod has 0 restarts

What you expected to happen: kube_pod_container_status_restarts_total metric returns 0 restarts while a pod has not restarted ever, and then returns the number of restarts if restarts happen

How to reproduce it (as minimally and precisely as possible):

• Query prometheus metrics in prometheus itself

Shows as such:

Expected to show as such:

Anything else we need to know?:

Environment:

• kube-state-metrics version:v2.14.0 (using kube-prometheus-stack)
• Kubernetes version (use kubectl version): 1.30 (EKS)
• Cloud provider or hardware configuration: Amazon
• Other info: prometheus:v2.55.1

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[richabanker]

Hi @trip-pguedes , why is the nil being returned here not sufficient in case of 0 pod restarts? The intnetion was to only emit non zero values for the metric to save resource consumption.

[trip-pguedes]

If you try and write a Grafana query based on this metric to ward when the first restart happens, the increase won't work, because it passes from Null value to 1, and if I alert based solely on number, it stays as 1 for the entirety of the pods life, which means it will keep alerting.

So far I couldn't create an alert that could work the increase or rate to track when a pod restarted.

Let me know if I am missing something

[trip-pguedes]

Hello, any news on this @richabanker ? No one else had this issue before?

[richabanker]

Sorry for the delay @trip-pguedes , I see that you're trying to setup an alert on the kube_pod_container_status_restarts_total metric which is a counter and the absence of 0s for the case when no restarts happen, makes it difficult to track the first increase seen in the metric. Which makes the increase(), delta(), rate() functions unusable.

@CatherineF-dev @rexagod seems like a missed usecase here, do you have concerns on emitting 0s in the case of no pod restarts here?

[richabanker]

cc @dgrisonnet

[richabanker]

Not a promQL expert myself, but wondering if its possible to just default the no_data / null value to 0 in the promQL and use that for your alert?

[trip-pguedes]

Was not able to with ("OR vector(0)") for example
sum by (pod) (kube_pod_container_status_restarts_total) OR vector(0)

Edit: Changed the query as I was using another metric, not the one discussed in this thread

[trip-pguedes]

Any developments?

[trip-pguedes]

@richabanker @dgrisonnet any developments?

[trip-pguedes]

@richabanker @dgrisonnet any developments?