[Feature Request] Support detailed disruption reason detection via new metric kube_pod_status_disruption_reason

[kincoy]

What would you like to be added:
Introduce a new metric kube_pod_status_disruption_reason that captures Pod disruption causes by reading the reason from pod.status.conditions (specifically where type == "DisruptionTarget"). This metric should support these possible reasons:

podDisruptionConditionReasons = [
  "PreemptionByScheduler",
  "DeletionByTaintManager",
  "EvictionByEvictionAPI",
  "DeletionByPodGC",
  "TerminationByKubelet"
]

These are the complete set of disruption reasons are defined in the official Kubernetes documentation on pod disruption conditions.

Why is this needed:
The existing kube_pod_status_reason metric is fundamentally limited and often returns zero for all reason labels—even in cases where pods are clearly being evicted. The root cause lies in its implementation: it only matches pod.status.reason == "Evicted", which is only set when eviction is initiated directly by the kubelet (e.g., due to resource pressure or shutdown). This logic completely overlooks evictions triggered via the Eviction API (e.g., using kubectl drain), which instead set pod.status.condition.reason = "EvictionByEvictionAPI" but leave pod.status.reason unchanged, resulting in a constant metric value of zero.

By extending to all known disruption reasons from Kubernetes (as listed above), the proposed metric enables users to accurately identify and monitor diverse Pod disruption events—whether they’re triggered by the Kubelet, scheduler preemption, taint manager, garbage collection, or Eviction API.

Describe the solution you'd like:

• Metric definition: kube_pod_status_disruption_reason (Gauge) that captures disruption cause via pod.status.conditions where type equals “DisruptionTarget.”
• Supported reason values:
  • PreemptionByScheduler
  • DeletionByTaintManager
  • EvictionByEvictionAPI
  • DeletionByPodGC
  • TerminationByKubelet
• Backward compatibility: Retain the legacy kube_pod_status_reason for existing dashboards and alerts, while clarifying in docs when to prefer the new metric.
• Implementation guidance:
  • In KSM implementation, inspect pod.status.conditions for DisruptionTarget conditions and emit the corresponding reason label.
  • Add unit/integration tests to simulate each disruption scenario.
  • Update metrics.md documentation with clear guidance, usage examples, and migration notes.

Additional context:
related issues:

• kube_pod_status_reason is 0 for all reasons #2612
• https://stackoverflow.com/questions/76653767/prometheus-metric-kube-pod-status-reason-returns-hundreds-of-results-but-all-wi

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kincoy]

Additional Verification: kubelet-triggered eviction via evictionHard updates pod.status.reason but API-based eviction does not

To clearly distinguish between eviction mechanisms and verify their observable behaviors, I conducted the following experiment comparing Kubelet-triggered and Eviction API–triggered evictions.

• Kubelet-triggered eviction:

  1. Trigger: Set a strict eviction threshold using Kubelet’s evictionHard (e.g., memory.available: "5Gi") and induce resource pressure to exceed it. This forces eviction due to node pressure.
  
  2. Oberverse Pod Status: The evicted pod had its pod.status.reason field correctly set to "Evicted":

  

  3. Oberverse KMS: The metric kube_pod_status_reason{reason="Evicted"} was correctly recorded and had a value of 1:
  
  4. Conclusion: Evictions triggered by the kubelet correctly populate pod.status.reason, and this is properly reflected by the existing kube_pod_status_reason metric.

• API-triggered eviction (via Eviction API, e.g., kubectl drain):

  1. Trigger: execute kubectl drain .

  

  2. Oberverse Pod Status: The pod was evicted, but this time the pod.status.reason field remained unset. Instead, a DisruptionTarget condition appeared with a reason such as "EvictionByEvictionAPI"

  

  

  3. Oberverse KMS: In this case, kube_pod_status_reason{reason="Evicted"} remained 0, indicating that this eviction path is not captured

  

  
  4. Conclusion: Evictions triggered via the Eviction API (e.g., kubectl drain) do not populate pod.status.reason, and thus the current kube_pod_status_reason metric remains ineffective in these cases.

[kincoy]

I’d be happy to contribute a PR for this if the proposed approach makes sense to the maintainers.
Let me know if there’s anything you’d like me to adjust before moving forward!

[sidewinder12s]

This would be so useful with more dynamic schedulers like Karpenter.

[dshackith]

This would be so useful with more dynamic schedulers like Karpenter.

This was the a reason #2612 was opened. Other related issues (don't mention Karpenter)
#1843
#2116