Tracking time when job fails, or pod terminates after failure

[sleepyfoodie]

What would you like to be added:
Thanks for taking look at this. Apologies if this already exists. I tried using all of these metrics to find the end time for failed jobs. But it doesn't look like the time when a job fails gets recorded.

• kube_pod_completion_time
• kube_job_status_completion_time
• kube_pod_container_status_last_terminated_timestamp

This is the cronjob I was running for testing:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: failing-random-half
spec:
  schedule: "*/2 * * * 1-5"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: failing-random-half
            image: busybox:1.28
            imagePullPolicy: IfNotPresent
            command: ["/bin/sh", "-c", "x=$((RANDOM % 2)); echo $x; echo 'this job fails if it returns 1'; echo 'sleeping for 60-120 seconds'; echo 'exit code:', $x, '🏃‍♀️‍➡️'; echo '💀'; echo 'job namespace: default'; sleep $((60 + $((RANDOM % 60)))); exit $x"]
          restartPolicy: Never

Why is this needed:
I'm working on a dashboard that tracks failed jobs.

Describe the solution you'd like
I'm looking for the AGE for the STATUS Failed ones, from the kubectl get jobs command, which returns this:

NAME                           STATUS     COMPLETIONS   DURATION   AGE
failing-random-half-29241256   Failed     0/1           158m       158m
failing-random-half-29241408   Complete   1/1           5m14s      6m59s

or AGE from ubectl get pods -A for the ones with STATUS Error

NAMESPACE         NAME                                                         READY   STATUS      RESTARTS        AGE
default           failing-random-half-29241256-2n2lg                           0/1     Error       0               140m
default           failing-random-half-29241256-5ntbj                           0/1     Error       0               154m
default           failing-random-half-29241256-9sk6g                           0/1     Error       0               157m
default           failing-random-half-29241256-h6tzh                           0/1     Error       0               147m
default           failing-random-half-29241256-jgpz9                           0/1     Error       0               156m

or the Finished time like Wed, 06 Aug 2025 06:37:46 -0400 from this command
kubectl describe pod failing-random-half-29241256-2n2lg

which returns this:

    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      x=$((RANDOM % 2)); echo $x; echo 'this job fails if it returns 1'; echo 'sleeping for 60-120 seconds'; echo 'exit code:', $x, '🏃‍♀️‍➡️'; echo '💀'; echo 'job namespace: default'; sleep $((60 + $((RANDOM % 60)))); exit $x
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 06 Aug 2025 06:35:48 -0400
      Finished:     Wed, 06 Aug 2025 06:37:46 -0400
    Ready:          False
    Restart Count:  0
    Environment:    <none>

Additional context

[assu-2000]

hello @sleepyfoodie ,

No need to apologize! All metrics are documented at the repo level docs/metrics//workload/object with all information you may need and their description. for this case, please refer here docs/metrics/workload/pod-metrics.md

You're correct that kube_job_status_completion_time is not emitted for failed jobs. This is because KSM faithfully reflects the state of the k8s API. The Job resource in k8s only populates its .status.completionTime field upon successful completion. for a Job that has failed, this field remains empty. so since the field is empty in the API object, KSM has no data to create the metric from.

The solution is to get the termination time from the job's pod. so the only metric needed cloud be the kube_pod_container_status_last_terminated_timestamp.

The source code (internal/store/pod.go-L380) generates this metric's value directly from the pod's .status.containerStatuses.lastTerminationState.terminated.finishedAt field.

So, would you mind sharing a screenshot or the actual output for kube_pod_container_status_last_terminated_timestamp in your case? Since you mentioned you've tried it, it’d help to see what exactly is being emitted on your side bcz this is the metric that’s supposed to hold the timestamp info we’re after.

[dgrisonnet]

/triage accepted

[sleepyfoodie]

hi @assu-2000, thanks so much for your time 🙏

So I created a second cronjob that is exactly the same, except I changed restartPolicy to OnFailure. And I am getting data in kube_pod_container_status_last_terminated_timestamp this time 👀. Here's some screenshot of the cronjob that had no data.

Here's an example of a job (spawned by a cronjob) that was from a kubectl command which failed:

Running this query, it will show records of failure on the job:

kube_job_status_failed{cluster="random-bash-fail", namespace="default", job_name="failing-random-half-29244002", reason!=""} > 0

Running this query, I can also see that there's data on when job completes:

kube_job_status_completion_time{cluster="random-bash-fail", namespace="default", job_name=~"failing-random-half-2924453.*"}

Using this query, I wasn't able to get any data from kube_pod_container_status_last_terminated_timestamp 🤔 In fact, it looks like nothing from this job gets recorded.

kube_pod_container_status_last_terminated_timestamp{namespace="default", cluster="random-bash-fail", pod=~"failing-random-half.*"}

The only difference I can see in this cron job from my other cron jobs is that the restart policy is set to never:

restartPolicy: Never

Could restartPolicy: Never be the cause?

[assu-2000]

@sleepyfoodie , you're welcome !
Glad to see you got the metrics you were looking for.

Of course, it did affect . the k8s API only populates the container’s lastTerminationState sub-status if it has already been restarted at least once (i.e if restartCount > 0).

So, if the container has never restarted (e.g, successful termination on the first run, or a Never restart policy), this field remains empty.
Thanks for your contribution to the community. In case you need help, you can ping anytime , we're here.

@dgrisonnet , this issue is resolved and can be closed.

[k8s-ci-robot]

@assu-2000: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

@sleepyfoodie , you're welcome !
Glad to see you got the metrics you were looking for.

Of course, it did affect . the k8s API only populates the container’s lastTerminationState sub-status if it has already been restarted at least once (i.e if restartCount > 0).

So, if the container has never restarted (e.g, successful termination on the first run, or a Never restart policy), this field remains empty.
Thanks for your contribution to the community. In case you need help, you can ping anytime , we're here.

/close
This issue is resolved and can be closed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.