Skip to content

Commit d028549

Browse files
authored
Merge pull request #43773 from alculquicondor/job-finalizer-troubleshoot
Troubleshooting for job pod finalizers
2 parents b9eef4a + 46df889 commit d028549

File tree

2 files changed

+33
-0
lines changed

2 files changed

+33
-0
lines changed

content/en/docs/concepts/workloads/controllers/job.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -938,6 +938,11 @@ creates Pods with the finalizer `batch.kubernetes.io/job-tracking`. The
938938
controller removes the finalizer only after the Pod has been accounted for in
939939
the Job status, allowing the Pod to be removed by other controllers or users.
940940

941+
{{< note >}}
942+
See [My pod stays terminating](/docs/tasks/debug-application/debug-pods) if you
943+
observe that pods from a Job are stucked with the tracking finalizer.
944+
{{< /note >}}
945+
941946
### Elastic Indexed Jobs
942947

943948
{{< feature-state for_k8s_version="v1.27" state="beta" >}}

content/en/docs/tasks/debug/debug-application/debug-pods.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,34 @@ There are three things to check:
6969
* Try to manually pull the image to see if the image can be pulled. For example,
7070
if you use Docker on your PC, run `docker pull <image>`.
7171

72+
73+
#### My pod stays terminating
74+
75+
If a Pod is stuck in the `Terminating` state, it means that a deletion has been
76+
issued for the Pod, but the control plane is unable to delete the Pod object.
77+
78+
This typically happens if the Pod has a [finalizer](/docs/concepts/overview/working-with-objects/finalizers/)
79+
and there is an [admission webhook](/docs/reference/access-authn-authz/extensible-admission-controllers/)
80+
installed in the cluster that prevents the control plane from removing the
81+
finalizer.
82+
83+
To identify this scenario, check if your cluster has any
84+
ValidatingWebhookConfiguration or MutatingWebhookConfiguration that target
85+
`UPDATE` operations for `pods` resources.
86+
87+
If the webhook is provided by a third-party:
88+
- Make sure you are using the latest version.
89+
- Disable the webhook for `UPDATE` operations.
90+
- Report an issue with the corresponding provider.
91+
92+
If you are the author of the webhook:
93+
- For a mutating webhook, make sure it never changes immutable fields on
94+
`UPDATE` operations. For example, changes to containers are usually not allowed.
95+
- For a validating webhook, make sure that your validation policies only apply
96+
to new changes. In other words, you should allow Pods with existing violations
97+
to pass validation. This allows Pods that were created before the validating
98+
webhook was installed to continue running.
99+
72100
#### My pod is crashing or otherwise unhealthy
73101

74102
Once your pod has been scheduled, the methods described in

0 commit comments

Comments
 (0)