fix: pod with a restart policy of Never or OnFailure stuck at 'Progressing' (#15317) #709

RoelofKuijpers · 2025-04-09T07:58:23Z

This implementation extends the health condition check for pods.
Previously the assumption was that Pods with restart policy of Never or OnFailure are hooks with a finite life, these were considered as Progressing instead of Healthy. However, this logic does not apply when the pod is managed by an operator (e.g., Flink operator) and therefore has a restart policy of Never.
We introduce a new annotation which existence is checked when the pod is Running, that allows for skipping this logic on restart policy.

codecov · 2025-04-09T08:07:32Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.34%. Comparing base (8849c3f) to head (b216058).
⚠️ Report is 55 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #709      +/-   ##
==========================================
- Coverage   54.26%   47.34%   -6.93%     
==========================================
  Files          64       64              
  Lines        6164     6537     +373     
==========================================
- Hits         3345     3095     -250     
- Misses       2549     3187     +638     
+ Partials      270      255      -15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

drewhemm · 2025-04-09T09:05:23Z

This looks like a good approach to the problem.

drewhemm · 2025-04-09T09:19:45Z

The pod manifest needs the following to pass the tests:

Compute and storage resources defined
The alpine tag needs to use something other than latest, e.g. 3.21
Add automountServiceAccountToken: false to the pod spec, as per the Kubernetes docs

pkg/health/testdata/pod-running-restart-never-with-ignore-annotation.yaml

RoelofKuijpers · 2025-04-09T10:08:54Z

@drewhemm I have made the changes you suggested to get a Quality Gate pass

drewhemm · 2025-04-09T11:09:45Z

Cool, looks like the last blocking issue is the commit sign off.

drewhemm · 2025-04-09T11:46:03Z

A non-blocking issue has been flagged by SonarQube, probably best to resolve it as follows:

resources:
  requests:
    ephemeral-storage: "100Mi"

sivchari

LGTM. left nits.

pkg/health/health_pod.go

sonarqubecloud · 2025-04-21T16:32:47Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sivchari

LGTM

Liammarwood · 2025-05-14T21:49:07Z

This needs to be merged to solve lots of subsequent bugs that have been raised.

Liammarwood · 2025-05-28T02:31:44Z

@christianh814 would you be able to give this PR a review?

RoelofKuijpers · 2025-06-11T09:21:27Z

@crenshaw-dev would you be able to give this PR a review? Much appreciated!

nitishfy · 2025-07-03T11:01:30Z

pkg/health/health_pod.go

@@ -12,6 +12,10 @@ import (
 	"github.com/argoproj/gitops-engine/pkg/utils/kube"
 )

+const (
+	AnnotationIgnoreRestartPolicy = "argocd.argoproj.io/ignore-restart-policy"


I'd suggest putting it in a separate file -

gitops-engine/pkg/sync/common/types.go

Line 14 in 093aef0

AnnotationSyncWave = "argocd.argoproj.io/sync-wave"

@nitishfy Thanks for your comment. I have moved this to types.go as you suggested. Indeed better.

pkg/health/health_pod.go

nitishfy

Docs required!

Signed-off-by: Roelof Kuijpers <[email protected]>

…sses the checks of the Quality Gate Signed-off-by: Roelof Kuijpers <[email protected]>

Signed-off-by: Roelof Kuijpers <[email protected]>

improve code readability Co-authored-by: sivchari <[email protected]> Signed-off-by: Roelof Kuijpers <[email protected]>

Signed-off-by: Roelof Kuijpers <[email protected]>

sonarqubecloud · 2025-07-29T12:14:42Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

RoelofKuijpers · 2025-07-29T12:20:02Z

Docs required!

Have added info to the documentation now!

agaudreault · 2025-07-31T16:13:18Z

pkg/sync/doc.go

+A workaround is to use the annotation: argocd.argoproj.io/ignore-restart-policy: "true".
+
+When this annotation is set on the Pod resource, the controller will ignore the `restartPolicy` and consider the Pod *Running* as a valid healthy state.


This behavior, combined with hooks, would be problematic since hook would be Healthy before their completions.

It is good to explain the current behavior. But not should be part of the health check documentation, and not the hooks.

agaudreault · 2025-07-31T16:19:25Z

pkg/health/health_pod.go

+		policy := pod.Spec.RestartPolicy
+		// If the pod has the AnnotationIgnoreRestartPolicy annotation or its restart policy is Always,
+		// then treat it as a long-running pod and check its health status.
+		if _, ok := pod.Annotations[common.AnnotationIgnoreRestartPolicy]; ok || policy == corev1.RestartPolicyAlways {


I don't think this feature is necessary with the usage of argocd.argoproj.io/ignore-healthcheck: 'true'. The pod will show running, because that is the behavior that is configured in the Pod and we should have a consistent behaviour with Kubernetes.

If a controller is creating resource on which we cannot assume application healthiness reliably, then we should exclude those.

RoelofKuijpers requested a review from a team as a code owner April 9, 2025 07:58

RoelofKuijpers force-pushed the 15317 branch from 4db3279 to c6f17b7 Compare April 9, 2025 08:00

RoelofKuijpers mentioned this pull request Apr 9, 2025

App health stuck at "Progressing" forever when pod has restartPolicy: Never argoproj/argo-cd#15317

Open

3 tasks

github-advanced-security bot found potential problems Apr 9, 2025

View reviewed changes

pkg/health/testdata/pod-running-restart-never-with-ignore-annotation.yaml Fixed Show fixed Hide fixed

pkg/health/testdata/pod-running-restart-never-with-ignore-annotation.yaml Fixed Show fixed Hide fixed

RoelofKuijpers force-pushed the 15317 branch 2 times, most recently from 444a326 to 84a1039 Compare April 9, 2025 11:41

sivchari reviewed Apr 21, 2025

View reviewed changes

pkg/health/health_pod.go Outdated Show resolved Hide resolved

RoelofKuijpers force-pushed the 15317 branch 2 times, most recently from f6e7045 to 61ddfd1 Compare April 21, 2025 16:32

RoelofKuijpers requested a review from sivchari April 21, 2025 17:28

sivchari approved these changes Apr 29, 2025

View reviewed changes

nitishfy reviewed Jul 3, 2025

View reviewed changes

RoelofKuijpers force-pushed the 15317 branch from bc580d6 to 8c32116 Compare July 3, 2025 22:48

nitishfy reviewed Jul 29, 2025

View reviewed changes

pkg/health/health_pod.go Show resolved Hide resolved

nitishfy suggested changes Jul 29, 2025

View reviewed changes

RoelofKuijpers force-pushed the 15317 branch 2 times, most recently from 3de2f0b to ffeef76 Compare July 29, 2025 12:10

RoelofKuijpers added 3 commits July 29, 2025 14:14

fix: stuck at 'Progressing' #15317

9ba91f3

Signed-off-by: Roelof Kuijpers <[email protected]>

Update health_test.go

b0a3219

Signed-off-by: Roelof Kuijpers <[email protected]>

Update pod-running-restart-never-with-ignore-annotation.yaml so it pa…

07cb6a0

…sses the checks of the Quality Gate Signed-off-by: Roelof Kuijpers <[email protected]>

RoelofKuijpers and others added 5 commits July 29, 2025 14:14

add storage request

9a9eb19

Signed-off-by: Roelof Kuijpers <[email protected]>

Update pkg/health/health_pod.go

75c2fe7

improve code readability Co-authored-by: sivchari <[email protected]> Signed-off-by: Roelof Kuijpers <[email protected]>

Improve code readability fix

2c955b6

Signed-off-by: Roelof Kuijpers <[email protected]>

Move annotation into common package

8ef020a

Signed-off-by: Roelof Kuijpers <[email protected]>

Add documentation

b216058

Signed-off-by: Roelof Kuijpers <[email protected]>

RoelofKuijpers force-pushed the 15317 branch from b5e5c7e to b216058 Compare July 29, 2025 12:14

RoelofKuijpers requested a review from nitishfy July 29, 2025 12:24

agaudreault reviewed Jul 31, 2025

View reviewed changes

		A workaround is to use the annotation: argocd.argoproj.io/ignore-restart-policy: "true".

		When this annotation is set on the Pod resource, the controller will ignore the `restartPolicy` and consider the Pod Running as a valid healthy state.

fix: pod with a restart policy of Never or OnFailure stuck at 'Progressing' (#15317) #709

Are you sure you want to change the base?

fix: pod with a restart policy of Never or OnFailure stuck at 'Progressing' (#15317) #709

Uh oh!

Conversation

RoelofKuijpers commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

drewhemm commented Apr 9, 2025

Uh oh!

drewhemm commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RoelofKuijpers commented Apr 9, 2025

Uh oh!

drewhemm commented Apr 9, 2025

Uh oh!

drewhemm commented Apr 9, 2025

Uh oh!

sivchari left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sonarqubecloud bot commented Apr 21, 2025

Quality Gate passed

Uh oh!

sivchari left a comment

Choose a reason for hiding this comment

Uh oh!

Liammarwood commented May 14, 2025

Uh oh!

Liammarwood commented May 28, 2025

Uh oh!

RoelofKuijpers commented Jun 11, 2025

Uh oh!

nitishfy Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

RoelofKuijpers Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nitishfy left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Jul 29, 2025

Quality Gate passed

Uh oh!

RoelofKuijpers commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agaudreault Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

agaudreault Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

RoelofKuijpers commented Apr 9, 2025 •

edited

Loading

codecov bot commented Apr 9, 2025 •

edited

Loading

drewhemm commented Apr 9, 2025 •

edited

Loading

RoelofKuijpers commented Jul 29, 2025 •

edited

Loading