managed application shows outofsync / missing health after re-create from unauthorized delete

[drewbailey]

Describe the bug

If an application is deleted from a managed agent, it gets re-created, then wedged and never receives any more updates

Steps to reproduce the behaviour

1. Delete app on agent manually
2. Agent detects "unauthorized deletion" → recreates app
3. App was never unmanaged (since it was unauthorized deletion)
4. Recreate tries Manage() → fails "already managed"
5. Informer fires NewApp event for the recreated app
6. NewApp handler also tries Manage() → fails "already managed"

Expected behavior

I think unmanaging before re-creating might make sense? Or Manage() becomes idempotent

Additional context

llm summarized logs for clarify / remove environment info

20:43:23  DEBUG  cluster1 found in agent cache
20:43:23  DEBUG  Status update queued
20:43:23  DEBUG  Status update processed

20:43:23  INFO   Updating application (rv=777063671)
20:43:23  INFO   Application updated

20:44:01  INFO   Updating application (rv=777064217)
20:44:01  INFO   Application updated
20:44:01  DEBUG  Ignored stale resource version

20:44:41–20:44:42
           DEBUG  cluster1 repeatedly reconciled
           DEBUG  Status updates queued and processed

20:44:43  WARN   Delete event detected
20:44:43  WARN   Unauthorized deletion — recreating application
20:44:43  INFO   Application successfully recreated

20:44:43  WARN   App already managed (duplicate manage attempt)
20:44:43  ERROR  Cannot manage app that is already managed

20:44:43  DEBUG  Final status event processed

[drewbailey]

The application is actually updating, but it shows as OutOfSync / Missing in both the agent & principle, I think this is because the application controller isn't triggering auto-sync despite OutOfSync status.

the operationState shows a sync from before the delete:

  operationState:
    finishedAt: "2026-02-05T20:43:23Z"  # Before delete at 20:44:43
    phase: Succeeded

I wonder if in RevertUserInitiatedDeletion the operationState should be cleared?

[jannfis]

Hm. I actually could not reproduce the issue. I see the same errors in the log, but the sync between principal and agent for that (recreated) app does not seem to be affected.

[drewbailey]

I tried again, one time it did re-sync 2 minutes later, and then another time it did not end up resyncing, it seems like indeterminant behavior based on if the application controller thinks the application needs to resync

time="2026-02-06T14:05:27Z" level=warning msg="Unauthorized deletion detected - recreating" application=project-example/argocd-agent-m-eks-c-zho-test-go event=DeleteApp kind= module=GrpcEvent resource=argocd-agent-m-eks-c-zho-test-go
time="2026-02-06T14:05:27Z" level=info msg="Recreated resource after unauthorized deletion" application=project-example/argocd-agent-m-eks-c-zho-test-go event=DeleteApp kind= module=GrpcEvent resource=argocd-agent-m-eks-c-zho-test-go
time="2026-02-06T14:05:27Z" level=debug msg="New app event" application=project-example/argocd-agent-m-eks-c-zho-test-go event=NewApp module=GrpcEvent
time="2026-02-06T14:05:27Z" level=warning msg="Could not manage app project-example/argocd-agent-m-eks-c-zho-test-go: resource project-example/argocd-agent-m-eks-c-zho-test-go is already managed" component=AppManager
time="2026-02-06T14:05:27Z" level=error msg="Cannot manage app that is already managed" application=project-example/argocd-agent-m-eks-c-zho-test-go event=NewApp module=GrpcEvent
time="2026-02-06T14:05:27Z" level=debug msg="Received a new event from stream" clientAddr="100.103.205.184:443" direction=Recv event_id=argocd-agent-m-eks-c-zho-test-go_71c5a60b-bf58-4f65-a422-41991a85f25a_300114058 module=StreamEvent resource_id=argocd-agent-m-eks-c-zho-test-go_71c5a60b-bf58-4f65-a422-41991a85f25a ty
time="2026-02-06T14:07:44Z" level=debug msg="Application argocd-agent-m-eks-c-zho-test-go is available in agent cache" application=project-example/argocd-agent-m-eks-c-zho-test-go component=RevertManagedAppChanges resourceVersion=300115992
time="2026-02-06T14:07:44Z" level=debug msg="Added event of type io.argoproj.argocd-agent.event.status-update to send queue" application=project-example/argocd-agent-m-eks-c-zho-test-go event=UpdateApp module=GrpcEvent sendq_len=1 sendq_name=default
time="2026-02-06T14:07:44Z" level=debug msg="Application argocd-agent-m-eks-c-zho-test-go is available in agent cache" application=project-example/argocd-agent-m-eks-c-zho-test-go component=RevertManagedAppChanges resourceVersion=300115994
time="2026-02-06T14:07:44Z" level=debug msg="Added event of type io.argoproj.argocd-agent.event.status-update to send queue" application=project-example/argocd-agent-m-eks-c-zho-test-go event=UpdateApp module=GrpcEvent sendq_len=1 sendq_name=default
time="2026-02-06T14:07:44Z" level=debug msg="Application argocd-agent-m-eks-c-zho-test-go is available in agent cache" application=project-example/argocd-agent-m-eks-c-zho-test-go component=RevertManagedAppChanges resourceVersion=300115996
time="2026-02-06T14:07:44Z" level=debug msg="Added event of type io.argoproj.argocd-agent.event.status-update to send queue" application=project-example/argocd-agent-m-eks-c-zho-test-go event=UpdateApp module=GrpcEvent sendq_len=1 sendq_name=default
time="2026-02-06T14:07:44Z" level=debug msg="Received a new event from stream" clientAddr="100.103.205.184:443" direction=Recv event_id=argocd-agent-m-eks-c-zho-test-go_22483314-b735-4c5c-993f-d6f9632e3486_300115994 module=StreamEvent resource_id=argocd-agent-m-eks-c-zho-test-go_22483314-b735-4c5c-993f-d6f9632e3486 ty
time="2026-02-06T14:07:45Z" level=debug msg="Received a new event from stream" clientAddr="100.103.205.184:443" direction=Recv event_id=argocd-agent-m-eks-c-zho-test-go_22483314-b735-4c5c-993f-d6f9632e3486_300115996 module=StreamEvent resource_id=argocd-agent-m-eks-c-zho-test-go_22483314-b735-4c5c-993f-d6f9632e3486 ty
time="2026-02-06T14:07:46Z" level=debug msg="Application argocd-agent-m-eks-c-zho-test-go is available in agent cache" application=project-example/argocd-agent-m-eks-c-zho-test-go component=RevertManagedAppChanges resourceVersion=300116020
time="2026-02-06T14:07:46Z" level=debug msg="Added event of type io.argoproj.argocd-agent.event.status-update to send queue" application=project-example/argocd-agent-m-eks-c-zho-test-go event=UpdateApp module=GrpcEvent sendq_len=1 sendq_name=default
time="2026-02-06T14:07:47Z" level=debug msg="Received a new event from stream" clientAddr="100.103.205.184:443" direction=Recv event_id=argocd-agent-m-eks-c-zho-test-go_8149bff0-c835-4e5c-9747-300c49a1e86b_778001503 module=StreamEvent resource_id=argocd-agent-m-eks-c-zho-test-go_8149bff0-c835-4e5c-9747-300c49a1e86b ty
time="2026-02-06T14:07:47Z" level=info msg="Updating application" app=project-example/argocd-agent-m-eks-c-zho-test-go method=UpdateApplication module=GrpcEvent resourceVersion=778001503
time="2026-02-06T14:07:47Z" level=info msg="Updated application" application=project-example/argocd-agent-m-eks-c-zho-test-go component=UpdateManaged resourceVersion=778001503

failed attempt

time="2026-02-06T14:15:59Z" level=warning msg="Unauthorized deletion detected - recreating" application=project-example/argocd-agent-m-eks-c-zho-test-go event=DeleteApp kind= module=GrpcEvent resource=argocd-agent-m-eks-c-zho-test-go
time="2026-02-06T14:15:59Z" level=debug msg="Received a new event from stream" clientAddr="100.103.205.184:443" direction=Recv event_id=argocd-agent-m-eks-c-zho-test-go_22483314-b735-4c5c-993f-d6f9632e3486_300123009 module=StreamEvent resource_id=argocd-agent-m-eks-c-zho-test-go_22483314-b735-4c5c-993f-d6f9632e3486 ty
pe=io.argoproj.argocd-agent.event.processed
time="2026-02-06T14:15:59Z" level=info msg="Recreated resource after unauthorized deletion" application=project-example/argocd-agent-m-eks-c-zho-test-go event=DeleteApp kind= module=GrpcEvent resource=argocd-agent-m-eks-c-zho-test-go
time="2026-02-06T14:15:59Z" level=warning msg="Could not manage app project-example/argocd-agent-m-eks-c-zho-test-go: resource project-example/argocd-agent-m-eks-c-zho-test-go is already managed" component=AppManager
time="2026-02-06T14:15:59Z" level=debug msg="New app event" application=project-example/argocd-agent-m-eks-c-zho-test-go event=NewApp module=GrpcEvent
time="2026-02-06T14:15:59Z" level=error msg="Cannot manage app that is already managed" application=project-example/argocd-agent-m-eks-c-zho-test-go event=NewApp module=GrpcEvent
time="2026-02-06T14:19:37Z" level=debug msg="Application argocd-agent-m-eks-c-zho-test-go is available in agent cache" application=project-example/argocd-agent-m-eks-c-zho-test-go component=RevertManagedAppChanges resourceVersion=300126104
time="2026-02-06T14:19:37Z" level=debug msg="Added event of type io.argoproj.argocd-agent.event.status-update to send queue" application=project-example/argocd-agent-m-eks-c-zho-test-go event=UpdateApp module=GrpcEvent sendq_len=1 sendq_name=default
time="2026-02-06T14:19:37Z" level=debug msg="Received a new event from stream" clientAddr="100.103.205.184:443" direction=Recv event_id=argocd-agent-m-eks-c-zho-test-go_3371ef46-c3fd-4182-a4cd-a937f3ab4322_300126104 module=StreamEvent resource_id=argocd-agent-m-eks-c-zho-test-go_3371ef46-c3fd-4182-a4cd-a937f3ab4322 ty
pe=io.argoproj.argocd-agent.event.processed

application controller

{"app-namespace":"project-example","application":"argocd-agent-m-eks-c-zho-test-go","level":"info","msg":"Comparing app state (cluster: , namespace: example-test)","project":"example","t
ime":"2026-02-06T14:19:36Z"}
{"app-namespace":"project-example","application":"argocd-agent-m-eks-c-zho-test-go","build_options_ms":0,"level":"info","manifests_ms":207,"msg":"GetRepoObjs stats","plugins_ms":0,"proje
ct":"example","repo_ms":0,"time":"2026-02-06T14:19:37Z","time_ms":207}
{"app-namespace":"project-example","application":"argocd-agent-m-eks-c-zho-test-go","level":"info","msg":"Already attempted sync: comparing synced revisions to [a82c3d820be7b762c1d32e224
9280a778ecf6fb[]","project":"example","time":"2026-02-06T14:19:37Z"}
{"app-namespace":"project-example","application":"argocd-agent-m-eks-c-zho-test-go","level":"info","msg":"Skipping auto-sync: most recent sync already to [a82c3d820be7b762c1d32e224e9280a
78ecf6fb[]","project":"example","time":"2026-02-06T14:19:37Z"}
{"app-namespace":"project-example","application":"argocd-agent-m-eks-c-zho-test-go","level":"info","msg":"Update successful","project":"example","time":"2026-02-06T14:19:37Z"}
{"app-namespace":"project-example","app_status_update_ms":0,"application":"argocd-agent-m-eks-c-zho-test-go","auto_sync_ms":0,"compare_app_state_ms":209,"comparison-level":2,"comparison_
with_nothing_ms":0,"dedup_ms":0,"dest-name":"in-cluster","dest-namespace":"example-test","dest-server":"","diff_ms":0,"git_ms":207,"health_ms":0,"level":"info","live_ms":0,"msg":"Reconci
liation completed","normalize_application_ms":0,"patch_ms":15,"persist_app_status_ms":15,"process_finalizers_ms":0,"project":"example","refresh_app_conditions_ms":0,"set_app_managed_reso
urces_ms":2,"setop_ms":0,"settings_ms":0,"sync_ms":0,"time":"2026-02-06T14:19:37Z","time_ms":227}

[jannfis]

When you deleted the Application on the spoke cluster, does it actually have a resources finalizer? I.e., did it delete the managed resources too?

[jannfis]

Ah, I think I know what happens here because I reproduced it.

• Application exists on a managed agent system
• Application has automated sync policy without self-heal
• Application has the resources finalizer
• Application gets deleted on the agent system
• Due to the finalizer, Application's resources get deleted too
• Agent detects the deletion, and recreates the Application
• Application is OutOfSync and Health == Missing, because the Applications resources do not exist in the cluster

Now, the auto-sync of Argo CD doesn't trigger because the last synced revision is the same as the current revision in Git. This is expected behaviour in Argo CD. From Argo CD's perspective, the resources have been deleted in the cluster.

One solution to this is to set self-heal in the sync policy, the resources that have been deleted will be reconciled back into the cluster after the Application has been deleted.

The question about modifying or removing the .status.operationState is a good one. I would assume that there might be side-effects when the Application is deleted in the middle of an actual operation in progress. Gotta do some research on that.

[jannfis]

Thinking about it, maybe we should just remove the complete .status field when re-recreating the application and let the application controller repopulate it.

EDIT: Probably a bad idea, because we lose sync history then too.

[mikeshng]

Can we force a sync in this case and, once it's completed, the spoke app becomes BAU?

[jannfis]

Hm, I'm not so sure about having a forced sync by default in this scenario.

There's a few things coming to mind to consider here:

• A sync window could prevent the application from syncing, creating unexpected outcomes.
• If the Application was deleted on the spoke by accident, and automatically synced back after recreation, the chances to spot the accidental deletion (and therefore, root cause) becomes more complicated to detect (it looks like as if nothing ever happened)
• Sync hooks would be run without user consent/involvement, which might not be intended and create situations

Likely, as the first step, the behaviour should be documented (i.e. if Application is deleted on a managed agent's spoke, it is recreated - if it had a resource finalizer, user has to perform manual corrective steps unless self-heal is enabled for the Application).

Then, we might think about giving the user a deliberate choice, e.g. --on-application-recreate=<action>, where <action> could be one of ignore (the default), resync (manually triggers a sync for the Application), clear-status (removes operation state from the Application's status to let autosync pick up the sync).

WDYT?

[mikeshng]

SGTM!

--on-application-recreate Do we need to be explicit that this only impacts managed mode?