Skip to content

Fix race condition when deleting an agent#2930

Merged
vigoo merged 3 commits intomainfrom
delete-race-condition-fix
Mar 4, 2026
Merged

Fix race condition when deleting an agent#2930
vigoo merged 3 commits intomainfrom
delete-race-condition-fix

Conversation

@vigoo
Copy link
Contributor

@vigoo vigoo commented Mar 4, 2026

The test for this was flaky.
The theory is that after confirmation of the interruption (rx.recv) but before switching to the rejecting deleting state (start_deleting) a new invocation could start, picked from the pending invocation queue but never marked as interrupted/finished.

Calling start_deleting immediately closes the agent for further incoming invocation before the previously running one is interrupted.

Also fixes another flaky test by properly waiting for the auto-update to finish.

@vigoo vigoo marked this pull request as ready for review March 4, 2026 18:57
.auto_update_worker(&worker_id, updated_component.revision, false)
.await?;

executor
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an alternative to this there is await_ready_to_process_commands in the worker (not exposed via api currently), which will wait until replay / update is finished

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

User-facing APIs like invocation supposed to work during those phases too (recovery, update) so this would only be useful for tests like this; on the other hand this kind of waiting based on worker status is what the CLI is also doing if you want to await an update. So I'd keep it like this

@vigoo vigoo merged commit 66ab14d into main Mar 4, 2026
27 checks passed
@vigoo vigoo deleted the delete-race-condition-fix branch March 4, 2026 19:26
@github-actions github-actions bot locked and limited conversation to collaborators Mar 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants