Requeue if create pod returns already exists error #4201
+4
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikola-jokic
requested review from
mumoshu,
toast-gear,
rentziass and
a team
as code owners
There was a problem hiding this comment.
Pull Request Overview
This PR addresses a race condition where the ephemeral runner controller attempts to create a pod that already exists but hasn't been received through the Kubernetes event watcher yet. The change adds error handling to detect "already exists" errors and requeues the reconciliation with a 5-second delay instead of logging an error.
- Adds detection for "already exists" errors when creating pods
- Implements a requeue mechanism with 5-second delay for the race condition
- Updates log message to be more descriptive about the pod creation process
| @@ -268,13 +268,15 @@ func (r *EphemeralRunnerReconciler) Reconcile(ctx context.Context, req ctrl.Requ | |||
| log.Error(err, "Failed to fetch the pod") | |||
| return ctrl.Result{}, err | |||
| } | |||
| log.Info("Ephemeral runner pod does not exist. Creating new ephemeral runner") | |||
There was a problem hiding this comment.
The log message is misleading. At this point, the code has determined the pod doesn't exist, but the new error handling suggests the pod might actually exist. Consider a more accurate message like 'Attempting to create new ephemeral runner pod'.
Suggested change
| log.Info("Ephemeral runner pod does not exist. Creating new ephemeral runner") | |
| log.Info("Attempting to create new ephemeral runner pod") |
Copilot uses AI. Check for mistakes.
nikola-jokic
added
the
gha-runner-scale-set
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are situations when the newly created pod is not yet delivered through the watcher, causing subsequent reconciliations to issue a second create pod event.
This change tests for the condition, logs at the info level, and re-queues the event in 5s.
It is not necessary to do it for secrets since the secrets are not watched; therefore, the
GETrequest was being issued directly.It will reduce the noise in the log while backing off the reconciliation for a bit.