0.14.2: AutoscalingListener can retain deleted EphemeralRunnerSet after runner template update

[DocX]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided (rendered from the official 0.14.2 charts and applied through Argo CD).

Controller Version

0.14.2

Deployment Method

ArgoCD

Checks

• This isn't a question or user support case.
• I've read the changelog and this is not an expected backward-incompatible change.

To Reproduce

1. Deploy gha-runner-scale-set-controller and gha-runner-scale-set 0.14.2 with --update-strategy=eventual, minRunners: 1, and maxRunners: 10.
2. Wait until the AutoscalingRunnerSet, AutoscalingListener, and EphemeralRunnerSet are healthy.
3. Change AutoscalingRunnerSet.spec.template.spec so ARC rolls the EphemeralRunnerSet. In our case, a GitOps deployment removed an obsolete runtimeClassName and changed the runner container security context.
4. Observe that the old EphemeralRunnerSet is deleted and a different EphemeralRunnerSet survives, while AutoscalingListener.spec.ephemeralRunnerSetName still names the deleted set.
5. The controller repeatedly recreates the listener pod from that unchanged AutoscalingListener. Every pod attempts its initial patch against the deleted EphemeralRunnerSet, exits, and enters a permanent recreate/crash loop.

This happened during an update to an existing scale set, rather than a fresh install.

Describe the bug

On ARC 0.14.2, an AutoscalingListener remained bound to a deleted EphemeralRunnerSet after a runner-template update. The surviving EphemeralRunnerSet had a different generated name, but the listener CR, config, and RBAC still referenced the deleted name.

Every listener restart failed identically:

Application returned an error: handling initial message failed: could not patch ephemeral runner set,
patch JSON: {"spec":{"patchID":0,"replicas":1}},
error: ephemeralrunnersets.actions.github.com "<scale-set>-qb49n" not found

The controller then deleted and recreated the listener pod, but did not repair or replace the stale AutoscalingListener CR, so the loop continued indefinitely and jobs could not be assigned.

The symptoms and stale-reference mechanism match #4489 very closely. That issue covered 0.14.0/0.14.1 fresh installs and was closed by stale automation without a linked fix. This report confirms the same failure on 0.14.2 and during an update to an existing AutoscalingRunnerSet.

Recovery/workaround:

1. Detect that AutoscalingListener.spec.ephemeralRunnerSetName is not the single active EphemeralRunnerSet.
2. Delete the stale AutoscalingListener CR.
3. With updateStrategy=eventual, ARC drains running jobs, creates the replacement EphemeralRunnerSet/listener, and recovers.

We currently run that check as a post-sync hook because the controller did not self-heal the stale reference.

Describe the expected behavior

After replacing an EphemeralRunnerSet, ARC should atomically create/update the listener against the surviving set, or reconcile and replace any listener whose spec.ephemeralRunnerSetName points to a missing/non-current set.

A listener pod restart should not repeatedly inherit a permanently stale generated resource name.

Additional Context

controllerVersion: 0.14.2
scaleSetChartVersion: 0.14.2
kubernetesVersion: v1.35.2
controllerArgs:
  - --auto-scaling-runner-set-only
  - --watch-single-namespace=<runner-namespace>
  - --runner-max-concurrent-reconciles=2
  - --update-strategy=eventual
scaleSet:
  minRunners: 1
  maxRunners: 10
observedState:
  listenerTarget: <scale-set>-qb49n
  listenerTargetExists: false
  activeEphemeralRunnerSet: <different generated name>
relatedIssues:
  - https://github.com/actions/actions-runner-controller/issues/4489
  - https://github.com/actions/actions-runner-controller/issues/4432

Controller Logs

Complete captured controller log window (the controller includes the terminated listener's full output in message):

https://gist.github.com/DocX/80b533f37c0bd1a5253fdd8514df6609

The first entry already contains the decisive listener error; the recreate loop continues throughout the capture.

Runner Pod Logs

No runner pod failure was involved. The failing pod was the AutoscalingListener, and its complete output is embedded in the controller termination messages in the gist above. Runner creation/assignment was blocked because the listener could not patch its referenced EphemeralRunnerSet.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[diogotorres97]

This looks like one instance of a more general problem: references in the AutoscalingListener spec are write-once and never reconciled.

ephemeralRunnerSetName is stamped into the listener at construction (controllers/actions.github.com/resourcebuilder.go:139) and read back when the listener pod's config is built (:191-192). Nothing in the controllers ever compares it against the EphemeralRunnerSet that currently exists, so once the surviving set has a different generated name, the listener is permanently pointed at a deleted object. The listener itself has no recovery path — every error return in cmd/ghalistener exits the process — so this manifests exactly as you describe: a permanent recreate/crash loop that no restart can clear.

The same shape causes #4370, with runnerScaleSetId as the stale field instead: it's likewise written once (resourcebuilder.go:136), and the controller only re-registers a scale set when the runner-scale-set-id annotation is absent or unparseable (autoscalingrunnerset_controller.go:186-198), never when it's valid-but-dead. In both cases the listener is only rebuilt if the values-hash or ListenerSpecHash() changes, and ListenerSpecHash() covers ars.Spec alone (apis/actions.github.com/v1alpha1/autoscalingrunnerset_types.go:357-361).

So a fix scoped to ephemeralRunnerSetName would leave the sibling bug in place. Reconciling the listener's references against live state — or making the rebuild trigger include them — would resolve both this and #4370.