Listener image is not updated on controller upgrade — only re-stamped when the AutoscalingRunnerSet spec changes

[jforest]

What happened

After an in-place helm upgrade of the controller and the gha-runner-scale-set releases from 0.14.1 → 0.14.2, the controller pod runs 0.14.2 but the runner-scale-set listener pods stay on 0.14.1. Listeners are only updated when something else changes the AutoscalingRunnerSet spec (e.g. a runner/dind image change); a controller/chart version bump alone leaves them stale indefinitely.

Expected

Upgrading the controller updates the listener image to match the controller version.

Actual mechanism

AutoscalingListener.spec.image is stamped at listener creation from the controller image (CONTROLLER_MANAGER_CONTAINER_IMAGE) and is not reconciled when the controller image later changes. A patch chart bump doesn't touch the AutoscalingRunnerSet spec, so the listener is never recreated → controller and listener drift apart, widening with each upgrade.

Reproduction

1. Install controller + a scale set at version X; listener pods run X.
2. In-place helm upgrade the controller and the scale-set release to X+1 (a patch with no ARS-affecting template change).
3. Controller pod → X+1; listener pods remain X. kubectl get autoscalinglistener -o yaml shows .spec.image still X.
4. Make any unrelated change to the AutoscalingRunnerSet spec (e.g. runner image) → listeners are recreated at the current controller image, confirming the image only propagates via ARS-spec changes.

Impact

Silent, ever-widening version skew between controller and listeners across upgrades, until it crosses a compatibility boundary; also risks running listeners on a version with known bugs while the controller is newer.

Acknowledging the documented limitation

The documented upgrade path is to uninstall the scale sets and controller (and remove CRDs), then reinstall — not an in-place helm upgrade — because "there is no support for upgrading or deleting CRDs with Helm, it is not possible to use Helm to upgrade ARC." #3726 was closed as "working as expected" on that basis, with a note that the upgrade process is planned to be fixed. In practice, GitOps/IaC tooling (Helm, Argo CD, Pulumi) performs in-place upgrades, and uninstall/reinstall (or multi-cluster HA to mask the downtime) is impractical for many operators — which is how the controller/listener skew arises silently.

Prior art: Karpenter solved the same Helm/CRD constraint

The cited blocker — Helm not upgrading CRDs — is the same one Karpenter faced, and Karpenter ships working solutions in the same ecosystem:

• A separate karpenter-crd Helm chart with the CRDs in templates/ (not crds/), so Helm manages and upgrades them in place across versions, with no uninstall. ARC could ship an equivalent CRD chart to make in-place upgrades feasible.
• Drift: Karpenter continuously reconciles running resources (nodes) back to their desired spec. Analogously, the controller could reconcile each AutoscalingListener.spec.image to its current image so listeners never lag, independent of ARS-spec changes.

Suggested fix

1. Provide an upgradable CRD chart (à la karpenter-crd) so in-place Helm upgrades are supported.
2. Have the controller reconcile AutoscalingListener.spec.image to its own current image on startup/upgrade.

Workaround

Delete the autoscalinglistener resources after each upgrade so the controller recreates them at the current image; or pin the listener image explicitly in the scale-set listenerTemplate so a chart-version bump changes the ARS spec.

Environment

• Controller: gha-runner-scale-set-controller 0.14.2
• Scale set: gha-runner-scale-set 0.14.2
• Listener: scaleset@v0.3.0
• Install method: in-place Helm upgrade (via Pulumi)
• Platform: Amazon EKS

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.