Skip to content

Autoscaling Runner Sets not scaling down after long jobsΒ #4193

@crystalthenR2R

Description

@crystalthenR2R

Checks

Controller Version

0.12.1

Deployment Method

Kustomize

Checks

  • This isn't a question or user support case (For Q&A and community support, go to Discussions).
  • I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

The issue is happening intermittently, so I haven't been able to create a simple example to reliably reproduce it.
It seems to happen more often with long running jobs (>5 hours).

Describe the bug

When the job finishes successfully, the existing runner pod is removed. The EphemeralRunnerSet then immediately creates a new runner pod, as desired replicas is set to 1. This new runner pod remains idle until another workflow uses it. If no jobs request that runner set the idle runner remains indefinitely, despite minRunners being set to 0.

Describe the expected behavior

We set minRunners: 0 when deploying the gha-runner-scale-set chart, so we expect the EphemeralRunnerSet should scale down to 0 after the workflow finishes.

Additional Context

Listener logs
The listener never receives the message for the completed job.
https://gist.github.com/crystalthenR2R/d3c779ceb2bc14b9fec3a3e6f3fb5fa7

Running a new job on the runner immediately triggers a scale down:
https://gist.github.com/crystalthenR2R/870c6760f38bd2170e725681ba3c7288

Unfortunately this is happening with our larger sized runners with long/infrequent jobs. So they sometimes stick around for a very long time consuming a large amount of resources.

Controller Logs

Controller logs
https://gist.github.com/crystalthenR2R/a9e3010a78042178828146c81fa85454

The runner (nrxkx-runner-2cwzh) is marked as Finished at 2025-07-31 20:05:50, and a new runner is immediately created (nrxkx-runner-gfphq)

Runner Pod Logs

n/a

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggha-runner-scale-setRelated to the gha-runner-scale-set modeneeds triageRequires review from the maintainers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions