Pull-based autoscaling causes unrelated scale-ups

[Langleu]

Hey,
I'm running v0.27.4 on AWS EKS and haven't found any related issue / discussion.

I've created RunnerDeployments and HorizontalScaler purely based on the TotalNumberOfQueuedAndInProgressWorkflowRuns metric to just pull-based react to a set of repositories.
What I noticed is that even though none of the repositories is using the runner label that I've defined, I still get runners spawned since the controller is reacting to unrelated workflow schedules.

I didn't create a bug ticket yet since I've no 100% replicable case yet but tried scaling it down to a single repository and even that ended up triggering the controller eventually with a ubuntu-latest label.

Does anyone know how one could debug it further? I didn't find any other setting to at least check from the logs that workflow x from repo y has triggered a scale-up due to being queued.
My current debug setup as mentioned is a single workflow with ubuntu-latest and the repo is tracked by my horizontalscaler. Eventually, it will trigger a scale-up of unrelated RunnerDeployments that don't have the ubuntu-latest label but it's not a 100% thing. Feels like an edge case or race condition that is happening.

The thing is that this scale-up triggers every single horizontalscaler / runnerdeployment that I've configured like this. I even had some unrelated ubuntu-latest runs stuck from 3 weeks ago, which ended up scaling those RunnerDeployments.
From reading the docs, my understanding is that the scaling would only happen based on the used labels and not any queued workflow.

Controller logs

2023-06-02T10:02:39Z	DEBUG	horizontalrunnerautoscaler	Suggested desired replicas of 1 by TotalNumberOfQueuedAndInProgressWorkflowRuns	{"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 1, "workflow_runs_unknown": 0, "namespace": "gha-runner", "kind": "runnerdeployment", "name": "aws-core-2-unstable", "horizontal_runner_autoscaler": "aws-core-2-unstable"}
2023-06-02T10:02:39Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 1	{"horizontalrunnerautoscaler": "gha-runner/aws-core-2-unstable", "suggested": 1, "reserved": 0, "min": 0, "max": 20}
2023-06-02T10:02:39Z	DEBUG	runnerdeployment	Updated runnerreplicaset due to spec change	{"runnerdeployment": "gha-runner/aws-core-2-unstable", "currentDesiredReplicas": 0, "newDesiredReplicas": 1, "currentEffectiveTime": "<nil>", "newEffectiveTime": "<nil>"}
2023-06-02T10:02:39Z	DEBUG	runnerreplicaset	Created replica(s)	{"runnerreplicaset": "gha-runner/aws-core-2-unstable-sl9bt", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "7589bb9f55", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2023-06-02T10:02:39Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "gha-runner/aws-core-2-unstable-sl9bt", "owner": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz", "pods": null}
2023-06-02T10:02:39Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "gha-runner/aws-core-2-unstable-sl9bt", "owner": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz", "pods": null}
2023-06-02T10:02:40Z	INFO	runner	Updated registration token	{"runner": "aws-core-2-unstable-sl9bt-lf7lz", "repository": ""}
2023-06-02T10:02:40Z	DEBUG	runnerreplicaset	Skipped reconcilation because owner is not synced yet	{"runnerreplicaset": "gha-runner/aws-core-2-unstable-sl9bt", "owner": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz", "pods": null}
2023-06-02T10:02:40Z	DEBUG	events	Successfully update registration token	{"type": "Normal", "object": {"kind":"Runner","namespace":"gha-runner","name":"aws-core-2-unstable-sl9bt-lf7lz","uid":"ca19f919-09a6-4093-b212-211efaff230f","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"3326456"}, "reason": "RegistrationTokenUpdated"}
2023-06-02T10:02:40Z	INFO	runner	Created runner pod	{"runner": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz", "repository": ""}
2023-06-02T10:02:40Z	DEBUG	events	Created pod 'aws-core-2-unstable-sl9bt-lf7lz'	{"type": "Normal", "object": {"kind":"Runner","namespace":"gha-runner","name":"aws-core-2-unstable-sl9bt-lf7lz","uid":"ca19f919-09a6-4093-b212-211efaff230f","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"3326457"}, "reason": "PodCreated"}
2023-06-02T10:02:43Z	DEBUG	horizontalrunnerautoscaler	Suggested desired replicas of 1 by TotalNumberOfQueuedAndInProgressWorkflowRuns	{"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 1, "workflow_runs_unknown": 0, "namespace": "gha-runner", "kind": "runnerdeployment", "name": "aws-core-2-unstable", "horizontal_runner_autoscaler": "aws-core-2-unstable"}
2023-06-02T10:02:43Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 1	{"horizontalrunnerautoscaler": "gha-runner/aws-core-2-unstable", "suggested": 1, "reserved": 0, "min": 0, "max": 20, "last_scale_up_time": "2023-06-02 10:02:39 +0000 UTC", "scale_down_delay_until": "2023-06-02T10:03:39Z"}
2023-06-02T10:03:40Z	DEBUG	horizontalrunnerautoscaler	Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns	{"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0, "namespace": "gha-runner", "kind": "runnerdeployment", "name": "aws-core-2-unstable", "horizontal_runner_autoscaler": "aws-core-2-unstable"}
2023-06-02T10:03:40Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 1	{"horizontalrunnerautoscaler": "gha-runner/aws-core-2-unstable", "suggested": 0, "reserved": 0, "min": 0, "max": 20, "last_scale_up_time": "2023-06-02 10:02:39 +0000 UTC", "scale_down_delay_until": "2023-06-02T10:03:39Z"}
2023-06-02T10:04:40Z	DEBUG	horizontalrunnerautoscaler	Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns	{"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0, "namespace": "gha-runner", "kind": "runnerdeployment", "name": "aws-core-2-unstable", "horizontal_runner_autoscaler": "aws-core-2-unstable"}
2023-06-02T10:04:40Z	DEBUG	horizontalrunnerautoscaler	Calculated desired replicas of 0	{"horizontalrunnerautoscaler": "gha-runner/aws-core-2-unstable", "suggested": 0, "reserved": 0, "min": 0, "max": 20}
2023-06-02T10:04:40Z	DEBUG	runnerdeployment	Updated runnerreplicaset due to spec change	{"runnerdeployment": "gha-runner/aws-core-2-unstable", "currentDesiredReplicas": 1, "newDesiredReplicas": 0, "currentEffectiveTime": "<nil>", "newEffectiveTime": "<nil>"}
2023-06-02T10:04:41Z	INFO	runnerpod	Runner has just been unregistered.	{"runnerpod": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz"}
2023-06-02T10:04:41Z	INFO	runnerpod	Runner pod is marked as already unregistered.	{"runnerpod": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz"}
2023-06-02T10:04:41Z	INFO	runner	Removed finalizer	{"runner": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz"}
2023-06-02T10:04:41Z	INFO	runnerpod	Runner pod is marked as already unregistered.	{"runnerpod": "gha-runner/aws-core-2-unstable-sl9bt-lf7lz"}
2023-06-02T10:04:42Z	DEBUG	horizontalrunnerautoscaler	Suggested desired replicas of 0 by TotalNumberOfQueuedAndInProgressWorkflowRuns	{"workflow_runs_completed": 0, "workflow_runs_in_progress": 0, "workflow_runs_queued": 0, "workflow_runs_unknown": 0, "namespace": "gha-runner", "kind": "runnerdeployment", "name": "aws-core-2-unstable", "horizontal_runner_autoscaler": "aws-core-2-unstable"}

HorizontalRunnerAutoscaler

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  labels:
    app: gha-runner-controller
  name: aws-core-2-unstable
  namespace: gha-runner
spec:
  maxReplicas: 20
  metrics:
  - repositoryNames:
    - repo1
    - repo2
    - repo3
    type: TotalNumberOfQueuedAndInProgressWorkflowRuns
  minReplicas: 0
  scaleDownDelaySecondsAfterScaleOut: 60
  scaleTargetRef:
    name: aws-core-2-unstable

RunnerDeployment

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  labels:
    app: gha-runner-controller
  name: aws-core-2-unstable
  namespace: gha-runner
spec:
  template:
    spec:
      env:
      - name: DISABLE_RUNNER_UPDATE
        value: 'true'
      ephemeral: true
      image: summerwind/actions-runner:v2.304.0-ubuntu-22.04
      labels:
      - aws-core-2-default
      nodeSelector:
        nodegroup: core-8-unstable
      organization: myorg
      resources:
        limits:
          cpu: '1.8'
          memory: 6Gi
        requests:
          cpu: '1.8'
          memory: 6Gi
      serviceAccountName: agent
      volumeMounts:
      - mountPath: /etc/arc/hooks/job-started.d/label-podself.sh
        name: actions-runner-agent-config
        subPath: label-podself.sh
      volumes:
      - configMap:
          defaultMode: 493
          name: actions-runner-agent-config
        name: actions-runner-agent-config

[Langleu]

Closing the discussion due to bug report creation - #2659

Added my own debugging to figure out what's going on.
TL;DR - GItHub API returns a valid but empty response, which is caught by the controller and due to callbacks scaled up rather than ignored.