Duplicated runners using TotalNumberOfQueuedAndInProgressWorkflowRuns?

[mattpopa]

While using autoscaling via TotalNumberOfQueuedAndInProgressWorkflowRuns, I'm are getting duplicated runners spawned up (can see the workqueue_retries_total raising during this time, using sum(rate(workqueue_retries_total{job="actions-runner-controller-metrics-service"}[$__interval])) by (name) and some runners are duplicated, e.g. when is demand for 1 runner we are getting 2 runners spawned (HRA setting desired to 1), 1 picking up the job and the other idling -- this is a problem because the idle runner blocks resources that can be used by a different type of runner, which might stay in pending state because of it.

this is my config for ARC

---
syncPeriod: 3s
metrics:
  port: 8080
  proxy:
    enabled: false
podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: /metrics
  prometheus.io/port: "8080"
resources:
   limits:
     cpu: 100m
     memory: 512Mi
   requests:
     cpu: 100m
     memory: 256Mi
nodeSelector:
  pool: cluster_plumbing

and this is my config for Deployment + HRA

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: self-hosted-large
  namespace: actions-runner-system
spec:
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
    spec:
      affinity:
        podAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: "runner-deployment-name"
                      operator: In
                      values:
                        - self-hosted-large
                topologyKey: "kubernetes.io/hostname"
      priorityClassName: high-priority
      serviceAccountName: github-actions-sa
      securityContext:
        # For Ubuntu 22.04 runner
        fsGroup: 1001
      organization: my-org
      image: summerwind/actions-runner-dind:v2.309.0-ubuntu-22.04
      imagePullPolicy: IfNotPresent
      ephemeral: true
      dockerEnabled: false
      dockerdWithinRunnerContainer: true
      nodeSelector:
        pool: runners
      containers:
      - name: runner
        resources:
          requests:
            memory: "10Gi"
            cpu: "3000m"
          limits:
            memory: "10Gi"
            cpu: "3000m"
      labels:
        - large
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  namespace: actions-runner-system
  name: self-hosted-large
spec:
  scaleDownDelaySecondsAfterScaleOut: 30
  scaleTargetRef:
    kind: RunnerDeployment
    name: self-hosted-large
  minReplicas: 0
  maxReplicas: 6
  metrics:
    - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
      repositoryNames:
        - frontend

using EKS 1.23, latest ARC chart + latest dependencies (actions-runner-controller-0.23.5 as of writing).

There seems to be a labeling mismatch, so that the idle runner can't be killed, e.g. this is an idle runner and the busy one doesn't show up in the logs

2023-10-11T08:11:08Z    INFO    runnerpod   Runner pod is annotated to wait for completion, and the runner container is not restarting  {"runnerpod": "actions-runner-system/self-hosted-large-dhb4s-7dp2m"}

2023-10-11T08:10:56Z    DEBUG   runnerpod   Failed to unregister runner before deleting the pod.    {"runnerpod": "actions-runner-system/self-hosted-large-dhb4s-7dp2m", "error": "failed to remove runner: DELETE https://api.github.com/orgs/site-assist/actions/runners/180419: 422 Bad request - Runner \"self-hosted-large-dhb4s-7dp2m\" is still running a job\" []"}

Did anyone else experienced this? Any debugging tips?

[mattpopa]

Does anyone else here uses TotalNumberOfQueuedAndInProgressWorkflowRuns scaling type?