Auto scaling with webhook

[uralsemih]

We have different HRA and RunnerDeployments deployed in different namespaces.

We are facing a long queued( ["Waiting for a runner to pick up this job"])

Might be a broad question but I would like to clarify one thing. What might be the reason, is there are lots of queues although maxReplica is set to 100, and only 20 pods/runners are available.We have enough node resources, and also cluster auto-scaling enabled.

$ kubectl get runner -n staging-8c-32gb --no-headers | wc -l
      20

1. Can it be related to limits for RunnerDeployment? Is it make sense to remove limits for cpu and ephemeral-storage?

      resources:
        limits:
           cpu: "16"
          ephemeral-storage: 250Gi
          memory: 32Gi

2. Here are the logs, not sure whether this is related with scaling
   Controller logs ->

2023-02-22T22:05:30Z	INFO	runnerdeployment	The newest runnerreplicaset is 100% available. Deleting old runnerreplicasets	{"runnerdeployment": "staging-8c-32gb/gha-staging-8c-32gb", "newest_runnerreplicaset": "staging-8c-32gb/gha-staging-8c-32gb-zbhzj", "newest_runnerreplicaset_replicas_ready": 11, "newest_runnerreplicaset_replicas_desired": 11, "old_runnerreplicasets_count": 2}
2023-02-22T22:05:35Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-bl52k"}
2023-02-22T22:05:35Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-nwxmn"}
2023-02-22T22:05:35Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-9rwfg"}
2023-02-22T22:05:35Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-mplvl-lrd6n"}
2023-02-22T22:06:02Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-vdv95-qskkt"}
2023-02-22T22:06:02Z	INFO	runnerpod	Runner pod is annotated to wait for completion, and the runner container is not restarting	{"runnerpod": "staging-8c-32gb/gha-staging-8c-32gb-vdv95-8t8vr"}

3. Can it be due to misconfiguration ?

This HRA for staging-8c-32gb

$ kubectl get hra gha-staging-8c-32gb-autoscaler -n staging-8c-32gb -o yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  annotations:
    meta.helm.sh/release-name: staging-8c-32gb
    meta.helm.sh/release-namespace: staging-8c-32gb
  creationTimestamp: "2023-02-02T15:06:09Z"
  generation: 44075
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gha-staging-8c-32gb-autoscaler
  namespace: staging-8c-32gb
  resourceVersion: "95488238"
  uid: 9ac79c7a-135b-4802-a8ca-aa45e6f47f78
spec:
  capacityReservations:
  - effectiveTime: "2023-02-22T22:00:55Z"
    expirationTime: "2023-02-22T22:30:55Z"
    replicas: 1
  maxReplicas: 100
  minReplicas: 10
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    name: gha-staging-8c-32gb
  scaleUpTriggers:
  - duration: 30m
    githubEvent:
      workflowJob: {}
status:
  desiredReplicas: 11
  lastSuccessfulScaleOutTime: "2023-02-22T21:58:15Z"

This is RunnerDeployment staging-8c-32gb

$ kubectl get runnerdeployment gha-staging-8c-32gb -n staging-8c-32gb -o yaml
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  annotations:
    meta.helm.sh/release-name: staging-8c-32gb
    meta.helm.sh/release-namespace: staging-8c-32gb
  creationTimestamp: "2022-10-26T17:57:43Z"
  generation: 372001
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gha-staging-8c-32gb
  namespace: staging-8c-32gb
  resourceVersion: "95488240"
  uid: 4c1cd628-8c0a-489a-bb6b-4475cae150db
spec:
  effectiveTime: "2023-02-22T22:00:55Z"
  replicas: 11
  selector: null
  template:
    metadata: {}
    spec:
      dockerVolumeMounts:
      - mountPath: /var/lib/docker
        name: docker-extra
      dockerdContainerResources: {}
      dockerdWithinRunnerContainer: true
      env:
      - name: RUNNER_FEATURE_FLAG_EPHEMERAL
        value: "true"
      - name: DISABLE_RUNNER_UPDATE
        value: "false"
      group: secondary-staging-8c-32gb
      image: prodweuadoacr.azurecr.io/github-runner-full:latest-stable
      imagePullSecrets:
      - name: artifactory-cred
      initContainers:
      - command:
        - sh
        - -c
        - chmod -R 606 /dev/kvm
        image: busybox
        name: kvm-permission
        resources: {}
        volumeMounts:
        - mountPath: /dev/kvm
          name: kvm-device
      labels:
      - staging-8c-32gb
      - weu
      - self-hosted
      - Linux
      - linux
      - X64
      - x64
      nodeSelector:
        agentpool: stgrunneri
      organization: tomtom-internal
      resources:
        limits:
          memory: 32Gi
          ephemeral-storage: 250Gi
          memory: 32Gi
        requests:
          cpu: "8"
          ephemeral-storage: 250Gi
          memory: 32Gi
      sidecarContainers:
      - command:
        - bash
        - -c
        - --
        - bash -c 'while true; do tail -f /runner/_diag/*.log; [ $? -ne 0 ] && sleep
          3; done;'
        image: bash:latest
        name: logger
        resources: {}
        volumeMounts:
        - mountPath: /runner/_diag
          name: runnerlogs
      volumeMounts:
      - mountPath: /runner/_diag
        name: runnerlogs
      - mountPath: /dev/kvm
        name: kvm-device
      volumes:
      - hostPath:
          path: /mnt/docker-extra
          type: DirectoryOrCreate
        name: docker-extra
      - hostPath:
          path: /dev/kvm
        name: kvm-device
      - emptyDir: {}
        name: runnerlogs
status:
  availableReplicas: 18
  desiredReplicas: 11
  readyReplicas: 18
  replicas: 18
  updatedReplicas: 11

$ kubectl get runnerdeployment --all-namespaces
NAMESPACE             NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
staging-05c-2gb       gha-staging-05c-2gb       2         2         2            2           119d
staging-1c-4gb        gha-staging-1c-4gb        2         2         2            2           119d
staging-2c-8gb        gha-staging-2c-8gb        2         2         2            2           119d
staging-4c-16gb       gha-staging-4c-16gb       3         3         3            3           119d
staging-8c-32gb-nav   gha-staging-8c-32gb-nav   1         1         1            1           119d
staging-8c-32gb       gha-staging-8c-32gb       10        20        11           20          119d

$ kubectl get hra --all-namespaces
NAMESPACE             NAME                                 MIN   MAX   DESIRED   SCHEDULE
staging-05c-2gb       gha-staging-05c-2gb-autoscaler       1     50    2
staging-1c-4gb        gha-staging-1c-4gb-autoscaler        1     50    2
staging-2c-8gb        gha-staging-2c-8gb-autoscaler        1     50    2
staging-4c-16gb       gha-staging-4c-16gb-autoscaler       1     50    3
staging-8c-32gb-nav   gha-staging-8c-32gb-nav-autoscaler   1     30    1
staging-8c-32gb       gha-staging-8c-32gb-autoscaler       10    100   10