HorizontalRunnerAutoscaler doesn't scale runners

[Edarjak]

Hi community,

I'm trying to use HorizontalRunnerAutoscaler but with no luck. The RunnerDeployment itself works fine when I specify replicas: , but with HRA it somehow doesn't work - it shows only 1 runner while it should create the minReplicas number that is described in the deployment. Here it is:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-actions-runners-small
  namespace: github-actions-runners-small
spec:
  template:
    spec:
      organization: myorg
      env:
        - name: GIT_SSL_CAINFO
          value: /etc/ssl/certs/root-ca.pem
      labels:
        - small
      group: Default
      resources:
        limits:
          cpu: "0.8"
          memory: "1.5Gi"
        requests:
          cpu: "0.4"
          memory: "1.5Gi"
      volumes:
        - name: ca-cert
          configMap:
            name: ca-cert
      volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs/root-ca.pem
          subPath: root-ca.pem
          readOnly: true
      containers:
        - name: docker
          image: my.private-reg.com/reg/docker:dind
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: github-actions-runners-small
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    name: github-actions-runners-small
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'   
    scaleDownThreshold: '0.3'   
    scaleUpFactor: '1.4'       
    scaleDownFactor: '0.7' 

In the controller log, there is not that much information about HRA, everything looks ok, perhaps I'm missing something...

Can someone give an advice on how to find out what is the root cause of that problem?

Thanks in advance,

P.S. The ARC version is 0.27.1

[kevholmes]

@Edarjak does webhook driven scaling work properly?

Also - I have noticed on rare occasions with webhook driven scaling if I delete/reapply my HRA and RD resources sometimes we get stuck with the minimum count and cannot scale up. Usually deleting the impacted HRA/RD resources, and rolling the controller and webhook pods can help. I have also had luck setting a new minimum count for the impacted HRA and that usually causes a scale-up of pods and resolves the issue for me.

[kevholmes]

Hi @kevholmes,

The issue mentioned here #2544 resolved. However, the controller still doesn't autoscale runners. Perhaps you know the way to debug HRA? The manager container doesn't output anything useful:

Awesome to hear @Edarjak - could you explain what you've done above in your test and what you're seeing in k8s? Eg do you get at least two pods online as your min but they do not scale up to your max when multiple jobs come in? I have seen behavior like this sometimes with Dependabot via RunnerSets but not regular workloads via RunnerDeployments. Also I believe there is some lag with this so you may need sustained workloads to trigger the scale up thru the metrics method vs workflowJob: {}:

  scaleUpTriggers:
  - githubEvent:
      workflowJob: {} # run on any workflow job request
    duration: "30m" # duration before we scale back by amount
    amount: 1

I don't have world-scale load by any means but with that kind of setup we are able to scale up/down without issue and users rarely need to wait for a runner pod from what I have seen for most CI tasks unless they're using test matrix strategies with a large number of jobs spun out of it.

[Edarjak]

Thank you for your reply!

could you explain what you've done above in your test and what you're seeing in k8s?

I created a RunnerDeployment with HRA (you can find it in the discussion description). As the next step, I applied my deployment and as a result, I saw that controller didn't create the default minReplicas number of runners - it creates only one runner for every HRA deployment. In case I remove HRA and describe some amount of replicas: it creates the desired number of runners.

Deployment that works:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-actions-runners-small
  namespace: github-actions-runners-small
spec:
  replicas: 5
  template:
    spec:
      organization: myorg
      env:
        - name: GIT_SSL_CAINFO
          value: /etc/ssl/certs/root-ca.pem
      labels:
        - small
      group: Default
      resources:
        limits:
          cpu: "0.8"
          memory: "1.5Gi"
        requests:
          cpu: "0.4"
          memory: "1.5Gi"
      volumes:
        - name: ca-cert
          configMap:
            name: ca-cert
      volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs/root-ca.pem
          subPath: root-ca.pem
          readOnly: true
      containers:
        - name: docker
          image: my.private-reg.com/reg/docker:dind

Deployment that doesn't work:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-actions-runners-small
  namespace: github-actions-runners-small
spec:
  template:
    spec:
      organization: myorg
      env:
        - name: GIT_SSL_CAINFO
          value: /etc/ssl/certs/root-ca.pem
      labels:
        - small
      group: Default
      resources:
        limits:
          cpu: "0.8"
          memory: "1.5Gi"
        requests:
          cpu: "0.4"
          memory: "1.5Gi"
      volumes:
        - name: ca-cert
          configMap:
            name: ca-cert
      volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs/root-ca.pem
          subPath: root-ca.pem
          readOnly: true
      containers:
        - name: docker
          image: my.private-reg.com/reg/docker:dind
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: github-actions-runners-small
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    name: github-actions-runners-small
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'   
    scaleDownThreshold: '0.3'   
    scaleUpFactor: '1.4'       
    scaleDownFactor: '0.7' 

Thanks,

[Edarjak]

Perhaps @mumoshu will have any thoughts?

[kalined]

Hi @Edarjak,

I think you are missing a namespace: key for the HRA because you specify a namespace: github-actions-runners-small for your RunnerDeployment. It means that if the namespace is missing on the HRA side it simply doesn't know where to apply your deployment. So, my recommendation try to use this:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-actions-runners-small
  namespace: github-actions-runners-small
spec:
  template:
    spec:
      organization: myorg
      env:
        - name: GIT_SSL_CAINFO
          value: /etc/ssl/certs/root-ca.pem
      labels:
        - small
      group: Default
      resources:
        limits:
          cpu: "0.8"
          memory: "1.5Gi"
        requests:
          cpu: "0.4"
          memory: "1.5Gi"
      volumes:
        - name: ca-cert
          configMap:
            name: ca-cert
      volumeMounts:
        - name: ca-cert
          mountPath: /etc/ssl/certs/root-ca.pem
          subPath: root-ca.pem
          readOnly: true
      containers:
        - name: docker
          image: my.private-reg.com/reg/docker:dind
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: github-actions-runners-small
  namespace: github-actions-runners-small
spec:
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    name: github-actions-runners-small
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'   
    scaleDownThreshold: '0.3'   
    scaleUpFactor: '1.4'       
    scaleDownFactor: '0.7' 

That should help.

Br,

[Edarjak]

Hi @EdgeSan,

You are right, that helped! Thanks, I'm closing this discussion.

Br,