DigitalOcean Vertical Scaling not working with HorizontalRunnerAutoscaler

[nselvitelli]

We are currently using a k8s cluster on Digital Ocean with vertical autoscaling enabled and HRA setup. The issue is encountered when we rerun approx. 60 action runs at one time to stress-test the system. Runners autoscale with no problem; however, the nodes do not vertically scale to meet demand (Our single node reaches 100% CPU usage, but no additional nodes are spawned). Would anyone mind guiding us in the right direction, please?

Our runnerdeployment.yaml file looks like this:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: auto-runner-deployment
spec:
  template:
    spec:
      organization: ORG-NAME
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: runner-autoscaler-percentage-runners-busy
spec:
  # Runners in the targeted RunnerDeployment won't be scaled down
  # for 5 minutes instead of the default 10 minutes now
  scaleDownDelaySecondsAfterScaleOut: 300
  scaleTargetRef:
    kind: RunnerDeployment
    # # In case the scale target is RunnerSet:
    # kind: RunnerSet
    name: auto-runner-deployment
  minReplicas: 1
  maxReplicas: 100
  metrics:
  - type: PercentageRunnersBusy
    scaleUpThreshold: '0.75'
    scaleDownThreshold: '0.25'
    scaleUpAdjustment: 10        # The scale up runner count added to desired count
    scaleDownAdjustment: 5      # The scale down runner count subtracted from the desired count

[simpson]

We are having the exact same issue but in Azure AKS. We have AKS set to autoscale but it basically chokes out the cluster as it never requests additional nodes

[nselvitelli]

We were able to get some success by restarting the controller-manager deployment and increasing its replica pods.