Webhook Autoscaling not correctly detecting HRA

[amruthta]

Public discussions question


Below are my RunnerDeployment with repository runner and HorizontalRunnerAutoscaler based of webhook, for which we are getting error:



apiVersion: actions.summerwind.dev/v1alpha1


kind: RunnerDeployment
metadata:
name: github-group-runners
namespace: runner
spec:
template:
spec:
repository: xyz-corp/job-okta-github-group-sync
serviceAccountName: deployment-sa
automountServiceAccountToken: true
labels:
- github-group-sync-runner

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
name: github-group-runners
namespace: runner
spec:
minReplicas: 0
maxReplicas: 10
scaleTargetRef:
kind: RunnerDeployment
# # In case the scale target is RunnerSet:
# kind: RunnerSet
name: github-group-runners
scaleUpTriggers:
- githubEvent:
workflowJob: {}
duration: "20s"


Logs of actions runner controller when triggered webhook which is suppose to run above created runner, below are the logs and some key aspects which are highlighted:




2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler Found 0 HRAs by key {"key": "xyz-corp/job-okta-github-group-sync"}
2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler Skipped scale target irrelevant to event {"eventOrganization": "xyz-corp", "eventEnterprise": "xyz", "scaleTargetKind": "RunnerDeployment", "scaleTargetGroup": "", "scaleTargetEnterprise": "", "scaleTargetOrganization": ""}
2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler Skipped scale target irrelevant to event {"eventOrganization": "xyz-corp", "eventEnterprise": "xyz", "scaleTargetKind": "RunnerDeployment", "scaleTargetGroup": "", "scaleTargetEnterprise": "", "scaleTargetOrganization": ""}
2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler Skipped scale target irrelevant to event {"eventOrganization": "xyz-corp", "eventEnterprise": "xyz", "scaleTargetKind": "RunnerDeployment", "scaleTargetGroup": "", "scaleTargetEnterprise": "", "scaleTargetOrganization": ""}
2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler no repository/organizational/enterprise runner found {"event": "workflow_job", "hookID": "624549042", "delivery": "62w45490-34235-13ee-34f8-418212wag129", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted", "github-group-sync-runner"], "repository.name": "job-okta-github-group-sync", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "completed", "workflowJob.runID": 45365342, "workflowJob.ID": 289290412, "repository": "xyz-corp/job-okta-github-group-sync", "organization": "xyz-corp", "enterprises": "xyz"}
2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler groups {"event": "workflow_job", "hookID": "624549042", "delivery": "62w45490-34235-13ee-34f8-418212wag129", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted", "github-group-sync-runner"], "repository.name": "job-okta-github-group-sync", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "completed", "workflowJob.runID": 45365342, "workflowJob.ID": 289290412, "groups": ""}
2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler no repository/organizational/enterprise runner found {"event": "workflow_job", "hookID": "624549042", "delivery": "62w45490-34235-13ee-34f8-418212wag129", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted", "github-group-sync-runner"], "repository.name": "job-okta-github-group-sync", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "completed", "workflowJob.runID": 45365342, "workflowJob.ID": 289290412, "repository": "xyz-corp/job-okta-github-group-sync", "organization": "xyz-corp", "enterprise": "xyz"}
2023-06-15T15:22:06Z DEBUG controllers.webhookbasedautoscaler Scale target not found. If this is unexpected, ensure that there is exactly one repository-wide or organizational runner deployment that matches this webhook event. If --watch-namespace is set ensure this is configured correctly. {"event": "workflow_job", "hookID": "624549042", "delivery": "62w45490-34235-13ee-34f8-418212wag129", "workflowJob.status": "completed", "workflowJob.labels": ["self-hosted", "github-group-sync-runner"], "repository.name": "job-okta-github-group-sync", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "completed", "workflowJob.runID": 45365342, "workflowJob.ID": 289290412}

Image reference of response on webhook:

Where as in the same cluster and same namespace I have another repository runner which works fine, with similar configuration:-

RunnerDeployment with repository runner and HorizontalRunnerAutoscaler based of webhook which has no error and works fine:


apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: example-runners
namespace: runner
spec:
template:
spec:
repository: xyz-corp/githubactions-poc
labels:
- example-runner

apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
name: example-runners
namespace: runner
spec:
minReplicas: 0
maxReplicas: 10
scaleTargetRef:
kind: RunnerDeployment
# # In case the scale target is RunnerSet:
# kind: RunnerSet
name: example-runners
scaleUpTriggers:
- githubEvent:
workflowJob: {}
duration: "10s"


logs for working repository runner in the same namespace:

2023-06-15T11:10:10Z DEBUG controllers.webhookbasedautoscaler Found 1 HRAs by key {"key": "xyz-corp/githubactions-poc"}
2023-06-15T11:10:10Z INFO controllers.webhookbasedautoscaler job scale up target is repository-wide runners {"event": "workflow_job", "hookID": "624549042", "delivery": "62w45490-34235-13ee-34f8-418212wag129", "workflowJob.status": "queued", "workflowJob.labels": ["example-runner"], "repository.name": "githubactions-poc", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "queued", "workflowJob.runID": 6510750705, "workflowJob.ID": 66511428714, "repository": "githubactions-poc"}
2023-06-15T11:10:10Z INFO controllers.webhookbasedautoscaler scaled example-runners by 1 {"event": "workflow_job", "hookID": "624549042", "delivery": "62w45490-34235-13ee-34f8-418212wag129", "workflowJob.status": "queued", "workflowJob.labels": ["example-runner"], "repository.name": "githubactions-poc", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "queued", "workflowJob.runID": 6510750705, "workflowJob.ID": 66511428714}
2023-06-15T11:10:10Z INFO controllers.webhookbasedautoscaler Starting batch worker
2023-06-15T11:10:13Z DEBUG controllers.webhookbasedautoscaler Patching hra example-runners for capacityReservations update {"before": 0, "expired": -1, "added": 1, "completed": 0, "after": 1}
2023-06-15T11:13:53Z DEBUG controllers.webhookbasedautoscaler Found 1 HRAs by key {"key": "xyz-corp/githubactions-poc"}
2023-06-15T11:13:53Z INFO controllers.webhookbasedautoscaler job scale up target is repository-wide runners {"event": "workflow_job", "hookID": "624549042", "delivery": "b607a840-0b6d-11ee-8f24-f77286165486", "workflowJob.status": "completed", "workflowJob.labels": ["example-runner"], "repository.name": "githubactions-poc", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "completed", "workflowJob.runID": 6510750705, "workflowJob.ID": 66511428714, "repository": "githubactions-poc"}
2023-06-15T11:13:53Z INFO controllers.webhookbasedautoscaler scaled example-runners by -1 {"event": "workflow_job", "hookID": "624549042", "delivery": "b607a840-0b6d-11ee-8f24-f77286165486", "workflowJob.status": "completed", "workflowJob.labels": ["example-runner"], "repository.name": "githubactions-poc", "repository.owner.login": "xyz-corp", "repository.owner.type": "Organization", "enterprise.slug": "xyz", "action": "completed", "workflowJob.runID": 6510750705, "workflowJob.ID": 66511428714}
2023-06-15T11:13:55Z DEBUG controllers.webhookbasedautoscaler Patching hra example-runners for capacityReservations update {"before": 1, "expired": 1, "added": 0, "completed": -1, "after": 0}

Image reference of response on webhook:

I am not sure what I am doing wrong, as both has them has similar configurations. I have tried multiple things mentioned in the open forum, like using "group: Default", deleting all other runners and only run single runner, deployed to other namespaces (remove from the previous one) and redeploying the Action runner controller. None of them worked.