webhook-driven autoscaling in a single controller, multi-organization setup

[kathleenfrench]

hey - i have a question about webhook driven autoscaling in the context of a multi-organization actions runner controller setup (GHES).

currently, i have a single actions runner controller configured to orchestrate runner deployments for many organizations. it’s been working well, but at the time of initial setup webhook driven autoscaling was still experimental so i went with a pull based metrics approach.

now that there’s greater support for webhook driven autoscaling and as the demand for scaling this implementation up continues to grow, migrating to that over pull-driven seems like a better route. (also since i’m working in GHES, we aren’t concerned with rate limiting).

what isn’t clear to me from the documentation is whether a single webhook deployment could serve a multi-organization, single controller setup, as opposed to needing a controller for every organization that has its own dedicated webhook.

i don’t see any immediate reason why a single webhook/controller model wouldn’t work for multiple organizations judging by the webhook autoscaling logic, but as this use case hasn’t come up in any existing issues/discussions i want to make sure i'm not misinterpreting the extent of webhook driven autoscaling support in this context.

also beyond it “working” i’m hoping to determine whether there’s potential limitations or performance drawbacks that would make it a less optimal implementation vs going with a controller for every organization with a dedicated webhook deployment.

thanks so much in advance for any further insight!

[mumoshu]

Hey @kathleenfrench!

a single webhook deployment could serve a multi-organization

It should work this way- You'd expose ARC's github-webhook-based-autoscaler's webhook endpoint via a single URL and set the URL as the webhook endpoint for every organization in your GHES instance and it should just work.

performance drawbacks

Honestly, I have never had a chance to test it myself in a large-scale env, but- I designed the webhook-based autoscaler to be horizontally scalable. If multiple instances of the webhook-based autoscaler tried to scale the same HRA concurrently it ended up with conflict and losing the webhook event in previous versions of ARC. But since 0.25.x it does batching and retrying so it's more reliable. than before.
If you encounter any perf issues when you scaled webhook-based autoscalers horizontally, please feel free to file a bug report.

[kathleenfrench]

hey @mumoshu - that's great! i couldn't identify any reasons why it wouldn't work, but wanted to make sure i wasn't missing anything before going through with the migration 😅

thanks for getting back to me!