Scaling up self-hosted GitHub Actions runners

[Eta0]

ml-containers uses a self-hosted GitHub Actions runner to build container images through CI. It is currently only capable of handling one job at a time, sequentially. As a consequence, complex builds with many variations such as ml-containers/torch are taking up to 7 hours per commit to finish their CI.

Very heavy commits slow down development, as it makes iteratively fixing bugs in a CI deployment impractical.

Either dedicating more resources to keep runners available or implementing some form of autoscaling like with actions-runner-controller may improve on the situation.

[todie]

Alright so after some false starts, a little research, and a chat with @ChandonPierre, I'm going to try registering an auto-scaling solution inside a virtual cluster. Falling back to punting with a large dedicated virtual server for this repository.

[salanki]

Let's spin up some more VMs, I Think they can simply be scaled up.

[todie]

Sounds like a plan. punting with vms 🏈