Feature: Add option to hibernate runners instead of terminating them when scaling down

[Bechir-Braham]

Hello,

As we have a big bazel monorepo some CI jobs require a long initialization phase that takes up to 20-30minutes before actually running the test. After this, next jobs picked up by the same runner can re-use the cache and execute quite fast. The issue here is that the cache is held in memory and terminating and then starting runners again can become time-costly.

A possible solution to this problem is to hibernate inactive runners instead of terminating them when scaling down. This can help re-use the in-memory cache for an acceptable extra storage cost. When scaling up if there's a hibernated runner then we should use it otherwise a new runner spins up.

Another possible approach would be very similar to this issue #4033. Scale up to the maximum, hibernate runners_maximum_count - idleCount and then for scale up we wake up some of the hibernated runners.

Do you see any blockers or risks to implement such a change?

[npalm]

I don't see any blockers to support this. but could require first some refactoring to make this possible and fit in the current implemention. I see several caveats. And the biest one is how to support ephemral runners, which security wise are in my point of view tye type that should be used. Since hibernate will stop the instance a new jit token needs to be request in the reboot process. I don't see why the scale-up can't do this. By before creating an instance see if any hibernated one can be reused and create a fresh JIT token. Othe caveat could be spot isntances.

So looing forward on how you would like to implement this.