Add option to disable pod affinity

[Wielewout]

In #212 pod affinity was added when the kube scheduler is enabled. While a way better default, it makes less optimal use of resources in the cluster (or even breaks? some setups #201 (comment)).

By default pod affinity remains set, but by setting ACTIONS_RUNNER_USE_POD_AFFINITY=false pod affinity rules will be skipped.

When disabling, the runner and workflow pod can then be scheduled on different nodes again. It is up to the user though to support RWX volumes in the cluster, a node selector for architecture if using a multi-arch cluster (on both the runner and workflow pod so they match), ...

[zchenyu]

Thanks! fwiw, I had the exact same implementation in my fork :)

[gigabyte132]

This would help us a lot in our setup as well, as without it breaks our GPU runners whenever there is no availability of GPUs on the node that the job pod runs (even though there are GPUs available on other nodes).

[oed-lipphausent]

This is exactly what we need ! :D

Currently, we have the problem that the runner pods are created as expected, but we always have to wait for the workflow pods. Since the introduction of node affinity, our nodes do not have enough resources to process every workflow pod for every runner pod.
This is why we originally switched to the Kube Scheduler, but since the change, we have the problem again.

The change with node affinity makes using the Kube Scheduler pointless for us. We deliberately chose this path so that we could use smaller nodes and scale the number of nodes to save costs.

However, since this PR has been open for more than a month, I wonder if it is realistic to expect this change to be considered in the near future.

I don't know who is responsible for this maybe, @nikola-jokic, but it would be nice to see some feedback on this PR

[nikola-jokic]

Hey everyone, we are currently working on a PR that will disable the affinity and volume mounts completely.

[oed-lipphausent]

@nikola-jokic, thank you for the update. I didn't know you were working on something like this, and I'm very excited to see how it develops. I think it's a nice idea.