Throttle Simultaneous Transform Jobs

[BenGalewsky]

We have seen cases where we submit 380 transform jobs at once (with each requesting 20 pods) we exhaust the K8s controller memory.

As a workaround, Chicago Admins increased the memory for this controller, but there will be cases in the future where we run into this again.

Ideally we have a queue that will restrict the number of simultaneous jobs and only submit new ones when we actually have capacity

[ponyisi]

To understand the issue a bit better - the limit is in the number of running jobs? deployments? or the number being created in a short time period? @ivukotic

[ivukotic]

it is both number of transformer deployments created quickly and total transformers. Former stresses controller, later prevents cluster from doing any other stuff (eg. create cron pods).

[ponyisi]

@ivukotic ok ... the former we can always invent some mechanism to throttle submissions, but the latter seems like a task for priority classes no?