[WIP] feat: add support for native retry policies

[dejanzele]

A detailed description can be found in the GitHub Issue #4683

[Sovietaced]

One thing we ran into recently is that there can also be collision on service names and ingress names. This would only affect folks opting into those features so can probably be done in a follow up pull request.

[Sovietaced]

Another issue that we see daily that seems to completely disrupt scheduling is the fact that there is no concept of a gang generation. This specifically seems to happen when the scheduler has performed a preemption on a gang.

Later the scheduler will do some logical schedule and preempt logic in the scheduler but it will see some old pods from the original gang schedule that were preempted still on the nodes and then we get error messages like

scheduler.go:202 scheduling cycle failure error="gang runner-1328d9002fde4882bcb-n3-0-n3-0-dn0-0 was partially evicted: 2 out of 3 jobs evicted" cycleNumber=1030057

I believe that it misinterprets real pods on the nodes from the previous gang generation as related to the logical schedule/preempt it does as part of the regular scheduling algorithm and then blows up.