Make executor clean up pods stuck terminating more quickly (#4697)

This config is used to determine when the executor needs to step in to
force kill a pod
- The timeout is in addition to the pods termination grace period second
 - The timeout is purely give k8s some time to act

Waiting 30s for k8s to take action before calling force kill ourselves
should be OK (we're only speeding up what k8s should be doing already)

This should make cleaning up pods stuck in terminating states that k8s
can't handle far more responsive

Signed-off-by: JamesMurkin <jamesmurkin@hotmail.com>

Author: JamesMurkin

config/executor/config.yaml

@@ -40,8 +40,8 @@ kubernetes:
   minimumPodAge: 3m
   failedPodExpiry: 10m
   maxTerminatedPods: 1000 # Should be lower than kube-controller-managed terminated-pod-gc-threshold (default 12500)
-  stuckTerminatingPodExpiry: 1m
-  podKillTimeout: 5m
+  stuckTerminatingPodExpiry: 30s
+  podKillTimeout: 30s
   minimumResourcesMarkedAllocatedToNonArmadaPodsPerNode:
     cpu: 1
     memory: 200Mi