Expose NodeSelector and Affinity field via SelfNodeRemediationConfig #274
Description
Currently, the SelfNodeRemediationConfig custom resource exposes a customDsTolerations which is useful for allowing the DaemonSet schedule on tainted nodes.
However, the lack of NodeSelector and Affinity means that it's not currently possible to achieve "every node except" behaviour, which is desirable in some cases.
For example, I have many nodegroups with constantly changing taints, so it's not really feasible for me to maintain a list of tolerations in customDsTolerations, and I want SNR to run on most nodes anyway. Consequently, my toleration is simply:
customDsTolerations: [{
operator: 'Exists'
}]However, there is one exception in that I don't want SNR to try and schedule on Fargate nodes, since they're designed to act as nodes running only a single workload pod. For most daemonsets this is easy enough to achieve:
affinity: {
nodeAffinity: {
requiredDuringSchedulingIgnoredDuringExecution: {
nodeSelectorTerms: [{
matchExpressions: [{
key: 'eks.amazonaws.com/compute-type',
operator: 'NotIn',
values: ['fargate'],
}]
}]
}
},
},But as SNR doesn't expose the ability to set affinity, we end up an unschedulable pod for each Fargate node:
kubectl get pods -A -l app.kubernetes.io/name=self-node-remediation | /bin/grep Pending
operators self-node-remediation-ds-5ktsn 0/1 Pending 0 18h
operators self-node-remediation-ds-5nmct 0/1 Pending 0 16h
operators self-node-remediation-ds-8cbxt 0/1 Pending 0 16h
operators self-node-remediation-ds-dw4tt 0/1 Pending 0 18h
operators self-node-remediation-ds-thnf8 0/1 Pending 0 0s
It also doesn't seem to be possible to patch the daemonset, because OLM will immediately revert it.
If there's another way around this, I'm open to suggestions!