Add/remove taint of a k8s node during HCloud node fault

[gemelen]

TL;DR

Add/remove a taint for a non-graceful node shutdown

Expected behavior

During an outage of a physical server that hosts any particular HCloud server a respective k8s node is detected of being unavailable, but not tainted as such. This degrades a k8s cluster, where some workloads are not available, but not moved to the healthy nodes. See non-graceful-node-shutdown.

Tainting a such node by the HCCM with the node.kubernetes.io/out-of-service and either NoExecute or NoSchedule effect would provide an automated handling of a fault.

[apricote]

Hey @gemelen,

this sounds like a very useful addition. Right now we already set the taint "node.cloudprovider.kubernetes.io/shutdown" if the server is shutdown (status == "off"). It would be very easy to extend this to another status, I asked internally if there is a clear signal in the API if the underlying host has an outage.

I do not think we can easily set the node.kubernetes.io/out-of-service taint:

• all documentation says that users should manually apply the taint and
• we are using the k8s.io/cloud-provider library to talk to Kubernetes, which currently does not set this taint.

[gemelen]

@apricote thanks for the feedback.

I understand that originally it is designated that a user (ie cluster administrator) should set it manually.

The problem is that it could take quite some time (and effort in case of an attempt to automate) for the flow:
physical node maintenance occurs ->
cloud account's contact email notified about that (one gets both a email and a notification in web UI) ->
action taken by human to taint the k8s node(s).

As one gets a notification already, then it sounds like at least there is some API in Cloud management system to get aware of it.

Should the HCCM take that to action on behalf of a user? I'd say yes, maybe as an opt-in configuration.