You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Kubernetes v1.24 introduces alpha support for [Non-Graceful Node Shutdown](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2268-non-graceful-shutdown). This feature allows stateful workloads to failover to a different node after the original node is shutdown or in a non-recoverable state such as the hardware failure or broken OS.
10
+
Kubernetes v1.24 introduces alpha support for [Non-Graceful Node Shutdown](https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/2268-non-graceful-shutdown). This feature allows stateful workloads to failover to a different node after the original node is shutdown or in a non-recoverable state such as hardware failure or broken OS.
11
11
12
12
## How is this different from Graceful Node Shutdown
13
13
14
14
You might have heard about the [Graceful Node Shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown) capability of Kubernetes,
15
15
and are wondering how the Non-Graceful Node Shutdown feature is different from that. Graceful Node Shutdown
16
-
allows Kubernetes to detect when a node is shutting down cleanly, and handle that situation appropriately.
16
+
allows Kubernetes to detect when a node is shutting down cleanly, and handles that situation appropriately.
17
17
A Node Shutdown can be "graceful" only if the node shutdown action can be detected by the kubelet ahead
18
18
of the actual shutdown. However, there are cases where a node shutdown action may not be detected by
19
19
the kubelet. This could happen either because the shutdown command does not trigger the systemd inhibitor
@@ -23,15 +23,15 @@ locks mechanism that kubelet relies upon, or because of a configuration error
23
23
Graceful node shutdown relies on Linux-specific support. The kubelet does not watch for upcoming
24
24
shutdowns on Windows nodes (this may change in a future Kubernetes release).
25
25
26
-
When a node is shutdown but without the kubelet detecting it, Pods on that node
27
-
also shut down ungracefully. For stateless apps, that's often not a problem (a ReplicaSet adds a new Pod once
28
-
the cluster detects that the affected node or Pod has failed). For stateful apps, the story is more complicated.
29
-
If you use a StatefulSet and have a Pod from that StatefulSet on a node that fails uncleanly, that affected Pod
30
-
will be marked as terminating; the StatefulSet cannot create a replacement Pod because the existing Pod
26
+
When a node is shutdown but without the kubelet detecting it, pods on that node
27
+
also shut down ungracefully. For stateless apps, that's often not a problem (a ReplicaSet adds a new pod once
28
+
the cluster detects that the affected node or pod has failed). For stateful apps, the story is more complicated.
29
+
If you use a StatefulSet and have a pod from that StatefulSet on a node that fails uncleanly, that affected pod
30
+
will be marked as terminating; the StatefulSet cannot create a replacement pod because the pod
31
31
still exists in the cluster.
32
32
As a result, the application running on the StatefulSet may be degraded or even offline. If the original, shut
33
-
down node comes up again, the kubelet on that original node reports in, deletes the existing Pods, and
34
-
the control plane makes a replacement Pod for that StatefulSet on a different running node.
33
+
down node comes up again, the kubelet on that original node reports in, deletes the existing pods, and
34
+
the control plane makes a replacement pod for that StatefulSet on a different running node.
35
35
If the original node has failed and does not come up, those stateful pods would be stuck in a
36
36
terminating status on that failed node indefinitely.
37
37
@@ -54,7 +54,7 @@ taint following a shutdown that the kubelet did not detect and handle in advance
54
54
can use that taint is when the node is in a non-recoverable state due to a hardware failure or a broken OS.
55
55
The values you set for that taint can be `node.kubernetes.io/out-of-service=nodeshutdown: "NoExecute"`
56
56
or `node.kubernetes.io/out-of-service=nodeshutdown:" NoSchedule"`.
57
-
Provided you have enabled the feature gate as I mentioned earlier, setting the out-of-service taint on a Node
57
+
Provided you have enabled the feature gate mentioned earlier, setting the out-of-service taint on a Node
58
58
means that pods on the node will be deleted unless if there are matching tolerations on the pods.
59
59
Persistent volumes attached to the shutdown node will be detached, and for StatefulSets, replacement pods will
60
60
be created successfully on a different running node.
Note: Before applying the out-of-service taint, you **must** verify that a node is already in shutdown or power off state (not in the middle of restarting), either because the user intentionally shut it down or the node is down due to hardware failures, OS issues, etc.
72
72
73
-
Once all the workload Pods that are linked to the out-of-service node are moved to a new running node, and the shutdown node has been recovered, you should remove
73
+
Once all the workload pods that are linked to the out-of-service node are moved to a new running node, and the shutdown node has been recovered, you should remove
74
74
that taint on the affected node after the node is recovered.
75
75
If you know that the node will not return to service, you could instead delete the node from the cluster.
0 commit comments