You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
for handling a [non-graceful node shutdown](/docs/concepts/architecture/nodes/#non-graceful-node-shutdown).
12
12
In Kubernetes v1.26, this feature moves to beta. This feature allows stateful workloads to failover to a different node after the original node is shut down or in a non-recoverable state, such as the hardware failure or broken OS.
13
13
14
-
## What is a Node Shutdown in Kubernetes
14
+
## What is a node shutdown in Kubernetes?
15
15
16
16
In a Kubernetes cluster, it is possible for a node to shut down. This could happen either in a planned way or it could happen unexpectedly. You may plan for a security patch, or a kernel upgrade and need to reboot the node, or it may shut down due to preemption of VM instances. A node may also shut down due to a hardware failure or a software problem.
17
17
@@ -22,7 +22,7 @@ A node shutdown could lead to workload failure if the node is not drained before
22
22
23
23
In the following, we will describe what is a graceful node shutdown and what is a non-graceful node shutdown.
24
24
25
-
## What is a Graceful Node Shutdown
25
+
## What is a _graceful_ node shutdown?
26
26
27
27
The kubelet's handling for a [graceful node shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown)
28
28
allows the kubelet to detect a node shutdown event, properly terminate the pods on that node,
@@ -31,7 +31,7 @@ and release resources before the actual shutdown.
31
31
are terminated after all the regular pods are terminated, to ensure that the
32
32
essential functions of an application can continue to work as long as possible.
33
33
34
-
## What is a Non-Graceful Node Shutdown
34
+
## What is a _non-graceful_ node shutdown?
35
35
36
36
A Node shutdown can be graceful only if the kubelet's _node shutdown manager_ can
37
37
detect the upcoming node shutdown action. However, there are cases where a kubelet
@@ -54,7 +54,7 @@ so your workload may struggle to self-heal if it was already at maximum scale.
54
54
(By the way: if the node that had done a non-graceful shutdown comes back up, the kubelet does delete
55
55
the old Pod, and the control plane can make a replacement.)
56
56
57
-
## What’s new in Beta
57
+
## What's new for the beta?
58
58
59
59
For Kubernetes v1.26, the non-graceful node shutdown feature is beta and enabled by default.
60
60
The `NodeOutOfServiceVolumeDetach`
@@ -70,7 +70,7 @@ On the instrumentation side, the kube-controller-manager reports two new metrics
70
70
`force_delete_pod_errors_total`
71
71
: number of errors encountered when attempting forcible Pod deletion (also resets on Pod garbage collection controller restart)
72
72
73
-
## How does it work
73
+
## How does it work?
74
74
75
75
In the case of a node shutdown, if a graceful shutdown is not working or the node is in a
76
76
non-recoverable state due to hardware failure or broken OS, you can manually add an `out-of-service`
@@ -92,7 +92,14 @@ Once all the workload pods that are linked to the out-of-service node are moved
92
92
93
93
Depending on feedback and adoption, the Kubernetes team plans to push the Non-Graceful Node Shutdown implementation to GA in either 1.27 or 1.28.
94
94
95
-
This feature requires a user to manually add a taint to the node to trigger the failover of workloads and remove the taint after the node is recovered. In the future, we plan to find ways to automatically detect and fence nodes that are shut down or in a non-recoverable state and fail their workloads over to another node.
95
+
This feature requires a user to manually add a taint to the node to trigger the failover of workloads and remove the taint after the node is recovered.
96
+
97
+
The cluster operator can automate this process by automatically applying the `out-of-service` taint
98
+
if there is a programmatic way to determine that the node is really shut down and there isn’t IO between
99
+
the node and storage. The cluster operator can then automatically remove the taint after the workload
100
+
fails over successfully to another running node and that the shutdown node has been recovered.
101
+
102
+
In the future, we plan to find ways to automatically detect and fence nodes that are shut down or in a non-recoverable state and fail their workloads over to another node.
0 commit comments