Skip to content

Commit 9175b3c

Browse files
committed
Address comments
1 parent 8367071 commit 9175b3c

File tree

1 file changed

+13
-6
lines changed

1 file changed

+13
-6
lines changed

content/en/blog/_posts/2022-12-16-non-graceful-node-shutdown-to-beta.md

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Kubernetes v1.24 [introduced](https://kubernetes.io/blog/2022/05/20/kubernetes-1
1111
for handling a [non-graceful node shutdown](/docs/concepts/architecture/nodes/#non-graceful-node-shutdown).
1212
In Kubernetes v1.26, this feature moves to beta. This feature allows stateful workloads to failover to a different node after the original node is shut down or in a non-recoverable state, such as the hardware failure or broken OS.
1313

14-
## What is a Node Shutdown in Kubernetes
14+
## What is a node shutdown in Kubernetes?
1515

1616
In a Kubernetes cluster, it is possible for a node to shut down. This could happen either in a planned way or it could happen unexpectedly. You may plan for a security patch, or a kernel upgrade and need to reboot the node, or it may shut down due to preemption of VM instances. A node may also shut down due to a hardware failure or a software problem.
1717

@@ -22,7 +22,7 @@ A node shutdown could lead to workload failure if the node is not drained before
2222

2323
In the following, we will describe what is a graceful node shutdown and what is a non-graceful node shutdown.
2424

25-
## What is a Graceful Node Shutdown
25+
## What is a _graceful_ node shutdown?
2626

2727
The kubelet's handling for a [graceful node shutdown](/docs/concepts/architecture/nodes/#graceful-node-shutdown)
2828
allows the kubelet to detect a node shutdown event, properly terminate the pods on that node,
@@ -31,7 +31,7 @@ and release resources before the actual shutdown.
3131
are terminated after all the regular pods are terminated, to ensure that the
3232
essential functions of an application can continue to work as long as possible.
3333

34-
## What is a Non-Graceful Node Shutdown
34+
## What is a _non-graceful_ node shutdown?
3535

3636
A Node shutdown can be graceful only if the kubelet's _node shutdown manager_ can
3737
detect the upcoming node shutdown action. However, there are cases where a kubelet
@@ -54,7 +54,7 @@ so your workload may struggle to self-heal if it was already at maximum scale.
5454
(By the way: if the node that had done a non-graceful shutdown comes back up, the kubelet does delete
5555
the old Pod, and the control plane can make a replacement.)
5656

57-
## Whats new in Beta
57+
## What's new for the beta?
5858

5959
For Kubernetes v1.26, the non-graceful node shutdown feature is beta and enabled by default.
6060
The `NodeOutOfServiceVolumeDetach`
@@ -70,7 +70,7 @@ On the instrumentation side, the kube-controller-manager reports two new metrics
7070
`force_delete_pod_errors_total`
7171
: number of errors encountered when attempting forcible Pod deletion (also resets on Pod garbage collection controller restart)
7272

73-
## How does it work
73+
## How does it work?
7474

7575
In the case of a node shutdown, if a graceful shutdown is not working or the node is in a
7676
non-recoverable state due to hardware failure or broken OS, you can manually add an `out-of-service`
@@ -92,7 +92,14 @@ Once all the workload pods that are linked to the out-of-service node are moved
9292

9393
Depending on feedback and adoption, the Kubernetes team plans to push the Non-Graceful Node Shutdown implementation to GA in either 1.27 or 1.28.
9494

95-
This feature requires a user to manually add a taint to the node to trigger the failover of workloads and remove the taint after the node is recovered. In the future, we plan to find ways to automatically detect and fence nodes that are shut down or in a non-recoverable state and fail their workloads over to another node.
95+
This feature requires a user to manually add a taint to the node to trigger the failover of workloads and remove the taint after the node is recovered.
96+
97+
The cluster operator can automate this process by automatically applying the `out-of-service` taint
98+
if there is a programmatic way to determine that the node is really shut down and there isn’t IO between
99+
the node and storage. The cluster operator can then automatically remove the taint after the workload
100+
fails over successfully to another running node and that the shutdown node has been recovered.
101+
102+
In the future, we plan to find ways to automatically detect and fence nodes that are shut down or in a non-recoverable state and fail their workloads over to another node.
96103

97104
## How can I learn more?
98105

0 commit comments

Comments
 (0)