Adding TSG for remediating NAKS stuck workloads due to power failure

In the event of power-failure, Undercloud nodes will be powered-off
causing NAKS nodes running on powered-off UC node become not-ready,
which will result in stateful workloads being stuck on those not-ready
nodes. This TSG helps to move those stuck workloads to healthy nodes.

Author: Nyeem Akhtar

articles/operator-nexus/TOC.yml

@@ -371,8 +371,11 @@
           href: troubleshoot-neighbor-group-creation-error.md
         - name: Troubleshoot NAKS Cluster Node Packet Loss
           href: troubleshoot-packet-loss.md
-        - name: Troubleshooting Nexus Kubernetes Cluster stuck (unable to reschedule) workloads due to power failure
-          href: troubleshoot-kubernetes-cluster-stuck-workloads-due-to-power-failure.md
+        - name: Troubleshoot Nexus Kubernetes Cluster stuck (unable to reschedule) workloads
+          expanded: false
+          items:
+            - name: Due To Bare Metal Machine Power Failure
+              href: troubleshoot-kubernetes-cluster-stuck-workloads-due-to-power-failure.md
 - name: FAQ
   href: azure-operator-nexus-faq.md
 - name: Reference

articles/operator-nexus/media/bmm-where-not-ready-naks-reside.png

@@ -4,92 +4,110 @@ description: Troubleshooting Nexus Kubernetes Cluster workloads stuck (unable to
 ms.service: azure-operator-nexus
 ms.custom: troubleshooting
 ms.topic: troubleshooting
-ms.date: 01/30/2025
+ms.date: 02/18/2025
 ms.author: nyeemakhtar
 author: mdnyeemakhtar
 ---
-# Troubleshooting stuck (unable to reschedule) workloads in a Nexus Kubernetes Cluster
+# Troubleshooting stuck (unable to reschedule) workloads in a Nexus Kubernetes Cluster due to power failure
 
-This guide provides detailed steps for troubleshooting issues related to stuck workloads on Nexus Kubernetes Cluster not-ready nodes. If you're experiencing these issues due to bare-metal node power failure, this guide helps you identify and resolve the problem.
+What is a stuck workload?
+
+A stuck workload is a pod that is unable to reschedule to another node in a Kubernetes Cluster due to the node being not-ready. This issue can happen due to many reasons, including node power failure.
+
+Kubernetes, by design, doesn't move workloads that are stateful in nature if the node they're running on becomes not-ready (for example, due to power failure). For more information, see [Kubernetes documentation](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#non-graceful-node-shutdown).
+
+This guide details troubleshooting steps for cases where workloads on a Nexus Kubernetes Cluster become stuck due to bare-metal machine power failures. It also explains how to restart a stuck pod so that it can be rescheduled on a different node.
 
 ## Prerequisites
 
 * Permissions to view Azure resources in the subscription where the Nexus Kubernetes Cluster is deployed
-* Access to Azure monitoring
 * Necessary permissions to make changes using `kubectl` commands in Nexus Kubernetes Cluster (for example, deleting nodes)
 
-## Symptoms
+## Diagnosing Stuck Workloads
 
-* Nexus Kubernetes Cluster nodes are not-ready
-* Pods that aren't daemon-set pods stuck on the not-ready nodes
+If you observe that your applications aren't responding as expected, your workloads might be stuck on the not-ready nodes. To diagnose stuck workloads on Nexus Kubernetes Cluster not-ready nodes, look for the following symptoms:
 
-## Cause
+* To check if Nexus Kubernetes Cluster nodes are not-ready, run the following `kubectl` command in Nexus Kubernetes Cluster:
 
-Kubernetes, by design, doesn't move workloads that are stateful in nature if the node they're running on becomes not-ready. For more information, see [Kubernetes documentation](https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/#non-graceful-node-shutdown).
+  ```bash
+  kubectl get nodes | awk '$2 != "Ready" {print $1, $2}' | column -t
+  ```
+  ![kubectl get nodes output](media/naks-nodes-not-ready.png)
 
-The Nexus Kubernetes Cluster not-ready node could be caused by a power failure on the bare-metal machine in the cluster. You can verify by checking the "Idrac Power On" metric in Azure monitoring. If you have alerts set up for this metric, you might have received an alert indicating that the bare-metal machines are powered off.
+  If the command returns no results, then all the nodes are ready. Sometimes, nodes take a few minutes to become not-ready after the power failure, so you might need to run the command again after a few minutes. If nodes continue to show as ready even after reasonable time (5-10 mins) and your applications are still not responding, then the issue might be different, contact support for further assistance.
 
-## Warning
+* To list pods stuck on the not-ready nodes, run the following `kubectl` command in Nexus Kubernetes Cluster:
 
-### Nexus Kubernetes Cluster nodes Rack Spread
+   ```bash
+   kubectl get nodes -o json | jq -r '.items[]
+  | select(.status.conditions[] | select(.type=="Ready") | .status != "True")
+  | .metadata.name' | xargs -I {} sh -c 'kubectl get pods --all-namespaces --field-selector spec.nodeName={} --no-headers | awk -v node={} "{print \$1, \$2, \$4, node}"' | sort -k1,1 | column -t -N "NAMESPACE,NAME,STATUS,NODE"
+   ```
+   This command returns pods (stateful and daemon-set) that are stuck on the not-ready nodes. Pods status might show `Pending`, `Terminating`, or `Running`.
 
-This guide requires deleting nodes from the Nexus Kubernetes Cluster. This action can cause Nexus Kubernetes Cluster nodes Rack Spread to be impacted.
+   ![stuck workload on not-ready nodes](media/naks-workload-stuck-on-not-ready-nodes.png)
 
-### Host-Path storage
+## Diagnosing Power Failure
 
-If the pods are configured to use host-path storage on the node, deleting the node deletes the data too.
+Once you have confirmed that the workloads are stuck on the not-ready nodes, the next step will help you diagnose if Nexus Kubernetes Cluster nodes are not-ready due to power failure of one or more bare-metal machines.
 
-## Solution
+To diagnose power failure on the bare-metal machine, look for the following symptoms:
 
-To resolve the issue, follow these steps:
+* To list bare-metal machines where not-ready Nexus Kubernetes Cluster nodes are running, run the following `kubectl` command in the Nexus Kubernetes Cluster:
 
-1. Check the "Idrac Power On" metric in Azure monitoring to verify that the bare-metal machine is powered off. Following is an example screenshot of the metric in Azure monitoring:
+   ```bash
+   kubectl get nodes -o json | jq -r '.items[]
+  | select(.status.conditions[] | select(.type=="Ready") | .status != "True")
+  | [.metadata.name, .metadata.labels["topology.kubernetes.io/baremetalmachine"]]
+  | @tsv' | column -t -s $'\t' -N "Node Name,BMM Name"
+   ```
+   This command returns the list of nodes and the bare-metal machine name where the nodes are running.
 
-   ![Idrac Power On metric in Azure monitoring](media/idrac-power-on-metric.png)
+   ![bmm names where node-ready nodes reside](media/bmm-where-not-ready-naks-reside.png)
 
-2. If the "Idrac Power On" metric indicates that the bare-metal machine is powered on (showing a value of 1), **don't proceed with the following steps**.
+* To list bare-metal machines in the cluster that are powered off, run the following command at cluster managed resource group level:
 
-3. If the "Idrac Power On" metric indicates that the bare-metal machine is powered off (showing a value of 0), take the following actions on all impacted Nexus Kubernetes Clusters within that environment:
+  ```bash
+  az networkcloud baremetalmachine list \
+  --subscription <subscription-id> \
+  --resource-group <managed-resource-group> \
+  --query "[? powerState == 'Off' && detailedStatus != 'Available'].{BMMName: name, ResourceGroup: resourceGroup, PowerState: powerState, DetailedStatus: detailedStatus}" \
+  -o table
+  ```
+  Replace `<subscription-id>` with the subscription ID and `<managed-resource-group>` with the managed resource group of the cluster where bare-metal resources reside.
 
-   * Note down the name of the bare-metal machine that is powered off and managed-resource-group name. In the example screenshot in step 1, the machine name is `a12345bcde1co1` and managed-resource-group name is `poc02-19cf0b39e1e5-HostedResources-540F8D4E`. If this bare-metal machine is a compute node, then proceed with the following steps.
-   * To find out impacted Nexus Kubernetes Clusters due to the powered-off bare-metal machine, run the following `az networkcloud` command:
+  for example:
+  ```bash
+  az networkcloud baremetalmachine list \
+  --subscription 00000000-0000-0000-0000-000000000000 \
+  --resource-group poc02-19cf0b39e1e5-HostedResources-540F8D4E \
+  --query "[? powerState == 'Off' && detailedStatus != 'Available'].{BMMName: name, ResourceGroup: resourceGroup, PowerState: powerState, DetailedStatus: detailedStatus}" \
+  -o table
+  ```
 
-     ```bash
-     az networkcloud kubernetescluster list \
-     --subscription <subscription-id> \
-     --query "[?extendedLocation.name && contains(extendedLocation.name, '<managed-resource-group>') && nodes[? bareMetalMachineId == \`null\` || contains(bareMetalMachineId, '<powered-off-baremetal-machine-name>') && detailedStatus != 'Running']].{ClusterName: name}" \
-     -o table
-     ```
-     Replace `<subscription-id>` with the subscription ID and `<managed-resource-group>` where the bare-metal machine is located and `<powered-off-baremetal-machine-name>` with the name of the powered-off bare-metal machine noted earlier.
+  ![powered off bare-metal machines](media/list-of-powered-off-bmms.png)
+  **Note:** This screenshot doesn't show subscription ID since it was already set in the Azure CLI session using `az account set --subscription <subscription-id>` command.
 
-     for example:
-     ```bash
-     az networkcloud kubernetescluster list \
-     --subscription 00000000-0000-0000-0000-000000000000 \
-     --query "[?extendedLocation.name && contains(extendedLocation.name, 'poc02-19cf0b39e1e5-HostedResources-540F8D4E') && nodes[? bareMetalMachineId == \`null\` || contains(bareMetalMachineId, 'a12345bcde1co1') && detailedStatus != 'Running']].{ClusterName: name}" \
-     -o table
-     ```
-     **Note:** The prior command might return Nexus Kubernetes Clusters that aren't impacted by the powered-off bare-metal machine. Further steps help you identify the impacted Nexus Kubernetes Clusters.
-   * If there are no Nexus Kubernetes Clusters in the list from prior command, then don't proceed with the next steps. However, if there are Nexus Kubernetes Clusters, run the following steps on each Nexus Kubernetes Cluster.
-   * Run following `kubectl` command in the Nexus Kubernetes Cluster to get the list of all the nodes that are running on powered off bare-metal machines:
+  If the command returns no results, rerun the command after a few minutes. If the command still returns no results after a reasonable time (5-10 mins) and your workloads are still stuck, then the issue might be different, contact support for further assistance.
 
-     ```bash
-     kubectl get nodes -l topology.kubernetes.io/baremetalmachine=<powered-off-baremetal-machine-name>
-     ```
-      Replace `<powered-off-baremetal-machine-name>` with the name of the powered-off bare-metal machine noted earlier.
+Cross check the powered off bare-metal machines names with the list of bare-metal machines where not-ready nodes are running. If the bare-metal machines where not-ready nodes are running are powered off, then the issue is due to power failure. Now, you can proceed to the next section to resolve the issue.
 
-      for example:
-      ```bash
-      kubectl get nodes -l topology.kubernetes.io/baremetalmachine=a12345bcde1co1
-      ```
-      ![kubectl get nodes output](media/naks-nodes-not-ready.png)
+## Warning
+
+### Nexus Kubernetes Cluster virtual machine (VM) placement
 
-      If prior command doesn't list any nodes, no further action is needed for the current Nexus Kubernetes Cluster.
+This guide requires deleting nodes from the Nexus Kubernetes Cluster. This action can cause Nexus Kubernetes Cluster VMs rack placement to be impacted. For more information, see [how the Nexus platform schedules a Nexus Kubernetes Cluster VM](./concepts-nexus-kubernetes-placement.md#how-the-nexus-platform-schedules-a-nexus-kubernetes-cluster-vm).
+
+### Host-Path storage
+
+If the pods are configured to use host-path storage on the node, deleting the node deletes the data too.
+
+## Solution
 
-      Don't proceed with the next steps until all of the nodes statuses are not-ready from prior command. If the nodes are ready, then wait and run the command again to check the status. When all the nodes statuses are not-ready, proceed to the next step.
+To move the stuck workloads to other nodes in the Nexus Kubernetes Cluster, you need to delete the Nexus Kubernetes Cluster nodes that are not-ready due to power failure. The workloads that are stuck on the not-ready nodes will be rescheduled to other nodes in the Nexus Kubernetes Cluster. Additionally, new nodes are created to replace the deleted nodes automatically if there's enough capacity available in the cluster.
 
-   * Double check by refreshing the "Idrac Power On" metric in Azure monitoring to verify that the bare-metal machine is still powered off. If the machine is showing as **powered on, don't proceed with the next steps**. If the machine is still showing as powered off, proceed to the next step.
-   * To delete the nodes that are not-ready, run following `kubectl` command in the Nexus Kubernetes Cluster:
+   * From prior steps, note down the names of the powered-off bare-metal machines where the not-ready nodes are running.
+   * To delete the nodes that are not-ready, run the following `kubectl` command in the Nexus Kubernetes Cluster **for each powered-off bare-metal machine**:
 
       ```bash
       kubectl delete node -l topology.kubernetes.io/baremetalmachine=<powered-off-baremetal-machine-name>
@@ -98,6 +116,6 @@ To resolve the issue, follow these steps:
 
       for example:
       ```bash
-      kubectl delete node -l topology.kubernetes.io/baremetalmachine=a12345bcde1co1
+      kubectl delete node -l topology.kubernetes.io/baremetalmachine=b37100gipc1co01
       ```
-   * After you deleted the nodes, the workloads that were stuck on the not-ready nodes will be rescheduled to other nodes in the Nexus Kubernetes Cluster. **Note, this process might take upwards of 30 minutes to complete**. Additionally, new nodes are created to replace the deleted nodes automatically if there's enough capacity available in the cluster.
+   * After you deleted the nodes, the workloads that were stuck on the not-ready nodes should be rescheduled to other nodes in the Nexus Kubernetes Cluster. Run prior `kubectl` command for all remaining powered-off bare-metal machines noted earlier.