Merge pull request #273229 from eak13/main

Updates for Feature 758537 Prevent Simultaneous Disruptive Actions Against KCP nodes

Author: JamesJBarnett

articles/operator-nexus/howto-baremetal-functions.md

@@ -1,36 +1,45 @@
 ---
-title: "Azure Operator Nexus: Platform Functions for Bare Metal Machines"
-description: Learn how to manage Bare Metal Machines (BMM).
-author: harish6724
-ms.author: harishrao
+title: "Azure Operator Nexus: Platform functions for bare metal machines"
+description: Learn how to manage bare metal machines (BMM).
+author: eak13
+ms.author: ekarandjeff
 ms.service: azure-operator-nexus
 ms.topic: how-to
-ms.date: 05/26/2023
+ms.date: 04/30/2024
 ms.custom: template-how-to, devx-track-azurecli
 ---
 
-# Manage lifecycle of Bare Metal Machines
+# Manage the lifecycle of bare metal machines
 
-This article describes how to perform lifecycle management operations on Bare Metal Machines (BMM). These steps should be used for troubleshooting purposes to recover from failures or when taking maintenance actions. The commands to manage the lifecycle of the BMM include:
+This article describes how to perform lifecycle management operations on bare metal machines (BMM). These steps should be used for troubleshooting purposes to recover from failures or when taking maintenance actions. The commands to manage the lifecycle of the BMM include:
 
-- Power off the BMM
+> [!CAUTION]
+> Do not perform any action against management servers without first consulting with Microsoft support personnel. Doing so could affect the integrity of the Operator Nexus Cluster.
+
+- **Power off the BMM**
 - Start the BMM
-- Restart the BMM
-- Make the BMM unschedulable or schedulable
-- Reimage the BMM
-- Replace the BMM
+- **Restart the BMM**
+- Make the BMM unschedulable (cordon without evacuate)
+- **Make the BMM unschedulable (cordon with evacuate)**
+- Make the BMM schedulable (uncordon)
+- **Reimage the BMM**
+- **Replace the BMM**
+
+> [!IMPORTANT]
+> Disruptive command requests against a Kubernetes Control Plane (KCP) node are rejected if there is another disruptive action command already running against another KCP node or if the full KCP is not available. This check is done to maintain the integrity of the Nexus instance and ensure multiple KCP nodes don't go down at once due to simultaneous disruptive actions. If multiple nodes go down, it will break the healthy quorum threshold of the Kubernetes Control Plane.
+>
+> The bolded actions in the above list are considered disruptive (Power off, Restart, Reimage, Replace). Cordon without evacuate is not considered disruptive. Cordon with evacuate is considered disruptive.
+>
+> As noted in the cautionary statement, running actions against management servers, especially KCP nodes, should only be done in consultation with Microsoft support personnel.
 
 ## Prerequisites
 
 1. Install the latest version of the
-   [appropriate CLI extensions](./howto-install-cli-extensions.md)
-1. Get the name of the resource group for the BMM
-1. Get the name of the bare metal machine that requires a lifecycle management operation
-1. Ensure that the target bare metal machine `poweredState` set to `On` and `readyState` set to `True`
-   1. This prerequisite is not applicable for the `start` command
-
-> [!CAUTION]
-> Actions against management servers should not be run without consultation with Microsoft support personnel. Doing so could affect the integrity of the Operator Nexus Cluster.
+   [appropriate CLI extensions](./howto-install-cli-extensions.md).
+1. Get the name of the resource group for the BMM.
+1. Get the name of the bare metal machine that requires a lifecycle management operation.
+1. Ensure that the target bare metal machine `poweredState` set to `On` and `readyState` set to `True`.
+   1. This prerequisite isn't applicable for the `start` command.
 
 ## Power off the BMM
 
@@ -81,9 +90,9 @@ az networkcloud baremetalmachine cordon \
 
 The `evacuate "True"` removes workloads from that node while `evacuate "False"` only prevents the scheduling of new workloads.
 
-## Make a BMM schedulable (uncordon)
+## Make a BMM "schedulable" (uncordon)
 
-You can make a BMM `schedulable` (usable) by executing the [`uncordon`](#make-a-bmm-schedulable-uncordon) command. All workloads in a `pending`
+You can make a BMM "schedulable" (usable) by executing the [`uncordon`](#make-a-bmm-schedulable-uncordon) command. All workloads in a `pending`
 state on the BMM are `restarted` when the BMM is `uncordoned`.
 
 ```azurecli
@@ -94,15 +103,14 @@ az networkcloud baremetalmachine uncordon \
 
 ## Reimage a BMM
 
-You can restore the runtime version on a BMM by executing `reimage` command. This process **redeploys** the runtime image on the target BMM and executes the steps to rejoin the cluster with the same identifiers. This action doesn't impact the tenant workload files on this BMM.
+You can restore the runtime version on a BMM by executing the `reimage` command. This process **redeploys** the runtime image on the target BMM and executes the steps to rejoin the cluster with the same identifiers. This action doesn't affect the tenant workload files on this BMM.
 As a best practice, make sure the BMM's workloads are drained using the [`cordon`](#make-a-bmm-unschedulable-cordon)
-command, with `evacuate "True"`, prior to executing the `reimage` command.
+command, with `evacuate "True"`, before executing the `reimage` command.
 
-> [!Warning]
-> Running more than one baremetalmachine replace or reimage command at the same time, or running a replace
-> at the same time as a reimage, will leave servers in a nonworking state. Make sure one replace / reimage
-> has fully completed before starting another one. In a future release, we plan to either add the ability
-> to replace multiple servers at once or have the command return an error when attempting to do so.
+> [!WARNING]
+> Running more than one `baremetalmachine replace` or `reimage` command at the same time, or running a `replace`
+> at the same time as a `reimage` will leave servers in a nonworking state. Make sure one `replace`/`reimage`
+> has fully completed before starting another one.
 
 ```azurecli
 az networkcloud baremetalmachine reimage \
@@ -112,13 +120,12 @@ az networkcloud baremetalmachine reimage \
 
 ## Replace BMM
 
-Use `Replace BMM` command when a server has encountered hardware issues requiring a complete or partial hardware replacement. After replacement of components such as motherboard or NIC replacement, the MAC address of BMM will change, however the IDrac IP address and hostname will remain the same.
+Use the `replace` command when a server encounters hardware issues requiring a complete or partial hardware replacement. After replacement of components such as motherboard or Network Interface Card (NIC) replacement, the MAC address of BMM will change, however the iDRAC IP address and hostname will remain the same.
 
-> [!Warning]
-> Running more than one baremetalmachine replace or reimage command at the same time, or running a replace
-> at the same time as a reimage, will leave servers in a nonworking state. Make sure one replace / reimage
-> has fully completed before starting another one. In a future release, we plan to either add the ability
-> to replace multiple servers at once or have the command return an error when attempting to do so.
+> [!WARNING]
+> Running more than one `baremetalmachine replace` or `reimage` command at the same time, or running a `replace`
+> at the same time as a `reimage` will leave servers in a nonworking state. Make sure one `replace`/`reimage`
+> has fully completed before starting another one.
 
 ```azurecli
 az networkcloud baremetalmachine replace \

articles/operator-nexus/troubleshoot-reboot-reimage-replace.md

@@ -1,32 +1,42 @@
 ---
 title: Troubleshoot Azure Operator Nexus server problems
-description: Troubleshoot cluster bare-metal machines with three Rs for Azure Operator Nexus.
+description: Troubleshoot cluster bare metal machines with Restart, Reimage, Replace for Azure Operator Nexus.
 ms.service: azure-operator-nexus
 ms.custom: troubleshooting
 ms.topic: troubleshooting
-ms.date: 06/12/2023
-author: JAC0BSMITH
-ms.author: jacobsmith
+ms.date: 04/24/2024
+author: eak13
+ms.author: ekarandjeff
 ---
 
 # Troubleshoot Azure Operator Nexus server problems
 
-This article describes how to troubleshoot server problems by using restart, reimage, and replace (three Rs) actions on Azure Operator Nexus bare-metal machines (BMMs). You might need to take these actions on your server for maintenance reasons, which causes a brief disruption to specific BMMs.
+This article describes how to troubleshoot server problems by using restart, reimage, and replace actions on Azure Operator Nexus bare metal machines (BMMs). You might need to take these actions on your server for maintenance reasons, which causes a brief disruption to specific BMMs.
 
 The time required to complete each of these actions is similar. Restarting is the fastest, whereas replacing takes slightly longer. All three actions are simple and efficient methods for troubleshooting.
 
+> [!CAUTION]
+> Do not perform any action against management servers without first consulting with Microsoft support personnel. Doing so could affect the integrity of the Operator Nexus Cluster.
+
 ## Prerequisites
 
 - Familiarize yourself with the capabilities referenced in this article by reviewing the [BMM actions](howto-baremetal-functions.md).
 - Gather the following information:
   - Name of the resource group for the BMM
   - Name of the BMM that requires a lifecycle management operation
 
+> [!IMPORTANT]
+> Disruptive command requests against a Kubernetes Control Plane (KCP) node are rejected if there is another disruptive action command already running against another KCP node or if the full KCP is not available.
+>
+> Restart, reimage and replace are all considered disruptive actions.
+>
+> This check is done to maintain the integrity of the Nexus instance and ensure multiple KCP nodes don't go down at once due to simultaneous disruptive actions. If multiple nodes go down, it will break the healthy quorum threshold of the Kubernetes Control Plane.
+
 ## Identify the corrective action
 
-When you're troubleshooting a BMM for failures and determining the best corrective action, it's important to understand the available options. Restarting or reimaging a BMM can be an efficient and effective way to fix problems or simply restore the software to a known-good place. This article provides direction on the best practices for each of the three Rs.
+When you're troubleshooting a BMM for failures and determining the best corrective action, it's important to understand the available options. Restarting or reimaging a BMM can be an efficient and effective way to fix problems or restore the software to a known-good place. Replacing a BMM might be required when one or more hardware components fail on the server. This article provides direction on the best practices for each of the three actions.
 
-Troubleshooting technical problems requires a systematic approach. One effective method is to start with the simplest and least invasive solution and work your way up to more complex and drastic measures, if necessary.
+Troubleshooting technical problems requires a systematic approach. One effective method is to start with the least invasive solution and works your way up to more complex and drastic measures, if necessary.
 
 The first step in troubleshooting is often to try restarting the device or system. Restarting can help to clear any temporary glitches or errors that might be causing the problem. If restarting doesn't solve the problem, the next step might be to try reimaging the device or system.
 
@@ -42,7 +52,7 @@ The restart typically is the starting point for mitigating a problem.
 
 ## Troubleshoot with a reimage action
 
-Reimaging a BMM is a process that you use to redeploy the image on the OS disk, without impact to the tenant data. This action executes the steps to rejoin the cluster with the same identifiers.
+Reimaging a BMM is a process that you use to redeploy the image on the OS disk, without affecting the tenant data. This action executes the steps to rejoin the cluster with the same identifiers.
 
 The reimage action can be useful for troubleshooting problems by restoring the OS to a known-good working state. Common causes that can be resolved through reimaging include recovery due to doubt of host integrity, suspected or confirmed security compromise, or "break glass" write activity.
 
@@ -54,14 +64,14 @@ Servers contain many physical components that can fail over time. It's important
 
 A hardware validation process is invoked to ensure the integrity of the physical host in advance of deploying the OS image. Like the reimage action, the tenant data isn't modified during replacement.
 
-As a best practice, cordon off and shut down the BMM in advance of physical repairs. When you're performing the following physical repair, a replace action isn't required because the BMM host will continue to function normally after the repair:  
+As a best practice, first issue a `cordon` command to remove the bare metal machine from workload scheduling and then shut down the BMM in advance of physical repairs.
 
-- Hot swappable power supply
+When you're performing a physical hot swappable power supply repair, a replace action isn't required because the BMM host will continue to function normally after the repair.
 
 When you're performing the following physical repairs, we recommend a replace action, though it isn't necessary to bring the BMM back into service:
 
 - CPU
-- DIMM
+- Dual In-Line Memory Module (DIMM)
 - Fan
 - Expansion board riser
 - Transceiver
@@ -73,7 +83,7 @@ When you're performing the following physical repairs, a replace action is requi
 - System board
 - SSD disk
 - PERC/RAID adapter
-- Mellanox NIC
+- Mellanox Network Interface Card (NIC)
 - Broadcom embedded NIC
 
 ## Summary