Merge pull request #302166 from g0r1v3r4/add-cluster-resourcehealth-tsg

Stacyrch140 · web-flow · commit 19469f916e8c · 2025-07-03T18:20:53.000-04:00
Add Cluster Connection Status Resource Health TSG
diff --git a/articles/operator-nexus/TOC.yml b/articles/operator-nexus/TOC.yml
@@ -249,7 +249,7 @@
           href: howto-restrict-serial-port-access-and-set-timeout-on-terminal-server.md
         - name: How to configure BGP prefix limit on Customer Edge (CE) devices for Azure Operator Nexus
           href: howto-configure-bgp-prefix-limit-on-customer-edge-devices.md
-        - name: BMP log streaming in Azure Operator Nexus Network Fabric 
+        - name: BMP log streaming in Azure Operator Nexus Network Fabric
           href: concepts-bmp-log-streaming.md
         - name: How to enable / disable BMP log streaming Azure Operator Nexus
           href: howto-enable-log-streaming.md
@@ -310,7 +310,6 @@
           href: howto-kubernetes-cluster-install-microsoft-defender.md
         - name: Kubernetes cluster features
           href: howto-kubernetes-cluster-features.md
-
     - name: Nexus Virtual Machine
       expanded: false
       items:
@@ -367,6 +366,11 @@
 - name: Troubleshooting
   expanded: true
   items:
+    - name: Resource Health
+      expanded: false
+      items:
+        - name: Troubleshoot Resource Health alerts
+          href: troubleshoot-resource-health-alerts.md
     - name: Network Fabric
       expanded: false
       items:
@@ -378,7 +382,16 @@
           href: troubleshoot-dns-issues.md
         - name: Troubleshoot TWAMP (UDP) not working
           href: troubleshoot-twamp-udp-not-working.md
-    - name: Cluster or BMM
+    - name: Cluster
+      expanded: false
+      items:
+        - name: Troubleshoot Accepted Cluster Resource
+          href: troubleshoot-accepted-cluster-hydration.md
+        - name: Troubleshoot Control Plane Quorum
+          href: troubleshoot-control-plane-quorum.md
+        - name: Troubleshoot Cluster heartbeat connection status disconnected
+          href: troubleshoot-cluster-heartbeat-connection-status-disconnected.md
+    - name: Bare Metal Machine
       expanded: false
       items:
         - name: Troubleshoot Bare Metal Server Problems
@@ -391,10 +404,6 @@
           href: troubleshoot-bare-metal-machine-degraded.md
         - name: Troubleshoot Warning status
           href: troubleshoot-bare-metal-machine-warning.md
-        - name: Troubleshoot Control Plane Quorum
-          href: troubleshoot-control-plane-quorum.md
-        - name: Troubleshoot Accepted Cluster Resource
-          href: troubleshoot-accepted-cluster-hydration.md
         - name: Troubleshoot Out of Memory Pods
           href: troubleshoot-memory-limits.md
     - name: Tenant Workload
diff --git a/articles/operator-nexus/includes/contact-support.md b/articles/operator-nexus/includes/contact-support.md
@@ -0,0 +1,20 @@
+---
+author: omarrivera
+ms.author: omarrivera
+ms.date: 07/02/2025
+ms.topic: include
+ms.service: azure-operator-nexus
+---
+
+## Still having issues?
+
+If the steps outlined didn't provide a path to resolve the issue or if you still have questions [contact support].
+Please, provide as much detail as possible about the issue you're experiencing, including any error messages or logs that may be relevant.
+This will help the support team to assist you more effectively.
+
+You can open a support request through the [Azure portal][contact support].
+
+For more information about support plans, see [Azure Support plans].
+
+[contact support]: https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade
+[Azure Support plans]: https://azure.microsoft.com/support/plans/response/
diff --git a/articles/operator-nexus/media/troubleshoot-cluster-heartbeat-connection-status/azure-portal-cluster-connection-status.png b/articles/operator-nexus/media/troubleshoot-cluster-heartbeat-connection-status/azure-portal-cluster-connection-status.png
diff --git a/articles/operator-nexus/media/troubleshoot-cluster-heartbeat-connection-status/cluster-connection-status-components-for-metric.png b/articles/operator-nexus/media/troubleshoot-cluster-heartbeat-connection-status/cluster-connection-status-components-for-metric.png
diff --git a/articles/operator-nexus/troubleshoot-cluster-heartbeat-connection-status-disconnected.md b/articles/operator-nexus/troubleshoot-cluster-heartbeat-connection-status-disconnected.md
@@ -0,0 +1,147 @@
+---
+title: Troubleshoot Azure Operator Nexus Cluster Heartbeat Connection Status shows Disconnected
+description: Provide steps to investigate and possibly resolve circumstances that are preventing the Cluster from sending heartbeats to the Cluster Manager.
+ms.service: azure-operator-nexus
+ms.custom: troubleshooting
+ms.topic: troubleshooting
+ms.date: 07/02/2025
+ms.author: omarrivera
+author: omarrivera
+---
+
+# Troubleshoot Cluster heartbeat connection status shows disconnected
+
+This guide describes steps to troubleshoot a Cluster with a `ClusterConnectionStatus` in `Disconnected` state.
+For a Cluster, the `ClusterConnectionStatus` represents the stability in the connection between the on-premises Cluster and its ability to reach the Cluster Manager.
+
+> [!IMPORTANT]
+> The `ClusterConnectionStatus` **doesn't** represent nor is it related to the health or connectivity of the Arc Connected Kubernetes Cluster.
+> The `ClusterConnectionStatus` indicates that the Cluster is successful in sending heartbeats and receiving acknowledgment from the Cluster Manager.
+
+[!include[prereqAzCLI](./includes/baremetal-machines/prerequisites-azure-cli-bare-metal-machine-actions.md)]
+
+## Understanding the Cluster connection status signal
+
+The `ClusterConnectionStatus` represents the ability of the on-premises Cluster to send heartbeats and receive acknowledgments from the Cluster Manager, indicating the health of the network connection between them.
+`ClusterConnectionStatus` is distinct from the connectivity of the Arc Connected Kubernetes Cluster, though network issues affect both.
+
+A Cluster resource has the property `ClusterConnectionStatus` set to the value `Connected` if the heartbeats are continuously received and acknowledged.
+The `ClusterConnectionStatus` becomes `Connected` once the Cluster is in a healthy state and network connectivity issues are resolved.
+The Cluster shows `Timeout` only as a transitional state between `Connected` and `Disconnected`.
+The Cluster `ClusterConnectionStatus` value becomes `Disconnected` if the Cluster Manager detects continuously missed heartbeats.
+Heartbeats are considered missed if they aren't received within or beyond the specified time thresholds.
+Once the Cluster is a healthy state and there no network connectivity issues, the `ClusterConnectionStatus` automatically moves to `Connected`
+
+During the Cluster deployment process, the Cluster is in an `Undefined` state until the Cluster is fully deployed and operational.
+
+The following table shows the possible values of `ClusterConnectionStatus` and their definitions:
+
+| Status         | Definition                                                                                                            |
+|----------------|-----------------------------------------------------------------------------------------------------------------------|
+| `Connected`    | Heartbeats received, indicates healthy Cluster and Cluster Manager connectivity                                       |
+| `Disconnected` | Heartbeats missed for **over 5 minutes**, indicates likely connectivity issue between Cluster Manager and Cluster     |
+| `Timeout`      | Heartbeats missed for **over 2 minutes but less than 5 minutes**, Cluster connectivity is uncertain possibly degraded |
+| `Undefined`    | Cluster not yet deployed or running a version without the heartbeats feature                                          |
+
+## Check the value of the Cluster's ClusterConnectionStatus property
+
+The value of `ClusterConnectionStatus` is visible in the Azure portal in the Cluster resource view.
+
+:::image type="content" source="media/troubleshoot-cluster-heartbeat-connection-status/azure-portal-cluster-connection-status.png" alt-text="Screenshot of ClusterConnectionStatus property as shown in the Azure portal." lightbox="media/troubleshoot-cluster-heartbeat-connection-status/azure-portal-cluster-connection-status.png":::
+
+Or, you can use the Azure CLI to see the value of `ClusterConnectionStatus`:
+
+```azurecli
+az networkcloud cluster show \
+  -g "$CLUSTER_RG" \
+  -n "$CLUSTER_NAME" \
+  --subscription "$SUBSCRIPTION_ID" \
+  --query "{ClusterConnectionStatus:clusterConnectionStatus}" \
+  --output table
+
+ClusterConnectionStatus
+-------------------------
+Connected
+```
+
+## Understanding the NexusClusterConnectionStatus metric
+
+Use Azure Resource Health to build alerts for Cluster health, as it provides a comprehensive and supported view of resource status.
+The `NexusClusterConnectionStatus` metric integrates into the Cluster's Azure Resource Health.
+If you use the `NexusClusterConnectionStatus` metric directly, understand how it functions and what it represents.
+
+The Cluster Manager, not the on-premises Cluster, emits the metric based on the `ClusterConnectionStatus` property.
+A pod running on the on-premises Cluster sends heartbeat message to the Cluster Manager through the infrastructure proxy.
+The metric emits a value of "1" for all time series. Starting from when the Cluster resource's connectionStatus is set for the first time.
+The metric emitting process never sends "0" values. Any "0" values seen in graphs are due to graphing tools filling gaps.
+The detection of state changes requires the Cluster Manager's reconciliation process to update the Cluster resource's `ClusterConnectionStatus` property accordingly.
+
+There might be a delay between the actual loss of heartbeats and the metric reflecting the `Disconnected` state, due to the reconciliation loop and other operational factors.
+The `NexusClusterConnectionStatus` metric is used as a health indicator for the Cluster, but delays in status changes can occur due to reconciliation timing and operational constraints.
+Timeout events can occur if heartbeats aren't received within a 2-minute threshold, but a single successful heartbeat resets the timer.
+The status can transition between Connected, Timeout, and `Disconnected` based on heartbeat activity.
+
+The image shows a general representation of the components responsible for emitting the `NexusClusterConnectionStatus` metric.
+
+:::image type="content" source="media/troubleshoot-Cluster-heartbeat-connection-status/cluster-connection-status-components-for-metric.png" alt-text="Diagram that shows the components responsible for emitting the NexusClusterConnectionStatus metric." lightbox="media/troubleshoot-Cluster-heartbeat-connection-status/cluster-connection-status-components-for-metric.png":::
+
+### ClusterConnectionStatus isn't the same as Arc Connected Cluster status
+
+The Cluster's `ClusterConnectionStatus` and Arc Connected Cluster status are separate signals and shouldn't be treated interchangeably.
+Although the two signals aren't related, both rely on network connectivity for the Cluster.
+It's possible for a Cluster to be Arc `Disconnected` but still have a Heartbeat Status of `Connected`.
+Both signals depend on network connectivity, but they serve different purposes and managed by different systems.
+
+## Common investigation steps
+
+Infrastructure networking issues, permission changes in the Managed Identity, or other issues that might not be obvious at first, affect the Cluster resource connection status.
+The following sections provide some common investigation steps and references to help troubleshoot.
+
+> [!IMPORTANT]
+> The `ClusterConnectionStatus` indicates general instability, not the root cause.
+> This guide provides general resource health checks that might help locate the problem or at least help collect information useful for customer support.
+
+### Cluster Network Fabric health and connectivity
+
+It's useful to start with the Network Fabric [controller][Network Fabric Controller] and [services][Network Fabric Services] resources.
+Verify the [network configuration][How to Configure Network Fabric] or any other network-related settings that might be affecting the connectivity.
+Verify the physical network setup including rack cabling, IP addresses, DNS settings, routing rules, firewall rules, etc.
+
+[How to Configure Network Fabric]: ./howto-configure-network-fabric.md
+[Network Fabric Controller]: ./concepts-network-fabric-controller.md
+[Network Fabric Services]: ./concepts-network-fabric-services.md
+
+Evaluate any configured monitoring or metrics for the Network Fabric resources.
+For more information, see the following links:
+
+- [Nexus Network Fabric configuration monitoring overview](./concepts-network-fabric-configuration-monitoring.md)
+- [How to configure diagnostic settings and monitor configuration differences in Nexus Network Fabric](./howto-configure-diagnostic-settings-monitor-configuration-differences.md)
+- [Azure Operator Nexus Network Fabric internal network BGP metrics](./concepts-internal-network-bgp-metrics.md)
+- [How to monitor interface In and Out packet rate for network fabric devices](./howto-monitor-interface-packet-rate.md)
+
+### Recent changes to the Managed Identity permissions
+
+Changes to the Managed Identity permissions for the Cluster Manager or Cluster can affect the Cluster's ability to authenticate against the Cluster Manager.
+The Managed Identities (MI) and their permissions are used for service-to-service authentication.
+A change in the permissions results in authentication failures for the heartbeat messages.
+Even when network connectivity is healthy the Cluster's `ClusterConnectionStatus` shows `Disconnected` when heartbeats aren't successfully received and acknowledged.
+
+### Check control-plane BareMetal Machines health
+
+The control-plane BareMetal Machines host the component that emits the heartbeats to the Cluster Manager.
+In most cases, the pods running on the control-plane reschedule automatically to a different BareMetal Machine within the control-plane node pool.
+However, if the BareMetal Machines aren't healthy, the pods can't reschedule and the Cluster is unable to send heartbeats.
+
+To check the BareMetal Machines, use the following command:
+
+```azurecli
+az networkcloud baremetalmachine list \
+  --resource-group "$CLUSTER_RG" \
+  --cluster-name "$CLUSTER_NAME" \
+  --subscription "$SUBSCRIPTION_ID" \
+  --output table
+```
+
+Review the status of the control-plane BareMetal Machines. If any are unhealthy or unavailable, investigate further or contact support.
+
+[!include[stillHavingIssues](./includes/contact-support.md)]
diff --git a/articles/operator-nexus/troubleshoot-control-plane-quorum.md b/articles/operator-nexus/troubleshoot-control-plane-quorum.md
@@ -1,10 +1,10 @@
 ---
-title: Troubleshoot control plane quorum loss
-description: Learn how to restore control plane quorum loss.
+title: Troubleshoot control plane quorum loss when multiple nodes are offline
+description: Learn how to restore control plane quorum loss when multiple nodes are offline.
 ms.topic: article
-ms.date: 01/18/2024
-author: matthewernst
+ms.date: 07/02/2025
 ms.author: matthewernst
+author: matternst7258
 ms.service: azure-operator-nexus
 ---
 
@@ -34,29 +34,29 @@ Follow the steps in this troubleshooting article when multiple control plane nod
    - Sign in to the identified server.
    - Ensure that the ironic-conductor service is present on this node by using `crictl ps -a |grep -i ironic-conductor`. Here's example output:
 
-        ~~~
-        testuser@<servername> [ ~ ]$ sudo crictl ps -a |grep -i ironic-conductor
-        <id>       <id>       6 hours ago       Running       ironic-conductor       0       <id>
-        ~~~
+```shell
+testuser@<servername> [ ~ ]$ sudo crictl ps -a |grep -i ironic-conductor
+<id>       <id>       6 hours ago       Running       ironic-conductor       0       <id>
+```
 
 1. Determine the integrated Dell remote access controller (iDRAC) IP of the server:
    - Run the command `az networkcloud cluster list -g <RG_Name>`.
    - The output of the command is JSON with the iDRAC IP.
 
-        ~~~
-        {
-                "bmcConnectionString": "redfish+https://xx.xx.xx.xx/redfish/v1/Systems/System.Embedded.1",
-                "bmcCredentials": {
-                  "username": "<username>"
-                },
-                "bmcMacAddress": "<bmcMacAddress>",
-                "bootMacAddress": "<bootMacAddress",
-                "machineDetails": "extraDetails",
-                "machineName": "<machineName>",
-                "rackSlot": <rackSlot>,
-                "serialNumber": "<serialNumber>"
-        },
-        ~~~
+```json
+{
+    "bmcConnectionString": "redfish+https://xx.xx.xx.xx/redfish/v1/Systems/System.Embedded.1",
+    "bmcCredentials": {
+      "username": "<username>"
+    },
+    "bmcMacAddress": "<bmcMacAddress>",
+    "bootMacAddress": "<bootMacAddress",
+    "machineDetails": "extraDetails",
+    "machineName": "<machineName>",
+    "rackSlot": "<rackSlot>",
+    "serialNumber": "<serialNumber>"
+},
+```
 
 1. Access the integrated iDRAC graphical user interface (GUI) by using the IP in your browser to shut down affected management servers.
 
diff --git a/articles/operator-nexus/troubleshoot-resource-health-alerts.md b/articles/operator-nexus/troubleshoot-resource-health-alerts.md
@@ -0,0 +1,38 @@
+---
+title: Troubleshoot Azure Operator Nexus resource health alerts
+titleSuffix: Azure Operator Nexus
+description: Find troubleshooting guides for platform-emitted resource health alerts.
+ms.service: azure-operator-nexus
+ms.custom: troubleshooting
+ms.topic: troubleshooting
+ms.date: 07/02/2025
+ms.author: omarrivera
+author: omarrivera
+---
+
+# Troubleshoot resource health alerts
+
+This guide provides a breakdown of the resource health alerts emitted by the Azure Operator Nexus platform.
+It includes a description of each alert and links to troubleshooting guides for each alert.
+
+Resource health alerts emitted by the platform to indicate the health of a particular resource.
+These alerts are generated based on the status of the resource and its dependencies.
+
+## Cluster
+
+| Resource Health Event Name                                                                             | Troubleshooting Guide                                                 |
+|--------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|
+| `1PExtensionsFailedInstall`                                                                            | [Requires to contact support](#please-contact-support) |
+| `ClusterHeartbeatConnectionStatusDisconnectedClusterManagerOperationsAreAffectedPossibleNetworkIssues` | [Troubleshoot Cluster heartbeat connection status shows disconnected] |
+| `ClusterHeartbeatConnectionStatusTimedoutPossiblePerformanceIssues`                                    | [Troubleshoot Cluster heartbeat connection status shows disconnected] |
+
+[Troubleshoot Cluster heartbeat connection status shows disconnected]: ./troubleshoot-cluster-heartbeat-connection-status-disconnected.md
+
+## Please contact support
+
+For some resource health alerts, troubleshooting guides are not available.
+If you encounter these alerts, it is recommended to [contact Azure support] for further assistance.
+
+[contact Azure support]: https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade
+
+[!include[stillHavingIssues](./includes/contact-support.md)]
