adding updates to the cluster heartbeat connection status

Author: g0r1v3r4

articles/operator-nexus/TOC.yml

@@ -310,7 +310,6 @@
           href: howto-kubernetes-cluster-install-microsoft-defender.md
         - name: Kubernetes cluster features
           href: howto-kubernetes-cluster-features.md
-
     - name: Nexus Virtual Machine
       expanded: false
       items:
@@ -378,7 +377,18 @@
           href: troubleshoot-dns-issues.md
         - name: Troubleshoot TWAMP (UDP) not working
           href: troubleshoot-twamp-udp-not-working.md
-    - name: Cluster or BMM
+    - name: Cluster
+      expanded: false
+      items:
+        - name: Troubleshoot Accepted Cluster Resource
+          href: troubleshoot-accepted-cluster-hydration.md
+        - name: Troubleshoot Control Plane Quorum
+          href: troubleshoot-control-plane-quorum.md
+        - name: Troubleshoot ETCD Cluster quorum loss and recovery
+          href: troubleshoot-etcd-cluster-possible-quorum-lost.md
+        - name: Troubleshoot Cluster heartbeat connection status disconnected
+          href: troubleshoot-cluster-heartbeat-connection-status-disconnected.md
+    - name: Bare Metal Machine
       expanded: false
       items:
         - name: Troubleshoot Bare Metal Server Problems
@@ -391,14 +401,8 @@
           href: troubleshoot-bare-metal-machine-degraded.md
         - name: Troubleshoot Warning status
           href: troubleshoot-bare-metal-machine-warning.md
-        - name: Troubleshoot Control Plane Quorum
-          href: troubleshoot-control-plane-quorum.md
-        - name: Troubleshoot Accepted Cluster Resource
-          href: troubleshoot-accepted-cluster-hydration.md
         - name: Troubleshoot Out of Memory Pods
           href: troubleshoot-memory-limits.md
-        - name: Troubleshoot Cluster heartbeat connection status disconnected
-          href: troubleshoot-cluster-heartbeat-connection-status-disconnected.md
         - name: Troubleshoot Bare Metal Machine in not ready state
           href: troubleshoot-bare-metal-machine-not-ready-state.md
     - name: Tenant Workload

articles/operator-nexus/includes/contact-support.md

@@ -1,10 +1,11 @@
 ---
 author: omarrivera
 ms.author: omarrivera
-ms.date: 10/09/2024
+ms.date: 04/28/2025
 ms.topic: include
 ms.service: azure-operator-nexus
 ---
+
 ## Still Having Issues?
 
 If the steps outlined didn't provide a path to resolve the issue or if you still have questions [contact support].

articles/operator-nexus/media/troubleshoot-cluster-heartbeat-connection-status/az-portal-cluster-connection-status.png

@@ -4,36 +4,38 @@ description: Provide steps to investigate and possibly resolve circumstances tha
 ms.service: azure-operator-nexus
 ms.custom: troubleshooting
 ms.topic: troubleshooting
-ms.date: 10/09/2024
+ms.date: 04/28/2025
 ms.author: omarrivera
 author: omarrivera
 ---
 # Troubleshoot Azure Operator Nexus Cluster Heartbeat Connection Status shows Disconnected
 
-This guide attempts to provide steps to troubleshoot a Cluster is shown to have `clusterConnectionStatus` with a value of `Disconnected`.
+This guide attempts to provide steps to troubleshoot a Cluster with a `clusterConnectionStatus` in `Disconnected` state.
+For a Cluster, the `ClusterConnectionStatus` represents the stability in the connection between the on-premises Cluster and its ability to reach the Cluster Manager.
+
+> [!IMPORTANT]
+> The `ClusterConnectionStatus` **doesn't** represent or is related to the health or connectivity of the Arc Connected Kubernetes Cluster.
+> The `ClusterConnectionStatus` indicates that the Cluster is successful in sending heartbeats and receiving acknowledgment from the Cluster Manager.
 
 > [!CAUTION]
-> The `ClusterConnectionStatus` is likely a symptom or signal and not the root cause and this guide will not be able to provide answers for all scenarios.
-> The focus and purpose of this guide is to provide common issues and signals that can be inspected to determine where the issue might be.
-## Understanding the Issue
+> The information the `ClusterConnectionStatus` provides is an indication of a symptom of instability, not the root cause.
+> This guide focuses on identifying basic signals and components that might help locate the problem but might not cover all scenarios.
 
-Cluster Managers ensure continuous Cluster network connectivity through a heartbeat agent running within the target Cluster.
-The cluster-heartbeat agent sends periodic HTTP messages to the Cluster Manager and expects an acknowledgment response as well.
-A Cluster has the property `ClusterConnectionStatus` which is set to the value `Connected` as the heartbeats are continuously received and acknowledged.
+[!include[prereq-az-cli](./includes/baremetal-machines/prerequisites-azure-cli-bare-metal-machine-actions.md)]
 
-The `ClusterConnectionStatus` becomes `Connected` once the cluster is in a healthy state and network connectivity issues are resolved.
-If the Cluster is expected to be healthy but the `ClusterConnectionStatus` remains in `Disconnected` state [contact support] after following the steps in this guide.
+## Understanding the ClusterConnectionStatus signal
 
-> [!IMPORTANT]
-> `ClusterConnectionStatus` is **not** the same as Arc Connected Kubernetes Clusters.
-The command can be used to see the value of `ClsuterConnectionStatus` and it is visible in Azure Portal in the Cluster resource's JSON view.
+The `ClusterConnectionStatus` represents the ability for the on-premises Cluster to successfully send heartbeats and receive acknowledgments from the Cluster Manager.
+The continuous heartbeat messages are meant to detect the network connection health between the on-premises Cluster and the corresponding Cluster Manager.
+The `ClusterConnectionStatus` **isn't** the same as the connectivity of the Arc Connected Kubernetes Cluster.
+If there's network related issues, it's possible that the Arc Connected Kubernetes Cluster might also be affected.
 
-```azurecli
-az networkcloud cluster show --subscription "$SUBSCRIPTION_ID" -g "$CLUSTER_RG" -n "$CLUSTER_NAME" --output table --query "{ClusterConnectionStatus:clusterConnectionStatus}"
-ClusterConnectionStatus
--------------------------
-Connected
-```
+A Cluster resource has the property `ClusterConnectionStatus` which is set to the value `Connected` as the heartbeats are continuously received and acknowledged.
+The `ClusterConnectionStatus` becomes `Connected` once the Cluster is in a healthy state and network connectivity issues are resolved.
+The Cluster shows `Timeout` only as a transitional state between `Connected` and `Disconnected`.
+The Cluster `ClusterConnectionStatus` value becomes `Disconnected` as Cluster Manager detects continuously missed heartbeats.
+
+During the Cluster deployment process, the Cluster is in `Undefined` state until the Cluster is fully deployed and operational.
 
 The following table shows which status is displayed depending on the state of the undercloud cluster:
 
@@ -42,7 +44,28 @@ The following table shows which status is displayed depending on the state of th
 | `Connected`    | Heartbeats received, indicates healthy cluster and cluster manager connectivity                                       |
 | `Disconnected` | Heartbeats missed for __over 5 minutes__, indicates likely connectivity issue between Cluster Manager and Cluster     |
 | `Timeout`      | Heartbeats missed for __over 2 minutes but less than 5 minutes__, cluster connectivity is uncertain possibly degraded |
-| `Undefined`    | Cluster not yet deployed or running a version without the heartbeats feature                                                      |
+| `Undefined`    | Cluster not yet deployed or running a version without the heartbeats feature                                          |
+
+## Check the ClusterConnectionStatus
+
+The value of `ClusterConnectionStatus` is visible in the Azure portal in the Cluster resource view.
+
+![!include[clusterConnectionStatus](./includes/cluster-connection-status.md)]
+
+Or, you can use the Azure CLI to see the value of `ClusterConnectionStatus`:
+
+```azurecli
+az networkcloud cluster show \
+  -g "$CLUSTER_RG" \
+  -n "$CLUSTER_NAME" \
+  --subscription "$SUBSCRIPTION_ID" \
+  --query "{ClusterConnectionStatus:clusterConnectionStatus}" \
+  --output table
+
+ClusterConnectionStatus
+-------------------------
+Connected
+```
 
 ## Basic Investigation Steps
 
@@ -55,6 +78,8 @@ TODO - what steps could be done here?
 - Are there recent changes to the Managed Identity permissions for the Cluster Manager or Cluster?
   - The Managed Identities (MI) and their permissions are used for service-to-service authentication. A change in the permissions results in authentication failures for the heartbeat messages. Cluster Managers must both receive and acknowledge heartbeats failure to do so will also result in a `ClusterConnectionStatus` of `Disconnected`.
 
+If the Cluster is expected to be healthy but the `ClusterConnectionStatus` remains in `Disconnected` state [contact support] after following the steps in this guide.
+
 [!include[stillHavingIssues](./includes/contact-support.md)]
 
 [contact support]: https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade