Merge pull request #290899 from v-saambe/main

Create Troubleshoot LACP Bonding

Author: v-dirichards

articles/operator-nexus/TOC.yml

@@ -39,19 +39,19 @@
         - name: Route Policy
           expanded: false
           items:
-          - name: Route Policy Overview
-            href: concepts-nexus-route-policies-overview.md
-          - name: IP Prefixes Overview
-            href: concepts-nexus-ip-prefix.md
+            - name: Route Policy Overview
+              href: concepts-nexus-route-policies-overview.md
+            - name: IP Prefixes Overview
+              href: concepts-nexus-ip-prefix.md
         - name: Isolation Domains
           expanded: false
           items:
-          - name: Isolation Domains overview
-            href: concepts-isolation-domain.md
-          - name: Isolation Domain configuration
-            href: reference-isolation-domain-configuration.md
-          - name: Technical requirements for Isolation Domains
-            href: reference-isolation-domain-technical-requirements.md
+            - name: Isolation Domains overview
+              href: concepts-isolation-domain.md
+            - name: Isolation Domain configuration
+              href: reference-isolation-domain-configuration.md
+            - name: Technical requirements for Isolation Domains
+              href: reference-isolation-domain-technical-requirements.md
         - name: Access Control Lists
           href: concepts-access-control-lists.md
     - name: Nexus Kubernetes Cluster
@@ -157,60 +157,61 @@
     - name: Network Fabric
       expanded: false
       items:
-          - name: Isolation Domain
-            href: howto-configure-isolation-domain.md
-          - name: Isolation Domain Configuration Examples
-            href: reference-isolation-domain-configuration-examples.md
-          - name: Network Fabric Route Policy
-            href: how-to-route-policy.md
-          - name: IP Prefixes
-            href: how-to-ip-prefixes.md
-          - name: Network Packet Broker
-            href: howto-configure-network-packet-broker.md
-          - name: Validate cables for Nexus Network Fabric
-            href: how-to-validate-cables.md
-          - name: Creating Access Control Lists (ACLs)
-            href: howto-create-access-control-list-for-network-to-network-interconnects.md
-          - name: Apply ACLs to Network-to-Network Interconnects (NNI)
-            href: howto-apply-access-control-list-to-network-to-network-interconnects.md
-          - name: Updating ACL on Network-for-Network Interconnects (NNI)
-            href: howto-update-access-control-list-for-network-to-network-interconnects.md
-          - name: Delete ACLs associated with Network-to-Network Interconnects (NNI)
-            href: howto-delete-access-control-list-network-to-network-interconnect.md
-          - name: How to Configure Diagnostic Settings and Monitor Configuration Differences in Nexus Network Fabric
-            href: howto-configure-diagnostic-settings-monitor-configuration-differences.md
-          - name: How to Delete L3 Isolation Domains in Azure Nexus Network Fabric
-            href: howto-delete-layer-3-isolation-domains.md
-          - name: How to monitor interface In and Out packet rate for network fabric devices
-            href: howto-monitor-interface-packet-rate.md
-          - name: How to Delete L3 Isolation Domains in Azure Nexus Network Fabric
-            href: howto-delete-layer-3-isolation-domains.md
-          - name: Cross-subscription deployments and required RBAC for Network Fabric
-            href: concepts-cross-subscription-deployments-required-rbac-for-network-fabric.md
-          - name: How to replace network devices in Azure Operator Nexus Network Fabric
-            href: howto-replace-network-devices.md
-          - name: How to put a device into maintenance mode
-            href: howto-put-device-in-maintenance-mode.md
-          - name: How to upgrade Network Fabric
-            href: howto-upgrade-nexus-fabric.md
+        - name: Isolation Domain
+          href: howto-configure-isolation-domain.md
+        - name: Isolation Domain Configuration Examples
+          href: reference-isolation-domain-configuration-examples.md
+        - name: Network Fabric Route Policy
+          href: how-to-route-policy.md
+        - name: IP Prefixes
+          href: how-to-ip-prefixes.md
+        - name: Network Packet Broker
+          href: howto-configure-network-packet-broker.md
+        - name: Validate cables for Nexus Network Fabric
+          href: how-to-validate-cables.md
+        - name: Creating Access Control Lists (ACLs)
+          href: howto-create-access-control-list-for-network-to-network-interconnects.md
+        - name: Apply ACLs to Network-to-Network Interconnects (NNI)
+          href: howto-apply-access-control-list-to-network-to-network-interconnects.md
+        - name: Updating ACL on Network-for-Network Interconnects (NNI)
+          href: howto-update-access-control-list-for-network-to-network-interconnects.md
+        - name: Delete ACLs associated with Network-to-Network Interconnects (NNI)
+          href: howto-delete-access-control-list-network-to-network-interconnect.md
+        - name: How to Configure Diagnostic Settings and Monitor Configuration Differences
+            in Nexus Network Fabric
+          href: howto-configure-diagnostic-settings-monitor-configuration-differences.md
+        - name: How to Delete L3 Isolation Domains in Azure Nexus Network Fabric
+          href: howto-delete-layer-3-isolation-domains.md
+        - name: How to monitor interface In and Out packet rate for network fabric devices
+          href: howto-monitor-interface-packet-rate.md
+        - name: How to Delete L3 Isolation Domains in Azure Nexus Network Fabric
+          href: howto-delete-layer-3-isolation-domains.md
+        - name: Cross-subscription deployments and required RBAC for Network Fabric
+          href: concepts-cross-subscription-deployments-required-rbac-for-network-fabric.md
+        - name: How to replace network devices in Azure Operator Nexus Network Fabric
+          href: howto-replace-network-devices.md
+        - name: How to put a device into maintenance mode
+          href: howto-put-device-in-maintenance-mode.md
+        - name: How to upgrade Network Fabric
+          href: howto-upgrade-nexus-fabric.md
     - name: Cluster
       expanded: false
       items:
         - name: BareMetal Actions
           expanded: false
           items:
-              - name: BareMetal BMM Access Setup
-                href: howto-baremetal-bmm-ssh.md
-              - name: BareMetal BMC Access Setup
-                href: howto-baremetal-bmc-ssh.md
-              - name: BareMetal Functions
-                href: howto-baremetal-functions.md
-              - name: BareMetal Run-Read Execution
-                href: howto-baremetal-run-read.md
-              - name: BareMetal Run-Data-Extract Execution
-                href: howto-baremetal-run-data-extract.md
-              - name: Running BareMetal actions directly with nexusctl
-                href: howto-baremetal-nexusctl.md
+            - name: BareMetal BMM Access Setup
+              href: howto-baremetal-bmm-ssh.md
+            - name: BareMetal BMC Access Setup
+              href: howto-baremetal-bmc-ssh.md
+            - name: BareMetal Functions
+              href: howto-baremetal-functions.md
+            - name: BareMetal Run-Read Execution
+              href: howto-baremetal-run-read.md
+            - name: BareMetal Run-Data-Extract Execution
+              href: howto-baremetal-run-data-extract.md
+            - name: Running BareMetal actions directly with nexusctl
+              href: howto-baremetal-nexusctl.md
     - name: Nexus Kubernetes cluster
       expanded: false
       items:
@@ -308,9 +309,11 @@
           href: troubleshoot-accepted-cluster-hydration.md
         - name: Troubleshoot Out of Memory Pods
           href: troubleshoot-memory-limits.md
+        - name: Troubleshoot LACP Bonding
+          href: troubleshoot-lacp-bonding.md
     - name: Storage Array
       expanded: false
-      items:
+      items: null
     - name: Tenant Workload
       expanded: false
       items:
@@ -327,7 +330,8 @@
           href: troubleshoot-internet-host-virtual-machine.md
         - name: Troubleshoot VM errors after BMM restart
           href: troubleshoot-vm-error-after-reboot.md
-        - name: Troubleshooting dual-stack configuration issues for Nexus Kubernetes cluster
+        - name: Troubleshooting dual-stack configuration issues for Nexus Kubernetes
+            cluster
           href: troubleshoot-kubernetes-cluster-dual-stack-configuration.md
 - name: FAQ
   href: azure-operator-nexus-faq.md

articles/operator-nexus/troubleshoot-lacp-bonding.md

@@ -0,0 +1,52 @@
+---
+title: "Azure Operator Nexus: Networking"
+description: Checking LACP Bonding on Physical Hosts.
+author: keithritchie73
+ms.author: keithritchie
+ms.service: azure-operator-nexus
+ms.custom: azure-operator-nexus
+ms.topic: troubleshooting
+ms.date: 11/15/2024
+---
+
+# Checking LACP Bonding on Physical Hosts
+
+On physical host startup, the two Mellanox cards are LACP bonded to a pair of Arista switches. If LACP isn't properly negotiated between the server's cards and the switches, it can cause strange packet loss or load balancing behavior. These errors might not be noticeable until a tenant workload attempts to pass traffic and is due to the hashing/load balancing nature of LACP.
+
+## Diagnosis
+
+If, LACP isn't negotiated correctly traffic loss can occur. But traffic can pass for some flows too. This behavior can manifest itself as a vm that can't get on the network, or even oam/storage outages.
+
+## Checking LACP Bonding
+
+To check the LACP bonding status on a physical host run the following command. For control plane hosts, use file 8a_pf_bond as there's only one Mellanox card on those hosts. For worker hosts, use either 4b_pf_bond or 98_pf_bond to check its two cards.
+
+```bash
+# cat /proc/net/bonding/8a_pf_bond
+```
+
+### Interpreting the results
+
+Key validations to check in the /proc/net/bonding/ output are:
+
+For Bond level (the top part):
+
+1. MII Status: up - Is the entire bond up
+2. LACP active: on - Is LACP active
+3. Aggregator ID: 1 - The top level aggregator ID should match both replicas. See each port for its aggregator ID.
+4. System MAC address: 42:56:86:9c:81:89 - Is there a System MAC defined. If a bond isn't negotiated this will be undefined or all zeros, e.g 00:00:00:00:00:00
+
+For each port:
+
+1. MII Status: up - Is the interface up
+2. Aggregator ID: 1 - Both replicas should have the same aggregator ID
+3. details partner lacp pdu: port state 61 - The value is a bit mask that represents the LACP negotiation state on that port. Generally 61 and 63 are what we want. [See](https://movingpackets.net/2017/10/17/decoding-lacp-port-state)
+
+### Fixing the issue
+
+The most common causes for these LACP issues are host/switch miswiring or mismatched LACP/MLAG configuration on the Arista switches. Investigate the situation by tracing out and repairing any wiring issues. If the wiring is correct, then determine if the switch LACP/MLAG configuration is incorrect.
+
+## Further information
+
+If you still have questions, [contact support](https://portal.azure.com/?#blade/Microsoft_Azure_Support/HelpAndSupportBlade).
+For more information about Support plans, see [Azure Support plans](https://azure.microsoft.com/support/plans/response/).