Merge pull request #18231 from sethmanheim/akstsg-sk

Add AKS Arc connectivity TSG

Author: sethmanheim

AKS-Arc/TOC.yml

@@ -187,6 +187,8 @@
       href: network-validation-errors.md
     - name: Network validation error due to .local domain
       href: network-validation-error-local.md
+    - name: BGP with FRR not working
+      href: connectivity-troubleshoot.md
   - name: Reference
     items:
     - name: Azure CLI

AKS-Arc/aks-troubleshoot.md

@@ -3,7 +3,7 @@ title: Troubleshoot common issues in AKS enabled by Azure Arc
 description: Learn about common issues and workarounds in AKS enabled by Arc.
 ms.topic: how-to
 author: sethmanheim
-ms.date: 04/30/2025
+ms.date: 06/18/2025
 ms.author: sethm 
 ms.lastreviewed: 04/01/2025
 ms.reviewer: abha
@@ -36,13 +36,15 @@ The following sections describe known issues for AKS enabled by Azure Arc:
 
 | AKS Arc operation | Issue |
 |------------------------|-------|
+| General network validation errors | [Troubleshoot network validation errors](network-validation-errors.md) |
 | Create validation      | [Control plane configuration validation errors](control-plane-validation-errors.md) |
 | Create validation      | [K8sVersionValidation error](cluster-k8s-version.md) |
 | Create validation      | [KubeAPIServer unreachable error](kube-api-server-unreachable.md) |
 | Network configuration issues | [Use diagnostic checker](aks-arc-diagnostic-checker.md) |
 | Kubernetes steady state   | [Resolve issues due to out-of-band deletion of storage volumes](delete-storage-volume.md) |
 | Release validation     | [Azure Advisor upgrade recommendation message](azure-advisor-upgrade.md) |
 | Network validation | [Network validation error due to .local domain](network-validation-error-local.md) |
+| BGP with FRR not working | [Troubleshoot BGP with FRR in AKS Arc environments](connectivity-troubleshoot.md) |
 
 ## Next steps
 

AKS-Arc/connectivity-troubleshoot.md

@@ -0,0 +1,82 @@
+---
+title: Troubleshoot BGP with FRR in AKS Arc environments
+description: Learn how to troubleshoot BGP connectivity issues when using MetalLB with FRR in AKS Arc deployments.
+author: sethmanheim
+ms.date: 06/19/2025
+ms.author: sethm
+ms.topic: troubleshooting
+ms.reviewer: srikantsarwa
+ms.lastreviewed: 06/19/2025
+
+---
+
+# BGP with FRR not working in AKS Arc environment
+
+This article helps you identify and resolve Border Gateway Protocol (BGP) connectivity issues when using MetalLB with Free Range Routing (FRR) in Azure Kubernetes Service (AKS) Arc environments.
+
+Use this guidance when BGP sessions fail to establish, external IP routing doesn't work correctly, or network connectivity to exposed services becomes unreliable in your AKS Arc deployment.
+
+## Symptoms
+
+In environments using MetalLB with FRR for BGP peering, you might experience the following issues:
+
+- BGP sessions are not established or keep flapping, a condition where the BGP session repeatedly goes up and down, causing route instability. This behavior can be due to network issues, misconfigurations, or hardware problems. It can result in degraded performance or loss of service availability.
+- Services of type `LoadBalancer` don't receive properly routed external IPs.
+- Advertised routes are missing or not propagated to upstream routers.
+- Network connectivity to exposed services is inconsistent or unavailable.
+
+These symptoms are often observed in specific hardware environments such as Hyper-Converged Infrastructure (HCI) or where strict network/security policies are enforced.
+
+## Mitigation
+
+If you encounter these issues with FRR, you can temporarily disable it using Azure CLI:
+
+```azurecli
+# Retrieve the object ID for the managed identity
+$objID = az ad sp list --filter "appId eq '087fca6e-4606-4d41-b3f6-5ebdf75b8b4c'" --query "[].id" --output tsv
+
+# Update the arcnetworking extension to disable FRR
+az k8s-extension update \
+  --cluster-name $clusterName \
+  -g $rgName \
+  --cluster-type connectedClusters \
+  --extension-type microsoft.arcnetworking \
+  --config "k8sRuntimeFpaObjectId=$objID" \
+  --config "metallb.speaker.frr.enabled=false" \
+  -n arcnetworking
+```
+
+## Troubleshooting steps
+
+Use the following steps to diagnose and resolve BGP issues with MetalLB and FRR in your AKS Arc environment.
+
+### Check BGP configuration
+
+```azurecli
+kubectl get ipaddresspools -A -o yaml
+kubectl get bgppeers.metallb.io -A -o yaml
+kubectl get bgpadvertisements -A -o yaml
+```
+
+### Collect logs from MetalLB speaker (FRR)
+
+```azurecli
+# Get the list of MetalLB speaker pods
+kubectl get pods -n kube-system
+
+# Speaker container logs
+kubectl logs -n kube-system arcnetworking-metallb-speaker-xxxxx -c speaker
+
+# FRR container logs
+kubectl logs -n kube-system arcnetworking-metallb-speaker-xxxxx -c frr
+```
+
+### Review TOR switch configuration
+
+- Configuration and logs from the top-of-rack (TOR) switch or upstream router might be necessary.
+- These logs are hardware/vendor-specific and not covered in this guide.
+
+## Next steps
+
+[Official MetalLB troubleshooting guide](https://metallb.universe.tf/troubleshooting/#with-frr)
+