Merge pull request #3707 from MicrosoftDocs/main638859674168990301sync_temp

For protected branch, push strategy should use PR and merge to target branch method to work around git push error

Author: learn-build-service-prod[bot]

AKS-Arc/TOC.yml

@@ -62,6 +62,8 @@
         href: create-clusters-terraform.md
       - name: Azure Resource Manager template
         href: resource-manager-quickstart.md
+      - name: REST API
+        href: aks-create-clusters-api.md
     - name: Networking
       items:
       - name: Create logical networks
@@ -187,6 +189,8 @@
       href: network-validation-errors.md
     - name: Network validation error due to .local domain
       href: network-validation-error-local.md
+    - name: BGP with FRR not working
+      href: connectivity-troubleshoot.md
   - name: Reference
     items:
     - name: Azure CLI

AKS-Arc/aks-create-clusters-api.md

@@ -0,0 +1,159 @@
+---
+title: Create Kubernetes clusters using REST APIs
+description: Learn how to create Kubernetes clusters in Azure Local using REST API for the Hybrid Container Service.
+ms.topic: how-to
+author: rcheeran
+ms.date: 06/19/2025
+ms.author: rcheeran 
+ms.lastreviewed: 06/19/2025
+ms.reviewer: rjaini
+---
+
+# Create Kubernetes clusters using the REST API
+
+[!INCLUDE [hci-applies-to-23h2](includes/hci-applies-to-23h2.md)]
+
+This article describes how to create Kubernetes clusters on Azure Local using the REST API. The Azure resource type for [AKS Arc provisioned clusters](/azure/templates/microsoft.hybridcontainerservice/provisionedclusterinstances?pivots=deployment-language-arm-template) is **"Microsoft.HybridContainerService/provisionedClusterInstances"**. This resource is an extension of the [connected clusters](/azure/templates/microsoft.kubernetes/connectedclusters?pivots=deployment-language-arm-template) resource type, **"Microsoft.Kubernetes/connectedClusters"**. Due to this dependency, you must first create a connected cluster resource before creating an AKS Arc resource.
+
+## Before you begin
+
+Before you begin, make sure you have the following details from your on-premises infrastructure administrator:
+
+- **Azure subscription ID**: The Azure subscription ID that Azure Local uses for deployment and registration.
+- **Custom Location ID**: The Azure Resource Manager ID of the custom location. The custom location is configured during the Azure Local cluster deployment. Your infrastructure admin should give you the Resource Manager ID of the custom location. This parameter is required in order to create Kubernetes clusters. If the infrastructure admin provides a custom location name and resource group name, you can also get the Resource Manager ID using the following command:
+
+  ```azurecli
+  az customlocation show --name "<custom location name>" --resource-group <azure resource group> --query "id" -o tsv
+  ```
+  
+- **Network ID**: The Azure Resource Manager ID of the Azure Local logical network you created [following these steps](aks-networks.md). Your admin should give you the ID of the logical network. This parameter is required in order to create Kubernetes clusters. If you know the resource group in which the logical network was created, you can also get the Azure Resource Manager ID using the following command:
+
+  ```azurecli
+  az stack-hci-vm network lnet show --name "<lnet name>" --resource-group <azure resource group> --query "id" -o tsv
+  ```
+  
+- **Create an SSH key pair**: Create an SSH key pair in Azure and store the private key file for troubleshooting and log collection purposes. For detailed instructions, see [Create and store SSH keys with the Azure CLI](/azure/virtual-machines/ssh-keys-azure-cli), or with the [Azure portal](/azure/virtual-machines/ssh-keys-portal).
+- To connect to the Kubernetes cluster from anywhere, create a Microsoft Entra group and add members to it. All the members in the Microsoft Entra group have cluster administrator access to the cluster. Make sure to add yourself as a member to the Microsoft Entra group. If you don't add yourself, you can't access the Kubernetes cluster using **kubectl**. For more information about creating Microsoft Entra groups and adding users, see [Manage Microsoft Entra groups and group membership](/entra/fundamentals/how-to-manage-groups).
+
+## Step 1: Create a connected cluster resource
+
+See the API definition for [connected clusters](/rest/api/hybridkubernetes/connected-cluster/create) and create a **PUT** request with the `kind` property set to `ProvisionedCluster`. The following example is a sample **PUT** request to create a connected cluster resource using the REST API:
+
+```http
+PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Kubernetes/connectedClusters/{connectedClusterName}?api-version=2024-01-01
+Content-Type: application/json
+Authorization: Bearer <access_token>
+
+{
+    "location": "<region>",
+    "identity": {
+        "type": "SystemAssigned"
+    },
+    "kind": "ProvisionedCluster",
+    "properties": {
+        "agentPublicKeyCertificate": "",
+        "azureHybridBenefit": "NotApplicable",
+        "distribution": "AKS",
+        "distributionVersion": "1.0",
+        "aadProfile": {
+          "enableAzureRBAC": true,
+          "adminGroupObjectIDs": [
+            "<entra-group-id>"
+          ],
+          "tenantID": "<tenant-id>"
+    },
+  }
+}
+```
+
+Replace all placeholder values with your actual details. For more information, see the [connected clusters API documentation](/rest/api/hybridkubernetes/connected-cluster/create).
+
+## Step 2: Create a provisioned cluster resource
+
+See the API definition for [provisioned clusters](/rest/api/hybridcontainer/provisioned-cluster-instances/create-or-update). In this **PUT** call, pass the Azure Resource Manager identifier created in the previous step as the URI parameter. The following code is an example HTTP **PUT** request to create a provisioned cluster resource with only the required parameters:
+
+```http
+PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.HybridContainerService/provisionedClusterInstances/{clusterName}?api-version=2024-01-01-preview
+Content-Type: application/json
+Authorization: Bearer <access_token>
+
+{
+  "extendedLocation": {
+    "type": "CustomLocation",
+    "name": "<ARM ID of Custom Location>"
+  },
+  "properties": {
+    "controlPlane": {
+      "count": 1,
+      "vmSize": "Standard_A4_v2"
+    },
+    "agentPoolProfiles": [
+      {
+        "name": "default-nodepool-1",
+        "count": 1,
+        "vmSize": "Standard_A4_v2",
+        "osType": "Linux",
+      }
+    ],
+    "linuxProfile": {
+      "ssh": {
+        "publicKeys": [
+          {
+            "keyData": "<SSH public key>"
+          }
+        ]
+      }
+    },
+    "cloudProviderProfile": {
+      "infraNetworkProfile": {
+        "vnetSubnetIds": [
+          "<ARM ID of logical network>"
+        ]
+      }
+    },
+  }
+}
+
+```
+
+Replace the placeholder values with your actual details. For more information, see the [provisioned clusters API documentation](/rest/api/hybridcontainer/provisioned-cluster-instances/create-or-update).
+
+## Connect to the Kubernetes cluster
+
+Now you can connect to your Kubernetes cluster by running the `az connectedk8s proxy` command from your development machine. Make sure you sign in to Azure before running this command. If you have multiple Azure subscriptions, select the appropriate subscription ID using the [az account set](/cli/azure/account#az-account-set) command.
+
+This command downloads the **kubeconfig** of your Kubernetes cluster to your development machine and opens a proxy connection channel to your on-premises Kubernetes cluster. The channel is open for as long as the command runs. Let this command run for as long as you want to access your cluster. If it times out, close the CLI window, open a fresh one, and then run the command again.
+
+You must have Contributor permissions on the resource group that hosts the Kubernetes cluster in order to successfully run the following command:
+
+```azurecli
+az connectedk8s proxy --name $aksclustername --resource-group $resource_group --file .\aks-arc-kube-config
+```
+
+Expected output:
+
+```output
+Proxy is listening on port 47011
+Merged "aks-workload" as current context in .\\aks-arc-kube-config
+Start sending kubectl requests on 'aks-workload' context using
+kubeconfig at .\\aks-arc-kube-config
+Press Ctrl+C to close proxy.
+```
+
+Keep this session running and connect to your Kubernetes cluster from a different terminal or command prompt. Verify that you can connect to your Kubernetes cluster by running the `kubectl get` command. This command returns a list of the cluster nodes:
+
+```azurecli
+kubectl get node -A --kubeconfig .\aks-arc-kube-config
+```
+
+The following example output shows the node you created in the previous steps. Make sure the node status is **Ready**:
+
+```output
+NAME             STATUS ROLES                AGE VERSION
+moc-l0ttdmaioew  Ready  control-plane,master 34m v1.24.11
+moc-ls38tngowsl  Ready  <none>               32m v1.24.11
+```
+
+## Next steps
+
+[AKS Arc overview](overview.md)

AKS-Arc/aks-troubleshoot.md

@@ -3,7 +3,7 @@ title: Troubleshoot common issues in AKS enabled by Azure Arc
 description: Learn about common issues and workarounds in AKS enabled by Arc.
 ms.topic: how-to
 author: sethmanheim
-ms.date: 04/30/2025
+ms.date: 06/18/2025
 ms.author: sethm 
 ms.lastreviewed: 04/01/2025
 ms.reviewer: abha
@@ -36,13 +36,15 @@ The following sections describe known issues for AKS enabled by Azure Arc:
 
 | AKS Arc operation | Issue |
 |------------------------|-------|
+| General network validation errors | [Troubleshoot network validation errors](network-validation-errors.md) |
 | Create validation      | [Control plane configuration validation errors](control-plane-validation-errors.md) |
 | Create validation      | [K8sVersionValidation error](cluster-k8s-version.md) |
 | Create validation      | [KubeAPIServer unreachable error](kube-api-server-unreachable.md) |
 | Network configuration issues | [Use diagnostic checker](aks-arc-diagnostic-checker.md) |
 | Kubernetes steady state   | [Resolve issues due to out-of-band deletion of storage volumes](delete-storage-volume.md) |
 | Release validation     | [Azure Advisor upgrade recommendation message](azure-advisor-upgrade.md) |
 | Network validation | [Network validation error due to .local domain](network-validation-error-local.md) |
+| BGP with FRR not working | [Troubleshoot BGP with FRR in AKS Arc environments](connectivity-troubleshoot.md) |
 
 ## Next steps
 

AKS-Arc/connectivity-troubleshoot.md

@@ -0,0 +1,82 @@
+---
+title: Troubleshoot BGP with FRR in AKS Arc environments
+description: Learn how to troubleshoot BGP connectivity issues when using MetalLB with FRR in AKS Arc deployments.
+author: sethmanheim
+ms.date: 06/19/2025
+ms.author: sethm
+ms.topic: troubleshooting
+ms.reviewer: srikantsarwa
+ms.lastreviewed: 06/19/2025
+
+---
+
+# BGP with FRR not working in AKS Arc environment
+
+This article helps you identify and resolve Border Gateway Protocol (BGP) connectivity issues when using MetalLB with Free Range Routing (FRR) in Azure Kubernetes Service (AKS) Arc environments.
+
+Use this guidance when BGP sessions fail to establish, external IP routing doesn't work correctly, or network connectivity to exposed services becomes unreliable in your AKS Arc deployment.
+
+## Symptoms
+
+In environments using MetalLB with FRR for BGP peering, you might experience the following issues:
+
+- BGP sessions are not established or keep flapping, a condition where the BGP session repeatedly goes up and down, causing route instability. This behavior can be due to network issues, misconfigurations, or hardware problems. It can result in degraded performance or loss of service availability.
+- Services of type `LoadBalancer` don't receive properly routed external IPs.
+- Advertised routes are missing or not propagated to upstream routers.
+- Network connectivity to exposed services is inconsistent or unavailable.
+
+These symptoms are often observed in specific hardware environments such as Hyper-Converged Infrastructure (HCI) or where strict network/security policies are enforced.
+
+## Mitigation
+
+If you encounter these issues with FRR, you can temporarily disable it using Azure CLI:
+
+```azurecli
+# Retrieve the object ID for the managed identity
+$objID = az ad sp list --filter "appId eq '087fca6e-4606-4d41-b3f6-5ebdf75b8b4c'" --query "[].id" --output tsv
+
+# Update the arcnetworking extension to disable FRR
+az k8s-extension update \
+  --cluster-name $clusterName \
+  -g $rgName \
+  --cluster-type connectedClusters \
+  --extension-type microsoft.arcnetworking \
+  --config "k8sRuntimeFpaObjectId=$objID" \
+  --config "metallb.speaker.frr.enabled=false" \
+  -n arcnetworking
+```
+
+## Troubleshooting steps
+
+Use the following steps to diagnose and resolve BGP issues with MetalLB and FRR in your AKS Arc environment.
+
+### Check BGP configuration
+
+```azurecli
+kubectl get ipaddresspools -A -o yaml
+kubectl get bgppeers.metallb.io -A -o yaml
+kubectl get bgpadvertisements -A -o yaml
+```
+
+### Collect logs from MetalLB speaker (FRR)
+
+```azurecli
+# Get the list of MetalLB speaker pods
+kubectl get pods -n kube-system
+
+# Speaker container logs
+kubectl logs -n kube-system arcnetworking-metallb-speaker-xxxxx -c speaker
+
+# FRR container logs
+kubectl logs -n kube-system arcnetworking-metallb-speaker-xxxxx -c frr
+```
+
+### Review TOR switch configuration
+
+- Configuration and logs from the top-of-rack (TOR) switch or upstream router might be necessary.
+- These logs are hardware/vendor-specific and not covered in this guide.
+
+## Next steps
+
+[Official MetalLB troubleshooting guide](https://metallb.universe.tf/troubleshooting/#with-frr)
+