Merge branch 'main' into arm-template-update

Author: v-sissondan

AKS-Arc/TOC.yml

@@ -151,24 +151,31 @@
       href: aks-troubleshoot.md
     - name: Control plane configuration validation errors
       href: control-plane-validation-errors.md
-    - name: Connectivity issues with MetalLB
-      href: load-balancer-issues.md 
     - name: K8sVersionValidation error
       href: cluster-k8s-version.md
     - name: Use diagnostic checker
       href: aks-arc-diagnostic-checker.md
     - name: KubeAPIServer unreachable error
       href: kube-api-server-unreachable.md
-    - name: Can't see VM SKUs on Azure portal
-      href: check-vm-sku.md
-    - name: Deleted AKS Arc cluster still visible on Azure portal
-      href: deleted-cluster-visible.md
+    - name: Can't create/scale AKS cluster due to image issues
+      href: gallery-image-not-usable.md
+    - name: Disk space exhaustion on control plane VMs
+      href: kube-apiserver-log-overflow.md
+    - name: Telemetry pod consumes too much memory and CPU  
+      href: telemetry-pod-resources.md
+    - name: Issues after deleting storage volumes
+      href: delete-storage-volume.md
     - name: Can't fully delete AKS Arc cluster with PodDisruptionBudget (PDB) resources
       href: delete-cluster-pdb.md
     - name: Azure Advisor upgrade recommendation
       href: azure-advisor-upgrade.md
-    - name: Issues after deleting storage volumes
-      href: delete-storage-volume.md
+    - name: Deleted AKS Arc cluster still visible on Azure portal
+      href: deleted-cluster-visible.md
+    - name: Can't see VM SKUs on Azure portal
+      href: check-vm-sku.md
+    - name: Connectivity issues with MetalLB
+      href: load-balancer-issues.md 
+
   - name: Reference
     items:
     - name: Azure CLI

AKS-Arc/aks-create-clusters-cli.md

@@ -4,7 +4,7 @@ description: Learn how to create Kubernetes clusters in Azure Local using Azure
 ms.topic: how-to
 ms.custom: devx-track-azurecli
 author: sethmanheim
-ms.date: 02/18/2025
+ms.date: 03/31/2025
 ms.author: sethm 
 ms.lastreviewed: 01/25/2024
 ms.reviewer: guanghu
@@ -44,7 +44,7 @@ az extension add -n connectedk8s --upgrade
 
 ## Create a Kubernetes cluster
 
-Use the `az aksarc create` command to create a Kubernetes cluster in AKS Arc. Make sure you sign in to Azure before running this command. If you have multiple Azure subscriptions, select the appropriate subscription ID using the [az account set](/cli/azure/account#az-account-set) command.
+Use the [`az aksarc create`](/cli/azure/aksarc#az-aksarc-create) command to create a Kubernetes cluster in AKS Arc. Make sure you sign in to Azure before you run this command. If you have multiple Azure subscriptions, select the appropriate subscription ID using the [`az account set`](/cli/azure/account#az-account-set) command. With the  `az aksarc create` command, we recommend that you use the `--validate` flag, which validates the input parameters that you intend to use. Once the input parameters are validated, you can run the  `az aksarc create` command without the `--validate` flag to create the Kubernetes cluster. 
 
 ```azurecli
 az aksarc create -n $aksclustername -g $resource_group --custom-location $customlocationID --vnet-ids $logicnetId --aad-admin-group-object-ids $aadgroupID --generate-ssh-keys 

AKS-Arc/aks-edge-howto-scale-out.md

@@ -15,6 +15,12 @@ Now that AKS Edge Essentials is installed on your primary machine, this article
 > [!CAUTION]
 > Scaling to additional nodes is an experimental feature.
 
+> [!IMPORTANT]
+> AKS Edge Essentials multi-machine deployment is currently in PREVIEW.
+> See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
+> Azure Kubernetes Service Edge Essentials previews are partially covered by customer support on a best-effort basis.
+
+
 ## Prerequisites
 
 - Set up your [scalable Kubernetes](aks-edge-howto-multi-node-deployment.md) cluster.

AKS-Arc/aks-networks.md

@@ -3,7 +3,7 @@ title: Create logical networks for Kubernetes clusters on Azure Local, version 2
 description: Learn how to create Arc-enabled logical networks for AKS.
 ms.topic: how-to
 author: sethmanheim
-ms.date: 03/21/2025
+ms.date: 04/01/2025
 ms.author: sethm 
 ms.lastreviewed: 04/01/2024
 ms.reviewer: abha
@@ -80,7 +80,7 @@ For static IP, the required parameters are as follows:
 | `--ip-allocation-method`   | The IP address allocation method. Supported values are `Static`. Usage: `--ip-allocation-method "Static"`. |
 | `--ip-pool-start`     | The start IP address of your IP pool. The address must be in range of the address prefix. Usage: `--ip-pool-start "10.220.32.18"`.  |
 | `--ip-pool-end`       | The end IP address of your IP pool. The address must be in range of the address prefix. Usage: `--ip-pool-end "10.220.32.38"`.  |
-| `--vlan`              | The VLAN ID. Usage: `--vlan 10`. This parameter is required, otherwise the default value of 0 results in an AKS Arc cluster creation failure.  |
+| `--vlan`              | The VLAN ID. Usage: `--vlan 10`. This parameter is optional. Specifies the VLAN ID (an int32 value) to use when creating the logical network.  |
 
 # [Azure portal](#tab/azureportal)
 

AKS-Arc/aks-troubleshoot.md

@@ -3,10 +3,10 @@ title: Troubleshoot common issues in AKS enabled by Azure Arc
 description: Learn about common issues and workarounds in AKS enabled by Arc.
 ms.topic: how-to
 author: sethmanheim
-ms.date: 02/28/2025
+ms.date: 04/01/2025
 ms.author: sethm 
-ms.lastreviewed: 02/27/2024
-ms.reviewer: guanghu
+ms.lastreviewed: 04/01/2025
+ms.reviewer: abha
 
 ---
 
@@ -20,18 +20,29 @@ To open a support request, see the [Get support](/azure/aks/hybrid/help-support)
 
 ## Known issues
 
-The following sections describe known issues and workarounds for AKS enabled by Azure Arc:
-
-- [Control plane configuration validation errors](control-plane-validation-errors.md)
-- [Connectivity issues with MetalLB](load-balancer-issues.md)
-- [K8sVersionValidation error](cluster-k8s-version.md)
-- [Use diagnostic checker](aks-arc-diagnostic-checker.md)
-- [KubeAPIServer unreachable error](kube-api-server-unreachable.md)
-- [Can't see VM SKUs on Azure portal](check-vm-sku.md)
-- [Deleted AKS Arc cluster still visible on Azure portal](deleted-cluster-visible.md)
-- [Can't fully delete AKS Arc cluster with PodDisruptionBudget (PDB) resources](delete-cluster-pdb.md)
-- [Azure Advisor upgrade recommendation message](azure-advisor-upgrade.md)
-- [Issues after deleting storage volume](delete-storage-volume.md)
+The following sections describe known issues for AKS enabled by Azure Arc:
+
+| AKS Arc CRUD operation | Issue | Fix status |
+|------------------------|-------|------------|
+| AKS cluster create     | [Can't create AKS cluster or scale node pool because of issues with AKS Arc images](gallery-image-not-usable.md) | Partially fixed in 2503 release |
+| AKS steady state       | [AKS Arc telemetry pod consumes too much memory and CPU](telemetry-pod-resources.md) | Active
+| AKS steady state       | [Disk space exhaustion on control plane VMs due to accumulation of kube-apiserver audit logs](kube-apiserver-log-overflow.md) | Active
+| AKS cluster delete     | [Deleted AKS Arc cluster still visible on Azure portal](deleted-cluster-visible.md) | Active |
+| AKS cluster delete     | [Can't fully delete AKS Arc cluster with PodDisruptionBudget (PDB) resources](delete-cluster-pdb.md) | Fixed in 2503 release |
+| Azure portal           | [Can't see VM SKUs on Azure portal](check-vm-sku.md) | Fixed in 2411 release |
+| MetalLB Arc extension  | [Connectivity issues with MetalLB](load-balancer-issues.md) | Fixed in 2411 release | 
+
+
+## Guides to diagnose and troubleshoot Kubernetes CRUD failures
+
+| AKS Arc operation | Issue | 
+|------------------------|-------|
+| Create validation      | [Control plane configuration validation errors](control-plane-validation-errors.md) 
+| Create validation      | [K8sVersionValidation error](cluster-k8s-version.md)   
+| Create validation      | [KubeAPIServer unreachable error](kube-api-server-unreachable.md)  
+| Network configuration issues | [Use diagnostic checker](aks-arc-diagnostic-checker.md)
+| Kubernetes steady state   | [Resolve issues due to out-of-band deletion of storage volumes](delete-storage-volume.md)
+| Release validation     | [Azure Advisor upgrade recommendation message](azure-advisor-upgrade.md)
 
 ## Next steps
 

AKS-Arc/aks-whats-new-23h2.md

@@ -2,11 +2,11 @@
 title: What's new in AKS on Azure Local, version 23H2
 description: Learn about what's new in AKS on Azure Local, version 23H2.
 ms.topic: overview
-ms.date: 11/19/2024
+ms.date: 04/01/2025
 author: sethmanheim
 ms.author: sethm 
 ms.reviewer: guanghu
-ms.lastreviewed: 06/25/2024
+ms.lastreviewed: 04/01/2025
 
 ---
 
@@ -42,6 +42,23 @@ By integrating these components, Azure Arc offers a unified and efficient Kubern
 
 This section lists the new features and improvements in AKS Arc in each release of Azure Local, version 23H2.
 
+### Release 2503
+
+The following Kubernetes cluster deployment and management capabilities are available:
+
+- **Large VM SKUs for Kubernetes nodepools**: Added two new VM SKUs - `Standard_D32s_v3`: 32 vCPU, 128 GiB and `Standard_D16s_v3`: 16 vCPU, 64 GiB - to support larger nodepools on an AKS cluster. For more information about supported VM sizes, see [supported scale options](scale-requirements.md).
+- **Improved log collection experience**: Improved log collection for AKS control plane node VMs and nodepool VMs, with support for passing multiple IP addresses and SSH key or directory path. For more information, see [on-demand log collection](get-on-demand-logs.md) and [az aksarc get-logs CLI](/cli/azure/aksarc#az-aksarc-get-logs).
+- **Improved diagnosability**: The [Diagnostic Checker tool](aks-arc-diagnostic-checker.md) is automatically run in case of Kubernetes cluster create failure, and added new test cases.
+- **Improved Kubernetes cluster delete**: Fixed deletion issues; for example, due to [pod disruption budgets](delete-cluster-pdb.md?tabs=aks-on-azure-local).
+- **Improved AKS Arc image download**: Fixed issues with AKS Arc image downloads.
+- **Improved GPU support**: Improved error handling for Kubernetes cluster creation with GPU enabled nodepools. Fixed known issues with attaching persistent volumes on GPU enabled nodepools.
+
+To get started with these features in the 2503 release, make sure to update your [AKSArc CLI extension](/cli/azure/aksarc) to version 1.5.37 or higher.
+
+#### Supported Kubernetes versions for 2503
+
+The Kubernetes versions supported in the 2503 release are: 1.28.12, 1.28.14, 1.29.7, 1.29.9, 1.30.3 and 1.30.4.
+
 ### Release 2411
 
 The following Kubernetes cluster deployment and management capabilities are available:

AKS-Arc/delete-cluster-pdb.md

@@ -4,7 +4,7 @@ description: Learn how to troubleshoot when deleted workload cluster resources c
 ms.topic: troubleshooting
 author: sethmanheim
 ms.author: sethm
-ms.date: 12/12/2024
+ms.date: 04/01/2025
 ms.reviewer: leslielin
 
 ---
@@ -15,7 +15,15 @@ ms.reviewer: leslielin
 
 When you delete an AKS Arc cluster that has [PodDisruptionBudget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/) (PDB) resources, the deletion might fail to remove the PDB resources. By default, PDB is installed in the workload identity-enabled AKS Arc cluster.
 
-## Workaround
+## Mitigation
+
+This issue was fixed in [AKS on Azure Local, version 2503](aks-whats-new-23h2.md#release-2503).
+
+- **For deleting an AKS cluster** with a PodDisruptionBudget: If you're on an older build, please update to Azure Local, version 2503. Once you update to 2503, you can retry deleting the AKS cluster. File a support case if you're on the 2503 release and your AKS cluster is not deleted after at least one retry.
+- **For deleting a nodepool** with a PodDisruptionBudget: By design, the nodepool isn't deleted if a PodDisruptionBudget exists, to protect applications. Use the following workaround to delete the PDB resources and then retry deleting the nodepool.
+
+
+## Workaround for AKS Edge Essentials and older versions of AKS on Azure Local
 
 Before you delete the AKS Arc cluster, access the AKS Arc cluster's **kubeconfig** and delete all PDBs:
 

AKS-Arc/gallery-image-not-usable.md

@@ -0,0 +1,54 @@
+---
+title: Kubernetes cluster create or nodepool scale failing due to AKS Arc image issues  
+description: Learn about a known issue with Kubernetes cluster create or nodepool scale failing due to AKS Arc VHD image download issues.
+ms.topic: troubleshooting
+author: sethmanheim
+ms.author: sethm
+ms.date: 04/01/2025
+ms.reviewer: abha
+
+---
+
+# Can't create AKS cluster or scale node pool because of issues with AKS Arc images
+
+[!INCLUDE [hci-applies-to-23h2](includes/hci-applies-to-23h2.md)]
+
+## Symptoms
+
+You see the following error when you try to create the AKS cluster:
+
+```output
+Kubernetes version 1.29.4 is not ready for use on Linux. Please go to https://aka.ms/aksarccheckk8sversions for details of how to check the readiness of Kubernetes versions.
+```
+
+You might also see the following error when you try to scale a nodepool:
+
+```output
+error with code NodepoolPrecheckFailed occured: AksHci nodepool creation precheck failed. Detailed message: 1 error occurred:\n\t* rpc error: code = Unknown desc = GalleryImage not usable, health state degraded: Degraded
+```
+
+When you run `az aksarc get-versions`, you see the following errors:
+
+```output
+...
+              {
+
+                "errorMessage": "failed cloud-side provisioning image linux-cblmariner-0.4.1.11203 to cloud gallery: {\n  \"code\": \"ImageProvisionError\",\n  \"message\": \"force failed to deprovision existing gallery image: failed to delete gallery image linux-cblmariner-0.4.1.11203: rpc error: code = Unknown desc = sa659p1012: rpc error: code = Unavailable desc = connection error: desc = \\\"transport: Error while dialing: dial tcp 10.202.244.4:45000: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\\\"\",\n  \"additionalInfo\": [\n   {\n    \"type\": \"providerImageProvisionInfo\",\n    \"info\": {\n     \"ProviderDownload\": \"True\"\n    }\n   }\n  ],\n  \"category\": \"\"\n }",
+                "osSku": "CBLMariner",
+                "osType": "Linux",
+                "ready": false
+              },
+...
+```
+
+## Mitigation
+
+- This issue was fixed in [AKS on Azure Local, version 2503](aks-whats-new-23h2.md#release-2503).
+- Upgrade your Azure Local deployment to the 2503 build.
+- Once updated, confirm that the images have been downloaded successfully by running the `az aksarc get-versions` command.
+- For new AKS clusters: new AKS clusters should now be created successfully.
+- For scaling existing AKS clusters: scaling existing AKS clusters continues to encounter issues. Please file a support case.
+
+## Next steps
+
+[Known issues in AKS enabled by Azure Arc](aks-known-issues.md)

AKS-Arc/get-on-demand-logs.md

@@ -28,13 +28,13 @@ Before log collection, you must have the SSH key you obtained when you created t
 You can collect logs using IPs or the `kubeconfig` parameter. If an IP is used, it collects the log from a particular node. If `kubeconfig` is used, it collects logs from all cluster nodes. This command generates a .zip file on the local disk. For other parameters, see the [Az CLI reference](/cli/azure/aksarc/logs#az-aksarc-logs-hci).
 
 ```azurecli
-az aksarc logs hci --ip 192.168.200.25 --credentials-dir ./.ssh --out-dir ./logs
+az aksarc get-logs --ip 192.168.200.25 --credentials-dir ./.ssh --out-dir ./logs
 ```
 
 Or
 
 ```azurecli
-az aksarc logs hci --kubeconfig ./.kube/config --credentials-dir ./.ssh --out-dir ./logs
+az aksarc get-logs --kubeconfig ./.kube/config --credentials-dir ./.ssh --out-dir ./logs
 ```
 
 ## Send logs to Microsoft Support

AKS-Arc/kube-apiserver-log-overflow.md

@@ -0,0 +1,73 @@
+---
+title: Disk space exhaustion on the control plane VMs due to accumulation of kube-apiserver audit logs
+description: Learn about a known issue with disk space exhaustion on the control plane VMs due to accumulation of kube-apiserver audit logs.
+ms.topic: troubleshooting
+author: sethmanheim
+ms.author: sethm
+ms.date: 04/01/2025
+ms.reviewer: abha
+
+---
+
+# Disk space exhaustion on control plane VMs due to accumulation of kube-apiserver audit logs
+
+[!INCLUDE [hci-applies-to-23h2](includes/hci-applies-to-23h2.md)]
+
+## Symptoms
+
+If you're running kubectl commands and facing issues, you might see errors such as:
+
+```output
+kubectl get ns
+Error from server (InternalError): an error on the server ("Internal Server Error: \"/api/v1/namespaces?limit=500\": unknown") has prevented the request from succeeding (get namespaces)
+```
+
+When you SSH into the control plane VM, you might notice that your control plane VM ran out of disk space, specifically on the **/dev/sda2** partition. This is due to the accumulation of kube-apiserver audit logs in the **/var/log/kube-apiserver** directory, which can consume approximately 90 GB of disk space.
+
+```output
+clouduser@moc-laiwyj6tly6 [ /var/log/kube-apiserver ]$ df -h
+Filesystem      Size  Used Avail Use% Mounted on
+devtmpfs        4.0M     0  4.0M   0% /dev
+tmpfs           3.8G   84K  3.8G   1% /dev/shm
+tmpfs           1.6G  179M  1.4G  12% /run
+tmpfs           4.0M     0  4.0M   0% /sys/fs/cgroup
+/dev/sda2        99G   99G     0 100% /
+tmpfs           3.8G     0  3.8G   0% /tmp
+tmpfs           769M     0  769M   0% /run/user/1002
+clouduser@moc-laiwyj6tly6 [ /var/log/kube-apiserver ]$ sudo ls -l /var/log/kube-apiserver|wc -l
+890
+clouduser@moc-laiwyj6tly6 [ /var/log/kube-apiserver ]$ sudo du -h /var/log/kube-apiserver
+87G     /var/log/kube-apiserver
+```
+
+The issue occurs because the `--audit-log-maxbackup` value is set to 0. This setting allows the audit logs to accumulate without any limit, eventually filling up the disk. 
+
+## Mitigation
+
+To resolve the issue temporarily, you must manually clean up the old audit logs. Follow these steps:
+
+- SSH into the control plane virtual machine (VM) of your AKS Arc cluster.
+- Remove the old audit logs from the **/var/log/kube-apiserver** folder.
+- If you have multiple control plane nodes, you must repeat this process on each control plane VM.
+
+[SSH into the control plane VM](ssh-connect-to-windows-and-linux-worker-nodes.md) and navigate to the kube-apiserver logs directory:
+
+```bash
+cd /var/log/kube-apiserver
+```
+
+Remove the old audit log files:
+
+```bash
+rm audit-*.log
+```
+
+Exit the SSH session:
+
+```bash
+exit
+```
+
+## Next steps
+
+[Known issues in AKS enabled by Azure Arc](aks-known-issues.md)