Merge pull request #17811 from sethmanheim/akstsg4-30

AKS: add network validation error TSG

Author: sethmanheim

AKS-Arc/TOC.yml

@@ -180,7 +180,9 @@
     - name: Can't see VM SKUs on Azure portal
       href: check-vm-sku.md
     - name: Connectivity issues with MetalLB
-      href: load-balancer-issues.md 
+      href: load-balancer-issues.md
+    - name: Network validation error due to .local domain
+      href: network-validation-error-local.md
   - name: Reference
     items:
     - name: Azure CLI

AKS-Arc/aks-troubleshoot.md

@@ -3,7 +3,7 @@ title: Troubleshoot common issues in AKS enabled by Azure Arc
 description: Learn about common issues and workarounds in AKS enabled by Arc.
 ms.topic: how-to
 author: sethmanheim
-ms.date: 04/01/2025
+ms.date: 04/30/2025
 ms.author: sethm 
 ms.lastreviewed: 04/01/2025
 ms.reviewer: abha
@@ -25,24 +25,24 @@ The following sections describe known issues for AKS enabled by Azure Arc:
 | AKS Arc CRUD operation | Issue | Fix status |
 |------------------------|-------|------------|
 | AKS cluster create     | [Can't create AKS cluster or scale node pool because of issues with AKS Arc images](gallery-image-not-usable.md) | Partially fixed in 2503 release |
-| AKS steady state       | [AKS Arc telemetry pod consumes too much memory and CPU](telemetry-pod-resources.md) | Active
-| AKS steady state       | [Disk space exhaustion on control plane VMs due to accumulation of kube-apiserver audit logs](kube-apiserver-log-overflow.md) | Active
+| AKS steady state       | [AKS Arc telemetry pod consumes too much memory and CPU](telemetry-pod-resources.md) | Active |
+| AKS steady state       | [Disk space exhaustion on control plane VMs due to accumulation of kube-apiserver audit logs](kube-apiserver-log-overflow.md) | Active |
 | AKS cluster delete     | [Deleted AKS Arc cluster still visible on Azure portal](deleted-cluster-visible.md) | Active |
 | AKS cluster delete     | [Can't fully delete AKS Arc cluster with PodDisruptionBudget (PDB) resources](delete-cluster-pdb.md) | Fixed in 2503 release |
 | Azure portal           | [Can't see VM SKUs on Azure portal](check-vm-sku.md) | Fixed in 2411 release |
-| MetalLB Arc extension  | [Connectivity issues with MetalLB](load-balancer-issues.md) | Fixed in 2411 release | 
-
+| MetalLB Arc extension  | [Connectivity issues with MetalLB](load-balancer-issues.md) | Fixed in 2411 release |
 
 ## Guides to diagnose and troubleshoot Kubernetes CRUD failures
 
-| AKS Arc operation | Issue | 
+| AKS Arc operation | Issue |
 |------------------------|-------|
-| Create validation      | [Control plane configuration validation errors](control-plane-validation-errors.md) 
-| Create validation      | [K8sVersionValidation error](cluster-k8s-version.md)   
-| Create validation      | [KubeAPIServer unreachable error](kube-api-server-unreachable.md)  
-| Network configuration issues | [Use diagnostic checker](aks-arc-diagnostic-checker.md)
-| Kubernetes steady state   | [Resolve issues due to out-of-band deletion of storage volumes](delete-storage-volume.md)
-| Release validation     | [Azure Advisor upgrade recommendation message](azure-advisor-upgrade.md)
+| Create validation      | [Control plane configuration validation errors](control-plane-validation-errors.md) |
+| Create validation      | [K8sVersionValidation error](cluster-k8s-version.md) |
+| Create validation      | [KubeAPIServer unreachable error](kube-api-server-unreachable.md) |
+| Network configuration issues | [Use diagnostic checker](aks-arc-diagnostic-checker.md) |
+| Kubernetes steady state   | [Resolve issues due to out-of-band deletion of storage volumes](delete-storage-volume.md) |
+| Release validation     | [Azure Advisor upgrade recommendation message](azure-advisor-upgrade.md) |
+| Network validation | [Network validation error due to .local domain](network-validation-error-local.md) |
 
 ## Next steps
 

AKS-Arc/network-validation-error-local.md

@@ -0,0 +1,43 @@
+---
+title: Network validation error due to .local domain
+description: Learn how to troubleshoot network validation errors due to the .local domain.
+author: sethmanheim
+ms.author: sethm
+ms.topic: troubleshooting
+ms.date: 04/30/2025
+ms.reviewer: pradwivedi
+ms.lastreviewed: 04/30/2025
+
+---
+
+# Troubleshoot network validation error due to .local domain
+
+This article describes how to resolve the `Not able to connect to http://cloudagent.contoso.local:50000` error. This error occurs when you try to create and deploy an AKS on Azure Local cluster.
+
+## Symptoms
+
+You can deploy `.local` domains on Azure Local but might sometimes encounter failures during AKS scenarios, such as create, scale, update, upgrade, and delete. You might see the following error message:
+
+`Error: Network validation failed during cluster creation. Detailed message: Not able to connect to http://cloudagent.contoso.local:50000. Error returned: action failed after 5 attempts: Get "http://cloudagent.contoso.local:50000": dial tcp: lookup http://cloudagent.contoso.local: Temporary failure in name resolution`
+
+## Possible causes
+
+There are two possible causes for this error:
+
+1. Because `.local` is an officially reserved special-use domain name, host names with this top-level label are only resolvable via the multicast DNS name resolution protocol. Other mechanisms such as unicast DNS can also be used to resolve this name.
+
+   When a URL ending with `.local` for the failover cluster is used, a fully qualified domain name (FQDN) ending with `.local` is also used for the MOC cloud agent. The Azure Local 2503 release consists of various network validation tests. One of the tests tries to connect to the MOC cloud FQDN from the AKS Arc control plane VM. This specific test fails when the MOC cloud agent FQDN uses the `.local` domain name. This is because the **Go HTTP** client relies on standard DNS resolution, so it doesn't automatically resolve the `.local` address via mDNS.
+
+1. When the on-premises directory is synchronized with Microsoft 365, you must have a verified domain in Microsoft Entra ID. Only the user principal names (UPNs) that are associated with the on-premises Active Directory Domain Services (AD DS) domain are synchronized. However, any UPN that contains a non-routable domain, such as `.local` (for example, `[email protected]`), is synchronized to an `.onmicrosoft.com` domain (for example, `[email protected]`). For more information, see [Prepare a nonroutable domain for directory synchronization](/microsoft-365/enterprise/prepare-a-non-routable-domain-for-directory-synchronization?view=o365-worldwide&preserve-view=true).
+
+## Mitigation
+
+If you are on Azure Local 2503 or a later release, don't use `.local` in the domain name.
+
+Per the [possible cause #2](#possible-causes), if you currently use a `.local` domain for your user accounts in AD DS, we recommend that you change them to use a verified domain; for example, `[email protected]`, to properly synchronize with your Microsoft 365 domain.
+
+As a temporary mitigation, the checks for the `.local` domain are disabled in the Azure Local 2504 release. For more information, see [What's new in Azure Local, version 2504](/azure/azure-local/whats-new?view=azloc-2504&preserve-view=true).
+
+## Next steps
+
+[Troubleshoot issues in AKS enabled by Azure Arc](aks-troubleshoot.md)