Merge pull request #15317 from sethmanheim/akstsg6-13

Add new AKS Arc known issue

Author: sethmanheim

AKS-Hybrid/known-issues-arc.yml

@@ -7,15 +7,15 @@ metadata:
   ms.service: azure-stack
   ms.subservice: aks-hci
   ms.topic: faq
-  ms.date: 07/10/2023
+  ms.date: 6/13/2024
 
 title: Resolve errors when enabling or disabling Azure Arc on your AKS workload clusters in AKS enabled by Arc
 summary: |
-  **Applies to: AKS on Azure Stack HCI, AKS on Windows Server**
+  Applies to: AKS on Azure Stack HCI, AKS on Windows Server
   
-  This article describes errors you may encounter (and their workarounds) while connecting or disconnecting your AKS workload clusters to Azure Arc using the PowerShell cmdlets [Enable-AksHciArcConnection](./reference/ps/enable-akshciarcconnection.md) and [Disable-AksHciArcConnection](./reference/ps/disable-akshciarcconnection.md) in AKS Arc. For issues that are not covered in this article, see [Troubleshooting Arc enabled Kubernetes](/azure/azure-arc/kubernetes/troubleshooting).
+  This article describes errors you might encounter (and their workarounds) while connecting or disconnecting your AKS workload clusters to Azure Arc using the PowerShell cmdlets [`Enable-AksHciArcConnection`](./reference/ps/enable-akshciarcconnection.md) and [`Disable-AksHciArcConnection`](./reference/ps/disable-akshciarcconnection.md) in AKS Arc. For issues that are not covered in this article, see [Troubleshooting Arc enabled Kubernetes](/azure/azure-arc/kubernetes/troubleshooting).
   
-  You can also [open a support issue](./help-support.md) if none of the workarounds listed below apply to you.
+  You can also [open a support issue](./help-support.md) if none of the listed workarounds apply to you.
   
 sections:
   - name: Single section - ignored
@@ -24,7 +24,7 @@ sections:
           Error: "A workload cluster with the name 'my-aks-cluster' was not found"
 
         answer: | 
-          This error means that you have not created the workload cluster, or you have incorrectly spelled the name of the workload cluster. 
+          This error means that you have not created the workload cluster, or you incorrectly spelled the name of the workload cluster. 
           
           Run [Get-AksHciCluster](./reference/ps/get-akshcicluster.md) to ensure you have the correct name or that the cluster you want to connect to Arc exists.
           
@@ -42,10 +42,10 @@ sections:
           - Option 2: In PowerShell, check if the cluster has been successfully created by running the [Get-AksHciCluster](./reference/ps/get-akshcicluster.md) command, and then use [Enable-AksHciArcConnection](./reference/ps/enable-akshciarcconnection.md) to connect your cluster to Arc.
           
       - question: | 
-          Enable-AksHciArcConnection fails if Connect-AzAccount is used to sign in to Azure
+          `Enable-AksHciArcConnection` fails if `Connect-AzAccount` is used to sign in to Azure
 
         answer: | 
-          When you use [Connect-AzAccount](/powershell/module/az.accounts/connect-azaccount?view=azps-6.4.0&preserve-view=true) to sign in to Azure, you might set a different subscription as your default context than the one that you gave as an input to [Set-AksHciRegistration](./reference/ps/set-akshciregistration.md). When you then run [Enable-AksHciArcConnection](./reference/ps/enable-akshciarcconnection.md), the command expects the subscription used in `Set-AksHciRegistration`. However, `Enable-AksHciArcConnection` gets the default subscription set using the `Connect-AzAccount`, and therefore, might cause an error.
+          When you use [`Connect-AzAccount`](/powershell/module/az.accounts/connect-azaccount?view=azps-6.4.0&preserve-view=true) to sign in to Azure, you might set a different subscription as your default context than the one that you gave as an input to [`Set-AksHciRegistration`](./reference/ps/set-akshciregistration.md). When you then run [`Enable-AksHciArcConnection`](./reference/ps/enable-akshciarcconnection.md), the command expects the subscription used in `Set-AksHciRegistration`. However, `Enable-AksHciArcConnection` gets the default subscription set using the `Connect-AzAccount`, and therefore, might cause an error.
           
           To prevent this error, follow one of the options below:
           
@@ -58,7 +58,7 @@ sections:
         answer: | 
           This error usually points to one of the following issues:
           
-          - The clusters were created in an Azure VM in a virtualized environment, or you were deploying AKS on Azure Stack HCI on multiple levels of virtualization. 
+          - The clusters were created in an Azure VM in a virtualized environment, or you deployed AKS on Azure Stack HCI on multiple levels of virtualization. 
           - A slow internet caused the timeout.
           
           If one of the above scenarios applies to you, run [Disable-AksHciArcConnection](./reference/ps/disable-akshciarcconnection.md), and try connecting again. If the above scenario doesn't apply to you, [open a support issue](./help-support.md) for AKS on Azure Stack HCI.
@@ -68,75 +68,104 @@ sections:
         answer: | 
           This error indicates that your Kubernetes API server could not be reached. 
           
-          Try running the [Disable-AksHciArcConnection](./reference/ps/disable-akshciarcconnection.md) command again, and then go to the [Azure portal](https://portal.azure.com) to confirm that your `connectedCluster` resource has actually been deleted. You can also run `kubectl get ns -A` to confirm that the namespace, `azure-arc`, does not exist on your cluster.
-          
-      - question: | 
-          Error: 'Error while updating agents for enabling features'
-
-        answer: | 
-          If you enable the *custom location* and *cluster connect* features on an AKS cluster that is connected to Azure Arc, you may see the following error: 
-          
-          `Error while updating agents for enabling features. Please run "kubectl get pods -n azure-arc" to check the pods in case of timeout error. Error: Error: UPGRADE FAILED: timed out waiting for the condition`
-          
-          This is a known issue with the September release and is fixed in the October release. New AKS clusters created using the October release and connected to Arc using [Enable-AksHciArcConnection](./reference/ps/enable-akshciarcconnection.md) do not experience this issue. Update your AKS Arc deployment to the October release and then reconnect your existing clusters to Arc for a workaround to this issue.
-          
+          Try running the [`Disable-AksHciArcConnection`](./reference/ps/disable-akshciarcconnection.md) command again, and then go to the [Azure portal](https://portal.azure.com) to confirm that your `connectedCluster` resource was actually deleted. You can also run `kubectl get ns -A` to confirm that the namespace, `azure-arc`, does not exist on your cluster.
+                  
       - question: | 
-          Error: 'Connection to Azure failed. Please run 'Set-AksHciRegistration' and try again'
+          Error: "Connection to Azure failed. Please run 'Set-AksHciRegistration' and try again"
 
         answer: | 
           This error means that your login credentials to Azure have expired. 
           
-          Use [Set-AksHciRegistration](./reference/ps/set-akshciregistration.md) to log in to Azure before running the [Enable-AksHciArcConnection](./reference/ps/enable-akshciarcconnection.md) command again. When rerunning `Set-AksHciRegistration`, make sure you use the same subscription and resource group details you used when you first registered the AKS host to Azure for billing. If you rerun the command with a different subscription or resource group, they will not be registered. Once the subscription and resource group are set in `Set-AksHciRegistration`, they cannot be changed without uninstalling AKS Arc.
+          Use [`Set-AksHciRegistration`](./reference/ps/set-akshciregistration.md) to log in to Azure before running the [`Enable-AksHciArcConnection`](./reference/ps/enable-akshciarcconnection.md) command again. When rerunning `Set-AksHciRegistration`, make sure you use the same subscription and resource group details you used when you first registered the AKS host to Azure for billing. If you rerun the command with a different subscription or resource group, they will not be registered. Once the subscription and resource group are set in `Set-AksHciRegistration`, they cannot be changed without uninstalling AKS Arc.
           
       - question: | 
-          Error: ''My-Cluster' is not a valid cluster name. Names must be lowercase and match the regular expression pattern: '^[a-z0-9][a-z0-9-]*[a-z0-9]$''
+          Error: "'My-Cluster' is not a valid cluster name. Names must be lowercase and match the regular expression pattern: '^[a-z0-9][a-z0-9-]*[a-z0-9]$'"
 
         answer: | 
           This error indicates that the workload cluster does not follow the Kubernetes naming convention. 
           
           As the error suggests, make sure the cluster name is lowercase and matches the regular expression pattern: '^[a-z0-9][a-z0-9-]*[a-z0-9]$'.
           
       - question: | 
-          Error: 'addons.msft.microsoft "demo-arc-onboarding" already exists'
+          Error: "addons.msft.microsoft "demo-arc-onboarding" already exists"
 
         answer: | 
-          This error usually means that you have already connected your AKS cluster to Arc-enabled Kubernetes.
-          To confirm it's connected, go to the [Azure portal](https://portal.azure.com) and check under the subscription and resource group you provided when running [Set-AksHciRegistration](./reference/ps/set-akshciregistration.md) (if you've used default values) or [Enable-AksHciArcConnection](./reference/ps/enable-akshciarcconnection.md) (if you haven't used default values). You can also confirm if your AKS on Azure Stack HCI cluster is connected to Azure by running the [az connectedk8s show ](/cli/azure/connectedk8s#az-connectedk8s-show) Azure CLI command. If you do not see your workload cluster, run `Disable-AksHciArcConnection` and try again.
+          This error usually means that you already connected your AKS cluster to Arc-enabled Kubernetes.
+          To confirm it's connected, go to the [Azure portal](https://portal.azure.com) and check under the subscription and resource group you provided when you ran [`Set-AksHciRegistration`](./reference/ps/set-akshciregistration.md) (if you used default values) or [`Enable-AksHciArcConnection`](./reference/ps/enable-akshciarcconnection.md) (if you didn't use default values). You can also confirm if your AKS on Azure Stack HCI cluster is connected to Azure by running the [`az connectedk8s show` ](/cli/azure/connectedk8s#az-connectedk8s-show) Azure CLI command. If you don't see your workload cluster, run `Disable-AksHciArcConnection` and try again.
           
           
       - question: | 
-          Error: 'autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound"...'
+          Error: "autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound"..."
 
         answer: | 
-          The error below means that Azure could not find the `connectedCluster` ARM resource associated with your cluster:
+          This error means that Azure can't find the `connectedCluster` Azure Resource Manager resource associated with your cluster:
           
-          `autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The Resource 'Microsoft.Kubernetes/connectedClusters/my-workload-cluster' under resource group 'AKS-HCI2' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"]`
+          "autorest/azure: Service returned an error. Status=404 Code="ResourceNotFound" Message="The Resource 'Microsoft.Kubernetes/connectedClusters/my-workload-cluster' under resource group 'AKS-HCI2' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"]"
           
           You may encounter this error if: 
           
           - You supplied an incorrect resource group or subscription while running the `Disable-AksHciArcConnection` cmdlet. 
           - You manually deleted the resource on the Azure portal.
-          - ARM cannot find your Azure resource.
+          - Azure Resource Manager cannot find your Azure resource.
           
           To resolve this error, as indicated in the error message, see [resolve resource not found errors](/azure/azure-resource-manager/templates/error-not-found).
       - question: | 
-          Error: 'Cluster addons arc uninstall Error: namespaces "azure-arc" not found'
+          Error: "Cluster addons arc uninstall Error: namespaces "azure-arc" not found"
 
         answer: | 
-          This error usually means that you have already uninstalled Arc agents from your workload cluster, or you have manually deleted the `azure-arc` namespace using the `kubectl` command. 
+          This error usually means that you already uninstalled Arc agents from your workload cluster, or you manually deleted the `azure-arc` namespace using the `kubectl` command. 
           
-          Go to the [Azure portal](https://portal.azure.com) to confirm that you do not have any leaked resources. For example, verify that you do not see a `connectedCluster` resource in the subscription and resource group.
+          Go to the [Azure portal](https://portal.azure.com) to confirm that you don't have any leaked resources. For example, verify that you don't see a `connectedCluster` resource in the subscription and resource group.
           
       - question: | 
-          Error: 'Azure subscription is not properly configured'
+          Error: "Azure subscription is not properly configured"
 
         answer: | 
-          You may encounter this issue if you have not configured your Azure subscription with the Arc-enabled Kubernetes resource providers. We currently check that `Microsoft.Kubernetes` and `Microsoft.KubernetesConfiguration` are configured. 
+          You may encounter this issue if you haven't configured your Azure subscription with the Arc-enabled Kubernetes resource providers. We currently check that `Microsoft.Kubernetes` and `Microsoft.KubernetesConfiguration` are configured. 
           
           For more information about enabling these resource providers, see [Register providers for Arc-enabled Kubernetes](/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli#register-providers-for-azure-arc-enabled-kubernetes).
-          
-  
+      - question: | 
+          Error: "Unable to read ConfigMap 'azure-clusterconfig' in 'azure-arc' namespace"
 
+        answer: | 
+          You may encounter this issue when trying to re-enable the Arc connection on an AKS cluster after disabling an existing connection. The error is due to a change to the namespace in which Azure Arc secrets are stored.
+
+          The steps that lead to the error are:
+
+          1. Connect a workload cluster to Azure Arc with `Enable-AksHciArcConnection -name $clusterName`.
+          1. Disconnect the cluster from Azure Arc: `Disable-AksHciArcConnection -name $clusterName`.
+          1. Connect the workload cluster to Azure Arc with this command again: `Enable-AksHciArcConnection -name $clusterName`.
+
+          The error is:
+
+          ```output
+          returned a non zero exit code 1 [Error: Job azure-arc-onboarding terminated with Failed to run CLI command: Error from server (NotFound): namespaces "azure-arc"
+          not found
+          System.Management.Automation.RemoteException
+          ERROR: Unable to read ConfigMap 'azure-clusterconfig' in 'azure-arc' namespace:
+          Error Response: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"azure-clusterconfig\" not
+          found","reason":"NotFound","details":{"name":"azure-clusterconfig","kind":"configmaps"},"code":404}
+          System.Management.Automation.RemoteException
+          System.Management.Automation.RemoteException
+          : Job Failed Condition
+          ```
+
+          `Enable-AksHciArcConnection` always fails, and after you run `Disable-AksHciArcConnection`, there is a remaining secret in the azure-arc-release namespace. To check if the secret exists, you can run the following command and ensure that no secret is listed:
+
+          ```powershell
+          kubectl get secret -nazure-arc-release sh.helm.release.v1.azure-arc.v1
+          ```
+
+          To work around this issue, delete the azure-arc-release namespace after running `Disable-AksHciArcConnection`:
+
+          ```powershell
+          $clusterName = "<name of cluster>"
+          Get-AksHciCredential -name $clusterName
+          kubectl delete namespace azure-arc-release
+          Disable-AksHciArcConnection -name $clusterName
+          Enable-AksHciArcConnection -name $clusterName
+          ```
+  
 additionalContent: |
   ## Next steps