Fix merge conflicts

Author: pauljewellmsft

AKS-Hybrid/TOC.yml

@@ -64,8 +64,10 @@
           href: deploy-load-balancer-portal.md
         # - name: Troubleshoot issues
         #   href: load-balancer-troubleshoot.md
-    - name: Security and authentication
+    - name: Authentication and authorization
       items:
+      - name: Enable Microsoft Entra ID authentication for Kubernetes clusters
+        href: enable-authentication-microsoft-entra-id.md
       - name: Use Azure RBAC for Kubernetes authorization
         href: azure-rbac-23h2.md
       - name: Use Kubernetes RBAC with Microsoft Entra ID

AKS-Hybrid/aks-edge-troubleshoot-overview.md

@@ -4,7 +4,7 @@ description: Learn about common issues and workarounds in AKS Edge Essentials.
 author: rcheeran
 ms.author: rcheeran
 ms.topic: conceptual
-ms.date: 01/08/2024
+ms.date: 07/22/2024
 ms.custom: template-concept
 ---
 
@@ -31,7 +31,7 @@ Get-ExecutionPolicy
 if ((Get-ExecutionPolicy) -ne "RemoteSigned") { Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process -Force }
 ```
 
-### Low disk space causes precached container images to be deleted
+### Low disk space causes previously cached container images to be deleted
 
 When the node runs out of disk space, some of the preloaded images are garbage collected by the `containerd` runtime. In this case, first free up some disk space, and then run the following command to pull the cached images again:
 
@@ -48,9 +48,9 @@ This script checks for the missing images and reimports them as needed.
    Workaround: If the pods aren't cleaned up, run the following commands to manually clean up the existing Azure Arc-related resources before trying to reconnect again:
 
    ```powershell
-       kubectl delete ns azure-arc
-       kubectl delete clusterrolebinding azure-arc-operator
-       kubectl delete secret sh.helm.release.v1.azure-arc.v1
+   kubectl delete ns azure-arc
+   kubectl delete clusterrolebinding azure-arc-operator
+   kubectl delete secret sh.helm.release.v1.azure-arc.v1
    ```
 
 2. Issue: Azure Arc connectivity doesn't work in a proxy environment.
@@ -63,7 +63,32 @@ This script checks for the missing images and reimports them as needed.
 
 ### Failed to get nodeagent certificate: Not Found
 
-Check the network adapter configuration. During deployment, AKS Edge Essentials needs an adapter that's enabled and has the correct IP address, subnet, and default gateway. These values are automatically populated in a DHCP environment. If you're setting manually, ensure all three are set. In many cases the default gateway isn't set, which results in this error.
+Check the network adapter configuration. During deployment, AKS Edge Essentials needs an adapter that's enabled and has the correct IP address, subnet, and default gateway. These values are automatically populated in a DHCP environment. If you're setting manually, ensure all three are set. In many cases, the default gateway isn't set, which results in this error.
+
+## Kubernetes
+
+### Kube-vip pod continuously restarts
+
+In some scenarios, the **kube-vip** pod loops and restarts continuously.
+
+#### Validation
+
+To confirm that the scenario you're encountering is the same issue documented in this article, check that the kube-vip pod in the **kube-system** namespace has a high number of restarts by running the following command:
+
+```bash
+kubectl get pods –n kube-system
+```
+
+#### Cause
+
+There are a few different reasons why the kube-vip pod might be constantly restarting. These causes include:
+
+- Using an OS disk that is not backed with an SSD disk, or a premium SSD disk when using an Azure VM. You can [review the hardware requirements here](aks-edge-system-requirements.md#hardware-requirements).
+- Disk latency is too high. If the disk latency is greater than 10 ms, it can result in request timeouts, leader loss, and potential cluster instability. You can [review the hardware requirements for etcd here](https://etcd.io/docs/v3.4/op-guide/hardware/).
+
+#### Resolution
+
+To mitigate this issue, review your underlying storage infrastructure to ensure that it meets the performance requirements for etcd and AKS Edge Essentials. Also, consider using premium SSD-backed storage or optimizing your storage configuration for performance.
 
 ## Next steps
 

AKS-Hybrid/azure-rbac-23h2.md

@@ -6,8 +6,8 @@ ms.custom: devx-track-azurecli
 author: sethmanheim
 ms.author: sethm
 ms.reviewer: leslielin
-ms.date: 07/10/2024
-ms.lastreviewed: 07/10/2024
+ms.date: 07/26/2024
+ms.lastreviewed: 07/26/2024
 
 # Intent: As an IT Pro, I want to use Azure RBAC to authenticate connections to my AKS clusters over the Internet or on a private network.
 # Keyword: Kubernetes role-based access control AKS Azure RBAC AD
@@ -33,7 +33,6 @@ Before you begin, make sure you have the following prerequisites:
 - AKS on Azure Stack HCI 23H2 currently supports enabling Azure RBAC only during Kubernetes cluster creation. You can't enable Azure RBAC after the Kubernetes cluster is created.
 - Install the latest version of the **aksarc** and **connectedk8s** Azure CLI extensions. Note that you need to run the **aksarc** extension version 1.1.1 or later to enable Azure RBAC. Run `az --version` to find the current version. If you need to install or upgrade Azure CLI, see [Install Azure CLI](/cli/azure/install-azure-cli).
 
-
   ```azurecli
   az extension add --name aksarc
   az extension add --name connectedk8s
@@ -53,7 +52,7 @@ Before you begin, make sure you have the following prerequisites:
   - New role assignments can take up to five minutes to propagate and be updated by the authorization server.
 - Once Azure RBAC is enabled, you can access your Kubernetes cluster with the given permissions using either direct mode or proxy mode.
   - To access the Kubernetes cluster directly using the `az aksarc get-credentials` command, you need the **Microsoft.HybridContainerService/provisionedClusterInstances/listUserKubeconfig/action**, which is included in the **Azure Kubernetes Service Arc Cluster User** role permission.
-  - To access the Kubernetes cluster from anywhere with a proxy mode using`az connectedk8s proxy` command, you need the **Microsoft.Kubernetes/connectedClusters/listClusterUserCredential/action**, which is included in **Azure Arc-enabled Kubernetes Cluster User** role permission. Meanwhile, you need to verify that the agents and the machine performing the onboarding process meet the network requirements specified in [Azure Arc-enabled Kubernetes network requirements](/azure/azure-arc/kubernetes/network-requirements?tabs=azure-cloud#details).
+  - To access the Kubernetes cluster from anywhere with a proxy mode using the `az connectedk8s proxy` command, or from the Azure portal, you need the **Microsoft.Kubernetes/connectedClusters/listClusterUserCredential/action** action, which is included in the **Azure Arc-enabled Kubernetes Cluster User** role permission. Meanwhile, you must verify that the agents and the machine performing the onboarding process meet the network requirements specified in [Azure Arc-enabled Kubernetes network requirements](/azure/azure-arc/kubernetes/network-requirements?tabs=azure-cloud#details).
 - To use **kubectl**, you can access it using either Azure RBAC or the AAD Admin Group.
   - To use kubectl with Azure RBAC, you need the **Azure Arc Kubernetes Viewer** role scoped to the connected cluster resource.
   - To use kubectl with the AAD Admin Group, you don't need any specific role, but you must ensure you are in one of the groups in the **add-admin-group** list of the connected cluster resource.
@@ -167,7 +166,7 @@ Run the following steps on another client device:
      ```azurecli
      az connectedk8s proxy -n $CLUSTER_NAME -g $RESOURCE_GROUP
      ```
-     
+
      > [!NOTE]
      > This command opens the proxy and blocks the current shell.
 

AKS-Hybrid/enable-authentication-microsoft-entra-id.md

@@ -0,0 +1,77 @@
+---
+title: Enable Microsoft Entra authentication for Kubernetes clusters
+description: Learn how to enable Microsoft Entra ID on Azure Kubernetes Service with kubelogin and authenticate Azure users with credentials or managed roles.
+author: sethmanheim
+ms.author: sethm 
+ms.lastreviewed: 07/26/2024
+ms.reviewer: abha
+ms.topic: how-to
+ms.custom:
+  - devx-track-azurecli
+ms.date: 07/26/2024
+
+# Intent: As an IT Pro, I need to learn how to enable Microsoft Entra ID authentication for Kubernetes clusters
+# Keyword: Microsoft Entra ID
+---
+
+# Enable Microsoft Entra authentication for Kubernetes clusters
+
+Applies to: AKS on Azure Stack HCI 23H2
+
+AKS enabled by Azure Arc simplifies the authentication process with Microsoft Entra ID integration. For authorization, cluster administrators can configure Kubernetes role-based access control (Kubernetes RBAC) or Azure role-based access control (Azure RBAC) based on the directory group membership of the Microsoft Entra ID integration.
+
+Microsoft Entra authentication is provided to AKS Arc clusters with OpenID Connect. OpenID Connect is an identity layer built on top of the OAuth 2.0 protocol. For more information about OpenID Connect, see the [OpenID Connect documentation](/entra/identity-platform/v2-protocols-oidc). For more information about the Microsoft Entra integration flow, see the [Microsoft Entra documentation](concepts-security-access-identity.md#microsoft-entra-integration).
+
+This article describes how to enable and use Microsoft Entra ID authentication for Kubernetes clusters.
+
+## Before you begin
+
+- This configuration requires that you have a Microsoft Entra group for your cluster. This group is registered as an admin group on the cluster to grant admin permissions. If you don't have an existing Microsoft Entra group, you can create one using the [`az ad group create`](/cli/azure/ad/group#az_ad_group_create) command.
+- To create or update a Kubernetes cluster, you need the **Azure Kubernetes Service Arc Contributor** role.
+- To access the Kubernetes cluster directly using the [`az aksarc get-credentials`](/cli/azure/aksarc#az-aksarc-get-credentials) command and download the kubeconfig file, you need the **Microsoft.HybridContainerService/provisionedClusterInstances/listUserKubeconfig/action**, which is included in the **Azure Kubernetes Service Arc Cluster User** role permission.
+- Once your Microsoft Entra group is enabled with admin access to your AKS cluster, this Microsoft Entra group can interact with Kubernetes clusters. You must install [**kubectl**](https://kubernetes.io/docs/tasks/tools/) and [**kubelogin**](https://azure.github.io/kubelogin/install.html).
+- Integration can't be disabled once added. You can still use [`az aksarc update`](/cli/azure/aksarc#az-aksarc-update) to update the `aad-admin-group-object-ids` if needed.
+
+## Enable Microsoft Entra authentication for Kubernetes cluster
+
+### Create a new cluster with Microsoft Entra authentication
+
+1. Create an Azure resource group using the [`az group create`](/cli/azure/group#az-group-create) command:
+
+   ```azurecli
+   az group create --name $resource_group --location centralus
+   ```
+
+1. Create an AKS Arc cluster and enable admin access for your Microsoft Entra group using the `--aad-admin-group-object-ids` parameter in the [`az aksarc create`](/cli/azure/aksarc#az-aksarc-create) command:
+
+    ```azurecli
+    az aksarc create -n $aks_cluster_name -g $resource_group --custom-location $customlocationID --vnet-ids $logicnetId --aad-admin-group-object-ids $aadgroupID --generate-ssh-keys --control-plane-ip $controlplaneIP
+    ```
+
+### Use an existing cluster with Microsoft Entra authentication
+
+Enable Microsoft Entra authentication on your existing Kubernetes cluster using the `--aad-admin-group-object-ids` parameter in the [`az aksarc update`](/cli/azure/aksarc#az-aksarc-update) command. Make sure to set your admin group to retain access on your cluster:
+
+  ```azurecli
+  az aksarc update -n $aks_cluster_name -g $resource_group --aad-admin-group-object-ids $aadgroupID
+  ```
+
+## Access your Microsoft Entra-enabled cluster
+
+1. Get the user credentials to access your cluster using the [`az aksarc get-credentials`](/cli/azure/aksarc#az-aksarc-get-credentials) command. You need the **Microsoft.HybridContainerService/provisionedClusterInstances/listUserKubeconfig/action**, which is included in the **Azure Kubernetes Service Arc Cluster User** role permission:
+
+   ```azurecli
+   az aksarc get-credentials --resource-group $resource_group --name $aks_cluster_name
+   ```
+
+1. View the nodes in the cluster with the `kubectl get nodes` command and follow the instructions to sign in. You need to be in the Microsoft Entra ID group specified with the AKS cluster when you pass the `--aad-admin-group-object-ids $aadgroupID` parameter:
+
+   ```azurecli
+   kubectl get nodes
+   ```
+
+## Next steps
+
+- [Access and identity options for AKS enabled by Azure Arc](concepts-security-access-identity.md)
+- [Microsoft Entra integration with Kubernetes RBAC](kubernetes-rbac-23h2.md)
+- [Use Azure role-based access control (RBAC) for Kubernetes authorization](azure-rbac-23h2.md)

AKS-Hybrid/known-issues-installation.yml

@@ -8,7 +8,7 @@ metadata:
   ms.subservice: aks-hci
   ms.custom: devx-track-azurepowershell
   ms.topic: faq
-  ms.date: 07/11/2024
+  ms.date: 07/26/2024
 title: Resolve issues and errors during an AKS Arc installation
 summary: |
   **Applies to: AKS on Azure Stack HCI, AKS on Windows Server**
@@ -522,7 +522,28 @@ sections:
           This error indicates that the cloud service's IP address is not a part of the cluster network and doesn't match any of the cluster networks that have the `client and cluster communication` role enabled.
           
           To resolve this issue, run [Get-ClusterNetwork](/powershell/module/failoverclusters/get-clusternetwork?view=windowsserver2019-ps&preserve-view=true) where `Role` equals `ClusterAndClient`. Then, on one of the cluster nodes, select the name, address, and address mask to verify that the IP address provided for the `-cloudServiceIP` parameter of [New-AksHciNetworkSetting](./reference/ps/new-akshcinetworksetting.md) matches one of the displayed networks.
-  
+      - question: | 
+          The Enable-AksHciArcConnection cmdlet generates a warning indicating that GetServicePrincipals has insufficient privileges to enable custom locations
+
+        answer: | 
+          `Enable-AksHciArcConnection` can connect an AKS cluster to Azure, but it shows the following warning when the customer uses a service principal for authentication:
+          
+          ```shell
+          WARNING: Error occurred while executing GetServicePrincipals
+          Code: Authorization_RequestDenied
+          Message: Insufficient privileges to complete the operation.
+          RequestId: <removed>
+          DateTimeStamp: <removed>
+          HttpStatusCode: Forbidden
+          HttpStatusDescription: Forbidden
+          HttpResponseStatus: Completed
+          WARNING: Custom locations has not been enabled on the AKS-HCI cluster. To enable custom locations manually, visit aka.ms/enable-custom-location
+          ```
+          
+          The current behavior of Arc onboarding is to enable custom locations by default. To enable custom locations, the **GetServicePrincipals** action is performed in the context of the logged-in Azure user. If the user (or SPN) does not have sufficient permissions to be able to do this, the command issues a warning that these permissions don't exist, and therefore the Custom Locations feature won't be enabled.
+          
+          If you don't want Custom Locations to be enabled, you can safely ignore this warning, as this does not affect cluster onboarding to Arc. On the other hand, if you do need Custom Locations to be enabled, you must grant the necessary permissions to the user (or SPN).
+
 
 additionalContent: |
   ## Next steps