Merge pull request #227082 from jiaochenlu/update-230102

Update Kubernetes changes

Author: v-regandowner

articles/machine-learning/how-to-attach-kubernetes-to-workspace.md

@@ -16,30 +16,57 @@ ms.custom: build-spring-2022, cliv2, sdkv2, event-tier1-build-2022
 
 Once AzureML extension is deployed on AKS or Arc Kubernetes cluster, you can attach the Kubernetes cluster to AzureML workspace and create compute targets for ML professionals to use. 
 
-Some key considerations when attaching Kubernetes cluster to AzureML workspace:
-  * If you need to access Azure resource securely from your training script, you can specify a [managed identity](./how-to-identity-based-service-authentication.md) for Kubernetes compute target during attach operation.
-  * If you plan to have different compute target for different project/team, you can specify Kubernetes namespace for the compute target to isolate workload among different teams/projects.
-  * For the same Kubernetes cluster, you can attach it to the same workspace multiple times and create multiple compute targets for different project/team/workload.
+## Prerequisites
+
+Attaching a Kubernetes cluster to AzureML workspace can flexibly support many different scenarios, such as the shared scenarios with multiple attachments, model training scripts accessing Azure resources, and the authentication configuration of the workspace. But you need to pay attention to the following prerequisites.
+
+#### Multi-attach and workload isolation
+
+**One cluster to one workspace, creating multiple compute targets**
+  * For the same Kubernetes cluster, you can attach it to the same workspace multiple times and create multiple compute targets for different projects/teams/workloads.
+
+**One cluster to multiple workspaces**
   * For the same Kubernetes cluster, you can also attach it to multiple workspaces, and the multiple workspaces can share the same Kubernetes cluster.
 
-### Prerequisite
 
-Azure Machine Learning workspace defaults to having a system-assigned managed identity to access Azure ML resources. The steps are completed if the system assigned default setting is on. 
+If you plan to have different compute targets for different projects/teams, you can specify the existed **Kubernetes namespace** in your cluster for the compute target to **isolate workload** among different teams/projects. 
+
+> [!IMPORTANT]
+>
+> The namespace you plan to specify when attaching the cluster to AzureML workspace should be previously created in your cluster.
+
+#### Securely access Azure resource from training script
 
+ If you need to access Azure resource securely from your training script, you can specify a [managed identity](./how-to-identity-based-service-authentication.md) for Kubernetes compute target during attach operation.
 
-Otherwise, if a user-assigned managed identity is specified in Azure Machine Learning workspace creation, the following role assignments need to be granted to the managed identity manually before attaching the compute.
+#### Attach to workspace with user-assigned managed identity
 
-|Azure resource name |Role to be assigned|Description|
+Azure Machine Learning workspace defaults to having a system-assigned managed identity to access Azure ML resources. The steps are completed if the system assigned default setting is on. 
+
+Otherwise, if a [user-assigned managed identity is specified in Azure Machine Learning workspace creation](../machine-learning/how-to-identity-based-service-authentication.md#user-assigned-managed-identity), the following role assignments need to be granted to the managed identity manually before attaching the compute.
+
+|Azure resource name |Roles to be assigned|Description|
 |--|--|--|
 |Azure Relay|Azure Relay Owner|Only applicable for Arc-enabled Kubernetes cluster. Azure Relay isn't created for AKS cluster without Arc connected.|
-|Kubernetes - Azure Arc or Azure Kubernetes Service|Reader|Applicable for both Arc-enabled Kubernetes cluster and AKS cluster.|
+|Kubernetes - Azure Arc or Azure Kubernetes Service|Reader <br> Kubernetes Extension Contributor <br> Azure Kubernetes Service Cluster Admin |Applicable for both Arc-enabled Kubernetes cluster and AKS cluster.|
 
-Azure Relay resource is created during the extension deployment under the same Resource Group as the Arc-enabled Kubernetes cluster.
 
+> [!TIP]
+>
+> Azure Relay resource is created during the extension deployment under the same Resource Group as the Arc-enabled Kubernetes cluster.
+
+> [!NOTE]
+>
+> * If the "Kubernetes Extension Contributor" role permission is not available, the cluster attachment fails with "extension not installed" error.
+> * If the "Azure Kubernetes Service Cluster Admin" role permission is not available, the cluster attachment fails with "internal server" error.
+
+## How to attach a Kubernetes cluster to AzureML workspace
+
+We support two ways to attach a Kubernetes cluster to AzureML workspace, using Azure CLI or studio UI.
 
 ### [Azure CLI](#tab/cli)
 
-[!INCLUDE [cli v2](../../includes/machine-learning-cli-v2.md)]
+[!INCLUDE [CLI v2](../../includes/machine-learning-CLI-v2.md)]
 
 The following commands show how to attach an AKS and Azure Arc-enabled Kubernetes cluster, and use it as a compute target with managed identity enabled.
 
@@ -58,6 +85,7 @@ az ml compute attach --resource-group <resource-group-name> --workspace-name <wo
 Set the `--type` argument to `Kubernetes`. Use the `identity_type` argument to enable `SystemAssigned` or `UserAssigned` managed identities.
 
 > [!IMPORTANT]
+>
 > `--user-assigned-identities` is only required for `UserAssigned` managed identities. Although you can provide a list of comma-separated user managed identities, only the first one is used when you attach your cluster.
 >
 > Compute attach won't create the Kubernetes namespace automatically or validate whether the kubernetes namespace existed. You need to verify that the specified namespace exists in your cluster, otherwise, any AzureML workloads submitted to this compute will fail.  
@@ -98,7 +126,7 @@ To access Azure Container Registry (ACR) for a Docker image, and a Storage Accou
 ### Assign managed identity
 - You can assign a managed identity to the compute in the compute attach step.
 - If the compute has already been attached, you can update the settings to use a managed identity in Azure Machine Learning studio.
-    - Go to [Azure Machine Learning studio](https://ml.azure.com). Select __Compute__, __Attached compute__, and select your attached compute.
+    - Go to [Azure Machine Learning studio](https://ml.azure.com). Select **Compute**, **Attached compute**, and select your attached compute.
     - Select the pencil icon to edit managed identity.
 
     :::image type="content" source="media/how-to-attach-kubernetes-to-workspace/edit-identity.png" alt-text="Screenshot of updating identity of the Kubernetes compute from Azure portal.":::
@@ -117,11 +145,11 @@ If you are using the Azure portal to assign roles and have a **system-assigned m
 
 If you have user-assigned managed identity, select **Managed identity** to find the target identity.
 
-You can use Managed Identity to pull images from Azure Container Registry. Grant the __AcrPull__ role to the compute Managed Identity. For more information, see [Azure Container Registry roles and permissions](../container-registry/container-registry-roles.md).
+You can use Managed Identity to pull images from Azure Container Registry. Grant the **AcrPull** role to the compute Managed Identity. For more information, see [Azure Container Registry roles and permissions](../container-registry/container-registry-roles.md).
 
 You can use a managed identity to access Azure Blob:
-- For read-only purpose, __Storage Blob Data Reader__ role should be granted to the compute managed identity.
-- For read-write purpose, __Storage Blob Data Contributor__ role should be granted to the compute managed identity.
+- For read-only purpose, **Storage Blob Data Reader** role should be granted to the compute managed identity.
+- For read-write purpose, **Storage Blob Data Contributor** role should be granted to the compute managed identity.
 
 ## Next steps
 

articles/machine-learning/how-to-deploy-kubernetes-extension.md

@@ -32,20 +32,20 @@ In this article, you can learn:
 * Or an Arc Kubernetes cluster is up and running. Follow instructions in [connect existing Kubernetes cluster to Azure Arc](../azure-arc/kubernetes/quickstart-connect-cluster.md).
   * If the cluster is an Azure RedHat OpenShift Service (ARO) cluster or OpenShift Container Platform (OCP) cluster, you must satisfy other prerequisite steps as documented in the [Reference for configuring Kubernetes cluster](./reference-kubernetes.md#prerequisites-for-aro-or-ocp-clusters) article.
 * For production purposes, the Kubernetes cluster must have a minimum of **4 vCPU cores and 14-GB memory**. For more information on resource detail and cluster size recommendations, see [Recommended resource planning](./reference-kubernetes.md).
-* Cluster running behind an outbound proxy server or firewall needs extra [network configurations](./how-to-access-azureml-behind-firewall.md).
+* Cluster running behind an **outbound proxy server** or **firewall** needs extra [network configurations](./how-to-access-azureml-behind-firewall.md#scenario-use-kubernetes-compute).
 * Install or upgrade Azure CLI to version 2.24.0 or higher.
 * Install or upgrade Azure CLI extension `k8s-extension` to version 1.2.3 or higher.
 
 
 ## Limitations
 
 - [Using a service principal with AKS](../aks/kubernetes-service-principal.md) is **not supported** by Azure Machine Learning. The AKS cluster must use a **managed identity** instead. Both **system-assigned managed identity** and **user-assigned managed identity** are supported. For more information, see [Use a managed identity in Azure Kubernetes Service](../aks/use-managed-identity.md).
+    -  When your AKS cluster used service principal is converted to use Managed Identity, before installing the extension, all node pools need to be deleted and recreated, rather than updated directly.
 - [Disabling local accounts](../aks/managed-aad.md#disable-local-accounts) for AKS is **not supported**  by Azure Machine Learning. When the AKS Cluster is deployed, local accounts are enabled by default.
 - If your AKS cluster has an [Authorized IP range enabled to access the API server](../aks/api-server-authorized-ip-ranges.md), enable the AzureML control plane IP ranges for the AKS cluster. The AzureML control plane is deployed across paired regions. Without access to the API server, the machine learning pods can't be deployed. Use the [IP ranges](https://www.microsoft.com/download/confirmation.aspx?id=56519) for both the [paired regions](../availability-zones/cross-region-replication-azure.md) when enabling the IP ranges in an AKS cluster.
+- Azure Machine Learning does not support attaching an AKS cluster cross subscription. If you have an AKS cluster in a different subscription, you must first [connect it to Azure-Arc](../azure-arc/kubernetes/quickstart-connect-cluster.md) and specify in the same subscription as your Azure Machine Learning workspace.
 - Azure Machine Learning does not guarantee support for all preview stage features in AKS. For example, [Azure AD pod identity](../aks/use-azure-ad-pod-identity.md) is not supported.
 - If you've previously followed the steps from [AzureML AKS v1 document](./v1/how-to-create-attach-kubernetes.md) to create or attach your AKS as inference cluster, use the following link to [clean up the legacy azureml-fe related resources](./v1/how-to-create-attach-kubernetes.md#delete-azureml-fe-related-resources) before you continue the next step.
-- We currently don't support attaching your AKS cluster across subscription, which means that your AKS cluster must be in the same subscription as your workspace. 
-   - The workaround to meet your cross-subscription requirement is to first connect AKS to Azure-ARC and then attach this ARC-Kubernetes resource.
 
 
 ## Review AzureML extension configuration settings