Merge pull request #205478 from CocoWang-wql/patch-6

liljack17 · web-flow · commit 7b2f4333a971 · 2022-08-02T09:08:27.000-07:00
KMS GA doc.
diff --git a/articles/aks/use-kms-etcd-encryption.md b/articles/aks/use-kms-etcd-encryption.md
@@ -1,77 +1,50 @@
 ---
-title: Use KMS etcd encryption in Azure Kubernetes Service (AKS) (Preview)
+title: Use KMS etcd encryption in Azure Kubernetes Service (AKS) 
 description: Learn how to use kms etcd encryption with Azure Kubernetes Service (AKS)
 services: container-service
 ms.topic: article
-ms.date: 06/06/2022
+ms.date: 07/26/2022
 
 ---
 
-# Add KMS etcd encryption to an Azure Kubernetes Service (AKS) cluster (Preview)
+# Add KMS etcd encryption to an Azure Kubernetes Service (AKS) cluster 
 
 This article shows you how to enable encryption at rest for your Kubernetes data in etcd using Azure Key Vault with Key Management Service (KMS) plugin. The KMS plugin allows you to:
 
 * Use a key in Key Vault for etcd encryption
 * Bring your own keys
 * Provide encryption at rest for secrets stored in etcd
+* Rotate the keys in Key Vault
 
 For more information on using the KMS plugin, see [Encrypting Secret Data at Rest](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/).
 
-[!INCLUDE [preview features callout](./includes/preview/preview-callout.md)]
-
 ## Before you begin
 
 * An Azure subscription. If you don't have an Azure subscription, you can create a [free account](https://azure.microsoft.com/free).
-* [Azure CLI installed](/cli/azure/install-azure-cli).
-
-### Install the `aks-preview` Azure CLI
-
-You also need the *aks-preview* Azure CLI extension version 0.5.58 or later. Install the *aks-preview* Azure CLI extension by using the [az extension add][az-extension-add] command. Or install any available updates by using the [az extension update][az-extension-update] command.
-
-```azurecli-interactive
-# Install the aks-preview extension
-az extension add --name aks-preview
-# Update the extension to make sure you have the latest version installed
-az extension update --name aks-preview
-```
-
-### Register the `AzureKeyVaultKmsPreview` preview feature
-
-To use the feature, you must also enable the `AzureKeyVaultKmsPreview` feature flag on your subscription.
-
-Register the `AzureKeyVaultKmsPreview` feature flag by using the [az feature register][az-feature-register] command, as shown in the following example:
+* Azure CLI version 2.39.0 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][install-azure-cli].
 
-```azurecli-interactive
-az feature register --namespace "Microsoft.ContainerService" --name "AzureKeyVaultKmsPreview"
-```
-
-It takes a few minutes for the status to show *Registered*. Verify the registration status by using the [az feature list][az-feature-list] command:
-
-```azurecli-interactive
-az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AzureKeyVaultKmsPreview')].{Name:name,State:properties.state}"
-```
-
-When ready, refresh the registration of the *Microsoft.ContainerService* resource provider by using the [az provider register][az-provider-register] command:
-
-```azurecli-interactive
-az provider register --namespace Microsoft.ContainerService
-```
+> [!WARNING]
+> KMS only supports Konnectivity. You could use `kubectl get po -n kube-system` to check whether there is 'konnectivity-agent-xxx' pod running. 
 
 ## Limitations
 
 The following limitations apply when you integrate KMS etcd encryption with AKS:
 
-* Disabling of the KMS etcd encryption feature.
-* Changing of key ID, including key name and key version.
 * Deletion of the key, Key Vault, or the associated identity.
 * KMS etcd encryption doesn't work with System-Assigned Managed Identity. The keyvault access-policy is required to be set before the feature is enabled. In addition, System-Assigned Managed Identity isn't available until cluster creation, thus there's a cycle dependency.
 * Using more than 2000 secrets in a cluster.
 * Bring your own (BYO) Azure Key Vault from another tenant.
+* Change associated Azure Key Vault model (public, private) if KMS is enabled. For [changing associated key vault mode][changing-associated-key-vault-mode], you need to disable and enable KMS again.
+* Stop/satrt cluster which is enabled KMS with private key vault.
+
+KMS supports [public key vault][Enable-KMS-with-public-key-vault] and [private key vault][Enable-KMS-with-private-key-vault] now. 
 
-## Create a KeyVault and key
+## Enable KMS with public key vault
+
+### Create a key vault and key
 
 > [!WARNING]
-> Deleting the key or the Azure Key Vault is not supported and will cause your cluster to become unstable.
+> Deleting the key or the Azure Key Vault is not supported and will cause the secrets to be unrecoverable in the cluster.
 > 
 > If you need to recover your Key Vault or key, see the [Azure Key Vault recovery management with soft delete and purge protection](../key-vault/general/key-vault-recovery.md?tabs=azure-cli) documentation.
 
@@ -96,7 +69,7 @@ echo $KEY_ID
 
 The above example stores the Key ID in *KEY_ID*.
  
-## Create a user-assigned managed identity
+### Create a user-assigned managed identity
 
 Use `az identity create` to create a User-assigned managed identity.
 
@@ -122,28 +95,28 @@ echo $IDENTITY_RESOURCE_ID
 
 The above example stores the value of the Identity Resource ID in *IDENTITY_RESOURCE_ID*.
 
-## Assign permissions (decrypt and encrypt) to access key vault
+### Assign permissions (decrypt and encrypt) to access key vault
 
 Use `az keyvault set-policy` to create an Azure KeyVault policy.
 
 ```azurecli-interactive
 az keyvault set-policy -n MyKeyVault --key-permissions decrypt encrypt --object-id $IDENTITY_OBJECT_ID
 ```
 
-## Create an AKS cluster with KMS etcd encryption enabled
+### Create an AKS cluster with KMS etcd encryption enabled
 
-Create an AKS cluster using the [az aks create][az-aks-create] command with the `--enable-azure-keyvault-kms` and `--azure-keyvault-kms-key-id` parameters to enable KMS etcd encryption.
+Create an AKS cluster using the [az aks create][az-aks-create] command with the `--enable-azure-keyvault-kms`, `--azure-keyvault-kms-key-vault-network-access` and `--azure-keyvault-kms-key-id` parameters to enable KMS etcd encryption.
 
 ```azurecli-interactive
-az aks create --name myAKSCluster --resource-group MyResourceGroup --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID
+az aks create --name myAKSCluster --resource-group MyResourceGroup --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-vault-network-access "Public" --azure-keyvault-kms-key-id $KEY_ID
 ```
 
-## Update an exiting AKS cluster to enable KMS etcd encryption
+### Update an exiting AKS cluster to enable KMS etcd encryption
 
-Use [az aks update][az-aks-update] with the `--enable-azure-keyvault-kms` and `--azure-keyvault-kms-key-id` parameters to enable KMS etcd encryption on an existing cluster.
+Use [az aks update][az-aks-update] with the `--enable-azure-keyvault-kms`, `--azure-keyvault-kms-key-vault-network-access` and `--azure-keyvault-kms-key-id` parameters to enable KMS etcd encryption on an existing cluster.
 
 ```azurecli-interactive
-az aks update --name myAKSCluster --resource-group MyResourceGroup --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID
+az aks update --name myAKSCluster --resource-group MyResourceGroup --enable-azure-keyvault-kms --azure-keyvault-kms-key-vault-network-access "Public" --azure-keyvault-kms-key-id $KEY_ID
 ```
 
 Use below command to update all secrets. Otherwise, the old secrets aren't encrypted. 
@@ -152,6 +125,182 @@ Use below command to update all secrets. Otherwise, the old secrets aren't encry
 kubectl get secrets --all-namespaces -o json | kubectl replace -f -
 ```
 
+> [!NOTE]
+> For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
+
+### Rotate the existing keys 
+After changing the key ID (including key name and key version), you could use [az aks update][az-aks-update] with the `--enable-azure-keyvault-kms`, `--azure-keyvault-kms-key-vault-network-access` and `--azure-keyvault-kms-key-id` parameters to rotate the exitsing keys of KMS.
+
+> [!WARNING]
+> Remember to update all secrets after key rotation. Otheriwse, the secrets will be unaccessable if the old keys are not existing or working.
+
+```azurecli-interactive
+az aks update --name myAKSCluster --resource-group MyResourceGroup  --enable-azure-keyvault-kms --azure-keyvault-kms-key-vault-network-access "Public" --azure-keyvault-kms-key-id $NEW_KEY_ID 
+```
+
+Use below command to update all secrets. Otherwise, the old secrets are still encrypted with the previous key. 
+
+```azurecli-interactive
+kubectl get secrets --all-namespaces -o json | kubectl replace -f -
+```
+
+> [!NOTE]
+> For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
+
+## Enable KMS with private key vault
+
+If you enable KMS with private key vault, AKS will create a private endpoint and private link in the node resource group automatically. The key vault will be added a private endpoint connection with the AKS cluster.
+
+### Create a private key vault and key
+
+> [!WARNING]
+> Deleting the key or the Azure Key Vault is not supported and will cause the secrets to be unrecoverable in the cluster.
+> 
+> If you need to recover your Key Vault or key, see the [Azure Key Vault recovery management with soft delete and purge protection](../key-vault/general/key-vault-recovery.md?tabs=azure-cli) documentation.
+
+
+Use `az keyvault create` to create a priate KeyVault.
+
+```azurecli
+az keyvault create --name MyKeyVault --resource-group MyResourceGroup --public-network-access Disabled
+```
+
+Without private endpoint, it's not supported to create or update keys in private key vault. To manage private key vault, you could refer to [Integrate Key Vault with Azure Private Link](../key-vault/general/private-link-service.md).
+
+### Create a user-assigned managed identity
+
+Use `az identity create` to create a User-assigned managed identity.
+
+```azurecli
+az identity create --name MyIdentity --resource-group MyResourceGroup
+```
+
+Use `az identity show` to get Identity Object ID.
+
+```azurecli
+IDENTITY_OBJECT_ID=$(az identity show --name MyIdentity --resource-group MyResourceGroup --query 'principalId' -o tsv)
+echo $IDENTITY_OBJECT_ID
+```
+
+The above example stores the value of the Identity Object ID in *IDENTITY_OBJECT_ID*.
+
+Use `az identity show` to get Identity Resource ID.
+
+```azurecli
+IDENTITY_RESOURCE_ID=$(az identity show --name MyIdentity --resource-group MyResourceGroup --query 'id' -o tsv)
+echo $IDENTITY_RESOURCE_ID
+```
+
+The above example stores the value of the Identity Resource ID in *IDENTITY_RESOURCE_ID*.
+
+### Assign permissions (decrypt and encrypt) to access key vault
+
+Use `az keyvault set-policy` to create an Azure KeyVault policy.
+
+```azurecli-interactive
+az keyvault set-policy -n MyKeyVault --key-permissions decrypt encrypt --object-id $IDENTITY_OBJECT_ID
+```
+
+For private key vault, the AKS needs *Key Vault Contributor*  role to create private link between private key vault and cluster.
+
+```azurecli-interactive
+az role assignment create --role "Key Vault Contributor" --assignee-object-id $IDENTITY_OBJECT_ID --assignee-principal-type "ServicePrincipal" --scope $KEYVAULT_RESOURCE_ID
+```
+
+### Create an AKS cluster with private key vault and enable KMS etcd encryption 
+
+Create an AKS cluster using the [az aks create][az-aks-create] command with the `--enable-azure-keyvault-kms`, `--azure-keyvault-kms-key-id`, `--azure-keyvault-kms-key-vault-network-access` and `--azure-keyvault-kms-key-vault-resource-id` parameters to enable KMS etcd encryption with private key vault.
+
+```azurecli-interactive
+az aks create --name myAKSCluster --resource-group MyResourceGroup --assign-identity $IDENTITY_RESOURCE_ID --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID
+```
+
+### Update an exiting AKS cluster to enable KMS etcd encryption with private key vault
+
+Use [az aks update][az-aks-update] with the `--enable-azure-keyvault-kms`, `--azure-keyvault-kms-key-id`, `--azure-keyvault-kms-key-vault-network-access` and `--azure-keyvault-kms-key-vault-resource-id` parameters to enable KMS etcd encryption on an existing cluster with private key vault.
+
+```azurecli-interactive
+az aks update --name myAKSCluster --resource-group MyResourceGroup --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $KEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID
+```
+
+Use below command to update all secrets. Otherwise, the old secrets aren't encrypted. 
+
+```azurecli-interactive
+kubectl get secrets --all-namespaces -o json | kubectl replace -f -
+```
+
+> [!NOTE]
+> For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
+
+### Rotate the existing keys 
+After changing the key ID (including key name and key version), you could use [az aks update][az-aks-update] with the `--enable-azure-keyvault-kms`, `--azure-keyvault-kms-key-id`, `--azure-keyvault-kms-key-vault-network-access` and `--azure-keyvault-kms-key-vault-resource-id` parameters to rotate the existing keys of KMS.
+
+> [!WARNING]
+> Remember to update all secrets after key rotation. Otherwise, the secrets will be unaccessable if the old keys are not existing or working.
+
+```azurecli-interactive
+az aks update --name myAKSCluster --resource-group MyResourceGroup  --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $NewKEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID
+```
+
+Use below command to update all secrets. Otherwise, the old secrets are still encrypted with the previous key. 
+
+```azurecli-interactive
+kubectl get secrets --all-namespaces -o json | kubectl replace -f -
+```
+
+> [!NOTE]
+> For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
+
+## Update key vault mode
+
+> [!NOTE]
+> To change a different key vault with different mode (public, private), you could run `az aks update` directly. To change the mode of attached key vault, you need to diable KMS and re-enable it with new key vault ids. 
+
+Below are the steps about how to migrate the attached public key vault to private mode.
+
+### Disable KMS on the cluster
+
+Use below command to disable the KMS on existing cluster and release the key vault.
+
+```azurecli-interactive
+az aks update --name myAKSCluster --resource-group MyResourceGroup --disable-azure-keyvault-kms
+```
+
+### Change key vault mode
+
+Update the key vault from public to private.
+
+```azurecli-interactive
+az keyvault update --name MyKeyVault --resource-group MyResourceGroup --public-network-access Disabled
+```
+
+### Enable KMS on the cluster with updated key vault
+
+Use below command to re-enable the KMS with updated private key vault.
+
+```azurecli-interactive
+az aks update --name myAKSCluster --resource-group MyResourceGroup  --enable-azure-keyvault-kms --azure-keyvault-kms-key-id $NewKEY_ID --azure-keyvault-kms-key-vault-network-access "Private" --azure-keyvault-kms-key-vault-resource-id $KEYVAULT_RESOURCE_ID
+```
+
+After configuring KMS, you could enable [diagnostic-settings for key vault to check the encryption logs](../key-vault/general/howto-logging.md).
+
+## Disable KMS
+
+Use below command to disable KMS on existing cluster. 
+
+```azurecli-interactive
+az aks update --name myAKSCluster --resource-group MyResourceGroup --disable-azure-keyvault-kms
+```
+
+Use below command to update all secrets. Otherwise, the old secrets are still encrypted with the previous key. 
+
+```azurecli-interactive
+kubectl get secrets --all-namespaces -o json | kubectl replace -f -
+```
+
+> [!NOTE]
+> For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
+
 <!-- LINKS - Internal -->
 [aks-support-policies]: support-policies.md
 [aks-faq]: faq.md
@@ -167,3 +316,7 @@ kubectl get secrets --all-namespaces -o json | kubectl replace -f -
 [az-feature-list]: /cli/azure/feature#az_feature_list
 [az-provider-register]: /cli/azure/provider#az_provider_register
 [az-aks-update]: /cli/azure/aks#az_aks_update
+[Enable-KMS-with-public-key-vault]: use-kms-etcd-encryption.md#enable-kms-with-public-key-vault
+[Enable-KMS-with-private-key-vault]: use-kms-etcd-encryption.md#enable-kms-with-private-key-vault
+[changing-associated-key-vault-mode]: use-kms-etcd-encryption.md#update-key-vault-mode
+[install-azure-cli]: /cli/azure/install-azure-cli
