Merge pull request #3554 from MicrosoftDocs/main638773275454263201sync_temp

For protected branch, push strategy should use PR and merge to target branch method to work around git push error

Author: learn-build-service-prod[bot]

AKS-Arc/TOC.yml

@@ -222,14 +222,20 @@
       href: aks-edge-howto-deploy-azure-iot.md
     - name: Offline installation
       href: aks-edge-howto-offline-install.md
-    - name: Access TPM secrets
-      href: aks-edge-howto-access-tpm.md
     - name: Additional configuration
       href: aks-edge-howto-more-configs.md
     - name: Use GPU acceleration
       href: aks-edge-gpu.md
-    - name: Configure Workload Identity
-      href: aks-edge-workload-identity.md
+    - name: Security
+      items:
+      - name: Configure Workload Identity
+        href: aks-edge-workload-identity.md      
+      - name: Access TPM secrets
+        href: aks-edge-howto-access-tpm.md
+      - name: Enable secret encryption with the KMS plugin
+        href: aks-edge-howto-secret-encryption.md
+      - name: Use the Key Manager for Kubernetes extension
+        href: aks-edge-howto-key-manager.md        
     - name: Update AKS Edge Essentials
       items:
       - name: Update online

AKS-Arc/aks-edge-concept-networking.md

@@ -4,7 +4,7 @@ description: Basic networking concepts for AKS Edge Essentials
 author: sethmanheim
 ms.author: sethm
 ms.topic: conceptual
-ms.date: 07/11/2024
+ms.date: 03/10/2025
 ms.custom: template-concept
 ---
 
@@ -99,7 +99,7 @@ It's possible to check the DNS servers being used for both Linux and Windows nod
     Invoke-AksEdgeNodeCommand -NodeType Linux -command "resolvectl status"
     ```
 
-    The command output shows a list of the DNS servers configured for each Linux VM interfaces. In particular, it's important to check the **eth0** interface status, which is the default interface for the AKS EE VM communication. Also, make sure to check the IP addresses of the **Current DNS Server** and **DNS Servers** fields of the list. If there's no IP address, or the IP address isn't a valid DNS server IP address, then the DNS service won't work.
+    The command output shows a list of the DNS servers configured for each Linux VM interfaces. In particular, it's important to check the **eth0** interface status, which is the default interface for the AKS Edge Essentials VM communication. Also, make sure to check the IP addresses of the **Current DNS Server** and **DNS Servers** fields of the list. If there's no IP address, or the IP address isn't a valid DNS server IP address, then the DNS service won't work.
 
 - For Windows VM nodes:
 

AKS-Arc/aks-edge-deployment-config-json.md

@@ -4,7 +4,7 @@ description: Description of deployment configuration JSON parameters in AKS Edge
 author: sethmanheim
 ms.author: sethm
 ms.topic: conceptual
-ms.date: 07/11/2024
+ms.date: 03/10/2025
 ms.custom: template-concept
 ---
 
@@ -21,6 +21,7 @@ You can find the complete JSON schema file at `C:\Program Files\AksEdge\aksedge-
 | `DeploymentType` |[`SingleMachineCluster` / `ScalableCluster`]| Specifies deployment type. In `ScalableCluster`, you can add more machines to the cluster infrastructure. | `SingleMachineCluster` |Single-machine and full deployment|
 | `Init.ServiceIPRangeStart` |IPv4 address `A.B.C.x`.|Reserved IP start address for your Kubernetes services. This IP range must be free on your subnet **A.B.C.0**.| None |Single-machine and full deployment|
 | `Init.ServiceIPRangeSize` |`[0-127]`|Number of reserved IP start addresses for your Kubernetes services. Based on the size, we allocate a range of free IP addresses on your subnet. | `0` |Single-machine and full deployment|
+| `Init.KmsPlugin.Enable` | Boolean | Specifies whether the KMS plugin is enabled or not. | false | Single-machine and full deployment|
 | `Join.ClusterJoinToken` |String|`Reserved` | None |Full deployment only|
 | `Join.DiscoveryTokenHash` |String|`Reserved`| None |Full deployment only|
 | `Join.CertificateKey` |String|`Reserved`| None |Full deployment only|

AKS-Arc/aks-edge-howto-connect-to-arc.md

@@ -4,14 +4,17 @@ description: Connect your AKS Edge Essentials clusters to Arc
 author: sethmanheim
 ms.author: sethm
 ms.topic: how-to
-ms.date: 09/27/2024
+ms.date: 03/10/2025
 ms.custom: template-how-to
 ---
 
 # Connect your AKS Edge Essentials cluster to Arc
 
 This article describes how to connect your AKS Edge Essentials cluster to [Azure Arc](/azure/azure-arc/kubernetes/overview) so that you can monitor the health of your cluster on the Azure portal. If your cluster is connected to a proxy, you can use the scripts provided in the GitHub repo to connect your cluster to Arc [as described here](./aks-edge-howto-more-configs.md).
 
+> [!IMPORTANT]
+> Starting with the AKS Edge Essentials 1.10.868.0 release, the `Arc` section of the config file is required. The Azure Arc connection occurs automatically after you run `New-AksEdgeDeployment` to deploy an AKS Edge Essentials cluster.
+
 ## Prerequisites
 
 - Before connecting to Arc, infrastructure administrators who are the owner or contributor role of the subscription will have to:

AKS-Arc/aks-edge-howto-key-manager.md

@@ -0,0 +1,116 @@
+---
+title: Use Key Manager for Kubernetes clusters on AKS Edge Essentials (preview)
+description: Learn how to use the Key Manager for Kubernetes extension to rotate service account keys in Azure Kubernetes Service (AKS) Edge Essentials clusters.
+ms.topic: how-to
+author: sethmanheim
+ms.author: sethm
+ms.date: 03/10/2025
+ms.reviewer: leslielin
+---
+
+# How-to: use Key Manager for Kubernetes on an AKS Edge Essentials cluster (preview)
+
+The [Kubernetes service account](https://kubernetes.io/docs/concepts/security/service-accounts/) is a a non-human account that provides a unique identity within a Kubernetes cluster. Service account tokens serve important security and authentication functions in Kubernetes.
+
+In AKS Edge Essentials, *service account tokens* enable workload pods to authenticate and access Azure resources through [*workload identity federation*](aks-edge-workload-identity.md). Key Manager for Kubernetes is an Azure Arc extension that automates the rotation of the signing key used to issue these service account tokens. The rotation reduces the risk of token misuse and improves the overall security posture of the cluster.
+
+The following table compares the default behavior with and without using the Key Manager for Kubernetes extension:
+
+|   Behavior                              | By default, without the Key Manager extension                                                                                                | With the Key Manager extension                                                        |
+|----------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|
+| Automated service account key rotation | By default, Kubernetes doesn't automatically rotate service account signing keys. Instead, it uses the same key indefinitely to sign tokens. | Once enabled, the service account signing key is rotated automatically every 45 days. |
+| Service account signing key validity   | Unlimited                                                                                                                                    | 90 days                                                                               |
+
+> [!IMPORTANT]
+> These preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. Azure Kubernetes Service Edge Essentials previews are partially covered by customer support on a best-effort basis.
+
+> [!NOTE]
+> During the preview, the Key Manager for Kubernetes is only available for AKS Edge Essentials K3s version 1.30.6 or later single control plane node deployments with Arc connectivity. It's not compatible with other Arc-enabled Kubernetes distributions.
+
+## Before you begin
+
+Before you begin, ensure you have the following prerequisites:
+
+- An AKS Edge Essentials K3s cluster with Arc connectivity. If you plan to use Azure IoT Operations with AKS Edge Essentials, follow this [Quickstart guide](aks-edge-howto-deploy-azure-iot.md#create-an-arc-enabled-cluster) to create your cluster.
+- To enable TLS for intracluster log communication, the `cert-manager` and `trust-manager` tools are required.
+  - If you plan to [use Azure IoT Operations](/azure/iot-operations/get-started-end-to-end-sample/quickstart-deploy#deploy-azure-iot-operations), deploy it before installing the Key Manager for Kubernetes extension, since Azure IoT Operations installs its own copy of these applications by default.
+  - To verify if `cert-manager` and `trust-manager` are installed, run the following command::
+
+    ```bash
+    kubectl get pods -n cert-manager
+    ```
+
+    If they are installed, you can see their pods in a running state.
+
+  - If `cert-manager` and `trust-manager` are not present, follow the documentation to:
+
+    1. Install [cert-manager](https://cert-manager.io/docs/installation/).
+    1. Install [trust-manager](https://cert-manager.io/docs/trust/trust-manager/installation/). While installing trust manager, set the `trust namespace` to `cert-manager`. For example:
+
+       ```bash
+       helm upgrade trust-manager jetstack/trust-manager --install --namespace cert-manager --set app.trust.namespace=cert-manager --wait
+       ```
+
+       `trust-manager` is used to distribute a trust bundle to components.
+
+- The Key Manager extension only works with [bounded service account tokens](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#manual-secret-management-for-serviceaccounts). It doesn't support legacy tokens with infinite lifetimes. If your workflow relies on legacy tokens, do not install this extension.
+- Bounded service account tokens have a default lifetime of one year. To rotate these tokens, this lifetime should be reduced to one day, which ensures that tokens are rapidly reissued and signed with newly rotated keys. To implement these changes, you must modify the `api-server` configuration by running the following commands:
+
+  ```powershell
+  $url = "https://raw.githubusercontent.com/Azure/AKS-Edge/refs/heads/main/tools/scripts/AksEdgeKeyManagerExtension/UpdateK3sConfigForKeyManager.ps1"
+  Invoke-WebRequest -Uri $url -OutFile .\UpdateK3sConfigForKeyManager.ps1
+  Unblock-File .\UpdateK3sConfigForKeyManager.ps1
+  Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process -Force
+   
+  .\UpdateK3sConfigForKeyManager.ps1
+  ```
+
+## Install the Key Manager for Kubernetes extension for service account key rotation
+
+> [!IMPORTANT]
+> After you install the Key Manager extension, the `api-server` is updated with the new service account token during token rotation. This process briefly makes the API server inaccessible while it restarts.
+
+Now run the following commands. Replace the variables with your specific resource group name and AKS cluster name:
+
+# [Azure PowerShell](#tab/powershell)
+
+```powershell
+New-AzKubernetesExtension -SubscriptionId $yoursubscriptionID -ResourceGroupName $resource_group_name -ClusterName $aks_cluster_name -ClusterType connectedClusters -Name "MySAkeymanager" -ExtensionType microsoft.arc.kuberneteskeymanager
+```
+
+# [Azure CLI](#tab/azurecli)
+
+```azurecli
+az k8s-extension create -g $resource_group_name -c $aks_cluster_name -t connectedClusters -n "MySAkeymanager" --extension-type microsoft.arc.kuberneteskeymanager 
+```
+
+---
+
+After you install the extension, you can view the **MySAkeymanager** extension in the Azure portal under the **Settings/Extensions** section of your Kubernetes cluster.
+
+## Remove key manager for Kubernetes extension
+
+You can uninstall the Key Manager extension using the [az k8s-extension delete](/cli/azure/k8s-extension#az-k8s-extension-delete) command:
+
+# [Azure PowerShell](#tab/powershell)
+
+```powershell
+Remove-AzKubernetesExtension -ResourceGroupName $resource_group_name -ClusterName $aks_cluster_name -ClusterType connectedClusters -Name "MySAkeymanager"
+```
+
+# [Azure CLI](#tab/azurecli)
+
+```azurecli
+az k8s-extension delete -g $resource_group_name -c $aks_cluster_name -t connectedClusters -n "MySAkeymanager"
+```
+
+---
+
+## Working with Azure IoT Operations
+
+If you installed the Key Manager for Kubernetes extension before you deployed Azure IoT Operations, you must follow the [Bring your own issuer](/azure/iot-operations/secure-iot-ops/concept-default-root-ca#bring-your-own-issuer) instructions, as Azure IoT Operations installs `cert-manager` and `trust-manager` by default.
+
+## Next steps
+
+- [Deploy and configure workload identity on an AKS cluster](/azure/aks/workload-identity-deploy-cluster)
+- [Configure Workload Identity on an AKS Edge Essentials cluster](aks-edge-workload-identity.md)

AKS-Arc/aks-edge-howto-multi-node-deployment.md

@@ -4,7 +4,7 @@ description: Describes how to create a cluster with multiple machines in AKS Edg
 author: sethmanheim
 ms.author: sethm
 ms.topic: how-to
-ms.date: 01/09/2025
+ms.date: 03/10/2025
 ms.custom: template-how-to
 ---
 
@@ -52,6 +52,11 @@ The key parameters to note for a scalable Kubernetes deployment are:
 > [!IMPORTANT]
 > The Kubernetes `pod cidr` is `10.42.0.0/16` for K3s and `10.244.0.0/24` for K8s. The Kubernetes `service cidr` is `10.43.0.0/16` for K3s and `10.96.0.0/12` for K8s.
 
+- `Arc`: This section is required. During the AKS Edge Essentials deployment, the Arc parameters are used to connect the AKS Edge Essentials cluster to Azure Arc. For more information about the required Arc parameters, see the [connect to Arc documentation](aks-edge-howto-connect-to-arc.md).
+  
+> [!IMPORTANT]
+> Starting with the AKS Edge Essentials 1.10.868.0 release, the `Arc` section of the config file is required. The Azure Arc connection occurs automatically during AKS Edge Essentials deployment.
+  
 - The `Network.NetworkPlugin` value by default is `flannel`. Flannel is the default CNI for a K3S cluster. In a K8S cluster, change the `NetworkPlugin` to `calico`.
 - In addition to the previous parameters, you can set the following parameters according to your deployment configuration, [as described here](aks-edge-deployment-config-json.md): `LinuxNode.CpuCount`, `LinuxNode.MemoryInMB`, `LinuxNode.DataSizeInGB`, `LinuxNode.Ip4Address`, `WindowsNode.CpuCount`, `WindowsNode.MemoryInMB`, `WindowsNode.Ip4Address`, `Init.ServiceIPRangeSize`, and `Network.InternetDisabled`.
 

AKS-Arc/aks-edge-howto-secret-encryption.md

@@ -0,0 +1,100 @@
+---
+title: Enable secret encryption on an AKS Edge Essentials cluster (preview)
+description: Learn how to enable the KMS plugin for AKS Edge Essentials clusters to encrypt secrets.
+author: sethmanheim
+ms.author: sethm
+ms.topic: how-to
+ms.date: 03/11/2025
+ms.custom: template-how-to
+ms.reviewer: leslielin
+---
+
+# Enable secret encryption on an AKS Edge Essentials cluster (preview)
+
+Following Kubernetes security best practices, it's recommended that you encrypt the Kubernetes secret store on AKS Edge Essentials clusters. You can perform this encryption by activating the *Key Management Service (KMS) plugin for AKS Edge Essentials*, which enables [encryption at rest for secrets](https://kubernetes.io/docs/concepts/configuration/secret/) stored in the etcd key-value store. It enables this encryption by generating a Key Encryption Key (KEK) and automatically rotating it every 30 days. The KEK is protected with administrator credentials and is accessible only to administrators.
+
+For more detailed information about using KMS, see the official [KMS provider documentation](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/).
+
+This article demonstrates how to activate the KMS plugin for AKS Edge Essentials clusters.
+
+> [!IMPORTANT]
+> The KMS plugin for AKS Edge Essentials is currently in PREVIEW. See the [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/) for legal terms that apply to Azure features that are in beta, preview, or otherwise not yet released into general availability.
+
+## Prerequisites
+
+The KMS plugin is supported for all AKS Edge Essentials clusters, version 1.10.868.0 and later.
+
+> [!NOTE]
+> The KMS plugin can only be used for single node clusters. The plugin can't be used with [experimental features, such as multi-node](aks-edge-system-requirements.md#experimental-or-prerelease-features).
+
+## Enable the KMS plugin
+
+In your [**aksedge-config.json** file](aks-edge-deployment-config-json.md), in the `Init` section, set `Init.KmsPlugin.Enable` to `true`:
+
+```json
+"Init": {
+ "KmsPlugin": {
+     "Enable": true
+  }
+}
+```
+
+The following output is displayed during deployment, showing that the KMS plugin is enabled:
+
+```output
+Preparing to install kms-plugin as encryption provider...
+```
+
+For deployment instructions, see [Single machine deployment](aks-edge-howto-single-node-deployment.md).
+
+> [!NOTE]
+> You can only enable or disable the KMS plugin when you create a new deployment. Once you set the flag, it can't be changed.
+
+## Verify that the KMS plugin is enabled
+
+To verify that the KMS plugin is enabled, run the following command and ensure that the health status of **kms-providers** is **OK**:
+
+```powershell
+kubectl get --raw='/readyz?verbose'
+```
+
+```output
+[+]ping ok
+[+]Log ok
+[+]etcd ok
+[+]kms-providers ok
+[+]poststarthook/start-encryption-provider-config-automatic-reload ok
+```
+
+To create secrets in AKS Edge Essentials clusters, see [Managing Secrets using kubectl](https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/#use-raw-data) in the Kubernetes documentation.
+
+If you encounter errors, see the [Troubleshooting](#troubleshooting) section.
+
+## Troubleshooting
+
+If there are errors with the KMS plugin, follow this procedure:
+
+1. Check that the AKS version is **1.10.868.0** or later. Use the following command to check the current version of AKS Edge Essentials:
+
+   ```powershell
+   Get-Command -Module AKSEdge | Format-Table Name, Version
+   ```
+
+   If the version is older, upgrade to the latest version. For more information, see [Upgrade an AKS cluster](aks-edge-howto-update.md).
+
+1. View the `readyz` API. If the problem persists, verify that the KMS plugin is enabled. See the [Verify that the KMS plugin is enabled](#verify-that-the-kms-plugin-is-enabled) section.
+
+   If you receive "**[-]**" before the `kms-providers` field, collect diagnostic logs for debugging. For more information, see [Get kubelet logs from cluster nodes](aks-get-kubelet-logs.md).
+
+1. Repair KMS. If there are still errors, the machine running the AKS Edge Essentials cluster might be paused or turned off for an extended period of time (over 30 days). To get KMS back into a healthy state, you can use the `Repair-Kms` command to restore any necessary tokens:
+
+   ```powershell
+   Repair-AksEdgeKms
+   ```
+
+1. If you still encounter errors, contact [Microsoft Customer Support](aks-edge-troubleshoot-overview.md) and [collect logs](aks-get-kubelet-logs.md).
+
+## Next steps
+
+- [Overview](aks-edge-overview.md)
+- [Uninstall AKS cluster](aks-edge-howto-uninstall.md)