AKS Arc: add article on encrypt etcd secrets

Author: sethmanheim

AKS-Arc/TOC.yml

@@ -76,6 +76,10 @@
           href: deploy-load-balancer-portal.md
         # - name: Troubleshoot issues
         #   href: load-balancer-troubleshoot.md
+    - name: Security
+      items:
+      - name: Encrypt etcd secrets
+        href: encrypt-etcd-secrets.md
     - name: AI and Machine Learning
       items:
       - name: Deploy an AI model with the AI toolchain operator

AKS-Arc/encrypt-etcd-secrets.md

@@ -0,0 +1,95 @@
+---
+title: Encrypt etcd secrets for Kubernetes clusters in AKS on Azure Local
+description: Learn how to encrypt etcd secrets in AKS on Azure Local.
+author: sethmanheim
+ms.topic: how-to
+ms.date: 04/10/2025
+ms.author: sethm 
+ms.lastreviewed: 04/10/2025
+ms.reviewer: aathipsa
+# Intent: As an IT Pro, I want to learn about encrypted etcd secrets and how they are used in my AKS deployment. 
+# Keyword: etcd secrets AKS Windows Server
+
+---
+
+# How to: Encrypt etcd secrets for Kubernetes clusters in AKS on Azure Local
+
+[!INCLUDE [hci-applies-to-23h2](includes/hci-applies-to-23h2.md)]
+
+A [*secret*](https://kubernetes.io/docs/concepts/configuration/secret/) in Kubernetes is an object that contains a small amount of sensitive data, such as passwords and SSH keys. In the Kubernetes API server, secrets are stored in *etcd*, which is a highly available key value store used as the Kubernetes backing store for all cluster data.
+
+Azure Kubernetes Service (AKS) on Azure Local comes with encryption of etcd secrets using a **Key Management Service (KMS) plugin**. All Kubernetes clusters in Azure Local have a built-in KMS plugin enabled by default. This plugin generates the [Key Encryption Key (KEK)](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/#kms-encryption-and-per-object-encryption-keys)
+and automatically rotates it every 30 days.
+
+This article describes how to verify that the data is encrypted. For more information, see the [official Kubernetes documentation for the KMS plugin](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/).
+
+> [!NOTE]
+> The KMS plugin currently uses the KMS v1 protocol.
+
+## Before you begin
+
+Before you begin, ensure that you have the following prerequisites:
+
+- To interact with Kubernetes clusters, you must install [**kubectl**](https://kubernetes.io/docs/tasks/tools/) and [**kubelogin**](https://azure.github.io/kubelogin/install.html).
+- To view or manage secrets, ensure you have the necessary entitlements to access them. Learn more from [Access and identity](concepts-security-access-identity.md#built-in-roles).
+
+## Access your Microsoft Entra-enabled cluster
+
+Get the user credentials to access your cluster using the [az aksarc get-credentials](/cli/azure/aksarc#az-aksarc-get-credentials) command. You need the **Microsoft.HybridContainerService/provisionedClusterInstances/listUserKubeconfig/action** resource, which is included in the **Azure Kubernetes Service Arc Cluster User** role permission:
+
+```azurecli
+az aksarc get-credentials --resource-group $resource_group --name $aks_cluster_name
+```
+
+## Verify that the KMS plugin is enabled
+
+To verify that the KMS plugin is enabled, run the following command and ensure that the health status of **kms-providers** is **OK**:
+
+```azurecli
+kubectl get --raw='/readyz?verbose'
+```
+
+```output
+[+]ping ok
+[+]Log ok
+[+]etcd ok
+[+]kms-providers ok
+[+]poststarthook/start-encryption-provider-config-automatic-reload ok
+```
+
+## Verify that the data is encrypted
+
+To verify that secrets and data has been encrypted using a KMS plugin, [see the Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted). You can use the following commands to verify that the data is encrypted:
+
+```azurecli
+kubectl exec --stdin --tty <etcd pod name> -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt get /registry/secrets/default/db-user-pass -w fields
+```
+
+- `kubectl exec`: This is the kubectl command used to execute a command inside a running pod. It allows you to run commands within the container of a pod.
+- `--stdin`: This flag allows you to send input (stdin) to the command you're running inside the pod. It's useful if you need to interact with the command, especially for commands that expect user input.
+- `--tty`: This flag allocates a TTY (terminal) for the command, making it behave like you're interacting with a terminal session. It's especially useful when you want to run interactive commands (like a shell) and see the output in a terminal-like environment.
+- `<etcd pod name>`: to find the etcd pod name, run the following command:
+
+  ```azurecli
+   kubectl get pods -n kube-system | findstr etcd-moc
+   ```
+
+- `-n kube-system`: This flag specifies the namespace where the pod is located. kube-system is the default namespace used by Kubernetes for system components, such as etcd, kube-dns, and other control plane services.
+
+```azurecli
+kubectl exec --stdin --tty etcd-moc-lrhdsg6jk1f -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt get /registry/secrets/default/db-user-pass -w fields
+```
+
+After you run the command, examine the `Value` field in the output in the terminal window. This output shows the value stored in etcd for this key, which is the encrypted value of the secret. The value is encrypted using a KMS plugin. The `k8s:enc:kms:v1:` prefix indicates that Kubernetes is using the KMS plugin to store the secret in an encrypted format.
+
+If you use the `kubectl describe secrets` command to retrieve secrets, it returns them in base64-encoded format, but unencrypted. The `kubectl describe` command retrieves the details of a Kubernetes resource via the API server, which manages encryption and decryption automatically. For sensitive data such as secrets, even if they are mounted on a pod, the API server ensures that they are decrypted when accessed. As a result, running the `kubectl describe` command does not display secrets in their encrypted form, but rather in their decrypted form if they are being used by a resource.
+
+## Troubleshooting
+
+If you encounter any errors with the KMS plugin, follow the procedure on the [Troubleshooting page](aks-troubleshoot.md) to troubleshoot the issue.
+
+## Next steps
+
+- [Encrypt etcd secrets for Kubernetes clusters in AKS on Windows Server](encrypt-secrets.md)
+- [Deploy a Linux application on a Kubernetes cluster](deploy-linux-application.md)
+- [Deploy a Windows Server application on a Kubernetes cluster](deploy-windows-application.md)