|
| 1 | +--- |
| 2 | +title: Encrypt etcd secrets for Kubernetes clusters in AKS on Azure Local |
| 3 | +description: Learn how to encrypt etcd secrets in AKS on Azure Local. |
| 4 | +author: sethmanheim |
| 5 | +ms.topic: how-to |
| 6 | +ms.date: 04/11/2025 |
| 7 | +ms.author: sethm |
| 8 | +ms.lastreviewed: 04/10/2025 |
| 9 | +ms.reviewer: khareanushka |
| 10 | +# Intent: As an IT Pro, I want to learn about encrypted etcd secrets and how they are used in my AKS deployment. |
| 11 | +# Keyword: etcd secrets AKS Windows Server |
| 12 | + |
| 13 | +--- |
| 14 | + |
| 15 | +# How to: Encrypt etcd secrets for Kubernetes clusters |
| 16 | + |
| 17 | +[!INCLUDE [hci-applies-to-23h2](includes/hci-applies-to-23h2.md)] |
| 18 | + |
| 19 | +A [*secret*](https://kubernetes.io/docs/concepts/configuration/secret/) in Kubernetes is an object that contains a small amount of sensitive data, such as passwords and SSH keys. In the Kubernetes API server, secrets are stored in *etcd*, which is a highly available key value store used as the Kubernetes backing store for all cluster data. |
| 20 | + |
| 21 | +Azure Kubernetes Service (AKS) on Azure Local comes with encryption of etcd secrets using a **Key Management Service (KMS) plugin**. All Kubernetes clusters in Azure Local have a built-in KMS plugin enabled by default. This plugin generates the [Key Encryption Key (KEK)](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/#kms-encryption-and-per-object-encryption-keys) |
| 22 | +and automatically rotates it every 30 days. |
| 23 | + |
| 24 | +This article describes how to verify that the data is encrypted. For more information, see the [official Kubernetes documentation for the KMS plugin](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/). |
| 25 | + |
| 26 | +> [!NOTE] |
| 27 | +> The KMS plugin currently uses the KMS v1 protocol. |
| 28 | +
|
| 29 | +## Before you begin |
| 30 | + |
| 31 | +Before you begin, ensure that you have the following prerequisites: |
| 32 | + |
| 33 | +- To interact with Kubernetes clusters, you must install [**kubectl**](https://kubernetes.io/docs/tasks/tools/) and [**kubelogin**](https://azure.github.io/kubelogin/install.html). |
| 34 | +- To view or manage secrets, ensure you have the necessary entitlements to access them. For more information, see [Access and identity](concepts-security-access-identity.md#built-in-roles). |
| 35 | + |
| 36 | +## Access your Microsoft Entra-enabled cluster |
| 37 | + |
| 38 | +Get the user credentials to access your cluster using the [az aksarc get-credentials](/cli/azure/aksarc#az-aksarc-get-credentials) command. You need the **Microsoft.HybridContainerService/provisionedClusterInstances/listUserKubeconfig/action** resource, which is included in the **Azure Kubernetes Service Arc Cluster User** role permission: |
| 39 | + |
| 40 | +```azurecli |
| 41 | +az aksarc get-credentials --resource-group $resource_group --name $aks_cluster_name |
| 42 | +``` |
| 43 | + |
| 44 | +## Verify that the KMS plugin is enabled |
| 45 | + |
| 46 | +To verify that the KMS plugin is enabled, run the following command and ensure that the health status of **kms-providers** is **OK**: |
| 47 | + |
| 48 | +```azurecli |
| 49 | +kubectl get --raw='/readyz?verbose' |
| 50 | +``` |
| 51 | + |
| 52 | +```output |
| 53 | +[+]ping ok |
| 54 | +[+]Log ok |
| 55 | +[+]etcd ok |
| 56 | +[+]kms-providers ok |
| 57 | +[+]poststarthook/start-encryption-provider-config-automatic-reload ok |
| 58 | +``` |
| 59 | + |
| 60 | +## Verify that the data is encrypted |
| 61 | + |
| 62 | +To verify that secrets and data has been encrypted using a KMS plugin, [see the Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/#verifying-that-the-data-is-encrypted). You can use the following commands to verify that the data is encrypted: |
| 63 | + |
| 64 | +```azurecli |
| 65 | +kubectl exec --stdin --tty <etcd pod name> -n kube-system --etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/server.key --cert /etc/kubernetes/pki/etcd/server.crt get /registry/secrets/default/db-user-pass -w fields |
| 66 | +``` |
| 67 | + |
| 68 | +- `kubectl exec`: This is the kubectl command used to execute a command inside a running pod. It enables you to run commands within the container of a pod. |
| 69 | +- `--stdin`: This flag enables you to send input (stdin) to the command you are running inside the pod. |
| 70 | +- `--tty`: This flag allocates a TTY (terminal) for the command, making it behave as though you're interacting with a terminal session. |
| 71 | +- `<etcd pod name>`: to find the etcd pod name, run the following command: |
| 72 | + |
| 73 | + ```azurecli |
| 74 | + kubectl get pods -n kube-system | findstr etcd-moc |
| 75 | + ``` |
| 76 | + |
| 77 | +- `-n kube-system`: Specifies the namespace where the pod is located. **kube-system** is the default namespace used by Kubernetes for system components, such as etcd and other control plane services. |
| 78 | +- `--etcdctl`: Reads the secret from etcd. Additional fields are used for authentication before you get access to etcd. |
| 79 | + |
| 80 | +The following fields are returned in the command output: |
| 81 | + |
| 82 | +```output |
| 83 | +"ClusterID" : <cluster id> |
| 84 | +"MemberID" : <member id> |
| 85 | +"Revision" : <revision number> |
| 86 | +"RaftTerm" : 2 |
| 87 | +"Key" : <path to the key> |
| 88 | +"CreateRevision" : <revision number at the time the key was created> |
| 89 | +"ModRevision" : <revision number at the time the key was modified> |
| 90 | +"Version" : <version of the key-value pair in etcd> |
| 91 | +"Value" : "k8s:enc:kms:v1:kms-plugin: <encrypted secret value>" |
| 92 | +"Lease" : <lease associated with the secret> |
| 93 | +"More" : <indicates if there are more results> |
| 94 | +"Count" : <number of key-value pairs returned> |
| 95 | +``` |
| 96 | + |
| 97 | +After you run the command, examine the `Value` field in the output in the terminal window. This output shows the value stored in the etcd secret store for this key, which is the encrypted value of the secret. The value is encrypted using a KMS plugin. The `k8s:enc:kms:v1:` prefix indicates that Kubernetes is using the KMS v1 plugin to store the secret in an encrypted format. |
| 98 | + |
| 99 | +> [!NOTE] |
| 100 | +> If you use the `kubectl describe secrets` command to retrieve secrets, it returns them in base64-encoded format, but unencrypted. The `kubectl describe` command retrieves the details of a Kubernetes resource via the API server, which manages encryption and decryption automatically. For sensitive data such as secrets, even if they are mounted on a pod, the API server ensures that they are decrypted when accessed. As a result, running the `kubectl describe` command does not display secrets in their encrypted form, but rather in their decrypted form if they are being used by a resource. |
| 101 | +
|
| 102 | +## Troubleshooting |
| 103 | + |
| 104 | +If you encounter any errors with the KMS plugin, follow the procedure on the [Troubleshooting page](aks-troubleshoot.md) to troubleshoot the issue. |
| 105 | + |
| 106 | +## Next steps |
| 107 | + |
| 108 | +- [Create Kubernetes clusters](aks-create-clusters-cli.md#deploy-the-application-and-load-balancer) |
| 109 | +- [Deploy a Linux application on a Kubernetes cluster](deploy-linux-application.md) |
| 110 | + |
0 commit comments