Skip to content

Commit 9df2b66

Browse files
schaffererinschaffererin
committed
Freshness pass: Certificate rotation in AKS
1 parent cb79b8e commit 9df2b66

File tree

1 file changed

+86
-75
lines changed

1 file changed

+86
-75
lines changed

‎articles/aks/certificate-rotation.md

Lines changed: 86 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -8,141 +8,152 @@ ms.date: 01/19/2023
88

99
# Certificate rotation in Azure Kubernetes Service (AKS)
1010

11-
Azure Kubernetes Service (AKS) uses certificates for authentication with many of its components. If you have a RBAC-enabled cluster built after March 2022, it's enabled with certificate auto-rotation. Periodically, you may need to rotate those certificates for security or policy reasons. For example, you may have a policy to rotate all your certificates every 90 days.
11+
Azure Kubernetes Service (AKS) uses certificates for authentication with many of its components. RBAC-enabled clusters created after March 2022 are enabled with certificate auto-rotation. You may need to periodically rotate those certificates for security or policy reasons. For example, you may have a policy to rotate all your certificates every 90 days.
1212

1313
> [!NOTE]
14-
> Certificate auto-rotation will *only* be enabled by default for RBAC enabled AKS clusters.
14+
> Certificate auto-rotation is *only* enabled by default for RBAC enabled AKS clusters.
1515
1616
This article shows you how certificate rotation works in your AKS cluster.
1717

1818
## Before you begin
1919

20-
This article requires that you are running the Azure CLI version 2.0.77 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].
20+
This article requires the Azure CLI version 2.0.77 or later. Run `az --version` to find the version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].
2121

2222
## AKS certificates, Certificate Authorities, and Service Accounts
2323

24-
AKS generates and uses the following certificates, Certificate Authorities, and Service Accounts:
24+
AKS generates and uses the following certificates, Certificate Authorities (CA), and Service Accounts (SA):
2525

26-
* The AKS API server creates a Certificate Authority (CA) called the Cluster CA.
26+
* The AKS API server creates a CA called the Cluster CA.
2727
* The API server has a Cluster CA, which signs certificates for one-way communication from the API server to kubelets.
28-
* Each kubelet also creates a Certificate Signing Request (CSR), which is signed by the Cluster CA, for communication from the kubelet to the API server.
28+
* Each kubelet creates a Certificate Signing Request (CSR), which is signed by the Cluster CA, for communication from the kubelet to the API server.
2929
* The API aggregator uses the Cluster CA to issue certificates for communication with other APIs. The API aggregator can also have its own CA for issuing those certificates, but it currently uses the Cluster CA.
30-
* Each node uses a Service Account (SA) token, which is signed by the Cluster CA.
30+
* Each node uses an SA token, which is signed by the Cluster CA.
3131
* The `kubectl` client has a certificate for communicating with the AKS cluster.
3232

33-
Certificates mentioned above are maintained by Microsoft, except the cluster certificate, which you have to maintain.
33+
All certificates mentioned in this section, except for the cluster certificate, are maintained by Microsoft.
3434

3535
> [!NOTE]
36-
> AKS clusters created prior to May 2019 have certificates that expire after two years. Any cluster created after May 2019 or any cluster that has its certificates rotated have Cluster CA certificates that expire after 30 years. All other AKS certificates, which use the Cluster CA for signing, will expire after two years and are automatically rotated during an AKS version upgrade which happened after 8/1/2021. To verify when your cluster was created, use `kubectl get nodes` to see the *Age* of your node pools.
36+
> AKS clusters created prior to May 2019 have certificates that expire after two years. Any cluster created after May 2019 or any cluster that has its certificates rotated have Cluster CA certificates that expire after 30 years. All other AKS certificates, which use the Cluster CA for signing, expired after two years and were automatically rotated during an AKS version upgrade which occurred after 8/1/2021.
3737
>
38-
> Additionally, you can check the expiration date of your cluster's certificate. For example, the following bash command displays the client certificate details for the *myAKSCluster* cluster in resource group *rg*:
39-
> ```console
40-
> kubectl config view --raw -o jsonpath="{.users[?(@.name == 'clusterUser_rg_myAKSCluster')].user.client-certificate-data}" | base64 -d | openssl x509 -text | grep -A2 Validity
41-
> ```
38+
> You can verify when your cluster was created using the `kubectl get nodes` command, which shows you the *Age* of your node pools.
4239
43-
To check expiration date of apiserver certificate, run the following command:
40+
## Check certificate expiration dates
4441

45-
```console
46-
curl https://{apiserver-fqdn} -k -v 2>&1 |grep expire
47-
```
42+
### Check cluster certificate expiration date
4843

49-
To check the expiration date of certificate on VMAS agent node, run the following command:
44+
* Check the expiration date of the cluster certificate using the `kubectl config view` command.
5045

51-
```azurecli
52-
az vm run-command invoke -g MC_rg_myAKSCluster_region -n vm-name --command-id RunShellScript --query 'value[0].message' -otsv --scripts "openssl x509 -in /etc/kubernetes/certs/apiserver.crt -noout -enddate"
53-
```
46+
```console
47+
kubectl config view --raw -o jsonpath="{.users[?(@.name == 'clusterUser_rg_myAKSCluster')].user.client-certificate-data}" | base64 -d | openssl x509 -text | grep -A2 Validity
48+
```
5449

55-
To check expiration date of certificate on one virtual machine scale set agent node, run the following command:
50+
### Check API server certificate expiration date
5651

57-
```azurecli
58-
az vmss run-command invoke --resource-group "MC_rg_myAKSCluster_region" --name "vmss-name " --command-id RunShellScript --instance-id 1 --scripts "openssl x509 -in /etc/kubernetes/certs/apiserver.crt -noout -enddate" --query "value[0].message"
59-
```
52+
* Check the expiration date of the API server certificate using the following `curl` command.
6053

61-
## Certificate Auto Rotation
54+
```console
55+
curl https://{apiserver-fqdn} -k -v 2>&1 |grep expire
56+
```
6257

63-
For AKS to automatically rotate non-CA certificates, the cluster must have [TLS Bootstrapping](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/) which has been enabled by default in all Azure regions.
58+
### Check VMAS agent node certificate expiration date
6459

65-
> [!NOTE]
66-
> If you have an existing cluster you have to upgrade that cluster to enable Certificate Auto-Rotation.
67-
> Do not disable bootstrap to keep your auto-rotation enabled.
60+
* Check the expiration date of the VMAS agent node certificate using the `az vm run-command invoke` command.
6861

69-
> [!NOTE]
70-
> If the cluster is in a stopped state during the auto certificate rotation only the control plane certificates are rotated. In this case the nodepool should be recreated, after certificate rotation, in order to initiate the nodepool certificate rotation.
62+
```azurecli-interactive
63+
az vm run-command invoke -g MC_rg_myAKSCluster_region -n vm-name --command-id RunShellScript --query 'value[0].message' -otsv --scripts "openssl x509 -in /etc/kubernetes/certs/apiserver.crt -noout -enddate"
64+
```
7165

72-
For any AKS clusters created or upgraded after March 2022 Azure Kubernetes Service will automatically rotate non-CA certificates on both the control plane and agent nodes within 80% of the client certificate valid time, before they expire with no downtime for the cluster.
66+
### Check Virtual Machine Scale Set agent node certificate expiration date
7367

74-
### How to check whether current agent node pool is TLS Bootstrapping enabled?
68+
* Check the expiration date of the Virtual Machine Scale Set agent node certificate using the `az vm run-command invoke` command.
7569

76-
To verify if TLS Bootstrapping is enabled on your cluster browse to the following paths:
70+
```azurecli-interactive
71+
az vmss run-command invoke --resource-group "MC_rg_myAKSCluster_region" --name "vmss-name" --command-id RunShellScript --instance-id 1 --scripts "openssl x509 -in /etc/kubernetes/certs/apiserver.crt -noout -enddate" --query "value[0].message"
72+
```
7773

78-
* On a Linux node: */var/lib/kubelet/bootstrap-kubeconfig* or */host/var/lib/kubelet/bootstrap-kubeconfig*
79-
* On a Windows node: *C:\k\bootstrap-config*
74+
## Certificate Auto Rotation
8075

81-
To access agent nodes, see [Connect to Azure Kubernetes Service cluster nodes for maintenance or troubleshooting][aks-node-access] for more information.
76+
For AKS to automatically rotate non-CA certificates, the cluster must have [TLS Bootstrapping](https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/), which is enabled by default in all Azure regions.
8277

8378
> [!NOTE]
84-
> The file path may change as Kubernetes version evolves in the future.
79+
>
80+
> * If you have an existing cluster, you have to upgrade that cluster to enable Certificate Auto Rotation.
81+
> * Don't disable Bootstrap to keep auto rotation enabled.
82+
> * If the cluster is in a stopped state during the auto certificate rotation, only the control plane certificates are rotated. In this case, you should recreate the node pool after certificate rotation to initiate the node pool certificate rotation.
83+
84+
For any AKS clusters created or upgraded after March 2022, Azure Kubernetes Service automatically rotates non-CA certificates on both the control plane and agent nodes within 80% of the client certificate valid time before they expire with no downtime for the cluster.
8585

86-
Once a region is configured, create a new cluster or upgrade an existing cluster with `az aks upgrade` to set that cluster for auto-certificate rotation. A control plane and node pool upgrade is needed to enable this feature.
86+
### How to check whether current agent node pool is TLS Bootstrapping enabled?
87+
88+
1. Verify if TLS Bootstrapping is enabled on your cluster by browsing to one to the following paths:
8789

88-
```azurecli
89-
az aks upgrade -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
90-
```
90+
* On a Linux node: */var/lib/kubelet/bootstrap-kubeconfig* or */host/var/lib/kubelet/bootstrap-kubeconfig*
91+
* On a Windows node: *C:\k\bootstrap-config*
9192

92-
### Limitation
93+
For more information, see [Connect to Azure Kubernetes Service cluster nodes for maintenance or troubleshooting][aks-node-access].
9394

94-
Certificate auto-rotation will only be enabled by default for RBAC enabled AKS clusters.
95+
> [!NOTE]
96+
> The file path may change as Kubernetes versions evolve.
97+
98+
2. Once a region is configured, create a new cluster or upgrade an existing cluster to set auto rotation for the cluster certificate. You need to upgrade the control plane and node pool to enable this feature.
9599

96100
## Manually rotate your cluster certificates
97101

98102
> [!WARNING]
99-
> Rotating your certificates using `az aks rotate-certs` will recreate all of your nodes, VM scale set and their Disks and can cause up to 30 minutes of downtime for your AKS cluster.
103+
> Rotating your certificates using `az aks rotate-certs` recreates all of your nodes, Virtual Machine Scale Sets and Disks and can cause up to *30 minutes of downtime* for your AKS cluster.
100104

101-
Use [az aks get-credentials][az-aks-get-credentials] to sign in to your AKS cluster. This command also downloads and configures the `kubectl` client certificate on your local machine.
105+
1. Connect to your cluster using the [`az aks get-credentials`][az-aks-get-credentials] command.
102106

103-
```azurecli
104-
az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
105-
```
107+
```azurecli-interactive
108+
az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
109+
```
106110

107-
Use `az aks rotate-certs` to rotate all certificates, CAs, and SAs on your cluster.
111+
2. Rotate all certificates, CAs, and SAs on your cluster using the [`az aks rotate-certs`][az-aks-rotate-certs] command.
108112

109-
```azurecli
110-
az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
111-
```
113+
```azurecli-interactive
114+
az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME
115+
```
112116

113-
> [!IMPORTANT]
114-
> It may take up to 30 minutes for `az aks rotate-certs` to complete. If the command fails before completing, use `az aks show` to verify the status of the cluster is *Certificate Rotating*. If the cluster is in a failed state, rerun `az aks rotate-certs` to rotate your certificates again.
117+
> [!IMPORTANT]
118+
> It may take up to 30 minutes for `az aks rotate-certs` to complete. If the command fails before completing, use `az aks show` to verify the status of the cluster is *Certificate Rotating*. If the cluster is in a failed state, rerun `az aks rotate-certs` to rotate your certificates again.
115119

116-
Verify that the old certificates aren't valid by running any `kubectl` command. If you haven't updated the certificates used by `kubectl`, you'll see an error similar to the following example:
120+
3. Verify the old certificates are no longer valid using any `kubectl` command, such as `kubectl get nodes`.
117121

118-
```console
119-
kubectl get nodes
120-
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca")
121-
```
122+
```azurecli-interactive
123+
kubectl get nodes
124+
```
122125

123-
To update the certificate used by `kubectl`, run the [az aks get-credentials][az-aks-get-credentials] command:
126+
If you haven't updated the certificates used by `kubectl`, you see an error similar to the following example output:
124127

125-
```azurecli
126-
az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME --overwrite-existing
127-
```
128+
```output
129+
Unable to connect to the server: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "ca")
130+
```
128131

129-
To verify the certificates have been updated, run the following [kubectl get][kubectl-get] command:
132+
4. Update the certificate used by `kubectl` using the [`az aks get-credentials`][az-aks-get-credentials] command with the `--overwrite-existing` flag.
130133

131-
```console
132-
kubectl get nodes
133-
```
134+
```azurecli-interactive
135+
az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME --overwrite-existing
136+
```
134137

135-
> [!NOTE]
136-
> If you have any services that run on top of AKS, you might need to update their certificates.
138+
5. Verify the certificates have been updated using the [`kubectl get`][kubectl-get] command.
139+
140+
```azurecli-interactive
141+
kubectl get nodes
142+
```
143+
144+
> [!NOTE]
145+
> If you have any services that run on top of AKS, you might need to update their certificates.
137146

138147
## Next steps
139148

140-
This article showed you how to automatically rotate your cluster's certificates, CAs, and SAs. You can see [Best practices for cluster security and upgrades in Azure Kubernetes Service (AKS)][aks-best-practices-security-upgrades] for more information on AKS security best practices.
149+
This article showed you how to automatically rotate your cluster certificates, CAs, and SAs. For more information, see [Best practices for cluster security and upgrades in Azure Kubernetes Service (AKS)][aks-best-practices-security-upgrades].
141150

151+
<!-- LINKS - internal -->
142152
[azure-cli-install]: /cli/azure/install-azure-cli
143153
[az-aks-get-credentials]: /cli/azure/aks#az_aks_get_credentials
144-
[az-get]: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
145-
[az-extension-add]: /cli/azure/extension#az_extension_add
146-
[az-extension-update]: /cli/azure/extension#az_extension_update
147154
[aks-best-practices-security-upgrades]: operator-best-practices-cluster-security.md
148155
[aks-node-access]: ./node-access.md
156+
[az-aks-rotate-certs]: /cli/azure/aks#az_aks_rotate_certs
157+
158+
<!-- LINKS - external -->
159+
[kubectl-get]: /docs/reference/generated/kubectl/kubectl-commands#get

0 commit comments

Comments
 (0)