You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Custom certificate authority (CA) in Azure Kubernetes Service (AKS) (preview)
12
12
13
-
Custom certificate authorities (CAs) allow you to establish trust between your Azure Kubernetes Service (AKS) cluster and your workloads, such as private registries, proxies, and firewalls. A Kubernetes secret is used to store the certificate authority's information, then it's passed to all nodes in the cluster.
13
+
AKS generates and uses the following certificates, Certificate Authorities (CAs), and Service Accounts (SAs):
14
14
15
-
This feature is applied per nodepool, so new and existing node pools must be configured to enable this feature.
15
+
* The AKS API server creates a CA called the Cluster CA.
16
+
* The API server has a Cluster CA, which signs certificates for one-way communication from the API server to kubelets.
17
+
* Each kubelet also creates a Certificate Signing Request (CSR), which is signed by the Cluster CA, for communication from the kubelet to the API server.
18
+
* The API aggregator uses the Cluster CA to issue certificates for communication with other APIs. The API aggregator can also have its own CA for issuing those certificates, but it currently uses the Cluster CA.
19
+
* Each node uses an SA token, which is signed by the Cluster CA.
20
+
* The `kubectl` client has a certificate for communicating with the AKS cluster.
21
+
22
+
You can also create custom certificate authorities, which allow you to establish trust between your Azure Kubernetes Service (AKS) clusters and workloads, such as private registries, proxies, and firewalls. A Kubernetes secret stores the certificate authority's information, and then it's passed to all nodes in the cluster. This feature is applied per node pool, so you need to enable it on new and existing node pools.
23
+
24
+
This article shows you how to create custom CAs and apply them to your AKS clusters.
16
25
17
26
## Prerequisites
18
27
19
-
* An Azure subscription. If you don't have an Azure subscription, you can create a [free account](https://azure.microsoft.com/free).
28
+
* An Azure subscription. If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free).
20
29
*[Azure CLI installed][azure-cli-install] (version 2.43.0 or greater).
21
30
* A base64 encoded certificate string or a text file with certificate.
22
31
23
32
## Limitations
24
33
25
-
This feature isn't currently supported for Windows node pools.
34
+
*This feature currently isn't supported for Windows node pools.
26
35
27
-
## Install the aks-preview Azure CLI extension
36
+
## Install the `aks-preview` Azure CLI extension
28
37
29
38
[!INCLUDE [preview features callout](includes/preview/preview-callout.md)]
30
39
31
-
To install the aks-preview extension, run the following command:
40
+
1. Install the aks-preview extension using the [`az extension add`][az-extension-add]command.
32
41
33
-
```azurecli
34
-
az extension add --name aks-preview
35
-
```
42
+
```azurecli
43
+
az extension add --name aks-preview
44
+
```
36
45
37
-
Run the following command to update to the latest version of the extension released:
46
+
2. Update to the latest version of the extension using the [`az extension update`][az-extension-update] command.
38
47
39
-
```azurecli
40
-
az extension update --name aks-preview
41
-
```
48
+
```azurecli
49
+
az extension update --name aks-preview
50
+
```
42
51
43
-
## Register the 'CustomCATrustPreview' feature flag
52
+
## Register the `CustomCATrustPreview` feature flag
44
53
45
-
Register the `CustomCATrustPreview` feature flag by using the [az feature register][az-feature-register] command, as shown in the following example:
54
+
1. Register the `CustomCATrustPreview` feature flag using the [`az feature register`][az-feature-register] command.
46
55
47
-
```azurecli
48
-
az feature register --namespace "Microsoft.ContainerService" --name "CustomCATrustPreview"
49
-
```
56
+
```azurecli
57
+
az feature register --namespace "Microsoft.ContainerService" --name "CustomCATrustPreview"
58
+
```
50
59
51
-
It takes a few minutes for the status to show *Registered*. Verify the registration status by using the [az feature show][az-feature-show] command:
60
+
It takes a few minutes for the status to show *Registered*.
52
61
53
-
```azurecli-interactive
54
-
az feature show --namespace "Microsoft.ContainerService" --name "CustomCATrustPreview"
55
-
```
62
+
2. Verify the registration status using the [`az feature show`][az-feature-show] command.
56
63
57
-
When the status reflects *Registered*, refresh the registration of the *Microsoft.ContainerService* resource provider by using the [az provider register][az-provider-register] command:
64
+
```azurecli
65
+
az feature show --namespace "Microsoft.ContainerService" --name "CustomCATrustPreview"
66
+
```
58
67
59
-
```azurecli-interactive
60
-
az provider register --namespace Microsoft.ContainerService
61
-
```
68
+
3. When the status reflects *Registered*, refresh the registration of the *Microsoft.ContainerService* resource provider using the [`az provider register`][az-provider-register] command.
62
69
63
-
## Two ways for custom CA installation on AKS node pools
70
+
```azurecli
71
+
az provider register --namespace Microsoft.ContainerService
72
+
```
64
73
65
-
Two ways of installing custom CAs on your AKS cluster are available. They're intended for different use cases, which are outlined below.
74
+
## Custom CA installation on AKS node pools
66
75
67
-
### Install CAs during node pool boot up
68
-
If your environment requires your custom CAs to be added to node trust store for correct provisioning,
69
-
text file containing up to 10 blank line separated certificates needs to be passed during
70
-
[az aks create][az-aks-create] or [az aks update][az-aks-update] operations.
76
+
### Install CAs on AKS node pools
71
77
72
-
Example command:
73
-
```azurecli
74
-
az aks create \
75
-
--resource-group myResourceGroup \
76
-
--name myAKSCluster \
77
-
--node-count 2 \
78
-
--enable-custom-ca-trust \
79
-
--custom-ca-trust-certificates pathToFileWithCAs
80
-
```
78
+
* If your environment requires your custom CAs to be added to node trust store for correct provisioning, you need to pass a text file containing up to 10 blank line separated certificates during [`az aks create`][az-aks-create] or [`az aks update`][az-aks-update] operations. Example text file:
81
79
82
-
Example file:
83
-
```
84
-
-----BEGIN CERTIFICATE-----
85
-
cert1
86
-
-----END CERTIFICATE-----
80
+
```txt
81
+
-----BEGIN CERTIFICATE-----
82
+
cert1
83
+
-----END CERTIFICATE-----
87
84
88
-
-----BEGIN CERTIFICATE-----
89
-
cert2
90
-
-----END CERTIFICATE-----
91
-
```
85
+
-----BEGIN CERTIFICATE-----
86
+
cert2
87
+
-----END CERTIFICATE-----
88
+
```
92
89
93
-
CAs will be added to node's trust store during node boot up process, allowing the node to, for example access a private registry.
90
+
#### Install CAs during node pool creation
94
91
95
-
#### CA rotation for availability during node pool boot up
96
-
To update CAs passed to cluster during boot up [az aks update][az-aks-update] operation has to be used.
97
-
```azurecli
98
-
az aks update \
99
-
--resource-group myResourceGroup \
100
-
--name myAKSCluster \
101
-
--custom-ca-trust-certificates pathToFileWithCAs
102
-
```
92
+
* Install CAs during node pool creation using the [`az aks create][az-aks-create] command and specifying your text file for the `--custom-ca-trust-certificates` parameter.
93
+
94
+
```azurecli
95
+
az aks create \
96
+
--resource-group myResourceGroup \
97
+
--name myAKSCluster \
98
+
--node-count 2 \
99
+
--enable-custom-ca-trust \
100
+
--custom-ca-trust-certificates pathToFileWithCAs
101
+
```
103
102
104
-
> [!NOTE]
105
-
> Running this operation will trigger a model update, to ensure that new nodes added during for example scale up operation have the newest CAs required for correct provisioning.
106
-
> This means that AKS will create additional nodes, drain currently existing ones, delete them and then replace them with nodes that have the new set of CAs installed.
103
+
#### CA rotation for availability during node pool boot up
107
104
105
+
* Update CAs passed to your cluster during boot up using the [`az aks update`][az-aks-update] command and specifying your text file for the `--custom-ca-trust-certificates` parameter.
108
106
109
-
### Install CAs once node pool is up and running
110
-
If your environment can be successfully provisioned without your custom CAs, you can provide the CAs using a secret deployed in the kube-system namespace.
111
-
This approach allows for certificate rotation without the need for node recreation.
107
+
```azurecli
108
+
az aks update \
109
+
--resource-group myResourceGroup \
110
+
--name myAKSCluster \
111
+
--custom-ca-trust-certificates pathToFileWithCAs
112
+
```
112
113
113
-
Create a [Kubernetes secret][kubernetes-secrets] YAML manifest with your base64 encoded certificate string in the `data` field. Data from this secret is used to update CAs on all nodes.
114
+
> [!NOTE]
115
+
> This operation triggers a model update, ensuring new nodes have the newest CAs required for correct provisioning. AKS creates additional nodes, drains existing ones, deletes them, and replaces them with nodes that have the new set of CAs installed.
114
116
115
-
You must ensure that:
116
-
* The secret is named `custom-ca-trust-secret`.
117
-
* The secret is created in the `kube-system` namespace.
117
+
### Install CAs after node pool creation
118
118
119
-
```yaml
120
-
apiVersion: v1
121
-
kind: Secret
122
-
metadata:
123
-
name: custom-ca-trust-secret
124
-
namespace: kube-system
125
-
type: Opaque
126
-
data:
127
-
ca1.crt: |
128
-
{base64EncodedCertStringHere}
129
-
ca2.crt: |
130
-
{anotherBase64EncodedCertStringHere}
131
-
```
119
+
If your environment can be successfully provisioned without your custom CAs, you can provide the CAs by deploying a secret in the `kube-system` namespace. This approach allows for certificate rotation without the need for node recreation.
132
120
133
-
To update or remove a CA, edit and apply the secret's YAML manifest. The cluster will poll for changes and update the nodes accordingly. This process may take a couple of minutes before changes are applied.
121
+
* Create a [Kubernetes secret][kubernetes-secrets] YAML manifest with your base64 encoded certificate string in the `data` field.
134
122
135
-
Sometimes containerd restart on the node might be required for the CAs to be picked up properly. If it appears like CAs aren't added correctly to your node's trust store, you can trigger such restart using the following command from node's shell:
123
+
```yaml
124
+
apiVersion: v1
125
+
kind: Secret
126
+
metadata:
127
+
name: custom-ca-trust-secret
128
+
namespace: kube-system
129
+
type: Opaque
130
+
data:
131
+
ca1.crt: |
132
+
{base64EncodedCertStringHere}
133
+
ca2.crt: |
134
+
{anotherBase64EncodedCertStringHere}
135
+
```
136
136
137
-
```systemctl restart containerd```
137
+
Data from this secret is used to update CAs on all nodes. Make sure the secret is named `custom-ca-trust-secret` and is created in the `kube-system` namespace. Installing CAs using the secret in the `kube-system` namespace allows for CA rotation without the need for node recreation. To update or remove a CA, you can edit and apply the YAML manifest. The cluster polls for changes and updates the nodes accordingly. It may take a couple minutes before changes are applied.
138
138
139
-
> [!NOTE]
140
-
> Installing CAs using the secret in the kube-system namespace will allow for CA rotation without need for node recreation.
139
+
> [!NOTE]
140
+
>
141
+
> containerd restart on the node might be required for the CAs to be picked up properly. If it appears like CAs aren't correctly added to your node's trust store, you can trigger a restart using the following command from node's shell:
142
+
>
143
+
> ```systemctl restart containerd```
141
144
142
145
## Configure a new AKS cluster to use a custom CA
143
146
144
-
To configure a new AKS cluster to use a custom CA, run the [az aks create][az-aks-create] command with the `--enable-custom-ca-trust` parameter.
147
+
* Configure a new AKS cluster to use a custom CA using the [`az aks create`][az-aks-create] command with the `--enable-custom-ca-trust` parameter.
148
+
149
+
```azurecli
150
+
az aks create \
151
+
--resource-group myResourceGroup \
152
+
--name myAKSCluster \
153
+
--node-count 2 \
154
+
--enable-custom-ca-trust
155
+
```
145
156
146
-
```azurecli
147
-
az aks create \
148
-
--resource-group myResourceGroup \
149
-
--name myAKSCluster \
150
-
--node-count 2 \
151
-
--enable-custom-ca-trust
152
-
```
157
+
## Configure a new AKS cluster to use a custom CA with CAs installed before node boots up
153
158
154
-
To configure a new AKS cluster to use custom CA with CAs installed before node boots up, run the [az aks create][az-aks-create] command with the `--enable-custom-ca-trust` and `--custom-ca-trust-certificates` parameters.
159
+
* Configure a new AKS cluster to use custom CA with CAs installed before the node boots up using the [`az aks create`][az-aks-create] command with the `--enable-custom-ca-trust` and `--custom-ca-trust-certificates` parameters.
155
160
156
-
```azurecli
157
-
az aks create \
158
-
--resource-group myResourceGroup \
159
-
--name myAKSCluster \
160
-
--node-count 2 \
161
-
--enable-custom-ca-trust \
162
-
--custom-ca-trust-certificates pathToFileWithCAs
163
-
```
161
+
```azurecli
162
+
az aks create \
163
+
--resource-group myResourceGroup \
164
+
--name myAKSCluster \
165
+
--node-count 2 \
166
+
--enable-custom-ca-trust \
167
+
--custom-ca-trust-certificates pathToFileWithCAs
168
+
```
164
169
165
170
## Configure an existing AKS cluster to have custom CAs installed before node boots up
166
171
167
-
To configure an existing AKS cluster to have your custom CAs added to node's trust store before it boots up, run [az aks update][az-aks-update] command with the `--custom-ca-trust-certificates` parameter.
172
+
* Configure an existing AKS cluster to have your custom CAs added to node's trust store before it boots up using the [`az aks update`][az-aks-update] command with the `--custom-ca-trust-certificates` parameter.
168
173
169
-
```azurecli
170
-
az aks update \
171
-
--resource-group myResourceGroup \
172
-
--name myAKSCluster \
173
-
--custom-ca-trust-certificates pathToFileWithCAs
174
-
```
174
+
```azurecli
175
+
az aks update \
176
+
--resource-group myResourceGroup \
177
+
--name myAKSCluster \
178
+
--custom-ca-trust-certificates pathToFileWithCAs
179
+
```
175
180
176
181
## Configure a new node pool to use a custom CA
177
182
178
-
To configure a new node pool to use a custom CA, run the [az aks nodepool add][az-aks-nodepool-add] command with the `--enable-custom-ca-trust` parameter.
179
-
180
-
```azurecli
181
-
az aks nodepool add \
182
-
--cluster-name myAKSCluster \
183
-
--resource-group myResourceGroup \
184
-
--name myNodepool \
185
-
--enable-custom-ca-trust \
186
-
--os-type Linux
187
-
```
188
-
189
-
If there are currently no other node pools with the feature enabled, cluster will have to reconcile its settings for
190
-
the changes to take effect. Before that happens, daemonset and pods, which install CAs won't appear on the cluster.
191
-
This operation will happen automatically as a part of AKS's reconcile loop.
192
-
You can trigger reconcile operation immediately by running the [az aks update][az-aks-update] command:
183
+
* Configure a new node pool to use a custom CA using the [`az aks nodepool add`][az-aks-nodepool-add] command with the `--enable-custom-ca-trust` parameter.
193
184
194
-
```azurecli
195
-
az aks update \
196
-
--resource-group myResourceGroup \
197
-
--name cluster-name
198
-
```
185
+
```azurecli
186
+
az aks nodepool add \
187
+
--cluster-name myAKSCluster \
188
+
--resource-group myResourceGroup \
189
+
--name myNodepool \
190
+
--enable-custom-ca-trust \
191
+
--os-type Linux
192
+
```
199
193
200
-
Once completed, the daemonset and pods will appear in the cluster.
194
+
If no other node pools with the feature enabled exist, the cluster has to reconcile its settings for the changes to take effect. This operation happens automatically as a part of AKS's reconcile loop. Before the operation, the daemon set and pods don't appear on the cluster. You can trigger an immediate reconcile operation using the [`az aks update`][az-aks-update] command. The daemon set and pods appear after the update completes.
201
195
202
196
## Configure an existing node pool to use a custom CA
203
197
204
-
To configure an existing node pool to use a custom CA, run the [az aks nodepool update][az-aks-nodepool-update] command with the `--enable-custom-trust-ca` parameter.
198
+
* Configure an existing node pool to use a custom CA using the [`az aks nodepool update`][az-aks-nodepool-update] command with the `--enable-custom-trust-ca` parameter.
205
199
206
-
```azurecli
207
-
az aks nodepool update \
208
-
--resource-group myResourceGroup \
209
-
--cluster-name myAKSCluster \
210
-
--name myNodepool \
211
-
--enable-custom-ca-trust
212
-
```
200
+
```azurecli
201
+
az aks nodepool update \
202
+
--resource-group myResourceGroup \
203
+
--cluster-name myAKSCluster \
204
+
--name myNodepool \
205
+
--enable-custom-ca-trust
206
+
```
213
207
214
-
If there are currently no other node pools with the feature enabled, cluster will have to reconcile its settings for
215
-
the changes to take effect. Before that happens, daemon set and pods, which install CAs won't appear on the cluster.
216
-
This operation will happen automatically as a part of AKS's reconcile loop.
217
-
You can trigger reconcile operation by running the following command:
218
-
219
-
```azurecli
220
-
az aks update -g myResourceGroup --name cluster-name
221
-
```
222
-
223
-
Once complete, the daemonset and pods will appear in the cluster.
208
+
If no other node pools with the feature enabled exist, the cluster has to reconcile its settings for the changes to take effect. This operation happens automatically as a part of AKS's reconcile loop. Before the operation, the daemon set and pods don't appear on the cluster. You can trigger an immediate reconcile operation using the [`az aks update`][az-aks-update] command. The daemon set and pods appear after the update completes.
224
209
225
210
## Troubleshooting
226
211
227
212
### Feature is enabled and secret with CAs is added, but operations are failing with X.509 Certificate Signed by Unknown Authority error
213
+
228
214
#### Incorrectly formatted certs passed in the secret
215
+
229
216
AKS requires certs passed in the user-created secret to be properly formatted and base64 encoded. Make sure the CAs you passed are properly base64 encoded and that files with CAs don't have CRLF line breaks.
230
-
Certificates passed to ```--custom-ca-trust-certificates``` option shouldn't be base64 encoded.
231
-
#### Containerd hasn't picked up new certs
232
-
From node's shell, run ```systemctl restart containerd```, once containerd is restarted, new certs will be properly picked up by the container runtime.
217
+
Certificates passed to ```--custom-ca-trust-certificates``` shouldn't be base64 encoded.
218
+
219
+
#### containerd hasn't picked up new certs
220
+
221
+
From the node's shell, run ```systemctl restart containerd```. Once containerd is restarts, the new certs are properly picked up by the container runtime.
233
222
234
223
## Next steps
235
224
236
225
For more information on AKS security best practices, see [Best practices for cluster security and upgrades in Azure Kubernetes Service (AKS)][aks-best-practices-security-upgrades].
0 commit comments