You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Custom certificate authorities (CAs) allow you to establish trust between your Azure Kubernetes Service (AKS) cluster and your workloads, such as private registries, proxies, and firewalls. A Kubernetes secret is used to store the certificate authority's information, then it's passed to all nodes in the cluster.
14
14
15
-
This feature is applied per nodepool, so new and existing nodepools must be configured to enable this feature.
15
+
This feature is applied per nodepool, so new and existing node pools must be configured to enable this feature.
16
16
17
17
[!INCLUDE [preview features note](./includes/preview/preview-callout.md)]
18
18
19
19
## Prerequisites
20
20
21
21
* An Azure subscription. If you don't have an Azure subscription, you can create a [free account](https://azure.microsoft.com/free).
22
-
*[Azure CLI installed][azure-cli-install].
23
-
* A base64 encoded certificate string.
22
+
*[Azure CLI installed][azure-cli-install] (version 2.43.0 or greater).
23
+
* A base64 encoded certificate string or a text file with certificate.
24
24
25
25
### Limitations
26
26
27
-
This feature isn't currently supported for Windows nodepools.
27
+
This feature isn't currently supported for Windows node pools.
28
28
29
29
### Install the `aks-preview` extension
30
30
31
-
You also need the *aks-preview* Azure CLI extensions version 0.5.72 or later. Install the *aks-preview* extension by using the [az extension add][az-extension-add] command, or install any available updates by using the [az extension update][az-extension-update] command.
31
+
You also need the *aks-preview* Azure CLI extensions version 0.5.119 or later. Install the *aks-preview* extension by using the [az extension add][az-extension-add] command, or install any available updates by using the [az extension update][az-extension-update] command.
32
32
33
33
```azurecli
34
34
# Install the aks-preview extension
@@ -58,44 +58,55 @@ Refresh the registration of the *Microsoft.ContainerService* resource provider b
58
58
az provider register --namespace Microsoft.ContainerService
59
59
```
60
60
61
-
## Configure a new AKS cluster to use a custom CA
61
+
## Two ways for custom CA installation on AKS node pools
62
62
63
-
To configure a new AKS cluster to use a custom CA, run the [az aks create][az-aks-create] command with the `--enable-custom-ca-trust` parameter.
63
+
Two ways of installing custom CAs on your AKS cluster are available. They're intended for different use cases, which are outlined below.
64
+
65
+
### Install CAs during node pool boot up
66
+
If your environment requires your custom CAs to be added to node trust store for correct provisioning,
67
+
text file containing up to 10 blank line separated certificates needs to be passed during
68
+
[az aks create][az-aks-create] or [az aks update][az-aks-update] operations.
64
69
70
+
Example command:
65
71
```azurecli
66
72
az aks create \
67
73
--resource-group myResourceGroup \
68
74
--name myAKSCluster \
69
75
--node-count 2 \
70
-
--enable-custom-ca-trust
76
+
--enable-custom-ca-trust \
77
+
--custom-ca-trust-certificates pathToFileWithCAs
71
78
```
72
79
73
-
## Configure a new nodepool to use a custom CA
74
-
75
-
To configure a new nodepool to use a custom CA, run the [az aks nodepool add][az-aks-nodepool-add] command with the `--enable-custom-ca-trust` parameter.
76
-
77
-
```azurecli
78
-
az aks nodepool add \
79
-
--cluster-name myAKSCluster \
80
-
--resource-group myResourceGroup \
81
-
--name myNodepool \
82
-
--enable-custom-ca-trust \
83
-
--os-type Linux
80
+
Example file:
84
81
```
82
+
-----BEGIN CERTIFICATE-----
83
+
cert1
84
+
-----END CERTIFICATE-----
85
85
86
-
## Configure an existing nodepool to use a custom CA
86
+
-----BEGIN CERTIFICATE-----
87
+
cert2
88
+
-----END CERTIFICATE-----
89
+
```
87
90
88
-
To configure an existing nodepool to use a custom CA, run the [az aks nodepool update][az-aks-nodepool-update] command with the `--enable-custom-trust-ca` parameter.
91
+
CAs will be added to node's trust store during node boot up process, allowing the node to, for example access a private registry.
89
92
93
+
#### CA rotation for availability during node pool boot up
94
+
To update CAs passed to cluster during boot up [az aks update][az-aks-update] operation has to be used.
90
95
```azurecli
91
-
az aks nodepool update \
96
+
az aks update \
92
97
--resource-group myResourceGroup \
93
-
--cluster-name myAKSCluster \
94
-
--name myNodepool \
95
-
--enable-custom-ca-trust
98
+
--name myAKSCluster \
99
+
--custom-ca-trust-certificates pathToFileWithCAs
96
100
```
97
101
98
-
## Create a Kubernetes secret with your CA information
102
+
> [!NOTE]
103
+
> Running this operation will trigger a model update, to ensure that new nodes added during for example scale up operation have the newest CAs required for correct provisioning.
104
+
> This means that AKS will create additional nodes, drain currently existing ones, delete them and then replace them with nodes that have the new set of CAs installed.
105
+
106
+
107
+
### Install CAs once node pool is up and running
108
+
If your environment can be successfully provisioned without your custom CAs, you can provide the CAs using a secret deployed in the kube-system namespace.
109
+
This approach allows for certificate rotation without the need for node recreation.
99
110
100
111
Create a [Kubernetes secret][kubernetes-secrets] YAML manifest with your base64 encoded certificate string in the `data` field. Data from this secret is used to update CAs on all nodes.
101
112
@@ -117,7 +128,106 @@ data:
117
128
{anotherBase64EncodedCertStringHere}
118
129
```
119
130
120
-
To update or remove a CA, edit and apply the YAML manifest. The cluster will poll for changes and update the nodes accordingly. This process may take a couple of minutes before changes are applied.
131
+
To update or remove a CA, edit and apply the secret's YAML manifest. The cluster will poll for changes and update the nodes accordingly. This process may take a couple of minutes before changes are applied.
132
+
133
+
Sometimes containerd restart on the node might be required for the CAs to be picked up properly. If it appears like CAs aren't added correctly to your node's trust store, you can trigger such restart using the following command from node's shell:
134
+
135
+
```systemctl restart containerd```
136
+
137
+
> [!NOTE]
138
+
> Installing CAs using the secret in the kube-system namespace will allow for CA rotation without need for node recreation.
139
+
140
+
## Configure a new AKS cluster to use a custom CA
141
+
142
+
To configure a new AKS cluster to use a custom CA, run the [az aks create][az-aks-create] command with the `--enable-custom-ca-trust` parameter.
143
+
144
+
```azurecli
145
+
az aks create \
146
+
--resource-group myResourceGroup \
147
+
--name myAKSCluster \
148
+
--node-count 2 \
149
+
--enable-custom-ca-trust
150
+
```
151
+
152
+
To configure a new AKS cluster to use custom CA with CAs installed before node boots up, run the [az aks create][az-aks-create] command with the `--enable-custom-ca-trust` and `--custom-ca-trust-certificates` parameters.
153
+
154
+
```azurecli
155
+
az aks create \
156
+
--resource-group myResourceGroup \
157
+
--name myAKSCluster \
158
+
--node-count 2 \
159
+
--enable-custom-ca-trust \
160
+
--custom-ca-trust-certificates pathToFileWithCAs
161
+
```
162
+
163
+
## Configure an existing AKS cluster to have custom CAs installed before node boots up
164
+
165
+
To configure an existing AKS cluster to have your custom CAs added to node's trust store before it boots up, run [az aks update][az-aks-update] command with the `--custom-ca-trust-certificates` parameter.
166
+
167
+
```azurecli
168
+
az aks update \
169
+
--resource-group myResourceGroup \
170
+
--name myAKSCluster \
171
+
--custom-ca-trust-certificates pathToFileWithCAs
172
+
```
173
+
174
+
## Configure a new node pool to use a custom CA
175
+
176
+
To configure a new node pool to use a custom CA, run the [az aks nodepool add][az-aks-nodepool-add] command with the `--enable-custom-ca-trust` parameter.
177
+
178
+
```azurecli
179
+
az aks nodepool add \
180
+
--cluster-name myAKSCluster \
181
+
--resource-group myResourceGroup \
182
+
--name myNodepool \
183
+
--enable-custom-ca-trust \
184
+
--os-type Linux
185
+
```
186
+
187
+
If there are currently no other node pools with the feature enabled, cluster will have to reconcile its settings for
188
+
the changes to take effect. Before that happens, daemonset and pods, which install CAs won't appear on the cluster.
189
+
This operation will happen automatically as a part of AKS's reconcile loop.
190
+
You can trigger reconcile operation immediately by running the [az aks update][az-aks-update] command:
191
+
192
+
```azurecli
193
+
az aks update \
194
+
--resource-group myResourceGroup \
195
+
--name cluster-name
196
+
```
197
+
198
+
Once completed, the daemonset and pods will appear in the cluster.
199
+
200
+
## Configure an existing node pool to use a custom CA
201
+
202
+
To configure an existing node pool to use a custom CA, run the [az aks nodepool update][az-aks-nodepool-update] command with the `--enable-custom-trust-ca` parameter.
203
+
204
+
```azurecli
205
+
az aks nodepool update \
206
+
--resource-group myResourceGroup \
207
+
--cluster-name myAKSCluster \
208
+
--name myNodepool \
209
+
--enable-custom-ca-trust
210
+
```
211
+
212
+
If there are currently no other node pools with the feature enabled, cluster will have to reconcile its settings for
213
+
the changes to take effect. Before that happens, daemon set and pods, which install CAs won't appear on the cluster.
214
+
This operation will happen automatically as a part of AKS's reconcile loop.
215
+
You can trigger reconcile operation by running the following command:
216
+
217
+
```azurecli
218
+
az aks update -g myResourceGroup --name cluster-name
219
+
```
220
+
221
+
Once complete, the daemonset and pods will appear in the cluster.
222
+
223
+
## Troubleshooting
224
+
225
+
### Feature is enabled and secret with CAs is added, but operations are failing with X.509 Certificate Signed by Unknown Authority error
226
+
#### Incorrectly formatted certs passed in the secret
227
+
AKS requires certs passed in the user-created secret to be properly formatted and base64 encoded. Make sure the CAs you passed are properly base64 encoded and that files with CAs don't have CRLF line breaks.
228
+
Certificates passed to ```--custom-ca-trust-certificates``` option shouldn't be base64 encoded.
229
+
#### Containerd hasn't picked up new certs
230
+
From node's shell, run ```systemctl restart containerd```, once containerd is restarted, new certs will be properly picked up by the container runtime.
121
231
122
232
## Next steps
123
233
@@ -137,4 +247,4 @@ For more information on AKS security best practices, see [Best practices for clu
0 commit comments