Skip to content

Commit 96d22c2

Browse files
v-albemiv-albemi
authored
Merge pull request #177506 from Nickomang/aks-cluster-extensions
AKS Cluster Extensions
2 parents d8ba3eb + 79905ab commit 96d22c2

File tree

3 files changed

+276
-0
lines changed

3 files changed

+276
-0
lines changed

‎articles/aks/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -402,6 +402,8 @@
402402
href: open-service-mesh-troubleshoot.md
403403
- name: Disable OSM
404404
href: open-service-mesh-disable-add-on.md
405+
- name: Use cluster extensions (preview)
406+
href: cluster-extensions.md
405407
- name: DevOps
406408
items:
407409
- name: Use Ansible to create AKS clusters

‎articles/aks/cluster-extensions.md

Lines changed: 254 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,254 @@
1+
---
2+
title: Cluster extensions for Azure Kubernetes Service (AKS) (preview)
3+
description: Learn how to deploy and manage the lifecycle of extensions on Azure Kubernetes Service (AKS)
4+
ms.service: container-service
5+
ms.date: 10/13/2021
6+
ms.topic: article
7+
author: nickomang
8+
ms.author: nickoman
9+
---
10+
11+
# Deploy and manage cluster extensions for Azure Kubernetes Service (AKS) (preview)
12+
13+
Cluster extensions provides an Azure Resource Manager driven experience for installation and lifecycle management of services like Azure Machine Learning (ML) on an AKS cluster. This feature enables:
14+
15+
* Azure Resource Manager-based deployment of extensions, including at-scale deployments across AKS clusters.
16+
* Lifecycle management of the extension (Update, Delete) from Azure Resource Manager.
17+
18+
In this article, you will learn about:
19+
> [!div class="checklist"]
20+
21+
> * How to create an extension instance.
22+
> * Available cluster extensions on AKS.
23+
> * How to view, list, update, and delete extension instances.
24+
25+
A conceptual overview of this feature is available in [Cluster extensions - Azure Arc-enabled Kubernetes][arc-k8s-extensions] article.
26+
27+
[!INCLUDE [preview features note](./includes/preview/preview-callout.md)]
28+
29+
## Prerequisites
30+
31+
* An Azure subscription. If you don't have an Azure subscription, you can create a [free account](https://azure.microsoft.com/free).
32+
* [Azure CLI](/cli/azure/install-azure-cli) version >= 2.16.0 installed.
33+
34+
### Register provider for cluster extensions
35+
36+
#### [Azure CLI](#tab/azure-cli)
37+
38+
1. Enter the following commands:
39+
40+
```azurecli-interactive
41+
az provider register --namespace Microsoft.KubernetesConfiguration
42+
az provider register --namespace Microsoft.ContainerService
43+
```
44+
45+
2. Monitor the registration process. Registration may take up to 10 minutes.
46+
47+
```azurecli-interactive
48+
az provider show -n Microsoft.KubernetesConfiguration -o table
49+
az provider show -n Microsoft.ContainerService -o table
50+
```
51+
52+
Once registered, you should see the `RegistrationState` state for these namespaces change to `Registered`.
53+
54+
#### [PowerShell](#tab/azure-powershell)
55+
56+
1. Enter the following commands:
57+
58+
```azurepowershell
59+
Register-AzResourceProvider -ProviderNamespace Microsoft.KubernetesConfiguration
60+
Register-AzResourceProvider -ProviderNamespace Microsoft.ContainerService
61+
```
62+
63+
1. Monitor the registration process. Registration may take up to 10 minutes.
64+
65+
```azurepowershell
66+
Get-AzResourceProvider -ProviderNamespace Microsoft.KubernetesConfiguration
67+
Get-AzResourceProvider -ProviderNamespace Microsoft.ContainerService
68+
```
69+
70+
Once registered, you should see the `RegistrationState` state for these namespaces change to `Registered`.
71+
72+
---
73+
74+
### Register the `AKS-ExtensionManager` preview features
75+
76+
To create an AKS cluster that can use cluster extensions, you must enable the `AKS-ExtensionManager` feature flag on your subscription.
77+
78+
Register the `AKS-ExtensionManager` feature flag by using the [az feature register][az-feature-register] command, as shown in the following example:
79+
80+
```azurecli-interactive
81+
az feature register --namespace "Microsoft.ContainerService" --name "AKS-ExtensionManager"
82+
```
83+
84+
It takes a few minutes for the status to show *Registered*. Verify the registration status by using the [az feature list][az-feature-list] command:
85+
86+
```azurecli-interactive
87+
az feature list -o table --query "[?contains(name, 'Microsoft.ContainerService/AKS-ExtensionManager')].{Name:name,State:properties.state}"
88+
```
89+
90+
When ready, refresh the registration of the *Microsoft.KubernetesConfiguration* and *Microsoft.ContainerService* resource providers by using the [az provider register][az-provider-register] command:
91+
92+
```azurecli-interactive
93+
az provider register --namespace Microsoft.KubernetesConfiguration
94+
az provider register --namespace Microsoft.ContainerService
95+
```
96+
97+
### Setup the Azure CLI extension for cluster extensions
98+
99+
> [!NOTE]
100+
> The minimum supported version for the `k8s-extension` Azure CLI extension is `1.0.0`. If you are unsure what version you have installed, run `az extension show --name k8s-extension` and look for the `version` field.
101+
102+
You will also need the `k8s-extension` Azure CLI extension. Install this by running the following commands:
103+
104+
```azurecli-interactive
105+
az extension add --name k8s-extension
106+
```
107+
108+
If the `k8s-extension` extension is already installed, you can update it to the latest version using the following command:
109+
110+
```azurecli-interactive
111+
az extension update --name k8s-extension
112+
```
113+
114+
## Currently available extensions
115+
116+
>[!NOTE]
117+
> Cluster extensions provides a platform for different extensions to be installed and managed on an AKS cluster. If you are facing issues while using any of these extensions, please open a support ticket with the respective service.
118+
119+
<!--
120+
| Extension | Description |
121+
| --------- | ----------- |
122+
-->
123+
124+
Currently, no extensions are available.
125+
126+
## Supported regions and Kubernetes versions
127+
128+
Cluster extensions can be used on AKS clusters in the regions listed in [Azure Arc enabled Kubernetes region support][arc-k8s-regions].
129+
130+
For supported Kubernetes versions, refer to the corresponding documentation for each extension.
131+
132+
## Usage of cluster extensions
133+
134+
> [!NOTE]
135+
> The samples provided in this article are not complete, and are only meant to showcase functionality. For a comprehensive list of commands and their parameters, please see the [az k8s-extension CLI reference][k8s-extension-reference].
136+
137+
### Create extensions instance
138+
139+
Create a new extension instance with `k8s-extension create`, passing in values for the mandatory parameters. The below command creates an Azure Machine Learning extension instance on your AKS cluster:
140+
141+
```azurecli
142+
az k8s-extension create --name aml-compute --extension-type Microsoft.AzureML.Kubernetes --scope cluster --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters --configuration-settings enableInference=True allowInsecureConnections=True
143+
```
144+
145+
> [!NOTE]
146+
> The Cluster Extensions service is unable to retain sensitive information for more than 48 hours. If the cluster extension agents don't have network connectivity for more than 48 hours and cannot determine whether to create an extension on the cluster, then the extension transitions to `Failed` state. Once in `Failed` state, you will need to run `k8s-extension create` again to create a fresh extension instance.
147+
148+
#### Required parameters
149+
150+
| Parameter name | Description |
151+
|----------------|------------|
152+
| `--name` | Name of the extension instance |
153+
| `--extension-type` | The type of extension you want to install on the cluster. For example: Microsoft.AzureML.Kubernetes |
154+
| `--cluster-name` | Name of the AKS cluster on which the extension instance has to be created |
155+
| `--resource-group` | The resource group containing the AKS cluster |
156+
| `--cluster-type` | The cluster type on which the extension instance has to be created. Specify `managedClusters` as it maps to AKS clusters|
157+
158+
#### Optional parameters
159+
160+
| Parameter name | Description |
161+
|--------------|------------|
162+
| `--auto-upgrade-minor-version` | Boolean property that specifies if the extension minor version will be upgraded automatically or not. Default: `true`. If this parameter is set to true, you cannot set `version` parameter, as the version will be dynamically updated. If set to `false`, extension will not be auto-upgraded even for patch versions. |
163+
| `--version` | Version of the extension to be installed (specific version to pin the extension instance to). Must not be supplied if auto-upgrade-minor-version is set to `true`. |
164+
| `--configuration-settings` | Settings that can be passed into the extension to control its functionality. They are to be passed in as space separated `key=value` pairs after the parameter name. If this parameter is used in the command, then `--configuration-settings-file` can't be used in the same command. |
165+
| `--configuration-settings-file` | Path to the JSON file having key value pairs to be used for passing in configuration settings to the extension. If this parameter is used in the command, then `--configuration-settings` can't be used in the same command. |
166+
| `--configuration-protected-settings` | These settings are not retrievable using `GET` API calls or `az k8s-extension show` commands, and are thus used to pass in sensitive settings. They are to be passed in as space separated `key=value` pairs after the parameter name. If this parameter is used in the command, then `--configuration-protected-settings-file` can't be used in the same command. |
167+
| `--configuration-protected-settings-file` | Path to the JSON file having key value pairs to be used for passing in sensitive settings to the extension. If this parameter is used in the command, then `--configuration-protected-settings` can't be used in the same command. |
168+
| `--scope` | Scope of installation for the extension - `cluster` or `namespace` |
169+
| `--release-namespace` | This parameter indicates the namespace within which the release is to be created. This parameter is only relevant if `scope` parameter is set to `cluster`. |
170+
| `--release-train` | Extension authors can publish versions in different release trains such as `Stable`, `Preview`, etc. If this parameter is not set explicitly, `Stable` is used as default. This parameter can't be used when `autoUpgradeMinorVersion` parameter is set to `false`. |
171+
| `--target-namespace` | This parameter indicates the namespace within which the release will be created. Permission of the system account created for this extension instance will be restricted to this namespace. This parameter is only relevant if the `scope` parameter is set to `namespace`. |
172+
173+
### Show details of an extension instance
174+
175+
View details of a currently installed extension instance with `k8s-extension show`, passing in values for the mandatory parameters:
176+
177+
```azurecli
178+
az k8s-extension show --name azureml --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters
179+
```
180+
181+
### List all extensions installed on the cluster
182+
183+
List all extensions installed on a cluster with `k8s-extension list`, passing in values for the mandatory parameters.
184+
185+
```azurecli
186+
az k8s-extension list --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters
187+
```
188+
189+
### Update extension instance
190+
191+
> [!NOTE]
192+
> Refer to documentation of the extension type (Eg: Azure ML) to learn about the specific settings under ConfigurationSetting and ConfigurationProtectedSettings that are allowed to be updated. For ConfigurationProtectedSettings, all settings are expected to be provided during an update of a single setting. If some settings are omitted, those settings would be considered obsolete and deleted.
193+
194+
Update an existing extension instance with `k8s-extension update`, passing in values for the mandatory parameters. The below command updates the auto-upgrade setting for an Azure Machine Learning extension instance:
195+
196+
```azurecli
197+
az k8s-extension update --name azureml --extension-type Microsoft.AzureML.Kubernetes --scope cluster --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters
198+
```
199+
200+
**Required parameters**
201+
202+
| Parameter name | Description |
203+
|----------------|------------|
204+
| `--name` | Name of the extension instance |
205+
| `--extension-type` | The type of extension you want to install on the cluster. For example: Microsoft.AzureML.Kubernetes |
206+
| `--cluster-name` | Name of the AKS cluster on which the extension instance has to be created |
207+
| `--resource-group` | The resource group containing the AKS cluster |
208+
| `--cluster-type` | The cluster type on which the extension instance has to be created. Specify `managedClusters` as it maps to AKS clusters|
209+
210+
**Optional parameters**
211+
212+
| Parameter name | Description |
213+
|--------------|------------|
214+
| `--auto-upgrade-minor-version` | Boolean property that specifies if the extension minor version will be upgraded automatically or not. Default: `true`. If this parameter is set to true, you cannot set `version` parameter, as the version will be dynamically updated. If set to `false`, extension will not be auto-upgraded even for patch versions. |
215+
| `--version` | Version of the extension to be installed (specific version to pin the extension instance to). Must not be supplied if auto-upgrade-minor-version is set to `true`. |
216+
| `--configuration-settings` | Settings that can be passed into the extension to control its functionality. Only the settings that require an update need to be provided. The provided settings would be replaced with the provided values. They are to be passed in as space separated `key=value` pairs after the parameter name. If this parameter is used in the command, then `--configuration-settings-file` can't be used in the same command. |
217+
| `--configuration-settings-file` | Path to the JSON file having key value pairs to be used for passing in configuration settings to the extension. If this parameter is used in the command, then `--configuration-settings` can't be used in the same command. |
218+
| `--configuration-protected-settings` | These settings are not retrievable using `GET` API calls or `az k8s-extension show` commands, and are thus used to pass in sensitive settings. When updating a setting, all settings are expected to be provided. If some settings are omitted, those settings would be considered obsolete and deleted. They are to be passed in as space separated `key=value` pairs after the parameter name. If this parameter is used in the command, then `--configuration-protected-settings-file` can't be used in the same command. |
219+
| `--configuration-protected-settings-file` | Path to the JSON file having key value pairs to be used for passing in sensitive settings to the extension. If this parameter is used in the command, then `--configuration-protected-settings` can't be used in the same command. |
220+
| `--scope` | Scope of installation for the extension - `cluster` or `namespace` |
221+
| `--release-train` | Extension authors can publish versions in different release trains such as `Stable`, `Preview`, etc. If this parameter is not set explicitly, `Stable` is used as default. This parameter can't be used when `autoUpgradeMinorVersion` parameter is set to `false`. |
222+
223+
### Delete extension instance
224+
225+
Delete an extension instance on a cluster with `k8s-extension delete`, passing in values for the mandatory parameters.
226+
227+
```azurecli
228+
az k8s-extension delete --name azureml --cluster-name <clusterName> --resource-group <resourceGroupName> --cluster-type managedClusters
229+
```
230+
231+
>[!NOTE]
232+
> The Azure resource representing this extension gets deleted immediately. The Helm release on the cluster associated with this extension is only deleted when the agents running on the Kubernetes cluster have network connectivity and can reach out to Azure services again to fetch the desired state.
233+
234+
<!-- when extensions are available, add this section
235+
## Next steps
236+
237+
Learn more about the cluster extensions currently available for AKS:
238+
239+
> [!div class="nextstepaction"]
240+
241+
-->
242+
243+
<!-- LINKS -->
244+
<!-- INTERNAL -->
245+
[arc-k8s-extensions]: ../azure-arc/kubernetes/conceptual-extensions.md
246+
[az-feature-register]: /cli/azure/feature#az_feature_register
247+
[az-feature-list]: /cli/azure/feature#az_feature_list
248+
[az-provider-register]: /cli/azure/provider#az_provider_register
249+
[azure-ml-overview]: <!-- need link -->
250+
[dapr-overview]: <!-- Not yet live -->
251+
[k8s-extension-reference]: /cli/azure/k8s-extension
252+
253+
<!-- EXTERNAL -->
254+
[arc-k8s-regions]: https://azure.microsoft.com/global-infrastructure/services/?products=azure-arc&regions=all

‎articles/aks/limit-egress-traffic.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,26 @@ The following FQDN / application rules are required for AKS clusters that have t
206206
| **`data.policy.azure.us`** | **`HTTPS:443`** | This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service. |
207207
| **`store.policy.azure.us`** | **`HTTPS:443`** | This address is used to pull the Gatekeeper artifacts of built-in policies. |
208208

209+
## Cluster extensions
210+
211+
### Required FQDN / application rules
212+
213+
The following FQDN / application rules are required for using cluster extensions on AKS clusters.
214+
215+
| FQDN | Port | Use |
216+
|-----------------------------------------------|-----------|----------|
217+
| **`<region>.dp.kubernetesconfiguration.azure.com`** | **`HTTPS:443`** | This address is used to fetch configuration information from the Cluster Extensions service and report extension status to the service.|
218+
| **`mcr.microsoft.com, *.data.mcr.microsoft.com`** | **`HTTPS:443`** | This address is required to pull container images for installing cluster extension agents on AKS cluster.|
219+
220+
#### Azure US Government Required FQDN / application rules
221+
222+
The following FQDN / application rules are required for using cluster extensions on AKS clusters.
223+
224+
| FQDN | Port | Use |
225+
|-----------------------------------------------|-----------|----------|
226+
| **`<region>.dp.kubernetesconfiguration.azure.us`** | **`HTTPS:443`** | This address is used to fetch configuration information from the Cluster Extensions service and report extension status to the service. |
227+
| **`mcr.microsoft.com, *.data.mcr.microsoft.com`** | **`HTTPS:443`** | This address is required to pull container images for installing cluster extension agents on AKS cluster.|
228+
209229
## Restrict egress traffic using Azure firewall
210230

211231
Azure Firewall provides an Azure Kubernetes Service (`AzureKubernetesService`) FQDN Tag to simplify this configuration.

0 commit comments

Comments
 (0)