You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Use Image Cleaner to clean up stale images on your Azure Kubernetes Service cluster (preview)
11
+
# Use Image Cleaner to clean up stale images on your Azure Kubernetes Service (AKS) cluster
12
12
13
-
It's common to use pipelines to build and deploy images on Azure Kubernetes Service (AKS) clusters. While great for image creation, this process often doesn't account for the stale images left behind and can lead to image bloat on cluster nodes. These images can present security issues as they may contain vulnerabilities. By cleaning these unreferenced images, you can remove an area of risk in your clusters. When done manually, this process can be time intensive, which Image Cleaner can mitigate via automatic image identification and removal.
13
+
It's common to use pipelines to build and deploy images on Azure Kubernetes Service (AKS) clusters. While great for image creation, this process often doesn't account for the stale images left behind and can lead to image bloat on cluster nodes. These images may contain vulnerabilities, which may create security issues. To remove security risks in your clusters, you can clean these unreferenced images. Manually cleaning images can be time intensive. Image Cleaner performs automatic image identification and removal, which mitigates the risk of stale images and reduces the time required to clean them up.
14
14
15
15
> [!NOTE]
16
16
> Image Cleaner is a feature based on [Eraser](https://azure.github.io/eraser).
17
-
> On an AKS cluster, the feature name and property name is `Image Cleaner` while the relevant Image Cleaner pods' names contain `Eraser`.
18
-
19
-
[!INCLUDE [preview features callout](./includes/preview/preview-callout.md)]
17
+
> On an AKS cluster, the feature name and property name is `Image Cleaner`, while the relevant Image Cleaner pods' names contain `Eraser`.
20
18
21
19
## Prerequisites
22
20
23
21
* An Azure subscription. If you don't have an Azure subscription, you can create a [free account](https://azure.microsoft.com/free).
24
-
*[Azure CLI][azure-cli-install] or [Azure PowerShell][azure-powershell-install] and the `aks-preview` 0.5.96 or later CLI extension installed.
25
-
* The `EnableImageCleanerPreview` feature flag registered on your subscription:
26
-
27
-
### [Azure CLI](#tab/azure-cli)
28
-
29
-
First, install the aks-preview extension by running the following command:
30
-
31
-
```azurecli
32
-
az extension add --name aks-preview
33
-
```
34
-
35
-
Run the following command to update to the latest version of the extension released:
36
-
37
-
```azurecli
38
-
az extension update --name aks-preview
39
-
```
40
-
41
-
Then register the `EnableImageCleanerPreview` feature flag by using the [az feature register][az-feature-register] command, as shown in the following example:
42
-
43
-
```azurecli-interactive
44
-
az feature register --namespace "Microsoft.ContainerService" --name "EnableImageCleanerPreview"
45
-
```
46
-
47
-
It takes a few minutes for the status to show *Registered*. Verify the registration status by using the [az feature show][az-feature-show] command:
22
+
* Azure CLI version 2.49.0 or later. Run `az --version` to find your version. If you need to install or upgrade, see [Install Azure CLI][azure-cli-install].
48
23
49
-
```azurecli-interactive
50
-
az feature show --namespace "Microsoft.ContainerService" --name "EnableImageCleanerPreview"
51
-
```
52
-
53
-
When the status reflects *Registered*, refresh the registration of the *Microsoft.ContainerService* resource provider by using the [az provider register][az-provider-register] command:
24
+
## Limitations
54
25
55
-
```azurecli-interactive
56
-
az provider register --namespace Microsoft.ContainerService
57
-
```
26
+
Image Cleaner doesn't yet support Windows node pools or AKS virtual nodes.
58
27
59
-
### [Azure PowerShell](#tab/azure-powershell)
28
+
##How Image Cleaner works
60
29
61
-
Register the `EnableImageCleanerPreview` feature flag by using the [Register-AzProviderPreviewFeature][register-azproviderpreviewfeature] cmdlet, as shown in the following example:
30
+
When you enable Image Cleaner, it deploys an `eraser-controller-manager` pod, which generates an `ImageList` CRD. The eraser pods running on each node clean up any unreferenced and vulnerable images according to the `ImageList`. A [trivy][trivy] scan helps determine vulnerability and flags images with a classification of `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL`. Image Cleaner automatically generates an updated `ImageList` based on a set time interval and can also be supplied manually. Once Image Cleaner generates an `ImageList`, it removes all images in the list from node VMs.
:::image type="content" source="./media/image-cleaner/image-cleaner.jpg" alt-text="Screenshot of a diagram showing ImageCleaner's workflow. The ImageCleaner pods running on the cluster can generate an ImageList, or manual input can be provided.":::
66
33
67
-
It takes a few minutes for the status to show *Registered*. Verify the registration status by using the [Get-AzProviderPreviewFeature][get-azproviderpreviewfeature] cmdlet:
With Image Cleaner, you can choose between manual and automatic mode and the following configuration options:
73
37
74
-
When ready, refresh the registration of the *Microsoft.ContainerService* resource provider by using the [Register-AzResourceProvider][register-azresourceprovider] command:
38
+
|Name|Description|Required|
39
+
|----|-----------|--------|
40
+
|`--enable-image-cleaner`|Enable the Image Cleaner feature for an AKS cluster|Yes, unless disable is specified|
41
+
|`--disable-image-cleaner`|Disable the Image Cleaner feature for an AKS cluster|Yes, unless enable is specified|
42
+
|`--image-cleaner-interval-hours`|This parameter determines the interval time (in hours) Image Cleaner uses to run. The default value for Azure CLI is one week, the minimum value is 24 hours and the maximum is three months.|Not required for Azure CLI, required for ARM template or other clients|
> After disabling Image Cleaner, the old configuration still exists. This means if you enable the feature again without explicitly passing configuration, the existing value is used instead of the default.
79
46
80
-
---
47
+
## Enable Image Cleaner on your AKS cluster
81
48
82
-
##Limitations
49
+
### Enable Image Cleaner on a new cluster
83
50
84
-
Image Cleaner does not support the following:
51
+
* Enable Image Cleaner on a new AKS cluster using the [`az aks create`][az-aks-create] command with the `--enable-image-cleaner` parameter.
85
52
86
-
* ARM64 node pools. For more information, see [Azure Virtual Machines with ARM-based processors][arm-vms].
87
-
* Windows node pools.
53
+
```azurecli-interactive
54
+
az aks create -g myResourceGroup -n myManagedCluster \
55
+
--enable-image-cleaner
56
+
```
88
57
89
-
##How Image Cleaner works
58
+
### Enable Image Cleaner on an existing cluster
90
59
91
-
When enabled, an `eraser-controller-manager` pod is deployed, which generates an `ImageList` CRD. The eraser pods running on each nodes will clean up the unreferenced and vulnerable images according to the ImageList. Vulnerability is determined based on a [trivy][trivy] scan, after which images with a `LOW`, `MEDIUM`, `HIGH`, or `CRITICAL` classification are flagged. An updated `ImageList` will be automatically generated by Image Cleaner based on a set time interval, and can also be supplied manually.
60
+
* Enable Image Cleaner on an existing AKS cluster using the [`az aks update`][az-aks-update] command.
92
61
62
+
```azurecli-interactive
63
+
az aks update -g myResourceGroup -n myManagedCluster \
64
+
--enable-image-cleaner
65
+
```
93
66
67
+
### Update the Image Cleaner interval on a new or existing cluster
94
68
95
-
Once an `ImageList` is generated, Image Cleaner will remove all the images in the list from node VMs.
69
+
* Update the Image Cleaner interval on a new or existing AKS cluster using the `--image-cleaner-interval-hours` parameter.
96
70
97
-
:::image type="content" source="./media/image-cleaner/image-cleaner.jpg" alt-text="Screenshot of a diagram showing ImageCleaner's workflow. The ImageCleaner pods running on the cluster can generate an ImageList, or manual input can be provided.":::
71
+
```azurecli-interactive
72
+
# Update the interval on a new cluster
73
+
az aks create -g myResourceGroup -n myManagedCluster \
74
+
--enable-image-cleaner \
75
+
--image-cleaner-interval-hours 48
76
+
# Update the interval on an existing cluster
77
+
az aks update -g myResourceGroup -n myManagedCluster \
78
+
--image-cleaner-interval-hours 48
79
+
```
98
80
99
-
## Configuration options
81
+
After you enable the feature, the `eraser-controller-manager-xxx` pod and `collector-aks-xxx` pod are deployed. The `eraser-aks-xxx` pod contains *three* containers:
100
82
101
-
In addition to choosing between manual and automatic mode, there are several options for Image Cleaner:
- **Collector container**: Collects nonrunning and unused images
85
+
- **Remover container**: Removes these images from cluster nodes
102
86
103
-
|Name|Description|Required|
104
-
|----|-----------|--------|
105
-
|--enable-image-cleaner|Enable the Image Cleaner feature for an AKS cluster|Yes, unless disable is specified|
106
-
|--disable-image-cleaner|Disable the Image Cleaner feature for an AKS cluster|Yes, unless enable is specified|
107
-
|--image-cleaner-interval-hours|This parameter determines the interval time (in hours) Image Cleaner will use to run. The default value for Azure CLI is one week, the minimum value is 24 hours and the maximum is three months.|Not required for Azure CLI, required for ARM template or other clients|
87
+
Image Cleaner generates an `ImageList` containing nonrunning and vulnerable images at the desired interval based on your configuration. Image Cleaner automatically removes these images from cluster nodes.
108
88
109
-
> [!NOTE]
110
-
> After disabling Image Cleaner, the old configuration still exists. This means that if you enable the feature again without explicitly passing configuration, the existing value will be used rather than the default.
89
+
## Manually remove images using Image Cleaner
111
90
112
-
## Enable Image Cleaner on your AKS cluster
91
+
1. Create an `ImageList` using the following example YAML named `image-list.yml`.
113
92
114
-
To create a new AKS cluster using the default interval, use [az aks create][az-aks-create]:
93
+
```yml
94
+
apiVersion: eraser.sh/v1alpha1
95
+
kind: ImageList
96
+
metadata:
97
+
name: imagelist
98
+
spec:
99
+
images:
100
+
- docker.io/library/alpine:3.7.3 # You can also use "*" to specify all non-running images
101
+
```
115
102
116
-
```azurecli-interactive
117
-
az aks create -g MyResourceGroup -n MyManagedCluster \
118
-
--enable-image-cleaner
119
-
```
103
+
2. Apply the `ImageList` to your cluster using the `kubectl apply` command.
120
104
121
-
To enable on an existing AKS cluster, use [az aks update][az-aks-update]:
105
+
```bash
106
+
kubectl apply -f image-list.yml
107
+
```
122
108
123
-
```azurecli-interactive
124
-
az aks update -g MyResourceGroup -n MyManagedCluster \
125
-
--enable-image-cleaner
126
-
```
109
+
Applying the `ImageList` triggers a job named `eraser-aks-xxx`, which causes Image Cleaner to remove the desired images from all nodes. Unlike the `eraser-aks-xxx` pod under autoclean with *three* containers, the eraser-pod here has only *one* container.
127
110
128
-
The `--image-cleaner-interval-hours` parameter can be specified at creation time or for an existing cluster. For example, the following command updates the interval for a cluster with Image Cleaner already enabled:
111
+
## Image exclusion list
129
112
130
-
```azurecli-interactive
131
-
az aks update -g MyResourceGroup -n MyManagedCluster \
132
-
--image-cleaner-interval-hours 48
133
-
```
113
+
Images specified in the exclusion list aren't removed from the cluster. Image Cleaner supports system and user-defined exclusion lists. It's not supported to edit the system exclusion list.
134
114
135
-
After the feature is enabled, the `eraser-controller-manager-xxx` pod and `collector-aks-xxx` pod will be deployed.
136
-
Based on your configuration, Image Cleaner will generate an `ImageList` containing non-running and vulnerable images at the desired interval. Image Cleaner will automatically remove these images from cluster nodes.
115
+
### Check the system exclusion list
137
116
138
-
## Manually remove images
117
+
* Check the system exclusion list using the following `kubectl get` command.
139
118
140
-
To manually remove images from your cluster using Image Cleaner, first create an `ImageList`. For example, save the following as `image-list.yml`:
119
+
```bash
120
+
kubectl get -n kube-system cm eraser-system-exclusion -o yaml
121
+
```
141
122
142
-
```yml
143
-
apiVersion: eraser.sh/v1alpha1
144
-
kind: ImageList
145
-
metadata:
146
-
name: imagelist
147
-
spec:
148
-
images:
149
-
- docker.io/library/alpine:3.7.3 # You can also use "*" to specify all non-running images
150
-
```
123
+
### Create a user-defined exclusion list
151
124
152
-
And apply it to the cluster:
125
+
1. Create a sample JSON file to contain excluded images.
153
126
154
-
```bash
155
-
kubectl apply -f image-list.yml
156
-
```
127
+
```bash
128
+
cat > sample.json <<EOF
129
+
{"excluded": ["excluded-image-name"]}
130
+
EOF
131
+
```
157
132
158
-
A job named `eraser-aks-xxx`will be triggered which causes Image Cleaner to remove the desired images from all nodes.
133
+
2. Create a `configmap` using the sample JSON file using the following `kubectl create` and `kubectl label` command.
To stop using Image Cleaner, you can disable it via the `--disable-image-cleaner` flag:
140
+
3. Verify the images are in the exclusion list using the following `kubectl logs` command.
163
141
164
-
```azurecli-interactive
165
-
az aks update -g MyResourceGroup -n MyManagedCluster
166
-
--disable-image-cleaner
167
-
```
142
+
```bash
143
+
kubectl logs -n kube-system <eraser-pod-name>
144
+
```
168
145
169
-
## Logging
146
+
## Image Cleaner image logs
170
147
171
-
Deletion image logs are stored in `eraser-aks-nodepool-xxx` pods for manually deleted images, and in `collector-aks-nodes-xxx` pods for automatically deleted images.
148
+
Deletion image logs are stored in `eraser-aks-nodepool-xxx` pods for manually deleted images and in `collector-aks-nodes-xxx` pods for automatically deleted images.
172
149
173
-
You can view these logs by running`kubectl logs <pod name> -n kubesystem`. However, this command may return only the most recent logs, since older logs are routinely deleted. To view all logs, follow these steps to enable the [Azure Monitor add-on](./monitor-aks.md) and use the Container Insights pod log table.
150
+
You can view these logs using the `kubectl logs <pod name> -n kubesystem` command. However, this command may return only the most recent logs, since older logs are routinely deleted. To view all logs, follow these steps to enable the [Azure Monitor add-on](./monitor-aks.md) and use the Container Insights pod log table.
174
151
175
-
1. Ensure that Azure monitoring is enabled on the cluster. For detailed steps, see [Enable Container Insights for AKS cluster](../azure-monitor/containers/container-insights-enable-aks.md#existing-aks-cluster).
152
+
1. Ensure Azure Monitoring is enabled on your cluster. For detailed steps, see [Enable Container Insights on AKS clusters](../azure-monitor/containers/container-insights-enable-aks.md#existing-aks-cluster).
176
153
177
-
1. Get the Log Analytics resource ID:
154
+
2. Get the Log Analytics resource ID using the [`az aks show`][az-aks-show] command.
178
155
179
156
```azurecli
180
-
az aks show -g <resourceGroupofAKSCluster> -n <nameofAksCluster>
157
+
az aks show -g myResourceGroup -n myManagedCluster
181
158
```
182
159
183
-
After a few minutes, the command returns JSON-formatted information about the solution, including the workspace resource ID:
160
+
After a few minutes, the command returns JSON-formatted information about the solution, including the workspace resource ID.
184
161
185
-
```json
162
+
```json
186
163
"addonProfiles": {
187
164
"omsagent": {
188
165
"config": {
@@ -191,11 +168,11 @@ You can view these logs by running `kubectl logs <pod name> -n kubesystem`. Howe
191
168
"enabled": true
192
169
}
193
170
}
194
-
```
171
+
```
195
172
196
-
1. In the Azure portal, search for the workspace resource ID, then select **Logs**.
173
+
3. In the Azure portal, search for the workspace resource ID, then select **Logs**.
197
174
198
-
1. Copy this query into the table, replacing `name` with either `eraser-aks-nodepool-xxx` (for manual mode) or `collector-aks-nodes-xxx` (for automatic mode).
175
+
4. Copy this query into the table, replacing `name` with either `eraser-aks-nodepool-xxx` (for manual mode) or `collector-aks-nodes-xxx` (for automatic mode).
199
176
200
177
```kusto
201
178
let startTimestamp = ago(1h);
@@ -217,23 +194,23 @@ You can view these logs by running `kubectl logs <pod name> -n kubesystem`. Howe
217
194
| order by TimeGenerated desc
218
195
```
219
196
220
-
1. Select **Run**. Any deleted image logs will appear in the **Results** area.
197
+
5. Select **Run**. Any deleted image logs appear in the **Results** area.
221
198
222
199
:::image type="content" source="media/image-cleaner/eraser-log-analytics.png" alt-text="Screenshot showing deleted image logs in the Azure portal." lightbox="media/image-cleaner/eraser-log-analytics.png":::
223
200
201
+
## Disable Image Cleaner
202
+
203
+
* Disable Image Cleaner on your cluster using the [`az aks update`][az-aks-update] command with the `--disable-image-cleaner` parameter.
204
+
205
+
```azurecli-interactive
206
+
az aks update -g myResourceGroup -n myManagedCluster \
0 commit comments