You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: AKS-Arc/aks-troubleshoot.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,10 +3,10 @@ title: Troubleshoot common issues in AKS enabled by Azure Arc
3
3
description: Learn about common issues and workarounds in AKS enabled by Arc.
4
4
ms.topic: how-to
5
5
author: sethmanheim
6
-
ms.date: 06/27/2025
6
+
ms.date: 07/17/2025
7
7
ms.author: sethm
8
-
ms.lastreviewed: 04/01/2025
9
-
ms.reviewer: abha
8
+
ms.lastreviewed: 07/17/2025
9
+
ms.reviewer: rcheeran
10
10
11
11
---
12
12
@@ -24,10 +24,10 @@ The following sections describe known issues for AKS enabled by Azure Arc:
24
24
25
25
| AKS Arc CRUD operation | Issue | Fix status |
26
26
|------------------------|-------|------------|
27
-
| AKS cluster create |[Can't create AKS cluster or scale node pool because of issues with AKS Arc images](gallery-image-not-usable.md)| Partially fixed in 2503 release |
28
-
| AKS steady state |[AKS Arc telemetry pod consumes too much memory and CPU](telemetry-pod-resources.md)| Active |
29
-
| AKS steady state |[Disk space exhaustion on control plane VMs due to accumulation of kube-apiserver audit logs](kube-apiserver-log-overflow.md)| Active |
30
27
| AKS cluster delete |[Deleted AKS Arc cluster still visible on Azure portal](deleted-cluster-visible.md)| Active |
28
+
| AKS steady state |[AKS Arc telemetry pod consumes too much memory and CPU](telemetry-pod-resources.md)| Fixed in 2507 release |
29
+
| AKS cluster create |[Can't create AKS cluster or scale node pool because of issues with AKS Arc images](gallery-image-not-usable.md)| Fixed in 2507 release |
30
+
| AKS steady state |[Disk space exhaustion on control plane VMs due to accumulation of kube-apiserver audit logs](kube-apiserver-log-overflow.md)| Fixed in 2507 release |
31
31
| AKS cluster upgrade |[AKS Arc cluster stuck in "Upgrading" state](cluster-upgrade-status.md)| Fixed in 2505 release |
32
32
| AKS cluster delete |[Can't fully delete AKS Arc cluster with PodDisruptionBudget (PDB) resources](delete-cluster-pdb.md)| Fixed in 2503 release |
33
33
| Azure portal |[Can't see VM SKUs on Azure portal](check-vm-sku.md)| Fixed in 2411 release |
@@ -43,7 +43,7 @@ The following sections describe known issues for AKS enabled by Azure Arc:
Copy file name to clipboardExpand all lines: AKS-Arc/gallery-image-not-usable.md
+6-5Lines changed: 6 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,8 +4,8 @@ description: Learn about a known issue with Kubernetes cluster create or nodepoo
4
4
ms.topic: troubleshooting
5
5
author: sethmanheim
6
6
ms.author: sethm
7
-
ms.date: 04/01/2025
8
-
ms.reviewer: abha
7
+
ms.date: 07/17/2025
8
+
ms.reviewer: rcheeran
9
9
10
10
---
11
11
@@ -43,9 +43,10 @@ When you run `az aksarc get-versions`, you see the following errors:
43
43
44
44
## Mitigation
45
45
46
-
- This issue was fixed in [AKS on Azure Local, version 2503](aks-whats-new-23h2.md#release-2503).
47
-
- Upgrade your Azure Local deployment to the 2503 build.
48
-
- Once updated, confirm that the images have been downloaded successfully by running the `az aksarc get-versions` command.
46
+
This issue was fixed in [AKS on Azure Local, version 2507](/azure/azure-local/whats-new?view=azloc-2507&preserve-view=true#features-and-improvements-in-2507). Upgrade your Azure Local deployment to the 2507 build.
47
+
48
+
- Upgrade your Azure Local deployment to the 2507 build.
49
+
- Once updated, confirm that the images were downloaded successfully by running the `az aksarc get-versions` command.
49
50
- For new AKS clusters: new AKS clusters should now be created successfully.
50
51
- For scaling existing AKS clusters: scaling existing AKS clusters continues to encounter issues. Please file a support case.
Copy file name to clipboardExpand all lines: AKS-Arc/kube-apiserver-log-overflow.md
+5-1Lines changed: 5 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: Learn about a known issue with disk space exhaustion on the control
4
4
ms.topic: troubleshooting
5
5
author: sethmanheim
6
6
ms.author: sethm
7
-
ms.date: 04/01/2025
7
+
ms.date: 07/17/2025
8
8
ms.reviewer: abha
9
9
10
10
---
@@ -44,6 +44,10 @@ The issue occurs because the `--audit-log-maxbackup` value is set to 0. This set
44
44
45
45
## Mitigation
46
46
47
+
This issue was fixed in [AKS on Azure Local, version 2507](/azure/azure-local/whats-new?view=azloc-2507&preserve-view=true#features-and-improvements-in-2507). Upgrade your Azure Local deployment to the 2507 build.
48
+
49
+
### Workaround for Azure Local versions 2503 or 2504
50
+
47
51
To resolve the issue temporarily, you must manually clean up the old audit logs. Follow these steps:
48
52
49
53
- SSH into the control plane virtual machine (VM) of your AKS Arc cluster.
| 1.30 |[2503](aks-whats-new-23h2.md#features-and-improvements)| Generally Available | TBD |
46
-
| 1.29 |[2503](aks-whats-new-23h2.md#features-and-improvements), [2411](aks-whats-new-23h2.md#features-and-improvements), [2408](aks-whats-new-23h2.md#release-2408)| Generally available | TBD |
47
-
| 1.28 |[2503](aks-whats-new-23h2.md#features-and-improvements), [2411](aks-whats-new-23h2.md#features-and-improvements), [2408](aks-whats-new-23h2.md#release-2408), [2405](aks-whats-new-23h2.md#release-2405)| No more patch versions/CVE updates |2503 release |
45
+
| 1.30 |[2507, 2503](aks-whats-new-23h2.md#features-and-improvements)| Generally Available | TBD |
46
+
| 1.29 |[2507, 2503](aks-whats-new-23h2.md#features-and-improvements), [2411](aks-whats-new-23h2.md#features-and-improvements), [2408](aks-whats-new-23h2.md#release-2408)| Generally available | TBD |
47
+
| 1.28 |[2507, 2503](aks-whats-new-23h2.md#features-and-improvements), [2411](aks-whats-new-23h2.md#features-and-improvements), [2408](aks-whats-new-23h2.md#release-2408), [2405](aks-whats-new-23h2.md#release-2405)| No more patch versions/CVE updates |2507 release |
48
48
| 1.27 |[2411](aks-whats-new-23h2.md#features-and-improvements), [2408](aks-whats-new-23h2.md#release-2408), [2405](aks-whats-new-23h2.md#release-2405), 2402 | No more patch versions/CVE updates | 2411 release |
49
49
| 1.26 |[2405](aks-whats-new-23h2.md#release-2405), 2402 and older | No more patch versions/CVE updates | 2405 release |
50
50
51
+
> [!NOTE]
52
+
> Kubernetes version 1.31 is planned for the next Azure Local release - 2508
53
+
51
54
### AKS Arc supported Kubernetes minor and patch versions per release
AKS defines a generally available (GA) version as a version that's available for download when deploying or updating AKS enabled by Arc. AKS supports three GA minor versions of Kubernetes:
66
+
AKS Arc defines a generally available (GA) version as a version that's available for download when deploying or updating clusters.
67
+
AKS Arc supports three GA minor versions:
63
68
64
-
* The latest GA minor version that is released for AKS (referred to as *N*).
65
-
* Two previous minor versions. Each supported minor version also supports stable patches.
66
-
67
-
AKS might also support preview versions, which are explicitly labeled as previews.
68
-
69
-
> [!NOTE]
70
-
> AKS uses safe deployment practices which involve gradual region deployment. This means it can take up to 10 business days for a new release or a new version to be available in all regions.
71
-
72
-
The supported window of Kubernetes versions on AKS is known as "N-2": (N (Latest release) - 2 (minor versions)).
69
+
* The latest GA version (N).
70
+
* The two previous minor versions (N-1 and N-2).
73
71
74
72
For example, if AKS introduces **1.30** today, support is provided for the following versions:
75
73
76
74
| New minor version | Supported Version List |
77
75
| ----------------- | ---------------------- |
78
-
| 1.30 | 1.30, 1.29, 1.28 |
76
+
| 1.30 | 1.30, 1.29, 1.28|
79
77
80
78
When a new minor version is introduced, the oldest minor version and patch releases supported are deprecated and removed. For example, the current supported version list is:
81
79
@@ -85,10 +83,7 @@ When a new minor version is introduced, the oldest minor version and patch relea
85
83
86
84
When AKS releases 1.30.\*, all the 1.27.\* versions are removed and go out of support in 30 days.
87
85
88
-
> [!NOTE]
89
-
> If you're running an unsupported Kubernetes version, you are asked to upgrade when requesting support for the cluster. Clusters running unsupported Kubernetes releases are not covered by the [AKS support policies](./support-policies.md).
90
-
91
-
In addition to this policy, AKS supports a maximum of two patch releases of a given minor version. Given the following supported versions:
86
+
In addition to this policy, AKS Arc supports a maximum of two patch releases of a given minor version. Given the following supported versions:
92
87
93
88
```output
94
89
Current Supported Version List
@@ -104,6 +99,13 @@ New Supported Version List
104
99
1.29.*9*, 1.29.*8*, 1.28.*11*, 1.28.*10*
105
100
```
106
101
102
+
AKS Arc reserves the right to deprecate patches if a critical CVE or security vulnerability is detected. For awareness on patch availability and any ad-hoc deprecation, see the version release notes.
103
+
104
+
AKS Arc might also support preview versions, which are explicitly labeled as previews.
105
+
106
+
> [!NOTE]
107
+
> If you're running an unsupported Kubernetes version, you are asked to upgrade when requesting support for the cluster. Clusters running unsupported Kubernetes releases are not covered by the [AKS Arc support policies](./support-policies.md).
108
+
107
109
### Supported `kubectl` versions
108
110
109
111
You can use one minor version older or newer of `kubectl` relative to your **kube-apiserver** version, consistent with the [Kubernetes support policy for kubectl](https://kubernetes.io/docs/setup/release/version-skew-policy/#kubectl).
@@ -115,30 +117,24 @@ To install or update your version of `kubectl`, run `az aks install-cli`.
115
117
116
118
For new minor versions of Kubernetes:
117
119
118
-
* AKS publishes a pre-announcement with the planned date of a new version release and respective old version deprecation in the [AKS release notes](https://aka.ms/aks-hci-relnotes) at least 30 days prior to removal.
120
+
* AKS Arc publishes a pre-announcement with the planned date of a new version release and respective old version deprecation in the [AKS release notes](https://aka.ms/aks-hci-relnotes) at least 30 days prior to removal.
119
121
* Users have 30 days from version removal to upgrade to a supported minor version release to continue receiving support.
120
122
121
-
For new patch versions of Kubernetes:
122
-
123
-
* Because of the urgent nature of patch versions, they can be introduced into the service as they become available.
124
-
* In general, AKS doesn't broadly communicate the release of new patch versions. However, AKS constantly monitors and validates available CVE patches to support them in AKS in a timely manner. If a critical patch is found or user action is required, AKS notifies users to upgrade to the newly available patch.
125
-
* Users have 30 days from a patch release's removal from AKS to upgrade to a supported patch and continue receiving support.
126
-
127
123
### Supported versions policy exceptions
128
124
129
-
AKS reserves the right to add or remove new/existing versions with one or more critical production-impacting bugs or security issues without advance notice.
125
+
AKS Arc reserves the right to add or remove new/existing versions with one or more critical production-impacting bugs or security issues without advance notice.
130
126
131
127
Specific patch releases may be skipped or rollout accelerated, depending on the severity of the bug or security issue.
132
128
133
129
## FAQ
134
130
135
131
### How does Microsoft notify me of new Kubernetes versions?
136
132
137
-
The AKS team publishes pre-announcements with planned dates of new Kubernetes versions in the AKS Arc documentation.
133
+
The AKS Arc team publishes pre-announcements with planned dates of new Kubernetes versions in the AKS Arc documentation.
138
134
139
135
### How often should I expect to upgrade Kubernetes versions to stay in support?
140
136
141
-
Starting with Kubernetes 1.19, the [open source community expanded support to one year](https://kubernetes.io/blog/2020/08/31/kubernetes-1-19-feature-one-year-support/). AKS commits to enabling patches and support matching the upstream commitments. For Kubernetes clusters on 1.19 and greater, you'll be able to upgrade a minimum of once a year to stay on a supported version.
137
+
Starting with Kubernetes 1.19, the [open source community expanded support to one year](https://kubernetes.io/blog/2020/08/31/kubernetes-1-19-feature-one-year-support/). AKS Arc commits to enabling patches and support matching the upstream commitments. For Kubernetes clusters on 1.19 and greater, you'll be able to upgrade a minimum of once a year to stay on a supported version.
142
138
143
139
### What happens when a user upgrades a Kubernetes cluster with a minor version that isn't supported?
144
140
@@ -156,11 +152,11 @@ Downgrades are not supported.
156
152
* The version you're running is outside of the supported versions list.
157
153
* You'll be asked to upgrade the cluster to a supported version when requesting support, unless you're within the 30-day grace period after version deprecation.
158
154
159
-
Additionally, AKS doesn't make any runtime (or other) guarantees for clusters outside of the supported versions list.
155
+
Additionally, AKS Arc doesn't make any runtime (or other) guarantees for clusters outside of the supported versions list.
160
156
161
157
### What happens when I scale a Kubernetes cluster with a minor version that isn't supported?
162
158
163
-
For minor versions not supported by AKS, scaling in or out should continue to work. Since there are no Quality of Service guarantees, we recommend upgrading to bring your cluster back into support.
159
+
For minor versions not supported by AKS Arc, scaling in or out should continue to work. Since there are no Quality of Service guarantees, we recommend upgrading to bring your cluster back into support.
164
160
165
161
### Can I skip multiple Kubernetes versions during a cluster upgrade?
This issue was fixed in [AKS on Azure Local, version 2507](/azure/azure-local/whats-new?view=azloc-2507&preserve-view=true#features-and-improvements-in-2507). Upgrade your Azure Local deployment to the 2507 build.
31
+
32
+
### Workaround for Azure Local versions 2506 and older
33
+
31
34
To resolve this issue, set default **resource limits** for the pods in the `kube-system` namespace.
32
35
33
-
### Important notes
36
+
####Important notes
34
37
35
38
- Verify if you have any pods in the **kube-system** namespace that might require more memory than the default limit setting. If so, adjustments might be needed.
36
39
- The **LimitRange** is applied to the **namespace**; in this case, the `kube-system` namespace. The default resource limits also apply to new pods that don't specify their own limits.
@@ -40,7 +43,7 @@ To resolve this issue, set default **resource limits** for the pods in the `kube
40
43
41
44
To proceed with setting the resource limits, you can run the following script. While the script uses `az aksarc get-credentials`, you can also use `az connectedk8s proxy` to get the proxy kubeconfig and access the Kubernetes cluster.
42
45
43
-
### Define the LimitRange YAML to set default CPU and memory limits
46
+
####Define the LimitRange YAML to set default CPU and memory limits
44
47
45
48
```powershell
46
49
# Set the $cluster_name and $resource_group of the aksarc cluster
@@ -76,7 +79,7 @@ sleep 5
76
79
kubectl get pods -l app=akshci-telemetry -n kube-system --kubeconfig "./kubeconfig-$cluster_name"
77
80
```
78
81
79
-
### Validate if the resource limits were applied correctly
82
+
####Validate if the resource limits were applied correctly
80
83
81
84
1. Check the resource limits in the pod's YAML configuration:
![learn-build-service-prod[bot]](https://avatars.githubusercontent.com/in/237442?v=4&size=40){kind=avatar}
0 commit comments