You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
title: AKS cluster upgrade fails with UnsatisfiablePDB error
3
+
description: Provides solutions to the UnsatisfiablePDB error when you try to upgrade an Azure Kubernetes Service (AKS) cluster.
4
+
ms.date: 10/27/2023
5
+
ms.reviewer: chiragpa, v-weizhu
6
+
ms.service: azure-kubernetes-service
7
+
ms.custom: sap:Create, Upgrade, Scale and Delete operations (cluster or nodepool)
8
+
#Customer intent: As an Azure Kubernetes Services (AKS) user, I want to troubleshoot an Azure Kubernetes Service cluster upgrade that failed because of a UnsatisfiablePDB error so that I can upgrade the cluster successfully.
9
+
---
10
+
11
+
# Error "UnsatisfiablePDB" when upgrading an AKS cluster
12
+
13
+
This article discusses how to identify and resolve the "UnsatisfiablePDB" error that might occur when you try to [upgrade an Azure Kubernetes Service (AKS) cluster](/azure/aks/upgrade-aks-cluster).
14
+
15
+
## Prerequisites
16
+
17
+
This article requires Azure CLI version 2.53.0 or a later version. Run `az --version` to find your installed version. If you need to install or upgrade the Azure CLI, see [How to install the Azure CLI](/cli/azure/install-azure-cli).
18
+
19
+
## Symptoms
20
+
21
+
An AKS cluster upgrade operation fails with the following error message:
22
+
23
+
> Code: UnsatisfiablePDB
24
+
> Message: 1 error occurred:
25
+
> \* PDB \<pdb-namespace>/\<pdb-name> has maxunavailble == 0 can't proceed with put operation
26
+
27
+
## Cause
28
+
29
+
Before starting an upgrade operation, AKS checks the cluster for any existing [Pod Disruption Budgets (PDBs)](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) that have the `maxUnavailable` parameter set to 0. Such PDBs are likely to block node drain operations. If node drain operations are blocked, the cluster upgrade operation can't complete successfully. This might potentially cause the cluster to be in a failed state.
30
+
31
+
After receiving the "UnsatisfiablePDB" error, you can confirm the PDB's status by running the following command:
32
+
33
+
```console
34
+
$ kubectl get pdb <pdb-name> -n <pdb-namespace>
35
+
```
36
+
37
+
The output of this command should be similar to the following one:
38
+
39
+
```output
40
+
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
41
+
<pdb-name> N/A 0 0 49s
42
+
```
43
+
44
+
If the value of `MAX UNAVAILABLE` is 0, the node drain fails during the upgrade process.
45
+
46
+
To resolve this issue, use one of the following solutions.
47
+
48
+
## Solution 1: Adjust the PDB's "maxUnavailable" parameter
49
+
50
+
> [!NOTE]
51
+
> Use this solution if you can edit the PDB resource directly.
52
+
53
+
1. Set the PDB's `maxUnavailable` parameter to `1` or a greater value. For more information, see [Specifying a PodDisruptionBudget](https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget).
54
+
2. Retry the AKS cluster upgrade operation.
55
+
56
+
## Solution 2: Back up, delete, and redeploy the PDB
57
+
58
+
> [!NOTE]
59
+
> Use this solution if directly editing the PDB resource isn't viable.
Copy file name to clipboardExpand all lines: support/azure/azure-kubernetes/extensions/custom-policy-prevents-aks-upgrade.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,7 @@ editor: v-jsitser
8
8
ms.reviewer: cssakscic, v-rekhanain, v-leedennis
9
9
ms.service: azure-kubernetes-service
10
10
ms.topic: troubleshooting-problem-resolution
11
+
ms.custom: sap:Extensions, Policies and Add-Ons
11
12
#Customer intent: As an Azure Kubernetes user, I want to troubleshoot why a Kubernetes upgrade fails when I'm using an Azure custom policy so that I can use the Azure Policy Add-on for an upgraded Azure Kubernetes Service cluster successfully.
12
13
---
13
14
# AKS upgrade fails when using an Azure custom policy
#Customer intent: As an Azure Kubernetes user, I want to do general troubleshooting on the Istio add-on so that I can use the Istio service mesh successfully.
12
13
---
13
14
# General troubleshooting of the Istio service mesh add-on
#Customer intent: As an Azure Kubernetes user, I want to troubleshoot the ingress gateway of the Istio add-on so that I can use the Istio service mesh successfully.
12
13
---
13
14
# Istio service mesh add-on ingress gateway troubleshooting
#Customer intent: As an Azure Kubernetes user, I want to troubleshoot the global mesh configuration (MeshConfig) of the Istio add-on so that I can use the Istio service mesh successfully.
12
13
---
13
14
# Istio service mesh add-on MeshConfig troubleshooting
#Customer intent: As an Azure Kubernetes user, I want to troubleshoot minor revision upgrades of the Istio add-on so that I can use the Istio service mesh successfully.
11
12
---
@@ -28,9 +29,10 @@ The following table lists various problems and the different scenarios and solut
28
29
29
30
| Scenario | Problem | Solution |
30
31
|--|--|--|
31
-
| Data plane workloads are dropped from the mesh. | Data plane and control plane revisions didn't correspond before you completed or rolled back an upgrade. | <p>Follow these steps:</p><ol> <li><p>Relabel namespaces that contain workloads by specifying the revision that's expected to exist after the upgrade completion or rollback. To do this, run the [kubectl label](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_label/) command:</p><pre>kubectl label namespace default istio.io/rev=asm-x-y --overwrite</pre></li> <li><p>Restart the corresponding workload deployments to trigger sidecar reinjection of the correct revision. To do this, run the [kubectl rollout restart](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart/) command:</p><pre>kubectl rollout restart deployment \<deployment name></pre></li> <li><p>Verify that the sidecar images exist. To do this, run the [kubectl get](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_get/) command:</p><pre>kubectl get pods --namespace \<namespace> --output yaml \| grep mcr.microsoft.com/oss/istio/proxyv2:</pre></li> </ol> |
32
+
| Data plane workloads are dropped from the mesh. | Data plane and control plane revisions didn't correspond before you completed or rolled back an upgrade. | <p>Follow these steps:</p><ol> <li><p>Relabel namespaces that contain workloads by specifying the revision that's expected to exist after the upgrade completion or rollback:</p><pre>kubectl label namespace default istio.io/rev=asm-x-y --overwrite</pre></li> <li><p>Restart the corresponding workload deployments using the [kubectl rollout restart](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_rollout/kubectl_rollout_restart/) command to trigger sidecar reinjection of the correct revision:</p><pre>kubectl rollout restart deployment \<deployment name></pre></li> <li><p>Verify that the sidecar images exist using the [kubectl get](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_get/) command:</p><pre>kubectl get pods --namespace \<namespace> --output yaml \| grep mcr.microsoft.com/oss/istio/proxyv2:</pre></li> </ol> |
32
33
| Control plane pods are in the pending state. | The pods lack capacity. | Verify the state of the pods by running the [kubectl describe](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_describe/) command. If capacity is the problem, you can scale up your cluster to add another node. For more information, see [Manually scale the node count in an Azure Kubernetes Service (AKS) cluster](/azure/aks/scale-cluster). |
33
-
| The [az aks mesh get-upgrades](/cli/azure/aks/mesh#az-aks-mesh-get-upgrades) command returns no available upgrades. | The newest Istio revision might be incompatible with the current AKS cluster version. | You can use the [az aks mesh get-revisions](/cli/azure/aks/mesh#az-aks-mesh-get-revisions) command to discover whether newer Istio revisions exist. The output includes a list of compatible cluster versions for each Istio revision. Therefore, you can determine whether a cluster upgrade is necessary. |
34
+
| The [az aks mesh get-upgrades](/cli/azure/aks/mesh#az-aks-mesh-get-upgrades) command returns no available upgrades. | The next Istio revision might be incompatible with the current AKS cluster version. | You can use the [az aks mesh get-revisions](/cli/azure/aks/mesh#az-aks-mesh-get-revisions) command to discover whether newer Istio revisions exist. The output includes a list of compatible cluster versions for each Istio revision. Therefore, you can determine whether a cluster upgrade is necessary. If _both_ mesh and cluster are no longer supported, upgrade the cluster version first, and then the mesh revision. To recover from this scenario, a cluster upgrade is permitted even if it's incompatible with the mesh revision. |
35
+
34
36
35
37
> [!NOTE]
36
38
> To avoid unintended behavior and broken functionality, and also make sure that you're receiving updates for security vulnerabilities, we strongly recommend that you upgrade to a supported and up-to-date [AKS version](/azure/aks/supported-kubernetes-versions) and Istio add-on revision. Remember that the add-on revision should also be within the supported Kubernetes version range for the given AKS cluster. As highlighted in the [Minor revision upgrade](/azure/aks/istio-upgrade#minor-revision-upgrade) section of the Istio upgrade article, you can run the `az aks mesh get-revisions` and `az aks mesh get-upgrades` commands to learn about available add-on revisions, upgrades, and compatibility information.
@@ -39,7 +41,10 @@ The following table lists various problems and the different scenarios and solut
39
41
40
42
- A downgrade to an older revision (outside the canary rollback process) isn't allowed.
41
43
42
-
- Skipping from one revision to a nonconsecutive revision is allowed only if AKS no longer supports both the current revision and the next upgrade revision. At this point, the only upgrade that's available to you is the lowest supported revision.
44
+
- Available upgrades for a revision depend on whether it's currently supported. For example, if `n` is the currently installed revision and `n+2` is the latest revision:
45
+
- If `n` is supported, you may upgrade to the next revision `n+1` or directly to the newest revision `n+2`.
46
+
- If both `n` and `n+1` (next consecutive) are unsupported, the only available upgrade is `n+2` (next supported).
47
+
- If `n` has been unsupported for a while, it's possible both of the next two consecutive revisions are unsupported. In this case, the only available upgrade is the lowest supported revision.
43
48
44
49
- The Istio `sidecar.istio.io/inject` label doesn't enable sidecar injection for the Istio add-on. You must use the `istio.io/rev` label when you label and relabel your namespaces during the canary upgrade.
#Customer intent: As an Azure Kubernetes user, I want to troubleshoot plug-in CA certificates in the Istio add-on so that I can use the Istio service mesh successfully.
11
12
---
12
13
# Istio service mesh add-on plug-in CA certificate troubleshooting
Copy file name to clipboardExpand all lines: support/azure/azure-kubernetes/extensions/pods-created-user-namespaces.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,6 +8,7 @@ editor: v-jsitser
8
8
ms.reviewer: cssakscic, v-rekhanain, v-leedennis
9
9
ms.service: azure-kubernetes-service
10
10
ms.topic: troubleshooting-problem-resolution
11
+
ms.custom: sap:Extensions, Policies and Add-Ons
11
12
#Customer intent: As an Azure Kubernetes user, I want to learn why pods can be created in user namespaces so that I can be assured that there isn't a security risk to my Azure Kubernetes Service (AKS) cluster.
0 commit comments