Merge pull request #8653 from MicrosoftDocs/main

Auto push to live 2025-04-07 02:00:02

Author: Deland-Han

support/azure/azure-kubernetes/create-upgrade-delete/pod-stuck-crashloopbackoff-mode.md

@@ -1,17 +1,31 @@
 ---
 title: Pod is stuck in CrashLoopBackOff mode
 description: Troubleshoot a scenario in which a pod is stuck in CrashLoopBackOff mode on an Azure Kubernetes Service (AKS) cluster.
-ms.date: 09/07/2023
+ms.date: 04/07/2025
 author: VikasPullagura-MSFT
 ms.author: vipullag
-editor: v-jsitser
-ms.reviewer: chiragpa, nickoman, cssakscic, v-leedennis
+editor: v-jsitser, addobres
+ms.reviewer: chiragpa, nickoman, cssakscic, v-leedennis, addobres
 ms.service: azure-kubernetes-service
 ms.custom: sap:Create, Upgrade, Scale and Delete operations (cluster or nodepool)
 ---
 # Pod is stuck in CrashLoopBackOff mode
 
-If a pod has a `CrashLoopBackOff` status, then the pod probably failed or exited unexpectedly, and the log contains an exit code that isn't zero. There are several possible reasons why your pod is stuck in `CrashLoopBackOff` mode. Consider the following options and their associated [kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) commands.
+If a pod has a `CrashLoopBackOff` status, then the pod probably failed or exited unexpectedly, and the log contains an exit code that isn't zero. Here are several possible reasons why your pod is stuck in `CrashLoopBackOff` mode:
+
+1. **Application failure**: The application inside the container crashes shortly after starting, often due to misconfigurations, missing dependencies, or incorrect environment variables.
+2. **Incorrect resource limits**: If the pod exceeds its CPU or memory resource limits, Kubernetes might kill the container. This issue can happen if resource requests or limits are set too low.
+3. **Missing or misconfigured ConfigMaps/Secrets**: If the application relies on configuration files or environment variables stored in ConfigMaps or Secrets but they're missing or misconfigured, the application might crash.
+4. **Image pull issues**: If there's an issue with the image (for example, it's corrupted or has an incorrect tag), the container might not start properly and fail repeatedly.
+5. **Init containers failing**: If the pod has init containers and one or more fail to run properly, the pod will restart.
+6. **Liveness/Readiness probe failures**: If liveness or readiness probes are misconfigured, Kubernetes might detect the container as unhealthy and restart it.
+7. **Application dependencies not ready**: The application might depend on services that aren't yet ready, such as databases, message queues, or other APIs.
+8. **Networking issues**: Network misconfigurations can prevent the application from communicating with necessary services, causing it to fail.
+9. **Invalid commands or arguments**: The container might be started with an invalid `ENTRYPOINT`, command, or argument, leading to a crash.
+
+For more information about the container status, see [Pod Lifecycle - Container states](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states).
+
+Consider the following options and their associated [kubectl](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) commands.
 
 | Option | kubectl command |
 |--|--|

support/azure/azure-kubernetes/storage/fail-to-mount-azure-disk-volume.md

@@ -1,7 +1,7 @@
 ---
-title: Unable to mount Azure disk volumes
+title: Unable to Mount Azure Disk Volumes
 description: Describes errors that occur when mounting Azure disk volumes fails, and provides solutions.
-ms.date: 09/06/2024
+ms.date: 03/22/2025
 author: genlin
 ms.author: genli
 ms.reviewer: chiragpa, akscsscic, v-weizhu
@@ -14,17 +14,18 @@ This article provides solutions for errors that cause the mounting of Azure disk
 
 ## Symptoms
 
-You're trying to deploy a Kubernetes resource such as a Deployment or a StatefulSet, in an Azure Kubernetes Service (AKS) environment. The deployment will create a pod that should mount a PersistentVolumeClaim (PVC) referencing an Azure disk.
+You're trying to deploy a Kubernetes resource, such as a Deployment or a StatefulSet, in an Azure Kubernetes Service (AKS) environment. The deployment creates a pod that should mount a PersistentVolumeClaim (PVC) that references an Azure disk.
 
-However, the pod stays in the **ContainerCreating** status. When you run the `kubectl describe pods` command, you may see one of the following errors, which causes the mounting operation to fail:
+However, the pod stays in the **ContainerCreating** status. When you run the `kubectl describe pods` command, you may see one of the following errors that cause the mounting operation to fail:
 
 - [Disk cannot be attached to the VM because it is not in the same zone as the VM](#error1)
 - [Client '\<client-ID>' with object id '\<object-ID>' doesn't have authorization to perform action over scope '\<disk name>' or scope is invalid](#error2)
 - [Volume is already used by pod](#error3)
 - [StorageAccountType UltraSSD_LRS can be used only when additionalCapabilities.ultraSSDEnabled is set](#error4)
 - [ApplyFSGroup failed for vol](#error5)
+- [Node(s) exceed max volume count](#error6)
 
-See the following sections for error details, possible causes and solutions.
+See the following sections for error details, possible causes, and solutions.
 
 ## <a id="error1"></a>Disk cannot be attached to the VM because it is not in the same zone as the VM
 
@@ -47,13 +48,13 @@ RawError:
 
 ### Cause: Disk and node hosting pod are in different zones
 
-In AKS, the default and other built-in StorageClasses for Azure disks use [locally redundant storage (LRS)](/azure/storage/common/storage-redundancy#locally-redundant-storage). These disks are deployed in [availability zones](/azure/aks/availability-zones). If you use the node pool in AKS with availability zones, and the pod is scheduled on a node that's in another availability zone different from the disk, you may get this error.
+In AKS, the default and other built-in storage classes for Azure disks use [locally redundant storage (LRS)](/azure/storage/common/storage-redundancy#locally-redundant-storage). These disks are deployed in [availability zones](/azure/aks/availability-zones). If you use the node pool in AKS together with availability zones, and the pod is scheduled on a node that's in another availability zone that's different from the disk, you might experience this error.
 
-To resolve this error, use one of the following solutions:
+To resolve this error, use one of the following solutions.
 
 ### Solution 1: Ensure disk and node hosting the pod are in the same zone
 
-To make sure the disk and node that hosts the pod are in the same availability zone, use [node affinity](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/).
+To make sure that the disk and node that host the pod are in the same availability zone, use [node affinity](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/).
 
 Refer to the following script as an example:
 
@@ -69,19 +70,19 @@ affinity:
         - <region>-Y
 ```
 
-\<region> is the region of the AKS cluster. `Y` represents the availability zone of the disk, for example, westeurope-3.
+\<region> is the region of the AKS cluster. `Y` represents the availability zone of the disk (for example, westeurope-3).
 
 ### Solution 2: Use zone-redundant storage (ZRS) disks
 
 [ZRS](/azure/storage/common/storage-redundancy#zone-redundant-storage) disk volumes can be scheduled on all zone and non-zone agent nodes. For more information, see [Azure disk availability zone support](/azure/aks/availability-zones#azure-disk-availability-zone-support).
 
-To use a ZRS disk, create a new storage class with `Premium_ZRS` or `StandardSSD_ZRS`, and then deploy the PersistentVolumeClaim (PVC) referencing the storage.
+To use a ZRS disk, create a storage class by using `Premium_ZRS` or `StandardSSD_ZRS`, and then deploy the PersistentVolumeClaim (PVC) that references the storage.
 
 For more information about parameters, see [Driver Parameters](/azure/aks/azure-csi-files-storage-provision#storage-class-parameters-for-dynamic-persistentvolumes)
 
 ### Solution 3: Use Azure Files
 
-[Azure Files](/azure/storage/files/storage-files-introduction) is mounted by using NFS or SMB throughout network and it's not associated with availability zones.
+[Azure Files](/azure/storage/files/storage-files-introduction) is mounted by using NFS or SMB throughout network. It's not associated with availability zones.
 
 For more information, see the following articles:
 
@@ -107,11 +108,11 @@ RawError:
 
 ### Cause: AKS identity doesn't have required authorization over disk
 
-AKS cluster's identity doesn't have the required authorization over the Azure disk. This issue occurs when the disk is created in another resource group other than the infrastructure resource group of the AKS cluster.
+AKS cluster's identity doesn't have the required authorization over the Azure disk. This issue occurs if the disk is created in a resource group other than the infrastructure resource group of the AKS cluster.
 
 ### Solution: Create role assignment that includes required authorization
 
-Create a role assignment that includes the authorization required as per the error. We recommend that you use a [Contributor](/azure/role-based-access-control/built-in-roles/general#contributor) role. If you want to use another built-in role, see [Azure built-in roles](/azure/role-based-access-control/built-in-roles).
+Create a role assignment that includes the authorization required per the error. We recommend that you use a [Contributor](/azure/role-based-access-control/built-in-roles/general#contributor) role. If you want to use another built-in role, see [Azure built-in roles](/azure/role-based-access-control/built-in-roles).
 
 To assign a Contributor role, use one of the following methods:
 
@@ -135,9 +136,9 @@ Here are details of this error:
 
 ### Cause: Disk is mounted to multiple pods hosted on different nodes
 
-An Azure disk can be mounted only as [ReadWriteOnce](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes), which makes it available to one node in AKS. That means it can be attached to only one node and mounted only to a pod hosted by that node. If you mount the same disk to a pod on another node, you'll get this error because the disk is already attached to a node.
+An Azure disk can be mounted only as [ReadWriteOnce](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes). This makes it available to one node in AKS. That means that it can be attached to only one node and mounted to only a pod that's hosted by that node. If you mount the same disk to a pod on another node, you experience this error because the disk is already attached to a node.
 
-### Solution: Ensure disk isn't mounted by multiple pods hosted on different nodes
+### Solution: Make sure disk isn't mounted by multiple pods hosted on different nodes
 
 To resolve this error, refer to [Multi-Attach error](https://github.com/andyzhangx/demo/blob/master/issues/azuredisk-issues.md#25-multi-attach-error).
 
@@ -163,11 +164,11 @@ desc = Attach volume "/subscriptions/<subscription-ID>/resourceGroups/<disk-reso
 
 ### Cause: Ultra disk is attached to node pool with ultra disks disabled
 
-This error indicates that an [ultra disk](/azure/virtual-machines/disks-enable-ultra-ssd) is trying to be attached to a node pool with ultra disks disabled. By default, an ultra disk is disabled on AKS node pools.
+This error indicates that an [ultra disk](/azure/virtual-machines/disks-enable-ultra-ssd) is trying to be attached to a node pool by having ultra disks disabled. By default, an ultra disk is disabled on AKS node pools.
 
 ### Solution: Create a node pool that can use ultra disks
 
-To use ultra disks on AKS, create a node pool with ultra disks support by using the `--enable-ultra-ssd` flag. For more information, see [Use Azure ultra disks on Azure Kubernetes Service](/azure/aks/use-ultra-disks).
+To use ultra disks on AKS, create a node pool that has ultra disks support by using the `--enable-ultra-ssd` flag. For more information, see [Use Azure ultra disks on Azure Kubernetes Service](/azure/aks/use-ultra-disks).
 
 ## <a id="error5"></a>ApplyFSGroup failed for vol
 
@@ -177,20 +178,45 @@ Here are details of this error:
 
 ### Cause: Changing ownership and permissions for large volume takes much time
 
-When there's a large number of files already present in the volume, if a `securityContext` with `fsGroup` is in place, this error may occur. When there are lots of files and directories under one volume, changing the group ID would consume much time. It's also mentioned in the Kubernetes official documentation [Configure volume permission and ownership change policy for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods):
+If there are many files already present in the volume, and if a `securityContext` that uses `fsGroup` exists, this error might occur. If there are lots of files and directories in one volume, changing the group ID would consume excessive time. Additionally, the Kubernetes official documentation [Configure volume permission and ownership change policy for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods) mentions this situation:
 
 "By default, Kubernetes recursively changes ownership and permissions for the contents of each volume to match the `fsGroup` specified in a Pod's `securityContext` when that volume is mounted. For large volumes, checking and changing ownership and permissions can take much time, slowing Pod startup. You can use the `fsGroupChangePolicy` field inside a `securityContext` to control the way that Kubernetes checks and manages ownership and permissions for a volume."
 
 ### Solution: Set fsGroupChangePolicy field to OnRootMismatch
 
-To resolve this error, we recommend that you set `fsGroupChangePolicy: "OnRootMismatch"` in the `securityContext` of a Deployment, a StatefulSet or a pod.
+To resolve this error, we recommend that you set `fsGroupChangePolicy: "OnRootMismatch"` in the `securityContext` of a Deployment, a StatefulSet, or a pod.
 
-OnRootMismatch: Only change permissions and ownership if permission and ownership of root directory doesn't match with expected permissions of the volume. This setting could help shorten the time it takes to change ownership and permission of a volume.
+OnRootMismatch: Change permissions and ownership only if permission and ownership of the root directory doesn't match the expected permissions of the volume. This setting could help shorten the time that it takes to change ownership and permission of a volume.
 
 For more information, see [Configure volume permission and ownership change policy for Pods](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods).
 
-## More information  
+## <a id="error6"></a>Node(s) exceed max volume count
+
+Here are details of this error:
+
+```output
+Events:
+Type   Reason      Age  From        Message
+----   ------      ---- ----        -------
+Warning FailedScheduling 25s  default-scheduler 0/8 nodes are available: 8 node(s) exceed max volume count. preemption: 0/8 nodes are available: 8 No preemption victims found for incoming pod..
+```
+### Cause: Maximum disk limit is reached
+
+The node has reached its maximum disk capacity. In AKS, the number of disks per node depends on the VM size that's configured for the node pool.
+
+### Solution
+
+To resolve the issue, use one of the following methods:
+
+- Add a new node pool with a VM size that supports more disk limit.
+- Scale the node pool.
+- Delete existing disks from the node.
+
+Additionally, make sure that the number of disks per node does not exceed the [Kubernetes default limits](https://kubernetes.io/docs/concepts/storage/storage-limits/#kubernetes-default-limits).
+
+## More information
 
 For more Azure Disk known issues, see [Azure disk plugin known issues](https://github.com/andyzhangx/demo/blob/master/issues/azuredisk-issues.md).
 
 [!INCLUDE [Azure Help Support](../../../includes/azure-help-support.md)]
+