Offline volume expansion fails with csi-drivers #491
Description
When we execute csi-drivers certification test scenario 'External Storage [Driver: csi-powerstore.dellemc.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works'"
In 1.13.1 and 1.13.2 versions of resizer sidecar it fails to expand the volume, and the above suite fails. Logs are attached below.
Failure logs:
podman run --net host -v /root/test/certification-test/:/data:z --rm -it registry.redhat.io/openshift4/ose-tests sh -c "KUBECONFIG=/data/kubeconfig.yaml TEST_CSI_DRIVER_FILES=/data/manifest.yaml /usr/bin/openshift-tests run-test 'External Storage [Driver: csi-powerstore.dellemc.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works'"
Mar 28 11:17:32.513: INFO: Driver loaded from path [/data/manifest.yaml]: &{DriverInfo:{Name:csi-powerstore.dellemc.com InTreePluginName: FeatureTag: MaxFileSize:0 SupportedSizeRange:{Max:16Gi Min:3Gi} SupportedFsType:map[:{} ext4:{} xfs:{}] SupportedMountOption:map[noatime:{}] RequiredMountOption:map[] Capabilities:map[RWX:false block:true controllerExpansion:true exec:true fsGroup:true nodeExpansion:true offlineExpansion:true onlineExpansion:true persistence:true pvcDataSource:true singleNodeVolume:false snapshotDataSource:true topology:true volumeLimits:false] RequiredAccessModes:[] TopologyKeys:[csi-powerstore.dellemc.com/xxxxxx.iscsi] NumAllowedTopologies:1 StressTestOptions:<nil> VolumeSnapshotStressTestOptions:<nil> PerformanceTestOptions:<nil>} StorageClass:{FromName:false FromFile:powerstore_iscsi.yaml FromExistingClassName:} SnapshotClass:{FromName:true FromFile: FromExistingClassName:} InlineVolumes:[] ClientNodeName: Timeouts:map[]}
Mar 28 11:17:32.555: INFO: Enabling in-tree volume drivers
Running Suite: OpenShift e2e suite - /
======================================
Random Seed: 1743160652 - will randomize all specs
Will run 1 of 1 specs
------------------------------
External Storage [Driver: csi-powerstore.dellemc.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works
k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:172
STEP: Creating a kubernetes client @ 03/28/25 11:17:33.557
STEP: Building a namespace api object, basename volume-expand @ 03/28/25 11:17:33.558
Mar 28 11:17:33.589: INFO: About to run a Kube e2e test, ensuring namespace/e2e-volume-expand-8164 is privileged
STEP: Waiting for a default service account to be provisioned in namespace @ 03/28/25 11:17:33.924
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/28/25 11:17:33.929
Mar 28 11:17:33.933: INFO: Creating resource for dynamic PV
Mar 28 11:17:33.933: INFO: Using claimSize:3Gi, test suite supported size:{ 1Gi}, driver(csi-powerstore.dellemc.com) supported size:{ 1Gi}
STEP: creating a StorageClass e2e-volume-expand-8164-e2e-sc866sm @ 03/28/25 11:17:33.934
STEP: creating a claim @ 03/28/25 11:17:33.942
Mar 28 11:17:33.942: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Mar 28 11:17:33.955: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [csi-powerstore.dellemc.comzjqbc] to have phase Bound
Mar 28 11:17:33.962: INFO: PersistentVolumeClaim csi-powerstore.dellemc.comzjqbc found but phase is Pending instead of Bound.
Mar 28 11:17:35.969: INFO: PersistentVolumeClaim csi-powerstore.dellemc.comzjqbc found but phase is Pending instead of Bound.
Mar 28 11:17:37.974: INFO: PersistentVolumeClaim csi-powerstore.dellemc.comzjqbc found and phase=Bound (4.018657814s)
STEP: Creating a pod with dynamically provisioned volume @ 03/28/25 11:17:37.984
STEP: Deleting the previously created pod @ 03/28/25 11:17:48.042
Mar 28 11:17:48.042: INFO: Deleting pod "pod-17232006-20ac-478b-a5c7-3aecd5fd20dd" in namespace "e2e-volume-expand-8164"
Mar 28 11:17:48.081: INFO: Wait up to 5m0s for pod "pod-17232006-20ac-478b-a5c7-3aecd5fd20dd" to be fully deleted
STEP: Expanding current pvc @ 03/28/25 11:17:52.097
Mar 28 11:17:52.097: INFO: currentPvcSize {{3221225472 0} {<nil>} 3Gi BinarySI}, newSize {{4294967296 0} {<nil>} BinarySI}
STEP: Waiting for cloudprovider resize to finish @ 03/28/25 11:17:52.114
STEP: Checking for conditions on pvc @ 03/28/25 11:17:54.128
Mar 28 11:19:54.143: INFO: Unexpected error: While waiting for pvc to have fs resizing condition:
<*errors.errorString | 0xc001be0350>:
error waiting for pvc "csi-powerstore.dellemc.comzjqbc" to have filesystem resize status: timed out waiting for the condition
{
s: "error waiting for pvc \"csi-powerstore.dellemc.comzjqbc\" to have filesystem resize status: timed out waiting for the condition",
}
[FAILED] in [It] - k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:219 @ 03/28/25 11:19:54.143
Mar 28 11:19:54.143: INFO: Deleting pod "pod-17232006-20ac-478b-a5c7-3aecd5fd20dd" in namespace "e2e-volume-expand-8164"
STEP: Deleting pod @ 03/28/25 11:19:54.147
Mar 28 11:19:54.147: INFO: Deleting pod "pod-17232006-20ac-478b-a5c7-3aecd5fd20dd" in namespace "e2e-volume-expand-8164"
STEP: Deleting pvc @ 03/28/25 11:19:54.15
Mar 28 11:19:54.150: INFO: Deleting PersistentVolumeClaim "csi-powerstore.dellemc.comzjqbc"
Mar 28 11:19:54.159: INFO: Waiting up to 5m0s for PersistentVolume csivol-6473806841 to get deleted
Mar 28 11:19:54.163: INFO: PersistentVolume csivol-6473806841 found and phase=Bound (3.768328ms)
Mar 28 11:19:59.169: INFO: PersistentVolume csivol-6473806841 was removed
STEP: Deleting sc @ 03/28/25 11:19:59.169
STEP: dump namespace information after failure @ 03/28/25 11:19:59.176
STEP: Collecting events from namespace "e2e-volume-expand-8164". @ 03/28/25 11:19:59.177
STEP: Found 13 events. @ 03/28/25 11:19:59.181
Mar 28 11:19:59.181: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for pod-17232006-20ac-478b-a5c7-3aecd5fd20dd: { } Scheduled: Successfully assigned e2e-volume-expand-8164/pod-17232006-20ac-478b-a5c7-3aecd5fd20dd to xyz.net
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:33 +0000 UTC - event for csi-powerstore.dellemc.comzjqbc: {csi-powerstore.dellemc.com_powerstore-controller-b578bb67-ht2d4_6985a0fa-7fc6-460b-8851-84ffcd001676 } Provisioning: External provisioner is provisioning volume for claim "e2e-volume-expand-8164/csi-powerstore.dellemc.comzjqbc"
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:33 +0000 UTC - event for csi-powerstore.dellemc.comzjqbc: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'csi-powerstore.dellemc.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:36 +0000 UTC - event for csi-powerstore.dellemc.comzjqbc: {csi-powerstore.dellemc.com_powerstore-controller-b578bb67-ht2d4_6985a0fa-7fc6-460b-8851-84ffcd001676 } ProvisioningSucceeded: Successfully provisioned volume csivol-6473806841
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:40 +0000 UTC - event for pod-17232006-20ac-478b-a5c7-3aecd5fd20dd: {attachdetach-controller } SuccessfulAttachVolume: AttachVolume.Attach succeeded for volume "csivol-6473806841"
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:46 +0000 UTC - event for pod-17232006-20ac-478b-a5c7-3aecd5fd20dd: {kubelet xyz.net} Created: Created container: write-pod
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:46 +0000 UTC - event for pod-17232006-20ac-478b-a5c7-3aecd5fd20dd: {kubelet xyz.net} Started: Started container write-pod
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:48 +0000 UTC - event for pod-17232006-20ac-478b-a5c7-3aecd5fd20dd: {kubelet xyz.net} Killing: Stopping container write-pod
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:52 +0000 UTC - event for csi-powerstore.dellemc.comzjqbc: {external-resizer csi-powerstore.dellemc.com } Resizing: External resizer is resizing volume csivol-6473806841
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:52 +0000 UTC - event for csi-powerstore.dellemc.comzjqbc: {volume_expand } ExternalExpanding: waiting for an external controller to expand this PVC
Mar 28 11:19:59.181: INFO: At 2025-03-28 11:17:53 +0000 UTC - event for csi-powerstore.dellemc.comzjqbc: {external-resizer csi-powerstore.dellemc.com } FileSystemResizeRequired: Require file system resize of volume on node
Mar 28 11:19:59.187: INFO: POD NODE PHASE GRACE CONDITIONS
Mar 28 11:19:59.187: INFO:
Mar 28 11:19:59.197: INFO: skipping dumping cluster info - cluster too large
STEP: Destroying namespace "e2e-volume-expand-8164" for this suite. @ 03/28/25 11:19:59.198
• [FAILED] [145.656 seconds]
External Storage [Driver: csi-powerstore.dellemc.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand [It] Verify if offline PVC expansion works
k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:172
[FAILED] While waiting for pvc to have fs resizing condition: error waiting for pvc "csi-powerstore.dellemc.comzjqbc" to have filesystem resize status: timed out waiting for the condition In [It] at: k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:219 @ 03/28/25 11:19:54.143
------------------------------
Summarizing 1 Failure:
[FAIL] External Storage [Driver: csi-powerstore.dellemc.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand [It] Verify if offline PVC expansion works
k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:219
Ran 1 of 1 Specs in 145.657 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 0 Skipped
fail [k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:219]: While waiting for pvc to have fs resizing condition: error waiting for pvc "csi-powerstore.dellemc.comzjqbc" to have filesystem resize status: timed out waiting for the condition
Ginkgo exit error 1: exit with code 1
**Test passes if we set the NodeExpansionRequired to false in the CSI driver controller**
`&csi.ControllerExpandVolumeResponse{CapacityBytes: requiredBytes, NodeExpansionRequired: false}, nil`
podman run --net host -v /root/test:/data:z --rm -it registry.redhat.io/openshift4/ose-tests sh -c "KUBECONFIG=/data/kubeconfig.yaml TEST_CSI_DRIVER_FILES=/data/manifest.yaml /usr/bin/openshift-tests run-test 'External Storage [Driver: csi-powerstore.dellemc.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works'"
Running Suite: OpenShift e2e suite - /
======================================
Random Seed: 1743165267 - will randomize all specs
Will run 1 of 1 specs
------------------------------
External Storage [Driver: csi-powerstore.dellemc.com] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works
k8s.io/[email protected]/test/e2e/storage/testsuites/volume_expand.go:172
STEP: Creating a kubernetes client @ 03/28/25 12:34:29.695
STEP: Building a namespace api object, basename volume-expand @ 03/28/25 12:34:29.696
Mar 28 12:34:29.754: INFO: About to run a Kube e2e test, ensuring namespace/e2e-volume-expand-7714 is privileged
STEP: Waiting for a default service account to be provisioned in namespace @ 03/28/25 12:34:30.069
STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 03/28/25 12:34:30.073
Mar 28 12:34:30.076: INFO: Creating resource for dynamic PV
Mar 28 12:34:30.076: INFO: Using claimSize:8Gi, test suite supported size:{ 1Gi}, driver(csi-powerstore.dellemc.com) supported size:{ 1Gi}
STEP: creating a StorageClass e2e-volume-expand-7714-e2e-scqkvtl @ 03/28/25 12:34:30.078
STEP: creating a claim @ 03/28/25 12:34:30.088
Mar 28 12:34:30.088: INFO: Warning: Making PVC: VolumeMode specified as invalid empty string, treating as nil
Mar 28 12:34:30.108: INFO: Waiting up to timeout=5m0s for PersistentVolumeClaims [csi-powerstore.dellemc.comjjmh5] to have phase Bound
Mar 28 12:34:30.112: INFO: PersistentVolumeClaim csi-powerstore.dellemc.comjjmh5 found but phase is Pending instead of Bound.
Mar 28 12:34:32.117: INFO: PersistentVolumeClaim csi-powerstore.dellemc.comjjmh5 found and phase=Bound (2.008353612s)
STEP: Creating a pod with dynamically provisioned volume @ 03/28/25 12:34:32.123
STEP: Deleting the previously created pod @ 03/28/25 12:35:00.207
Mar 28 12:35:00.207: INFO: Deleting pod "pod-c0874c3e-6471-41d5-8275-348bdc79b714" in namespace "e2e-volume-expand-7714"
Mar 28 12:35:00.215: INFO: Wait up to 5m0s for pod "pod-c0874c3e-6471-41d5-8275-348bdc79b714" to be fully deleted
STEP: Expanding current pvc @ 03/28/25 12:35:04.233
Mar 28 12:35:04.233: INFO: currentPvcSize {{8589934592 0} {<nil>} BinarySI}, newSize {{9663676416 0} {<nil>} BinarySI}
STEP: Waiting for cloudprovider resize to finish @ 03/28/25 12:35:04.252
STEP: Checking for conditions on pvc @ 03/28/25 12:35:06.263
STEP: Creating a new pod with same volume @ 03/28/25 12:35:06.268
STEP: Waiting for file system resize to finish @ 03/28/25 12:35:28.356
Mar 28 12:35:28.361: INFO: Deleting pod "pod-e3765943-b2d8-41af-b596-b09462df1b82" in namespace "e2e-volume-expand-7714"
Mar 28 12:35:28.370: INFO: Wait up to 5m0s for pod "pod-e3765943-b2d8-41af-b596-b09462df1b82" to be fully deleted
Mar 28 12:35:32.384: INFO: Deleting pod "pod-c0874c3e-6471-41d5-8275-348bdc79b714" in namespace "e2e-volume-expand-7714"
STEP: Deleting pod @ 03/28/25 12:35:32.389
Mar 28 12:35:32.389: INFO: Deleting pod "pod-c0874c3e-6471-41d5-8275-348bdc79b714" in namespace "e2e-volume-expand-7714"
STEP: Deleting pod2 @ 03/28/25 12:35:32.392
Mar 28 12:35:32.392: INFO: Deleting pod "pod-e3765943-b2d8-41af-b596-b09462df1b82" in namespace "e2e-volume-expand-7714"
STEP: Deleting pvc @ 03/28/25 12:35:32.396
Mar 28 12:35:32.396: INFO: Deleting PersistentVolumeClaim "csi-powerstore.dellemc.comjjmh5"
Mar 28 12:35:32.404: INFO: Waiting up to 5m0s for PersistentVolume csivol-17e1caa243 to get deleted
Mar 28 12:35:32.408: INFO: PersistentVolume csivol-17e1caa243 found and phase=Bound (3.782428ms)
Mar 28 12:35:37.416: INFO: PersistentVolume csivol-17e1caa243 found and phase=Released (5.011931702s)
Mar 28 12:35:42.423: INFO: PersistentVolume csivol-17e1caa243 found and phase=Released (10.018479679s)
Mar 28 12:35:47.428: INFO: PersistentVolume csivol-17e1caa243 found and phase=Released (15.023955639s)
Mar 28 12:35:52.434: INFO: PersistentVolume csivol-17e1caa243 was removed
STEP: Deleting sc @ 03/28/25 12:35:52.434
STEP: Destroying namespace "e2e-volume-expand-7714" for this suite. @ 03/28/25 12:35:52.442
• [82.768 seconds]
------------------------------
Ran 1 of 1 Specs in 82.769 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped