Stale Persistent Volumes in Released state when DataVolumes are deleted.

[anastheo]

Hello,

It seems that we have faced a similar problem in Openshift Virtualization cluster (v4.19) with HPE CSI Driver v3.0.1.

When a datavolume is deleted, then we can see several stale Persistent Volumes in Released state.

Primera Storage Type: A630 (FC)

Example:
Stale Persistent Volumes in Releases state (PVC and Datavolumes are deleted)

$ oc get pv -n prepaid-in | grep template
pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5   60Gi       RWX            Delete           Released      prepaid-in/am-tasccon3-template                                           hpe-primera84-sc   <unset>                          35d
pvc-21e34872-4bec-408d-878f-1d80aad735bb   50Gi       RWX            Delete           Released      prepaid-in/am-icsrds1-template                                            hpe-primera84-sc   <unset>                          2d1h
pvc-2456739f-9fc6-4fb2-a12b-42f4161ec4de   85Gi       RWX            Delete           Released      prepaid-in/am-mobin1-template                                             hpe-primera84-sc   <unset>                          134m
pvc-26fc76c2-eb37-44da-80e5-1d3ba774c1ec   80Gi       RWX            Delete           Released      prepaid-in/am-icsapp1-template                                            hpe-primera84-sc   <unset>                          2d2h
pvc-35271bfb-5319-4f79-af0a-c8a7be6c443e   85Gi       RWX            Delete           Released      prepaid-in/am-mobin4-template                                             hpe-primera84-sc   <unset>                          99d

HPE CSI Controller Error - Volume Deletion Failed

$ oc logs hpe-csi-controller-6956b6589-xp9wk -n hpe-storage | less
I0306 13:48:50.157298       1 controller.go:1272] "shouldDelete" PV="pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5"
I0306 13:48:50.157306       1 controller.go:1302] "shouldDelete is true" PV="pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5"
I0306 13:48:50.157310       1 controller.go:1146] "shouldDelete" PV="pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5"
I0306 13:48:50.157316       1 controller.go:1559] "Started" PV="pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5"
**E0306 13:48:50.160483       1 controller.go:1569] "Volume deletion failed" err="persistentvolume pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5 is still attached to node am-ocpphy10" PV="pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5"
I0306 13:48:50.160527**

Stale Persistent Volume - Deletion Warning

$ oc describe pv pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5 -n prepaid-in | grep Warning
  Warning  VolumeFailedDelete  35s (x14233 over 37d)  csi.hpe.com_am-ocpphy12_3356f240-c637-4840-8edb-4299c5010779  persistentvolume pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5 is still attached to node am-ocpphy10

Stale Volume Attachment

$ oc get volumeattachments -A | grep pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5
csi-a1565634abfabc6cecd6b0d9bb3d7571b5c6d310572467ad48127470c8f288ff   csi.hpe.com   pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5   am-ocpphy10   true       37d

Is there any clean work around we could use , until we have the permanent fix ? What is the recommeded order to delete the stale resources ?

Please , if you think that it's a seperate case, open a new issue based on my comment.

Thanks,
Anastasios

Originally posted by @anastheo in #518

[anastheo]

I opened a new case because the other one is similar but not exactly the same as this one.

If you think that the root cause is exactly the same please keep one of them.

Best Regards,
Anastasios

[anastheo]

Hello,

We have received a response from Red Hat support , and it seems that our case is related to the fact that kubelet service of physical servers, have missing vol_data.json file for these PV. More specifically:

PV in Release state

$ oc get pv pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: csi.hpe.com
    volume.kubernetes.io/provisioner-deletion-secret-name: hpe-primera84-backend-secret
    volume.kubernetes.io/provisioner-deletion-secret-namespace: hpe-storage
  creationTimestamp: "2026-01-28T12:54:02Z"
  finalizers:
  - external-provisioner.volume.kubernetes.io/finalizer
  - kubernetes.io/pv-protection
  - external-attacher/csi-hpe-com
  name: pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5
  resourceVersion: "380007661"
  uid: e318d615-6ffb-47a4-975d-020d89d609e7
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 60Gi
  claimRef:
    name: am-tasccon3-template
    namespace: prepaid-in
    resourceVersion: "379999652"
    uid: 4bbd2d59-a669-4b7a-946e-1931f65cd167
  csi:
    controllerExpandSecretRef:
      name: hpe-primera84-backend-secret
      namespace: hpe-storage
    controllerPublishSecretRef:
      name: hpe-primera84-backend-secret
      namespace: hpe-storage
    driver: csi.hpe.com
    nodePublishSecretRef:
      name: hpe-primera84-backend-secret
      namespace: hpe-storage
    nodeStageSecretRef:
      name: hpe-primera84-backend-secret
      namespace: hpe-storage
    volumeAttributes:
      accessProtocol: fc
      compression: "false"
      cpg: SSD_r6
      csi.storage.k8s.io/pv/name: pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5
      csi.storage.k8s.io/pvc/name: prime-4bbd2d59-a669-4b7a-946e-1931f65cd167
      csi.storage.k8s.io/pvc/namespace: prepaid-in
      description: Volume created by the HPE CSI Driver for Kubernetes
      hostEncryption: "false"
      provisioningType: tpvv
      snapCpg: SSD_r6
      storage.kubernetes.io/csiProvisionerIdentity: 1764149421750-1465-csi.hpe.com
    volumeHandle: pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5
  persistentVolumeReclaimPolicy: Delete
  storageClassName: hpe-primera84-sc
  volumeMode: Block
status:
  lastPhaseTransitionTime: "2026-01-28T13:04:34Z"
  **phase: Released #<-----------------------------------------------------

Volume Attachment remains in Openshift

apiVersion: storage.k8s.io/v1
kind: VolumeAttachment
metadata:
  annotations:
    csi.alpha.kubernetes.io/node-id: 5e95175d-fade-97c7-76f5-4b15521ccc28
  creationTimestamp: "2026-01-28T12:54:04Z"
  finalizers:
  - external-attacher/csi-hpe-com
  name: csi-a1565634abfabc6cecd6b0d9bb3d7571b5c6d310572467ad48127470c8f288ff
  resourceVersion: "379988176"
  uid: b92daab7-e58f-4b3e-b676-b43c72b18365
spec:
  attacher: csi.hpe.com
  nodeName: am-ocpphy10
  source:
    persistentVolumeName: pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5
status:
  attached: true
  attachmentMetadata:
    accessProtocol: fc
    lunId: "13"
    readOnly: "false"
    serialNumber: 60002ac000000000000001de0002a8cc
    targetNames: ""
    targetScope: group

Kubelet Reports that val_data.json is missing for this node

Mar 12 01:08:14.232322 am-ocpphy10 kubenswrapper[8982]: E0312 01:08:14.232140    8982 reconciler_common.go:289] "operationExecutor.UnmountDevice failed (controllerAttachDetachEnabled true) for volume \"pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5\" (UniqueName: \"kubernetes.io/csi/csi.hpe.com^pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5\") on node \"am-ocpphy10\" : UnmapDevice.NewUnmapper failed for volume \"pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5\" (UniqueName: \"kubernetes.io/csi/csi.hpe.com^pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5\") on node \"am-ocpphy10\" : kubernetes.io/csi: unmapper failed to load volume data file [/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5/data]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5/data/vol_data.json]: open /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5/data/vol_data.json: no such file or directory" err="UnmapDevice.NewUnmapper failed for volume \"pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5\" (UniqueName: \"kubernetes.io/csi/csi.hpe.com^pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5\") on node \"am-ocpphy10\" : kubernetes.io/csi: unmapper failed to load volume data file [/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5/data]: kubernetes.io/csi: failed to open volume data file [/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5/data/vol_data.json]: open /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-1ef92aec-8d3b-4aad-a380-7d8cf9bb45b5/data/vol_data.json: no such file or directory"

This indicates that kubelet is attempting to detach the CSI block device, but the metadata file (/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/.../vol_data.json) used to track the device mapping is missing on the node. So for some reason CSI deletes records for these CIS block before unmount of devices has been completed.

Best Regards,
Anastasios

[matthelm-hpe]

We have another report of a customer hitting this issue if not in the exact same vein.

They're running OpenShift Virtualization 4.19 / HPE CSI Driver 3.0.1 and missing the json metadata file on the node the volume was mounted on.

[anastheo]

Hello,

We are in exactly the same versions (OpenShift Virtualization 4.19 / HPE CSI Driver 3.0.1).

In order to fix the issue as a work-around (recommended by Red Hat support), we have rebooted each node at a time and the stale persistent volumes / volumeattachment have been released and cleaned up.

Best Regards,
Anastasios