Restore from volume snapshots not working on HPE CSI Driver 3.0.0 on Openshift 4.16

[saad0805]

After upgrade to CSI driver 3.0.0, we cannot create PVC from Volume Snapshots

:~|⇒ oc get volumesnapshotcontent -A
NAME READYTOUSE RESTORESIZE DELETIONPOLICY DRIVER VOLUMESNAPSHOTCLASS VOLUMESNAPSHOT VOLUMESNAPSHOTNAMESPACE AGE
snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3 true 161061273600 Delete csi.hpe.com hpe-snapshot elasticsearch-bss-master-elasticsearch-bss-master-0-snap test-ns 28m

|⇒ oc edit volumesnapshotcontent snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3
name: snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3
resourceVersion: "209518740"
uid: 2da35b58-1783-4497-b212-6884b6bcb04d
spec:
deletionPolicy: Delete
driver: csi.hpe.com
source:
volumeHandle: pvc-06160b6f-69b2-406a-9f86-c53b1dc32002
sourceVolumeMode: Filesystem
volumeSnapshotClassName: hpe-snapshot
volumeSnapshotRef:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
name: elasticsearch-bss-master-elasticsearch-bss-master-0-snap
namespace: test-ns
resourceVersion: "209498560"
uid: a5d0c29e-d1bb-464d-9cb2-44301e9f95b3
status:
creationTime: 1756877692000000000
readyToUse: true
restoreSize: 161061273600
snapshotHandle: snapshot-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3

oc get pv pvc-06160b6f-69b2-406a-9f86-c53b1dc32002
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-06160b6f-69b2-406a-9f86-c53b1dc32002 150Gi RWO Retain Bound test-ns/elasticsearch-bss-master-elasticsearch-bss-master-0 3par-sc 113d

:~|⇒ cat test-resto.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-restore
spec:
dataSource:
name: snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 150Gi

:~|⇒ oc apply -f test-resto.yaml
persistentvolumeclaim/test-restore created

:~|⇒ oc get pvc | grep test
test-restore Pending 3par-sc 4s

1:~|⇒ oc describe pvc test-restore
Name: test-restore
Namespace: test-ns
StorageClass: 3par-sc
Status: Pending
Volume:
Labels:
Annotations: volume.beta.kubernetes.io/storage-provisioner: csi.hpe.com
volume.kubernetes.io/storage-provisioner: csi.hpe.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
DataSource:
APIGroup: snapshot.storage.k8s.io
Kind: VolumeSnapshot
Name: snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3
Used By:
Events:
Type Reason Age From Message

---

Normal Provisioning 6s (x5 over 21s) csi.hpe.com_hpdboco13.XXXXXXXX_60d86492-2066-4337-9298-5aec95ad6b89 External provisioner is provisioning volume for claim "test-ns/test-restore"
Warning ProvisioningFailed 6s (x5 over 21s) csi.hpe.com_hpdboco13.XXXXXX_60d86492-2066-4337-9298-5aec95ad6b89 failed to provision volume with StorageClass "3par-sc": error getting handle for DataSource Type VolumeSnapshot by Name snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3: error getting snapshot snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3 from api server: volumesnapshots.snapshot.storage.k8s.io "snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3" not found
Normal ExternalProvisioning 4s (x3 over 21s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'csi.hpe.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

Even though the volume snapshoft exist, it says that it cant be find.
The volume exist on 3par side as well

[saad0805]

[datamattsson]

This manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-restore
spec:
  dataSource:
    name: snapcontent-a5d0c29e-d1bb-464d-9cb2-44301e9f95b3
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
  requests:
    storage: 150Gi

Should use the VolumeSnapshot name in .spec.dataSource.name, "elasticsearch-bss-master-elasticsearch-bss-master-0-snap".

[saad0805]

After i changed the name , i get the same error
~|⇒ cat test-resto.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-restore
spec:
dataSource:
name: elasticsearch-bss-master-elasticsearch-bss-master-0-snap
kind: VolumeSnapshot
apiGroup: snapshot.storage.k8s.io
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 150Gi

~|⇒ oc apply -f test-resto.yaml
persistentvolumeclaim/test-restore created

~|⇒ oc describe pvc test-restore
Name: test-restore
Namespace: test-ns
StorageClass: 3par-sc
Status: Pending
Volume:
Labels:
Annotations: volume.beta.kubernetes.io/storage-provisioner: csi.hpe.com
volume.kubernetes.io/storage-provisioner: csi.hpe.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
DataSource:
APIGroup: snapshot.storage.k8s.io
Kind: VolumeSnapshot
Name: elasticsearch-bss-master-elasticsearch-bss-master-0-snap
Used By:
Events:
Type Reason Age From Message

---

Normal ExternalProvisioning 11s (x2 over 19s) persistentvolume-controller Waiting for a volume to be created either by the external provisioner 'csi.hpe.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Normal Provisioning 4s (x5 over 19s) csi.hpe.com_hpdboco13.XXXXXXXX_60d86492-2066-4337-9298-5aec95ad6b89 External provisioner is provisioning volume for claim " test-ns/test-restore"
Warning ProvisioningFailed 4s (x5 over 19s) csi.hpe.com_hpdboco13.XXXXX_60d86492-2066-4337-9298-5aec95ad6b89 failed to provision volume with StorageClass "3par-sc": error getting handle for DataSource Type VolumeSnapshot by Name elasticsearch-bss-master-elasticsearch-bss-master-0-snap: error getting snapshot elasticsearch-bss-master-elasticsearch-bss-master-0-snap from api server: volumesnapshots.snapshot.storage.k8s.io "elasticsearch-bss-master-elasticsearch-bss-master-0-snap" not found

[datamattsson]

Are you in the same Namespace as the PVC and VolumeSnapshot?

[saad0805]

Yes im working in only one namespace.

[datamattsson]

Yes im working in only one namespace.

It doesn't look like it.

oc describe pvc test-restore
Name: test-restore
Namespace: bss-comarch-rec

kind: VolumeSnapshot
name: elasticsearch-bss-master-elasticsearch-bss-master-0-snap
namespace: test-ns

[saad0805]

actually its bss* everywhere but im just replacing it to publish here

[datamattsson]

What does the logs say from the 3PAR CSP? It doesn't look like the request is even hitting our driver because the error indicates the resource doesn't exist in the control plane.

[saad0805]

[saad0805]

CSP logs when doing a clone

ime="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112914 ] -- >>>>>>>>> Get Volume By Id " file="request_handler.go:218"
time="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112914 ] -- >>>>> Get Volume Cmd - Volume name/id: pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce" file="get_volume_cmd.go:43"
time="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112914 ] -- GET VOLUME BY ID: {"Mountpoint":"/var/lib/kubelet/plugins/hpe.com/mounts/pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce","config":{"background_operation_status":"","com
pression":"true","cpg":"CIE3PAR-1_OPENSHIFT_HORSPROD","provisioning_type":"tdvv","snap_cpg":"CIE3PAR-1_OPENSHIFT_HORSPROD"},"description":"Volume created by the HPE CSI Driver for Kubernetes","id":"pvc-6b4e46c5-c95e-4f84-
b5dc-8e08861f27ce","name":"pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce","published":true,"size":17179869184,"volume_group_id":""}" file="get_volume_cmd.go:91"
time="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112914 ] -- <<<<< Get Volume Cmd - Volume name/id: pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce" file="get_volume_cmd.go:95"
time="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112914 ] -- <<<<<< Get Volume By Id" file="request_handler.go:225"
time="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112915 ] -- >>>>> UnPublish Volume Cmd on array 10.241.37.20 for volume pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce and host 00f749a6-cdf3-6707-d197-5be67b5a5097" file="unpublish_volume_cmd
.go:35"
time="2025-08-23T12:07:15Z" level=info msg="Port Map Contains: map[string]string{"10.241.37.20":"8080"}" file="managed_client.go:126"
time="2025-08-23T12:07:15Z" level=info msg="Creating array client using 10.241.37.20:8080" file="managed_client.go:167"
time="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112915 ] -- Access protocol for volume pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce is fc" file="unpublish_volume_cmd.go:57"
time="2025-08-23T12:07:15Z" level=info msg="Raising panic with following message: Host details not found in CRD. Cannot delete VLUNS and Host." file="utils.go:205"
time="2025-08-23T12:07:15Z" level=error msg="[ REQUEST-ID 112915 ] -- Host details not found in CRD. Cannot delete VLUNS and Host." file="unpublish_volume_cmd.go:67"
time="2025-08-23T12:07:15Z" level=info msg="[ REQUEST-ID 112915 ] -- <<<<< UnPublish Volume Cmd on array 10.241.37.20 for volume pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce and host 00f749a6-cdf3-6707-d197-5be67b5a5097" file="panic.go:791"
time="2025-08-23T12:07:15Z" level=info msg="CspManager: Recovering from panic..." file="csp_manager.go:40"
time="2025-08-23T12:07:15Z" level=info msg="Sending following message to CSI driver: Host details not found in CRD. Cannot delete VLUNS and Host." file="utils.go:213"
time="2025-08-23T12:07:15Z" level=info msg="Error response to be sent is &models.ErrorsPayload{Errors:[]models.ErrorObject{models.ErrorObject{Code:"Not Found", Message:"Host details not found in CRD. Cannot delete VLUNS and Host."}}}" file
="utils.go:221"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112916 ] -- >>>>>>>>> Get Volume By Id " file="request_handler.go:218"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112916 ] -- >>>>> Get Volume Cmd - Volume name/id: pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce" file="get_volume_cmd.go:43"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112916 ] -- GET VOLUME BY ID: {"Mountpoint":"/var/lib/kubelet/plugins/hpe.com/mounts/pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce","config":{"background_operation_status":"","com
pression":"true","cpg":"CIE3PAR-1_OPENSHIFT_HORSPROD","provisioning_type":"tdvv","snap_cpg":"CIE3PAR-1_OPENSHIFT_HORSPROD"},"description":"Volume created by the HPE CSI Driver for Kubernetes","id":"pvc-6b4e46c5-c95e-4f84-
b5dc-8e08861f27ce","name":"pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce","published":true,"size":17179869184,"volume_group_id":""}" file="get_volume_cmd.go:91"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112916 ] -- <<<<< Get Volume Cmd - Volume name/id: pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce" file="get_volume_cmd.go:95"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112916 ] -- <<<<<< Get Volume By Id" file="request_handler.go:225"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112917 ] -- >>>>> UnPublish Volume Cmd on array 10.241.37.20 for volume pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce and host 00f749a6-cdf3-6707-d197-5be67b5a5097" file="unpublish_volume_cmd
.go:35"
time="2025-08-23T12:12:16Z" level=info msg="Port Map Contains: map[string]string{"10.241.37.20":"8080"}" file="managed_client.go:126"
time="2025-08-23T12:12:16Z" level=info msg="Creating array client using 10.241.37.20:8080" file="managed_client.go:167"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112917 ] -- Access protocol for volume pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce is fc" file="unpublish_volume_cmd.go:57"
time="2025-08-23T12:12:16Z" level=error msg="[ REQUEST-ID 112917 ] -- Host details not found in CRD. Cannot delete VLUNS and Host." file="unpublish_volume_cmd.go:67"
time="2025-08-23T12:12:16Z" level=info msg="Raising panic with following message: Host details not found in CRD. Cannot delete VLUNS and Host." file="utils.go:205"
time="2025-08-23T12:12:16Z" level=info msg="[ REQUEST-ID 112917 ] -- <<<<< UnPublish Volume Cmd on array 10.241.37.20 for volume pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce and host 00f749a6-cdf3-6707-d197-5be67b5a5097" file="panic.go:791"
time="2025-08-23T12:12:16Z" level=info msg="CspManager: Recovering from panic..." file="csp_manager.go:40"
time="2025-08-23T12:12:16Z" level=info msg="Sending following message to CSI driver: Host details not found in CRD. Cannot delete VLUNS and Host." file="utils.go:213"
time="2025-08-23T12:12:16Z" level=info msg="Error response to be sent is &models.ErrorsPayload{Errors:[]models.ErrorObject{models.ErrorObject{Code:"Not Found", Message:"Host details not found in CRD. Cannot delete VLUNS and Host."}}}" file
="utils.go:221"
time="2025-08-23T12:16:01Z" level=info msg="[ REQUEST-ID 112918 ] -- >>>>>>>>> Get Volume By Id " file="request_handler.go:218"
time="2025-08-23T12:16:01Z" level=info msg="[ REQUEST-ID 112918 ] -- >>>>> Get Volume Cmd - Volume name/id: pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce" file="get_volume_cmd.go:43"
time="2025-08-23T12:16:01Z" level=info msg="[ REQUEST-ID 112918 ] -- GET VOLUME BY ID: {"Mountpoint":"/var/lib/kubelet/plugins/hpe.com/mounts/pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce","config":{"background_operation_status":"","com
pression":"true","cpg":"CIE3PAR-1_OPENSHIFT_HORSPROD","provisioning_type":"tdvv","snap_cpg":"CIE3PAR-1_OPENSHIFT_HORSPROD"},"description":"Volume created by the HPE CSI Driver for Kubernetes","id":"pvc-6b4e46c5-c95e-4f84-
b5dc-8e08861f27ce","name":"pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce","published":true,"size":17179869184,"volume_group_id":""}" file="get_volume_cmd.go:91"
time="2025-08-23T12:16:01Z" level=info msg="[ REQUEST-ID 112918 ] -- <<<<< Get Volume Cmd - Volume name/id: pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce" file="get_volume_cmd.go:95"
time="2025-08-23T12:16:01Z" level=info msg="[ REQUEST-ID 112918 ] -- <<<<<< Get Volume By Id" file="request_handler.go:225"
time="2025-08-23T12:16:01Z" level=info msg="[ REQUEST-ID 112919 ] -- >>>>> UnPublish Volume Cmd on array 10.241.37.20 for volume pvc-6b4e46c5-c95e-4f84-b5dc-8e08861f27ce and host 00f749a6-cdf3-6707-d197-5be67b5a5097" file="unpublish_volume_cmd
.go:35"

[saad0805]

i can send you complete logs if needed

[datamattsson]

I don't quite understand, this:

time="2025-08-23T12:12:16Z" level=info msg="Sending following message to CSI driver: Host details not found in CRD. Cannot delete VLUNS and Host." file="utils.go:213"
time="2025-08-23T12:12:16Z" level=info msg="Error response to be sent is &models.ErrorsPayload{Errors:[]models.ErrorObject{models.ErrorObject{Code:"Not Found", Message:"Host details not found in CRD. Cannot delete VLUNS and Host."}}}" file
="utils.go:221"

... indicates a different issue.

It's not related to not be able to creating a PVC from a VolumeSnapshot AFAIK.

[saad0805]

without doing anything, its now Bound!

[datamattsson]

There's a new way we create clones in 3.0.0 which is very confusing as we keep sending errors to Kubernetes while we're tracking background operations in the CSI driver. I think this is just a side effect of that.

[saad0805]

thank you. ill keep on monitor and post here if I find out about any new issue. It seems the errors were not blocking and at some point it just carries on and do the job.