Unable to import volume using ontap-san-economy drivers

[sergeykuperman]

Hello,
we are currently working on DR flow for our aws fsx ontap filesystem solution,
our use-case is - we use ontap-san-economy drivers for 100 luns (PVC's) per flexVol - tagged and backed up using aws Backup service, then restored into new FSX filesystem (new cluster, new svm, volumes containing multiple LUNs imported from recovery points)
next logical step is recreate pvc+pv pairs in the new cluster:
we have those backed up using velero, current strategy is to take the LUN data from pv
internalID: /svm/svm-8b492990/flexvol/trident_lun_pool_trident_HUUIAUUPCE/lun/trident_pvc_551d8895_a01f_479c_9437_feba9edca656

take the new fsx filesystem backendUUID, and use tridentctl import volume:
https://docs.netapp.com/us-en/trident/trident-use/vol-import.html#ontap-san
docs specifically state support for ontap-san-economy storage drivers, but do not give example for single LUN import.
recreate the pvc template for the new cluster, e.g.:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: workspaces-ws-lpaav-data
  namespace: ws-ns-workspaces-ws-lpaav
  labels:
    workspace-resource: workspaces-ws-lpaav
    workspace.devx.sap.com/backup: "true"
    workspace.devx.sap.com/persistence-fsx: "true"
    wsid: workspaces-ws-lpaav
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: netapp-economy-encrypted

and import (restored from the restore point) volume(s) into trident using:
tridentctl -n trident import volume fs-04c800c498b492990-economy trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151 -f ws-ckv8p-pvc.yaml
but the command does not work:

seastranger@W-5CG3503TSZ:~/repos/esm/trident/ws$ tridentctl -n trident import volume fs-04c800c498b492990-economy   trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151   -f ws-ckv8p-pvc.yaml
Error: error communicating with Trident REST API; Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF
command terminated with exit code 1
Error: exit status 1

i see the following logs in the controller:

time="2025-12-10T15:41:41Z" level=debug msg="ImportVolume: created pending PVC." PVC="&PersistentVolumeClaim{ObjectMeta:{workspaces-ws-lpaav-data  ws-ns-workspaces-ws-lpaav  a9f94a12-8757-424e-922b-4fb48871dd7a 55716001 0 2025-12-10 15:41:41 +0000 UTC <nil> <nil> map[workspace-resource:workspaces-ws-lpaav workspace.devx.sap.com/backup:true workspace.devx.sap.com/persistence-fsx:true wsid:workspaces-ws-lpaav] map[trident.netapp.io/importBackendUUID:0591493b-1bfc-43f6-b010-4dae8c28b844 trident.netapp.io/importOriginalName:trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151 trident.netapp.io/notManaged:false volume.beta.kubernetes.io/storage-provisioner:csi.trident.netapp.io] [] [kubernetes.io/pvc-protection] [{trident_orchestrator Update v1 2025-12-10 15:41:41 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:trident.netapp.io/importBackendUUID\":{},\"f:trident.netapp.io/importOriginalName\":{},\"f:trident.netapp.io/notManaged\":{},\"f:volume.beta.kubernetes.io/storage-provisioner\":{}},\"f:labels\":{\".\":{},\"f:workspace-resource\":{},\"f:workspace.devx.sap.com/backup\":{},\"f:workspace.devx.sap.com/persistence-fsx\":{},\"f:wsid\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:resources\":{\"f:requests\":{\".\":{},\"f:storage\":{}}},\"f:storageClassName\":{},\"f:volumeMode\":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{10756292608 0} {<nil>} 10756292608 DecimalSI},},},VolumeName:,Selector:nil,StorageClassName:*netapp-economy-encrypted,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},}" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=info msg=">>>> waitForCachedPVByName" MaxElapsedTime=3m0s Name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a logLayer=rest_frontend namespace=ws-ns-workspaces-ws-lpaav pvc=workspaces-ws-lpaav-data requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:41Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:42Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:42Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:42Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:43Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:43Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:43Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:45Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:45Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:46Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:47Z" level=debug msg="REST API call received." Duration="9.635µs" Method=GET RequestURL=/trident/v1/version Route=GetVersion logLayer=rest_frontend requestID=c75ac54b-bbff-477b-aa05-25be7ab477a3 requestSource=REST workflow="trident_rest=logger"
time="2025-12-10T15:41:47Z" level=debug msg="REST API call complete." Duration="142.291µs" Method=GET RequestURL=/trident/v1/version Route=GetVersion StatusCode=200 logLayer=rest_frontend requestID=c75ac54b-bbff-477b-aa05-25be7ab477a3 requestSource=REST workflow="trident_rest=logger"
time="2025-12-10T15:41:48Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:48Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:48Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:53Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-a9f94a12-8757-424e-922b-4fb48871dd7a requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:53Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"
time="2025-12-10T15:41:53Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=eb5d93e0-9e0b-42b8-9c47-b6faa8f9acc2 requestSource=REST workflow="volume=import"

Could you please update the docs with example of volume import for single lun in multi-lun flexVol, using ontap-san-economy drivers ? Am i doing something wrong with my approach for volume/pvc restoration in separate cluster/fsx filesystem from aws vol backup ?

Thanks in advance

[YvosOnTheHub]

Hi
to import a SAN_ECO volume, you can use the following example:

cat << EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: lun-import2
  annotations:
    trident.netapp.io/importOriginalName: "scenario7_4_2/lun0"
    trident.netapp.io/importBackendUUID: "8a802f70-8782-4acf-bd24-8d287f9e3e6c"
    trident.netapp.io/notManaged: "false"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: storage-class-iscsi-economy
EOF

notice the OriginalName parameter contains both ONTAP volume name & LUN name

more details here:
https://github.com/YvosOnTheHub/LabNetApp/tree/master/Kubernetes_v6/Trident_Scenarios/Scenario07/4_iSCSI_Import#b-import-multiple-luns-with-ontap-san-economy--kubectl

[sergeykuperman]

This is exactly what i do:

seastranger@W-5CG3503TSZ:~/repos/esm/trident$ tridentctl -n trident import volume fs-04c800c498b492990-economy   trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656   -f ws/ws-w2yn2-pvc.yaml
Error: error communicating with Trident REST API; Post "http://127.0.0.1:8000/trident/v1/volume/import": EOF
command terminated with exit code 1
Error: exit status 1
seastranger@W-5CG3503TSZ:~/repos/esm/trident$ k get pvc -n  ws-ns-workspaces-ws-w2yn2 -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      trident.netapp.io/importBackendUUID: 0591493b-1bfc-43f6-b010-4dae8c28b844
      trident.netapp.io/importOriginalName: trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656
      trident.netapp.io/notManaged: "false"
      volume.beta.kubernetes.io/storage-provisioner: csi.trident.netapp.io
    creationTimestamp: "2025-12-11T09:37:40Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      workspace-resource: workspaces-ws-w2yn2
      workspace.devx.sap.com/backup: "true"
      workspace.devx.sap.com/persistence-fsx: "true"
      wsid: workspaces-ws-w2yn2
    name: workspaces-ws-w2yn2-data
    namespace: ws-ns-workspaces-ws-w2yn2
    resourceVersion: "55901928"
    uid: d4c8d0ad-4b38-48e3-87c5-b449e180587a
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: "10756292608"
    storageClassName: netapp-economy-encrypted
    volumeMode: Filesystem
  status:
    phase: Pending
kind: List
metadata:
  resourceVersion: ""

inside svm the lun does exist:

seastranger@W-5CG3503TSZ:~/repos/esm/trident$ k node-shell ip-10-250-20-191.eu-north-1.compute.internal
spawning "nsenter-5wm2cz" on "ip-10-250-20-191.eu-north-1.compute.internal"
If you don't see a command prompt, try pressing enter.
root@ip-10-250-20-191:/# ssh fsxadmin@management.fs-04c800c498b492990.fsx.eu-north-1.amazonaws.com
(fsxadmin@management.fs-04c800c498b492990.fsx.eu-north-1.amazonaws.com) Password:

Last login time: 12/11/2025 07:57:58
FsxId04c800c498b492990::> set -privilege advanced

Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel.
Do you want to continue? {y|n}: y

FsxId04c800c498b492990::*> lun show -vserver svm-8b492990 -fields path,serial,uuid,size
vserver      path                                                                                      size    serial       uuid
------------ ----------------------------------------------------------------------------------------- ------- ------------ ------------------------------------
svm-8b492990 /vol/trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 10.02GB lWB4V]ZLM96t e415b9e4-934a-4d8b-a977-6f983cc30377
svm-8b492990 /vol/trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656_snapshot_snapshot_cf8f0548_bd43_4aa3_a9ee_234348a63954
                                                                                                       10.02GB lWB4V]ZLM96u 9cb0035a-208f-4284-ba00-1c1e870f1f79
svm-8b492990 /vol/trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_63250213_5073_4e9c_aa30_4d1a3dd19151 10.02GB lWB4V]ZLM96v fd6d3f7c-5dc3-4688-ab88-c3bfc7df0d98
3 entries were displayed.

FsxId04c800c498b492990::*>

the backend uuid is correct:

seastranger@W-5CG3503TSZ:~/repos/esm/trident$ k get tridentbackend -n trident | grep fs-04c800c498b492990-economy
tbe-h8r2w   fs-04c800c498b492990-economy   0591493b-1bfc-43f6-b010-4dae8c28b844

the import fails without root cause in the controller log:

cat trident.log | grep d2510c24-e271-41b5-8874-5d7fbe9f4503 
time="2025-12-11T09:37:40Z" level=debug msg="REST API call received." Duration="7.458µs" Method=POST RequestURL=/trident/v1/volume/import Route=ImportVolume logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="trident_rest=logger"
time="2025-12-11T09:37:40Z" level=debug msg="Found requested frontend." logLayer=core name=k8s_csi_helper requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Orchestrator#GetVolumeForImport" backendName=fs-04c800c498b492990-economy logLayer=core requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST volumeID=trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Calculated data size after subtracting snapshot reserve from total size." dataSizeBytes=10756292608 logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST sizeWithoutSnapReserve=1.0756292608e+10 snapReserveMultiplier=1 snapshotReserve=0 totalSize=10756292608 workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Volume import determined volume size." claimSize="{{10756292608 0} {<nil>} 10756292608 DecimalSI}" data size=10756292608 logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST snapshot reserve=0 total size=10756292608 workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="ImportVolume: created pending PVC." PVC="&PersistentVolumeClaim{ObjectMeta:{workspaces-ws-w2yn2-data  ws-ns-workspaces-ws-w2yn2  d4c8d0ad-4b38-48e3-87c5-b449e180587a 55901928 0 2025-12-11 09:37:40 +0000 UTC <nil> <nil> map[workspace-resource:workspaces-ws-w2yn2 workspace.devx.sap.com/backup:true workspace.devx.sap.com/persistence-fsx:true wsid:workspaces-ws-w2yn2] map[trident.netapp.io/importBackendUUID:0591493b-1bfc-43f6-b010-4dae8c28b844 trident.netapp.io/importOriginalName:trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656 trident.netapp.io/notManaged:false volume.beta.kubernetes.io/storage-provisioner:csi.trident.netapp.io] [] [kubernetes.io/pvc-protection] [{trident_orchestrator Update v1 2025-12-11 09:37:40 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:trident.netapp.io/importBackendUUID\":{},\"f:trident.netapp.io/importOriginalName\":{},\"f:trident.netapp.io/notManaged\":{},\"f:volume.beta.kubernetes.io/storage-provisioner\":{}},\"f:finalizers\":{\".\":{},\"v:\\\"kubernetes.io/pvc-protection\\\"\":{}},\"f:labels\":{\".\":{},\"f:workspace-resource\":{},\"f:workspace.devx.sap.com/backup\":{},\"f:workspace.devx.sap.com/persistence-fsx\":{},\"f:wsid\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:resources\":{\"f:requests\":{\".\":{},\"f:storage\":{}}},\"f:storageClassName\":{},\"f:volumeMode\":{}}} }]},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{10756292608 0} {<nil>} 10756292608 DecimalSI},},},VolumeName:,Selector:nil,StorageClassName:*netapp-economy-encrypted,VolumeMode:*Filesystem,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},Status:PersistentVolumeClaimStatus{Phase:Pending,AccessModes:[],Capacity:ResourceList{},Conditions:[]PersistentVolumeClaimCondition{},AllocatedResources:ResourceList{},AllocatedResourceStatuses:map[ResourceName]ClaimResourceStatus{},CurrentVolumeAttributesClassName:nil,ModifyVolumeStatus:nil,},}" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=info msg=">>>> waitForCachedPVByName" MaxElapsedTime=3m0s Name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a logLayer=rest_frontend namespace=ws-ns-workspaces-ws-w2yn2 pvc=workspaces-ws-w2yn2-data requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:40Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:41Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:41Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:41Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:43Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:43Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:43Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:45Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:45Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:45Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:47Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:47Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:47Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:51Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:51Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:51Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:56Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:56Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:37:56Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:01Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:01Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:01Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:07Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:07Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:07Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:12Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:12Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:12Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:16Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:16Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:16Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:22Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:22Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:22Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:27Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:27Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:27Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:32Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:32Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:32Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:37Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:37Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:37Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:42Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:42Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:42Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:47Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:47Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:47Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:53Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:53Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:53Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:57Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:57Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:38:57Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:03Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:03Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:03Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:07Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:07Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:07Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:13Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:13Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:13Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:17Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:17Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:17Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:23Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:23Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:23Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:28Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:28Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:28Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:32Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:32Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:32Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:37Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:37Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:37Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:42Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:42Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:42Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:47Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:47Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:47Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:52Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:52Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:52Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:57Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:57Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:39:57Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:02Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:02Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:02Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:07Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:07Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:07Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:11Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:11Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:11Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:16Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:16Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:16Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:21Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:21Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:21Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:25Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:25Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:25Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:30Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:30Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:30Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:35Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:35Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:35Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=debug msg="Could not find cached PV object by name." logLayer=rest_frontend name=pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=debug msg=">>>> checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=debug msg="<<<< checkAndHandleUnrecoverableError" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=info msg="<<<< waitForCachedPVByName" logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="volume=import"
time="2025-12-11T09:40:39Z" level=error msg="error waiting for PV pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a; PV pvc-d4c8d0ad-4b38-48e3-87c5-b449e180587a was not in cache after 180.00 seconds" handler=ImportVolume logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="trident_rest=logger"
time="2025-12-11T09:40:39Z" level=debug msg="REST API call complete." Duration=2m59.457350493s Method=POST RequestURL=/trident/v1/volume/import Route=ImportVolume StatusCode=400 logLayer=rest_frontend requestID=d2510c24-e271-41b5-8874-5d7fbe9f4503 requestSource=REST workflow="trident_rest=logger"

Please advise on how i can troubleshoot this

[YvosOnTheHub]

I'am a bit confused

• the import process failed
• a PVC is in 'pending' state

can you confirm that the PVC did not exist before trying to import the LUN ?

[sergeykuperman]

the pvc did not exist, it was created as a result of the tridentctl volume import command, it receives the pvc yaml file (without annot) as param:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    workspace-resource: workspaces-ws-w2yn2
    workspace.devx.sap.com/backup: "true"
    workspace.devx.sap.com/persistence-fsx: "true"
    wsid: workspaces-ws-w2yn2
  name: workspaces-ws-w2yn2-data
  namespace: ws-ns-workspaces-ws-w2yn2
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: netapp-economy-encrypted

then the command:

tridentctl -n trident import volume fs-04c800c498b492990-economy   trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656   -f ws/ws-w2yn2-pvc.yaml

i am guessing the command adds the annotations and applies it in the cluster

[NA-Scott]

The annotations:

    trident.netapp.io/importOriginalName: "scenario7_4_2/lun0"
    trident.netapp.io/importBackendUUID: "8a802f70-8782-4acf-bd24-8d287f9e3e6c"
    trident.netapp.io/notManaged: "false"

replace the need for running the tridentctl import command

Using the annotation as suggested with the importOriginalName in the format of "volume/lun" you can import the lun you desire. With the import command you are using you are only specifying a volume and not telling Trident the name of the lun to use.

[sergeykuperman]

but i see that the annotation that is created on the PVC does contain both volume and lun:

    annotations:
      trident.netapp.io/importBackendUUID: 0591493b-1bfc-43f6-b010-4dae8c28b844
      trident.netapp.io/importOriginalName: trident_lun_pool_trident_HUUIAUUPCE/trident_pvc_551d8895_a01f_479c_9437_feba9edca656
      trident.netapp.io/notManaged: "false"

vol - trident_lun_pool_trident_HUUIAUUPCE
lun - trident_pvc_551d8895_a01f_479c_9437_feba9edca656
match what i see in svm (see output above)

so i do follow the documentation, and it does not work (tridentvolume and pv are not created in the cluster)
any ideas why this could happen ?

[sergeykuperman]

@NA-Scott i have tried creating PVC with the annotation as you suggest, this does not trigger the import flow output in controller log at all, only the command tridentctl volume import triggers the logs, and the failure.
applying PVC with the annotation does nothing as far as i can see

[clintonk]

Hi, @sergeykuperman. The PVC that you entered manually or via tridentctl should trigger a CSI call to Trident just like any other PVC. The annotations @NA-Scott mentioned are applied for you by tridentctl and let Trident know it is an import instead of a new PVC. I don't see anything obviously wrong with your PVC or tridentctl invocation. But your log indicates that the PVC request never made it back to Trident via K8s, the provisioner sidecar, and CSI. I would ensure the netapp-economy-encrypted storage class exists and points to Trident as its provisioner, and then check the provisioner sidecar logs and PVC events for more clues. You could also try creating a new PVC in the new FSxN filesystem to ensure it is created and bound correctly. If that doesn't yield anything, please open a support case so @NA-Scott can look into this further.

[sergeykuperman]

i saw the following event in the namespace where import is attempted:

0s          Normal   WaitForFirstConsumer   persistentvolumeclaim/workspaces-ws-w2yn2-data   waiting for first consumer to be created before binding

which makes sense since the storageclass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
  name: netapp-economy-encrypted
parameters:
  backendType: ontap-san-economy
  csi.storage.k8s.io/node-expand-secret-name: luks-passphrases-secret
  csi.storage.k8s.io/node-expand-secret-namespace: ${pvc.namespace}
  csi.storage.k8s.io/node-publish-secret-name: luks-passphrases-secret
  csi.storage.k8s.io/node-publish-secret-namespace: ${pvc.namespace}
  csi.storage.k8s.io/node-stage-secret-name: luks-passphrases-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ${pvc.namespace}
  fsType: ext4
  selector: storagetype=devspace-economy-encrypted
provisioner: csi.trident.netapp.io
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

is wait for first consumer, so i created the luks secret with the matching passphrase from the original lun (before it was backed up)
and a pod mounting this pvc, and then i got the following events:

seastranger@W-5CG3503TSZ:~/repos/esm/trident/ws$ k get events -n test -w
LAST SEEN   TYPE      REASON                   OBJECT                                           MESSAGE
3m55s       Normal    Scheduled                pod/test-pod                                     Successfully assigned test/test-pod to ip-10-250-19-210.eu-north-1.compute.internal
3m54s       Normal    SuccessfulAttachVolume   pod/test-pod                                     AttachVolume.Attach succeeded for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0"
24s         Warning   FailedMount              pod/test-pod                                     MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
14h         Normal    WaitForFirstConsumer     persistentvolumeclaim/workspaces-ws-w2yn2-data   waiting for first consumer to be created before binding
15m         Normal    WaitForFirstConsumer     persistentvolumeclaim/workspaces-ws-w2yn2-data   waiting for first consumer to be created before binding
7m51s       Normal    WaitForFirstConsumer     persistentvolumeclaim/workspaces-ws-w2yn2-data   waiting for first consumer to be created before binding
3m59s       Normal    Provisioning             persistentvolumeclaim/workspaces-ws-w2yn2-data   External provisioner is provisioning volume for claim "test/workspaces-ws-w2yn2-data"
3m59s       Normal    ExternalProvisioning     persistentvolumeclaim/workspaces-ws-w2yn2-data   Waiting for a volume to be created either by the external provisioner 'csi.trident.netapp.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
3m57s       Normal    ProvisioningSuccess      persistentvolumeclaim/workspaces-ws-w2yn2-data   provisioned a volume
3m57s       Normal    ProvisioningSucceeded    persistentvolumeclaim/workspaces-ws-w2yn2-data   Successfully provisioned volume pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0
0s          Warning   FailedMount              pod/test-pod                                     MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
0s          Warning   FailedMount              pod/test-pod                                     MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
0s          Warning   FailedMount              pod/test-pod                                     MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS
0s          Warning   FailedMount              pod/test-pod                                     MountVolume.MountDevice failed for volume "pvc-c11773aa-b2a0-4b17-be88-ccc5836b93a0" : rpc error: code = Internal desc = rpc error: code = Internal desc = failed to stage volume: LUN trident_pvc_c11773aa_b2a0_4b17_be88_ccc5836b93a0, device dm-0 already formatted with other filesystem: crypto_LUKS

and the logs in the provisioner, and a snippet from main controller i am attaching:

trident-prov.log

vol_log.txt

It seems that the volume is imported, and then attacher attempts to do luks format on it even tho it is already formatted in the imported volume.
when i create straight new pvc in the new filesystem everything works.

@clintonk you mean aws support ticket correct ?

Thanks

[clintonk]

K8s & CSI don't know anything about volume import, so PVCs annotated for import come in to Trident like any other PVC; your use of WaitForFirstConsumer explains why the import didn't get handled immediately. If you import a volume with Immediate binding mode, does everything work as expected? Or is there still an issue with LUKS format? I'm unsure how a LUKS issue would be limited to an import workflow. If you want to open a support ticket, I'd start with NetApp support.

[sergeykuperman]

no, it does not work, the binding mode only affects if the issue with LUKS format happens immediately, or when trying to attach PVC to the pod.
I have tried to create account here
https://mysupport.netapp.com/site/user/registration
To create support ticket,
but this site seems bugged, it does not let me past the email verification step, keeps asking for more codes, and sending them to email, after entering the codes i see a short message that the code has been verified, then i am redirected to the same page where i need to verify again in a loop