Error creating volume snapshot (Ceph → NFS) on KVM host with CloudStack 4.21

[jeanvetorello]

problem

The snapshot should be successfully created on the Ceph primary and then backed up to the NFS secondary storage.

Actual Behavior

The snapshot was created on Ceph successfully (rbd snap create worked).

The backup to the NFS secondary failed.

The KVM agent logs show that libvirt could not mount the NFS storage pool:

libvirt failed to mount storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 at /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7

Full stack trace excerpt:

com.cloud.utils.exception.CloudRuntimeException: libvirt failed to mount storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 at /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.checkNetfsStoragePoolMounted(LibvirtStorageAdaptor.java:314)
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:881)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:390)
...

Notes

The Ceph snapshot creation worked fine (Successfully connected to Ceph cluster and Attempting to create RBD snapshot ... succeeded).

The failure happens in the CopyCommand when CloudStack tries to back up the snapshot to NFS.

The mountpoint -q check returns exit code 32, causing libvirt to fail mounting the NFS path.

2025-09-25 13:48:59,362 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:634b008b) Request:Seq 40021-3747839314902384680: { Cmd , MgmtId: 206863102531824, via: 40021, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":"1","poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":"0","url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":"false"}},"name":"ROOT-205","size":"(8.00 GB) 8589934592","path":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeId":"635","vmName":"i-122-205-VM","accountId":"122","format":"RAW","provisioningType":"THIN","poolId":"1","id":"635","deviceId":"0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":"1","poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":"0","url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":"false"}},"kvmIncrementalSnapshot":"false","vmName":"i-122-205-VM","name":"test-issue","hypervisorType":"KVM","id":"119","quiescevm":"false","physicalSize":"0","accountId":"122","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-25 13:48:59,362 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:634b008b) Processing command: org.apache.cloudstack.storage.command.CreateObjectCommand
2025-09-25 13:48:59,363 DEBUG [storage.resource.StorageSubsystemCommandHandlerBase] (AgentRequest-Handler-2:[]) (logid:634b008b) Executing command CreateObjectCommand: [{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":1,"poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":0,"url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":false}},"name":"ROOT-205","size":8589934592,"path":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeId":635,"vmName":"i-122-205-VM","accountId":122,"format":"RAW","provisioningType":"THIN","poolId":1,"id":635,"deviceId":0,"hypervisorType":"KVM","directDownload":false,"deployAsIs":false,"checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":false},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":1,"poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":0,"url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":false}},"kvmIncrementalSnapshot":false,"vmName":"i-122-205-VM","name":"test-issue","hypervisorType":"KVM","id":119,"quiescevm":false,"physicalSize":0,"accountId":122,"followRedirects":false}},"wait":0,"bypassHostMaintenance":false}].
2025-09-25 13:48:59,364 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-2:[]) (logid:634b008b) Property [hypervisor.uri] has empty or null value. Using default value [null].
2025-09-25 13:48:59,364 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:634b008b) Looking for libvirtd connection at: qemu:///system
2025-09-25 13:48:59,366 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:634b008b) Trying to fetch storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 from libvirt
2025-09-25 13:48:59,366 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:634b008b) Looking for libvirtd connection at: qemu:///system
2025-09-25 13:48:59,372 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:634b008b) Successfully refreshed pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 Capacity: (500.00 GB) 536866717696 Used: (130.35 GB) 139962546579 Available: (310.54 GB) 333439266816
2025-09-25 13:48:59,373 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:634b008b) Trying to fetch storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 from libvirt
2025-09-25 13:48:59,373 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-2:[]) (logid:634b008b) Looking for libvirtd connection at: qemu:///system
2025-09-25 13:48:59,378 DEBUG [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:634b008b) Successfully refreshed pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 Capacity: (500.00 GB) 536866717696 Used: (130.35 GB) 139962546579 Available: (310.54 GB) 333439266816
2025-09-25 13:48:59,542 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-2:[]) (logid:634b008b) Successfully connected to Ceph cluster at 192.168.27.80:0
2025-09-25 13:48:59,591 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-2:[]) (logid:634b008b) Attempting to create RBD snapshot e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3@3778411c-9608-4165-9c91-926de3cda097
2025-09-25 13:49:00,612 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-2:[]) (logid:634b008b) Seq 40021-3747839314902384680: { Ans: , MgmtId: 206863102531824, via: 40021, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"cloudstack/e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3/3778411c-9608-4165-9c91-926de3cda097","kvmIncrementalSnapshot":"false","id":"0","quiescevm":"false","physicalSize":"0","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }
2025-09-25 13:49:02,058 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:634b008b) Request:Seq 40021-3747839314902384681: { Cmd , MgmtId: 206863102531824, via: 40021, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"cloudstack/e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3/3778411c-9608-4165-9c91-926de3cda097","volume":{"uuid":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":"1","poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":"0","url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":"false"}},"name":"ROOT-205","size":"(8.00 GB) 8589934592","path":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeId":"635","vmName":"i-122-205-VM","accountId":"122","format":"RAW","provisioningType":"THIN","poolId":"1","id":"635","deviceId":"0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":"1","poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":"0","url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":"false"}},"kvmIncrementalSnapshot":"false","vmName":"i-122-205-VM","name":"test-issue","hypervisorType":"KVM","id":"119","quiescevm":"false","physicalSize":"0","accountId":"122","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/122/635","volume":{"uuid":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":"1","poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":"0","url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":"false"}},"name":"ROOT-205","size":"(8.00 GB) 8589934592","path":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeId":"635","vmName":"i-122-205-VM","accountId":"122","format":"RAW","provisioningType":"THIN","poolId":"1","id":"635","deviceId":"0","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false","checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":"false"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.27.80/cloudstack","_role":"Image"}},"kvmIncrementalSnapshot":"false","vmName":"i-122-205-VM","name":"test-issue","hypervisorType":"KVM","id":"119","quiescevm":"false","physicalSize":"0","accountId":"122","followRedirects":"false"}},"executeInSequence":"false","options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}] }
2025-09-25 13:49:02,058 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:634b008b) Processing command: org.apache.cloudstack.storage.command.CopyCommand
2025-09-25 13:49:02,058 DEBUG [storage.resource.StorageSubsystemCommandHandlerBase] (AgentRequest-Handler-1:[]) (logid:634b008b) Executing command CopyCommand: [{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"cloudstack/e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3/3778411c-9608-4165-9c91-926de3cda097","volume":{"uuid":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":1,"poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":0,"url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":false}},"name":"ROOT-205","size":8589934592,"path":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeId":635,"vmName":"i-122-205-VM","accountId":122,"format":"RAW","provisioningType":"THIN","poolId":1,"id":635,"deviceId":0,"hypervisorType":"KVM","directDownload":false,"deployAsIs":false,"checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":false},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":1,"poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":0,"url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":false}},"kvmIncrementalSnapshot":false,"vmName":"i-122-205-VM","name":"test-issue","hypervisorType":"KVM","id":119,"quiescevm":false,"physicalSize":0,"accountId":122,"followRedirects":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/122/635","volume":{"uuid":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"21c57ab7-9630-3817-b953-e46a7efcf2b7","name":"primary-ceph","id":1,"poolType":"RBD","host":"192.168.27.80","path":"cloudstack","port":0,"url":"RBD://192.168.27.80/cloudstack/?ROLE=Primary&STOREUUID=21c57ab7-9630-3817-b953-e46a7efcf2b7","isManaged":false}},"name":"ROOT-205","size":8589934592,"path":"e6a6f8ce-6ac2-44d5-ae17-5161d9bc42d3","volumeId":635,"vmName":"i-122-205-VM","accountId":122,"format":"RAW","provisioningType":"THIN","poolId":1,"id":635,"deviceId":0,"hypervisorType":"KVM","directDownload":false,"deployAsIs":false,"checkpointPaths":[],"checkpointImageStoreUrls":[],"followRedirects":false},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.27.80/cloudstack","_role":"Image"}},"kvmIncrementalSnapshot":false,"vmName":"i-122-205-VM","name":"test-issue","hypervisorType":"KVM","id":119,"quiescevm":false,"physicalSize":0,"accountId":122,"followRedirects":false}},"executeInSequence":false,"options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":21600,"bypassHostMaintenance":false}].
2025-09-25 13:49:02,059 DEBUG [agent.properties.AgentPropertiesFileHandler] (AgentRequest-Handler-1:[]) (logid:634b008b) Property [hypervisor.uri] has empty or null value. Using default value [null].
2025-09-25 13:49:02,059 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-1:[]) (logid:634b008b) Looking for libvirtd connection at: qemu:///system
2025-09-25 13:49:02,060 DEBUG [kvm.storage.KVMStoragePoolManager] (AgentRequest-Handler-1:[]) (logid:634b008b) Get storage pool by uri: nfs://192.168.27.80/cloudstack
2025-09-25 13:49:02,061 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:634b008b) Attempting to create storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 (NetworkFilesystem) in libvirt
2025-09-25 13:49:02,061 DEBUG [kvm.resource.LibvirtConnection] (AgentRequest-Handler-1:[]) (logid:634b008b) Looking for libvirtd connection at: qemu:///system
2025-09-25 13:49:02,062 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:634b008b) Found existing defined storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7, using it.
2025-09-25 13:49:02,064 DEBUG [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:634b008b) Executing command [/bin/bash -c mountpoint -q /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7 ].
2025-09-25 13:49:02,069 WARN [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:634b008b) Execution of process [83938] for command [/bin/bash -c mountpoint -q /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7 ] failed.
2025-09-25 13:49:02,070 DEBUG [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:634b008b) Exit value of process [83938] for command [/bin/bash -c mountpoint -q /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7 ] is [32].
2025-09-25 13:49:02,070 WARN [utils.script.Script] (AgentRequest-Handler-1:[]) (logid:634b008b) Process [83938] for command [/bin/bash -c mountpoint -q /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7 ] encountered the error: [32].
2025-09-25 13:49:02,070 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-1:[]) (logid:634b008b) libvirt failed to mount storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 at /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7
2025-09-25 13:49:02,071 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-1:[]) (logid:634b008b) Failed to backup snapshot: com.cloud.utils.exception.CloudRuntimeException: libvirt failed to mount storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 at /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7
2025-09-25 13:49:02,072 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:634b008b) Seq 40021-3747839314902384681: { Ans: , MgmtId: 206863102531824, via: 40021, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: libvirt failed to mount storage pool 21c57ab7-9630-3817-b953-e46a7efcf2b7 at /mnt/21c57ab7-9630-3817-b953-e46a7efcf2b7","wait":"0","bypassHostMaintenance":"false"}}] }

versions

Environment

CloudStack Management Server: 4.21

CloudStack Agent (KVM): 4.21

Hypervisor: KVM on Ubuntu 22.04.5 LTS

Primary Storage: Ceph RBD (ceph version 19.2.3)

Secondary Storage: NFSv4 on cephnfs

The steps to reproduce the bug

Steps to Reproduce

Go to Volumes in the CloudStack UI.

Select a ROOT volume (on Ceph primary).

Click Take Snapshot

What to do about it?

Workaround Attempted

I tested with the patch from PR #11690
, which disables the mountpoint check for NFS pools.

With this patch applied, the snapshot creation on Ceph succeeds.

However, the copy to NFS secondary still fails with the same libvirt mount error.

[jeanvetorello]

I identified the issue: the mount was failing because of the NFS version.
CephFS does not support NFSv3. It only works with NFSv4.x
After forcing the client to use NFSv4.2, the mount succeeded and the process worked correctly.
secstorage.nfs.version = 4.2
thanks