Create template from snapshot parallel allocating same NFS secondary storage on agent

[rp-]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Storage

CLOUDSTACK VERSION

4.19.0

But I assume also 4.18 is affected

CONFIGURATION

I could recreate this with Linstor primary storage and the config option: lin.backup.snapshots disabled
I'm not sure yet which other storage combination could be used to trigger this.

OS / ENVIRONMENT

Ubuntu 22.04 non hyperconverged (2 storage nodes, 3 compute nodes (with agents)) setup.

SUMMARY

If I create 2 templates from snapshot (from different snapshots) at nearly the same time and the copy command is sent to the same agent,
both copy commands use the same storage pool object and the 1. finished will try to unmount
the still used storage pool (from the second copy command).
In the worst case the first copy command is finished while the second didn't even start yet to copy, resulting in the second copy command to write into the local mount directory.

STEPS TO REPRODUCE

Do both at the same time, with a primary storage that copies to secondary storage:
* Create a snapshot from volume1.
* Create a snapshot from volume2.

If the copy commands run on the same agent you would see the secondary storage created twice. e.g.:
2024-04-10 12:03:13,081 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1ec7c8ad) Attempting to create storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 (NetworkFilesystem) in libvirt
2024-04-10 12:03:13,089 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1ec7c8ad) Attempting to create storage pool e9e15545-fceb-3d76-be34-37dfaf5af384
2024-04-10 12:03:23,319 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:1fab4dd4) Attempting to create storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 (NetworkFilesystem) in libvirt
2024-04-10 12:03:43,257 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-1:null) (logid:1ec7c8ad) Attempting to remove storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 from libvirt
2024-04-10 12:03:48,862 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-4:null) (logid:1fab4dd4) Attempting to remove storage pool e9e15545-fceb-3d76-be34-37dfaf5af384 from libvirt

Attached are management and agent logs.
management.log
agent.log

EXPECTED RESULTS

Either running the copy commands in sequence or having a ref counted storage pool, that doesn't remove itself
while still used.

ACTUAL RESULTS

2 storage pool objects created, pointing to the same mount point.
The first finished would unmount the mount point for the other.

[DaanHoogland]

@rp- , I used the scenario and a script

cmk create template name=1st ostypeid=a3bf2482-06d8-11ef-881c-1e0048000312 snapshotid=639c5966-2c93-4c05-91c4-38922f84d981 &
cmk create template name=2nd ostypeid=a3bf2482-06d8-11ef-881c-1e0048000312 snapshotid=09f04474-c261-4ad2-9148-482b5ccc2c45 &

but didn't reproduce the issue. I think it is credible though. By the looks of your description it shouldn't even be storage specific, but maybe there is a timing constrained that makes me not reproduce it.

Do you have a fix/idea/design to deal with this?

[rp-]

@DaanHoogland
I think it is kinda storage related, by the storage motion strategy CloudStack takes.
In this specific Linstor case CloudStack has to first copy the Snapshot into the secondary storage as template (as the snapshot in this case is only Linstor storage)
First I thought this is the only storage motion strategy to trigger that, but apparently a Customer reported that they still have problems with NFS mounts not correctly mounted/unmounted.
So it might either be that other storage motions are also affected, or the Customer setup has some extra flaw which I can't figure out.

Well my idea would be the 2 points from the "Expected results", either it is done in sequence OR the storage pool are refcounted.

[DaanHoogland]

Well my idea would be the 2 points from the "Expected results", either it is done in sequence OR the storage pool are refcounted.

@rp- ,

• I do not like the in sequence idea as this is storage being copied over network this will no doubt be a performance hog.
• keeping a mount-count could work, meaning that the SSVM would have to keep a tally of how many requests are using a certain mount, and on umount just returning success without action when the count is not zero. Only to really un-mount once the it is. One catcha is that the mount functionality must be centralised and may not be called by commands, but only requested. If we guarantee this it will work as you describe
  €0,02

[btzq]

Hi @DaanHoogland @rp- ,

Just curious, does this issue happen with Ceph users? I havent heard of any so far,

So if its working with Ceph, means that Linstor is using a different method?

[DaanHoogland]

@rp- I am moving this to 4.19.2 (or later) as we do not have a way forward yet.

[rp-]

@DaanHoogland
Here would be some way to fix this, but no feedback yet if it helps:
LINBIT@e53ea85