kvm.storage.LibvirtStorageAdaptor - Can't find Storage Pool

[wverleger]

Hi,

We are running ACS 4.19.2.0 with KVM hypervisors (Ubuntu 24.04 LTS), using linstor as hyperconverged storage and nfs as secondary storage.

Whilde doing daily volume snapshots we are seeing this INFO/WARNING popping up in our agent.logs:

2025-04-28 01:51:41,027 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:75bed1e5) Attempting to create storage pool 638de611-0552-36a1-91ca-6df1c493dc8f (NetworkFilesystem) in libvirt
2025-04-28 01:51:41,029 WARN  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:75bed1e5) Storage pool 638de611-0552-36a1-91ca-6df1c493dc8f was not found running in libvirt. Need to create it.
2025-04-28 01:51:41,029 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:75bed1e5) Didn't find an existing storage pool 638de611-0552-36a1-91ca-6df1c493dc8f by UUID, checking for pools with duplicate paths
2025-04-28 01:51:41,035 WARN  [utils.script.Script] (agentRequest-Handler-3:null) (logid:75bed1e5) Execution of process [3003735] for command [/bin/bash -c mountpoint -q /mnt/638de611-0552-36a1-91ca-6df1c493dc8f ] failed.
2025-04-28 01:51:41,036 WARN  [utils.script.Script] (agentRequest-Handler-3:null) (logid:75bed1e5) Process [3003735] for command [/bin/bash -c mountpoint -q /mnt/638de611-0552-36a1-91ca-6df1c493dc8f ] encountered the error: [32].
2025-04-28 01:51:41,086 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:75bed1e5) Trying to fetch storage pool 638de611-0552-36a1-91ca-6df1c493dc8f from libvirt
2025-04-28 01:53:03,255 INFO  [resource.wrapper.LinstorBackupSnapshotCommandWrapper] (agentRequest-Handler-3:null) (logid:75bed1e5) Backup snapshot '/dev/mapper/vg_nvme-cs--0c049ab6--2861--4ee9--90f6--97039bee7680_00000_cs--c7bf25e1--0580--494e--a58f--b1cc9f86b67c' to '/mnt/638de611-0552-36a1-91ca-6df1c493dc8f/snapshots/33/675/k-1189-8_ROOT-642_20250428015134'
2025-04-28 01:53:05,197 INFO  [resource.wrapper.LinstorBackupSnapshotCommandWrapper] (agentRequest-Handler-3:null) (logid:75bed1e5) Backup shrunk /mnt/638de611-0552-36a1-91ca-6df1c493dc8f/snapshots/33/675/k-1189-8_ROOT-642_20250428015134 to actual size 53687091200
2025-04-28 01:53:05,197 INFO  [resource.wrapper.LinstorBackupSnapshotCommandWrapper] (agentRequest-Handler-3:null) (logid:75bed1e5) Actual file size for '/mnt/638de611-0552-36a1-91ca-6df1c493dc8f/snapshots/33/675/k-1189-8_ROOT-642_20250428015134' is 12752453632

The volume snaphots will be created but I was wondering what those messages mean?

[DaanHoogland]

@wverleger , the only question is if the snapshots appear on the secondary storage?

I can see only message of concern. That seems to be the second WARN, Execution of process [3003735] for command [/bin/bash -c mountpoint -q /mnt/638de611-0552-36a1-91ca-6df1c493dc8f ] failed.

but if the snapshot gets backed up, a retry must have been successful. It looks that way as the linstor driver knows to construct the path /mnt/638de611-0552-36a1-91ca-6df1c493dc8f/snapshots/33/675/k-1189-8_ROOT-642_20250428015134

cc @rp- , any comment?

[rp-]

The LinstorBackupSnapshotCommand would just try to get the nfs secondary storage:'

secondaryPool = storagePoolMgr.getStoragePoolByURI(dstDataStore.getUrl());

Why the first mount failed, I don't know as we had a customer with unmount problems, I added a few more safeguards and
checks around mounting nfs secondary storage.

But this seems a bit different, as the storage pool seems not be known by libvirt yet.

[wverleger]

Sometimes seeing messages like this in ACS:

Nevertheless the snapshot will be backed up.
Could this be related?

[DaanHoogland]

it might well be. Is there also a successful SNAPSHOT.CREATE event?

[wverleger]

Yes:

[DaanHoogland]

ok, it remains a minor issue, but is ignorable in your case as it seems @wverleger . The fact that the event-record is created for a failed event that is retried and succeeds afterwards is unnecessary but logical.

cc @rp-, thoughts?

[rp-]

It would be interesting why the mount sometimes fails.
Or is this happening all the time and only works on the second attempt?