Default local storage /var/lib/libvirt/images is getting added everytime cloudstack-agent is restarted

[kiranchavala]

ISSUE TYPE

Bug

COMPONENT NAME

Component: Bug

CLOUDSTACK VERSION

Cloudstack version 4.19.1.1, 4.19.1.2

Ubuntu 22.04, and other Os

SUMMARY

Default local storage /var/lib/libvirt/images is getting added everytime the agent is restarted

Steps to reproduce the behaviour

1. Enable the 'Enable local storage for User Instances' on the zone
2. Make sure the agent.properties doesn't contain any value of

local.storage.uuid=
local.storage.path=

Example

root@kvm2:/home/kiran# cat /etc/cloudstack/agent/agent.properties
#Storage
#Thu Oct 17 08:53:20 UTC 2024
cluster=1
pod=1
resource=com.cloud.hypervisor.kvm.resource.LibvirtComputingResource
private.network.device=cloudbr0
domr.scripts.dir=scripts/network/domr/kvm
guest.network.device=cloudbr0
keystore.passphrase=v5DA5u6vazg2NpZP
hypervisor.type=kvm
port=8250
zone=1
public.network.device=cloudbr0
host=192.168.55.216@static
guid=05b58647-269d-386b-a3e9-334037c001e7
LibvirtComputingResource.id=5
workers=5
iscsi.session.cleanup.enabled=false
vm.migrate.wait=3600

3. Restart the cloud stack agent

4. The local storage of /var/lib/libvirt/images is added

5. Restart the cloud stack agent again

https://docs.cloudstack.apache.org/en/4.19.1.1/adminguide/storage.html#using-multiple-local-storages-kvm-only

Logs

2024-10-17 08:43:10,714 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 1fd120df-566c-4480-a31d-cdfa3aad7344 (Filesystem) in libvirt
2024-10-17 08:43:10,714 DEBUG [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Looking for libvirtd connection at: qemu:///system
2024-10-17 08:43:10,716 WARN  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Storage pool 1fd120df-566c-4480-a31d-cdfa3aad7344 was not found running in libvirt. Need to create it.
2024-10-17 08:43:10,716 INFO  [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Didn't find an existing storage pool 1fd120df-566c-4480-a31d-cdfa3aad7344 by UUID, checking for pools with duplicate paths
2024-10-17 08:43:10,717 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Checking path of existing pool 9f638db4-0845-3250-bfc7-f6f8be657c7f against pool we want to create
2024-10-17 08:43:10,721 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 1fd120df-566c-4480-a31d-cdfa3aad7344
2024-10-17 08:43:10,721 DEBUG [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) <pool type='dir'>
<name>1fd120df-566c-4480-a31d-cdfa3aad7344</name>
<uuid>1fd120df-566c-4480-a31d-cdfa3aad7344</uuid>
<target>
<path>/var/lib/libvirt/images</path>
</target>
</pool>

Actual Behaviour

Cloudstack is adding the default local storage if there is no entry in the /etc/cloudstack/agent/agent.properties

Expected Behaviour

Cloudstack should not add the default local storage if there is no entry in the /etc/cloudstack/agent/agent.properties

[harikrishna-patnala]

@kiranchavala This seems to me like as per design, /var/lib/libvirt/images is the default path for local storage.

[harikrishna-patnala]

I saw it again @kiranchavala I think the bug is not adding the default path to the local storage but adding it multiple times. am I right!

[kiranchavala]

@harikrishna-patnala

As discussed we need to fix the following issue

1. Update the documentation

https://docs.cloudstack.apache.org/en/4.20.0.0/adminguide/storage.html#manually-adding-local-storage-pool

Mention that admin user should not remove the local.storage.uuid

2. Fix the issue , add a local storage with a incorrect mount point

2025-01-21 05:12:47,452 DEBUG [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:[]) (logid:1ee3d271) Attempting to create storage pool 4341d700-5211-4a3a-b843-0d829c5dc5f5
2025-01-21 05:12:47,452 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:[]) (logid:1ee3d271) /lcoal does not exists. Check local.storage.path in agent.properties.
2025-01-21 05:12:47,453 WARN  [cloud.agent.Agent] (agentRequest-Handler-3:[]) (logid:1ee3d271) Caught: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: 4341d700-5211-4a3a-b843-0d829c5dc5f5
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:788)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:392)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:386)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
        at com.cloud.agent.Agent.processRequest(Agent.java:686)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

3. When there are multiple local storage uuid, and admin user deletes one local storage from the UI.

Cloudstack should remove the entry in local.storage.uuid and local.storage.path.

The default uuid should not be touched

local.storage.uuid=a43943c1-1759-4073-9db1-bc0ea19203aa,f5b1220b-4446-42dc-a872-cffd281f9f8c
local.storage.path=/var/lib/libvirt/images,/var/lib/libvirt/images2