Unable to add more than 2 volumes or networks to a vm deployed from a UEFI image

[kiranchavala]

problem

Unable to add more than 2 volume or network to a vm deployed from a UEFI image

versions

ACS 4.20.1

The steps to reproduce the bug

1. Register a UEFI image Template

https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

2. Deploy the vm by enabling UEFI secure mode in advanced settings

3. Once the vm is in running state , attach multiple volumes , Exception observed

Logs

2025-07-16 04:28:28,947 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:) Attaching device: <disk  device='disk' type='file'>
<driver name='qemu' type='qcow2' cache='none' />
<source file='/mnt/7b881182-e6d4-3e65-bf10-662bf0530b84/c6ef61d1-300f-442a-86e5-e31f482b64ab'/>
<target dev='vdc' bus='virtio'/>
<serial>c6ef61d1300f442a86e5</serial>
</disk>

2025-07-16 04:28:28,953 WARN  [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:) Failed to attach device to i-2-653-VM: internal error: No more available PCI slots
2025-07-16 04:28:28,953 DEBUG [kvm.storage.KVMStorageProcessor] (AgentRequest-Handler-3:[]) (logid:) Failed to attach volume [id: 394, uuid: c6ef61d1-300f-442a-86e5-e31f482b64ab, name: test2, path: c6ef61d1-300f-442a-86e5-e31f482b64ab], due to org.libvirt.LibvirtException: internal error: No more available PCI slots

4. Try to attach multiple networks, Exception observed

2025-07-16 04:36:02,019 DEBUG [kvm.resource.BridgeVifDriver] (AgentRequest-Handler-1:[]) (logid:) Successfully executed process [2045246] for command [/usr/share/cloudstack-common/scripts/vm/network/vnet/modifyvlan.sh -v 2144 -p eth1 -b breth1-2144 -o add ].
2025-07-16 04:36:02,021 WARN  [resource.wrapper.LibvirtPlugNicCommandWrapper] (AgentRequest-Handler-1:[]) (logid:)  Plug Nic failed due to org.libvirt.LibvirtException: internal error: No more available PCI slots org.libvirt.LibvirtException: internal error: No more available PCI slots
        at org.libvirt.ErrorHandler.processError(Unknown Source)
        at org.libvirt.ErrorHandler.processError(Unknown Source)

Workaround

Change the following settings on the vm after stopping

nicAdapter to e1000 or rtl813

rootDiskController to scsi or sata

https://prnt.sc/9wSS5uX0tVYX

What to do about it?

If the Template is a UEFI enabled , Cloudstack should check if the template is UEFI enabled or not and populated the necessary settings for root disk and nic adapter

yum install libguestfs-tools

virt-inspector --format=raw -a noble-server-cloudimg-amd64.img 2>/dev/null | grep -i 'true'

virt-inspector -a AlmaLinux-8-GenericCloud-UEFI-latest.x86_64.qcow2 2>/dev/null | grep -i 'true'

[weizhouapache]

this may be similar to #7490 (fixed by #9823)

[DaanHoogland]

@kiranchavala can you check this again?

this may be similar to #7490 (fixed by #9823)

that would mean 4.19.2 ?!

[weizhouapache]

@kiranchavala can you check this again?

this may be similar to #7490 (fixed by #9823)

that would mean 4.19.2 ?!

similar, not the same issue 😄

[tcp-dw]

I have tested attachment of multiple disks on a VM in KVM environment and getting the same error on a LAB with ACS v4.20.2.

1. I did a test with UEFI VM with virtio-scsi (OS Type: Other PV virtio-scsi 64bit) with 12 DATA disks (13 disks with ROOT included). It has successfully attached 13 disks in two scsi controllers. During attaching the 13th disk DATA-13 I got the following error:
   Failed to attach volume DATA-13 to VM XXXXXX; org.libvirt.LibvirtException: internal error: No more available PCI slots
2. Changed UEFI VM OS Type to AlmaLinux 8 (legagy virtio-blk driver) and added the 13th DATA disk successfully but by attaching the 14th DATA disk I got the same error as above:
   Failed to attach volume DATA-14 to VM XXXXXX; org.libvirt.LibvirtException: internal error: No more available PCI slots.

[weizhouapache]

I have tested attachment of multiple disks on a VM in KVM environment and getting the same error on a LAB with ACS v4.20.2.

1. I did a test with UEFI VM with virtio-scsi (OS Type: Other PV virtio-scsi 64bit) with 12 DATA disks (13 disks with ROOT included). It has successfully attached 13 disks in two scsi controllers. During attaching the 13th disk DATA-13 I got the following error:
   Failed to attach volume DATA-13 to VM XXXXXX; org.libvirt.LibvirtException: internal error: No more available PCI slots
2. Changed UEFI VM OS Type to AlmaLinux 8 (legagy virtio-blk driver) and added the 13th DATA disk successfully but by attaching the 14th DATA disk I got the same error as above:
   Failed to attach volume DATA-14 to VM XXXXXX; org.libvirt.LibvirtException: internal error: No more available PCI slots.

@tcp-dw
can you share some logs in agent.log in both cases ?

can you also test the steps (1) stop the vm , (2) attach volumes (3) and start the vm ?

[tcp-dw]

Sure, you will find attached Host-agent-and-daemon.log (with respective sections from host's agent.log and daemon.log during the tests reported above). Host is running qemu_v5.2.0/libvirt_v7.0.0.

Today, I did the following actions:

1. I stopped this VM changed OS-type back to "Other PV virtio-scsi" and added more disks until the maximum of allowed data disks of 32 (33 total with Root including). Error message from UI was: "Failed to attach volume. The specified VM already has the maximum number of data disks (32) attached. Please specify another VM.".

2. I started this VM from UI and curiously this VM with 33 disks has been successfully started! Then stopped it, changed OS-type to "AlmaLinux 8" and started again successfully. In both OS-types OS can see 33 block devices in total (sd* in vistio-scsi and vd* in virtio-block mode).

3. Then while it is running I removed most DATA disks and left 12 DATA disks (13 total). Stopped VM, changed back to virtio-scsi and started VM. Then tried again adding data disks while running. Added DATA-13, DATA-14, ...,DATA-19 but during adding DATA-20 I got again the error "Failed to attach volume. Failed to attach volume DATA-20 to VM ACS-LAB-NET-8-ROUTED-VPC; org.libvirt.LibvirtException: internal error: No more available PCI slots". I have attached logs from both management server (ACS-CreateAndAttach-DATA-20.log) and host (Host-agent-and-daemon-2ndtest.log) for this last action.

I hope that it is useful.

Host-agent-and-daemon.log

ACS-CreateAndAttach-DATA-20.log

Host-agent-and-daemon-2ndtest.log