[CloudStack 4.20.1.0]Extra lsilogic SCSI controller on aarch64 causes last disk unrecognized

[gw769]

problem

Environment

• CloudStack Version: 4.20.1.0
• CPU Architecture: aarch64
• Hypervisor: KVM
• UEFI Configuration (relevant for aarch64 VM boot, from /etc/cloudstack/agent/uefi.properties):

# CloudStack Agent UEFI Configuration for ARM64
# This file configures UEFI boot support for ARM64 virtual machines

# Secure boot mode with Microsoft keys
guest.nvram.template.secure=/usr/share/AAVMF/AAVMF_VARS.ms.fd
guest.loader.secure=/usr/share/AAVMF/AAVMF_CODE.ms.fd

# Secure boot mode without Microsoft keys
guest.nvram.template.secboot=/usr/share/AAVMF/AAVMF_VARS.fd
guest.loader.secboot=/usr/share/AAVMF/AAVMF_CODE.secboot.fd

# Standard UEFI mode (default)
guest.nvram.template.legacy=/usr/share/AAVMF/AAVMF_VARS.fd
guest.loader.legacy=/usr/share/AAVMF/AAVMF_CODE.fd

# No secure boot mode (explicitly disabled)
guest.nvram.template.nosecboot=/usr/share/AAVMF/AAVMF_VARS.fd
guest.loader.nosecboot=/usr/share/AAVMF/AAVMF_CODE.no-secboot.fd

# NVRAM storage path
guest.nvram.path=/var/lib/libvirt/qemu/nvram/

Problem Description

When attaching multiple disks to an aarch64 VM, CloudStack incorrectly adds an extra lsilogic SCSI controller after allocating virtio-scsi controllers for every 7 disks. This causes the last disk to be unrecognized after VM reboot.

Key Evidence: The second SCSI controller switches from virtio-scsi to lsilogic after reboot.

Reproduction Steps

1. Deploy an aarch64 VM with 5 disks → all recognized correctly under virtio-scsi controller
2. Hot-add a 6th disk while VM running → temporarily works, creates second virtio-scsi controller
3. Shutdown and restart the VM → second controller becomes lsilogic, 6th disk disappears

Expected Behavior

All disks assigned to virtio-scsi controllers, no lsilogic controller added.

Actual Behavior (with Evidence)

After reboot - WRONG
<controller type='scsi' index='1' model='lsilogic'>
Should be:  <controller type='scsi' index='1' model='virtio-scsi'>

Related Code

PR #9823 (bug at the end of implementation)

Attachments

Step 1: Initial State (5 disks attached)
root@NODE159:/var/log/cloudstack/agent# virsh dumpxml i-2-971-VM  | grep controller
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='4'/>
      <address type='drive' controller='0' bus='0' target='0' unit='5'/>
      <address type='drive' controller='0' bus='0' target='0' unit='6'/>
    <controller type='usb' index='0' model='qemu-xhci'>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='virtio-serial' index='0'>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='7' model='pcie-to-pci-bridge'>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
    </controller>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>

Step 2: Hot-add 6th disk while VM running,witch device id is 7
root@NODE159:/var/log/cloudstack/agent# virsh dumpxml i-2-971-VM  | grep controller
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='4'/>
      <address type='drive' controller='0' bus='0' target='0' unit='5'/>
      <address type='drive' controller='0' bus='0' target='0' unit='6'/>
      <address type='drive' controller='1' bus='0' target='0' unit='0'/>
    <controller type='usb' index='0' model='qemu-xhci'>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
    </controller>
    <controller type='scsi' index='1' model='virtio-scsi'>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='virtio-serial' index='0'>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='7' model='pcie-to-pci-bridge'>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
    </controller>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
Step 3: Reboot VM
root@NODE159:/var/log/cloudstack/agent# virsh dumpxml i-2-971-VM  | grep controller
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='4'/>
      <address type='drive' controller='0' bus='0' target='0' unit='5'/>
      <address type='drive' controller='0' bus='0' target='0' unit='6'/>
      <address type='drive' controller='1' bus='0' target='0' unit='0'/>
    <controller type='usb' index='0' model='qemu-xhci'>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='scsi' index='1' model='lsilogic'>    <!-- BUG: Should be virtio-scsi -->
    </controller>
    <controller type='virtio-serial' index='0'>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='7' model='pcie-to-pci-bridge'>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
    </controller>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
root@NODE159:/var/log/cloudstack/agent# 

versions

• CloudStack Version: 4.20.1.0
• CPU Architecture: aarch64
• Hypervisor: KVM

The steps to reproduce the bug

1.Initial State 5 disks attached
2.Hot-add 6th disk while VM running
3.reboot VM
...

What to do about it?

No response

[gw769]

Hi all,

I've found a temporary workaround for this issue (see #12090) using CloudStack's libvirt VM XML transformation hook. Here's the properly formatted details:

Workaround Approach

Use the libvirt-vm-xml-transformer.groovy hook to force-inject 4 persistent virtio-scsi controllers (index=1~4) into the VM's XML. This prevents the unwanted lsilogic controller from being added after reboot.

Steps to Apply

1. Create/Update the Hook File
   Create/overwrite the hook at:

   /etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy

2. Add the Workaround Script
   Paste this code (properly wrapped in a Groovy code block):

   import org.apache.logging.log4j.Logger
   
   class VmWwnHook {
       /**
        * Transform XML by injecting required virtio-scsi controllers
        * @param log Logger instance for logging
        * @param xml Original XML string to process
        * @return Modified XML (or original on exception)
        */
       public String transform(Logger log, String xml) {
           try {
               log.info("[VM-WWN-HOOK] Starting: Inject virtio-scsi controllers")
               log.debug("[VM-WWN-HOOK] Original XML:\n${xml}")  
               String modifiedXml = xml
               modifiedXml = injectVirtioScsiControllers(log, modifiedXml)
   
               log.debug("[VM-WWN-HOOK] Modified XML:\n${modifiedXml}")
               return modifiedXml
           } catch (Exception e) {
               log.error("[VM-WWN-HOOK] Execution error: ${e.getMessage()}", e)
               return xml
           }
       }
   
       /**
        * Force inject 4 virtio-scsi controllers (index=1~4), skip if exists
        * @param log Logger instance
        * @param xml Original XML string
        * @return XML with virtio-scsi controllers injected
        */
       private String injectVirtioScsiControllers(Logger log, String xml) {
           try {
               log.info("[VM-WWN-HOOK] Injecting virtio-scsi controllers (index=1~4)")
               String modifiedXml = xml
               String controllersToInject = ""
   
               // Check & inject controllers 1-4
               for (int i = 1; i <= 4; i++) {
                   String checkStr = "controller type='scsi' index='${i}'"
                   if (!modifiedXml.contains(checkStr)) {
                       controllersToInject += """
   <controller type='scsi' index='${i}' model='virtio-scsi'>
     <driver queues='4'/>
   </controller>"""
                       log.info("[VM-WWN-HOOK] Injecting missing controller (index=${i})")
                   } else {
                       log.info("[VM-WWN-HOOK] Controller (index=${i}) exists - skipping")
                   }
               }
   
               // Inject into XML
               if (!controllersToInject.isEmpty()) {
                   def existingScsiMatcher = modifiedXml =~ /(?s)(<controller type='scsi'.*?<\/controller>)/
                   if (existingScsiMatcher.find()) {
                       String lastController = existingScsiMatcher[existingScsiMatcher.count - 1][1]
                       modifiedXml = modifiedXml.replace(lastController, lastController + "\n" + controllersToInject)
                       log.info("[VM-WWN-HOOK] Injected after existing controllers")
                   } else {
                       modifiedXml = modifiedXml.replace("<devices>", "<devices>\n" + controllersToInject)
                       log.info("[VM-WWN-HOOK] Injected into <devices> tag")
                   }
               } else {
                   log.info("[VM-WWN-HOOK] All controllers exist - no injection needed")
               }
               return modifiedXml
           } catch (Exception e) {
               log.error("[VM-WWN-HOOK] Injection error: ${e.getMessage()}", e)
               return xml
           }
       }
   }
   
   def vmWwnHook = new VmWwnHook()

Verification

After applying:

• Reboot the VM, then run virsh dumpxml <VM_NAME> | grep controller
• All SCSI controllers will show model='virtio-scsi' (no more lsilogic), and all disks remain recognized.

root@NODE159:/etc/cloudstack/agent/hooks# virsh dumpxml i-2-971-VM | grep controll
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='4'/>
      <address type='drive' controller='0' bus='0' target='0' unit='5'/>
      <address type='drive' controller='0' bus='0' target='0' unit='6'/>
      <address type='drive' controller='1' bus='0' target='0' unit='0'/>
    <controller type='usb' index='0' model='qemu-xhci'>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
    </controller>
    <controller type='scsi' index='1' model='virtio-scsi'>
    </controller>
    <controller type='scsi' index='2' model='virtio-scsi'>
    </controller>
    <controller type='scsi' index='3' model='virtio-scsi'>
    </controller>
    <controller type='scsi' index='4' model='virtio-scsi'>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='virtio-serial' index='0'>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='10' model='pcie-to-pci-bridge'>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
    </controller>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>