NAS B&R KVM (ACS 4.20) Restore & Attach Issue

[yyzkevin]

I am able to take backups of instances, and use "Restore Instance backup" to restore an instance while it is shut down.

However, if I use the "Restore and attach backup volume" to attach a backup to another instance to recover files, what happens is it attaches the qcow2 file as raw, which I would expect is not the intended behavior?

Test environment is ACS 4.20, all KVM / NFS.

[rajujith]

@yyzkevin, what is the source primary storage type? Is it NFS?

[yyzkevin]

@yyzkevin, what is the source primary storage type? Is it NFS?

Yes primary & secondary are both NFS.

[yyzkevin]

cloudstack/plugins/backup/nas/src/main/java/org/apache/cloudstack/backup/NASBackupProvider.java

Line 256 in 39c5641

public Pair<Boolean, String> restoreBackedUpVolume(Backup backup, String volumeUuid, String hostIp, String dataStoreUuid, Pair<String, VirtualMachine.State> vmNameAndState) {

I notice that the format does not get set on the new volume, resulting in a null value, leading to the attachment as raw?

[abh1sar]

Hi @yyzkevin,

I’m looking into this and would appreciate a few details to help debug:
• Could you share the exact error message you’re seeing in CloudStack, and any relevant logs from /var/log/cloudstack/agent/agent.log on the KVM host?
• What OS is the instance running?
• You mentioned the restored volume appears in raw format — how did you verify that? I restored a similar volume and it showed as qcow2. Could you run qemu-img info on it to confirm?
• Please also share the output of virsh domblklist <instance_name> (e.g., i-2-3-VM).

Also, if the destination instance was running during restore, there could be hot-plugging issues. Could you try the same with the instance in a stopped state?

Thanks!

[yyzkevin]

I do not receive any error, everything looks as expected the backup is used to create a volume on the primary storage and it is attached to the instance (running or stopped). I initially noticed that the attached disk had no partition table, and when I looked at start of the device (vdb) I could see it was actually a qcow2 label seen from within the guest.

This is what is seen on the hypervisor, which is Ubuntu 24.04.2 LTS.

root@cs-node1:/# virsh qemu-monitor-command i-2-89-VM --hmp 'info block'
libvirt-2-format: /mnt/12490f2a-e712-308b-9479-0bcc3d465d8b/67f72268-35b4-4964-98d3-a5298bd958c8 (qcow2)
    Attached to:      /machine/peripheral/virtio-disk0/virtio-backend
    Cache mode:       writeback

ide0-1-0: [not inserted]
    Attached to:      ide0-1-0
    Removable device: not locked, tray closed

libvirt-5-format: /mnt/12490f2a-e712-308b-9479-0bcc3d465d8b/63423044-52bd-40ef-9d7c-ec1939ceaf04 (raw)
    Attached to:      /machine/peripheral/virtio-disk1/virtio-backend
    Cache mode:       writeback, direct

View from within the guest, see the actual qcow2.

root@SpeedTest:~# sudo dd if=/dev/vdb bs=512 count=1 | hexdump -C
00000000  51 46 49 fb 00 00 00 03  00 00 00 00 00 00 00 00  |QFI.............|
00000010  00 00 00 00 00 00 00 10  00 00 00 00 e0 00 00 00  |................|
00000020  00 00 00 00 00 00 00 07  00 00 00 00 00 03 00 00  |................|
00000030  00 00 00 00 00 01 00 00  00 00 00 01 00 00 00 00  |................|

When I look in the volumes table I see its format is NULL.

+----+---------------------------+--------+--------------------------------------+
| id | name                      | format | path                                 |
+----+---------------------------+--------+--------------------------------------+
| 93 | ROOT-89                   | QCOW2  | 67f72268-35b4-4964-98d3-a5298bd958c8 |
| 94 | RestoredVol-THE GOOD DISK | NULL   | 30a4d04a-d356-4fa7-b6f7-9eeeb1b75cb9 |
| 95 | RestoredVol-THE GOOD DISK | NULL   | 7d8bdde0-3362-48df-a8c9-c9cf21c11d00 |
| 96 | RestoredVol-ROOT-86       | NULL   | 63423044-52bd-40ef-9d7c-ec1939ceaf04 |
+----+---------------------------+--------+--------------------------------------+

[abh1sar]

There is a bug in CloudStack where the disk format isn’t correctly set in the volumes table. As a result, the domain XML generated by CloudStack specifies raw, which causes virsh qemu-monitor-command to report the format as raw, even though the actual image is qcow2 (as confirmed by you and qemu-img info below).

I’ve been able to reproduce this and will raise a PR to fix the incorrect DB entry. Just to confirm—do you see any other issues related to this?

[root@dev1-b-kvm-host1 ~]# virsh qemu-monitor-command i-2-6-VM --hmp 'info block'
libvirt-3-format: /mnt/67bf274c-346c-35ac-badb-afb6857dcb9e/287ede4b-1308-4fde-984b-9b5c3f4f532e (qcow2)
    Attached to:      ide0-0-0
    Cache mode:       writeback, direct
    Backing file:     /mnt/67bf274c-346c-35ac-badb-afb6857dcb9e/a1cf2027-2593-4295-a373-a7dc03f60299 (chain depth: 1)

libvirt-7-format: /mnt/79a4ac66-84c9-36c0-ac98-9b29c9c827e1/4c69665d-19b4-4f57-8788-0cd1cef0af07 (raw)
    Attached to:      /machine/peripheral/virtio-disk2/virtio-backend
    Cache mode:       writeback, direct

[root@dev1-b-kvm-host1 ~]# qemu-img info /mnt/79a4ac66-84c9-36c0-ac98-9b29c9c827e1/4c69665d-19b4-4f57-8788-0cd1cef0af07
image: /mnt/79a4ac66-84c9-36c0-ac98-9b29c9c827e1/4c69665d-19b4-4f57-8788-0cd1cef0af07
file format: qcow2
virtual size: 5 GiB (5368709120 bytes)
disk size: 169 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

[yyzkevin]

@abh1sar I have not seen other issues related to it yet but I will continue testing.

[abh1sar]

I am closing this discussion. @yyzkevin please open more issues if you come across any.