Description
Live backups of VMs using a Ceph RBD datastore execute the complete backup preparation script on the host selected from the backup datastore's BRIDGE_LIST (LAST_BRIDGE).
When the VM is running on a different host, filesystem quiescing operations such as virsh domfsfreeze, domfsthaw, suspend, and resume are executed on the bridge instead of the VM hypervisor. Since the libvirt domain does not exist on the bridge, the guest filesystem is not frozen.
Additionally, failures from domfsfreeze and suspend can be masked by their use inside an && list followed by a successful trap command. The backup may therefore complete successfully without filesystem quiescing.
To Reproduce
- Configure a Ceph RBD datastore for the VM disks.
- Configure a Restic backup datastore with a
BRIDGE_LIST.
- Run a VM on a KVM host that is not the selected backup bridge.
- Install and enable QEMU Guest Agent in the VM.
- Optionally configure a QEMU Guest Agent fsfreeze hook to log freeze and thaw operations.
- Create a backup job with
FS_FREEZE = "AGENT".
- Execute a full or incremental live backup.
- Verify that the backup completes, but the QEMU Guest Agent hook is not called.
- Check the backup logs and observe that
virsh domfsfreeze is executed on LAST_BRIDGE, where the domain does not exist.
Expected behavior
Filesystem quiescing operations must be executed on the hypervisor where the VM is running.
The backup must not create a Ceph snapshot until the guest filesystem has been successfully frozen. If freeze, thaw, suspend, or resume fails, the error must be propagated and the backup must not be reported as successful.
The interaction between the VM hypervisor and BRIDGE_LIST should be explicitly defined. Possible approaches include running the complete live backup on the VM hypervisor or coordinating freeze/thaw on the hypervisor while keeping export and upload operations on the selected bridge.
Details
- Affected Component: Storage
- Hypervisor: KVM
- Version: 7.0.1, 7.4.0, master
Additional context
The Ceph prebackup_live script selects its execution host as follows:
rhost = LAST_BRIDGE || vm_hypervisor
It then runs the complete generated backup script on rhost. That script includes both Ceph operations and local libvirt operations.
Ceph RBD commands work from the bridge because the storage is shared, but virsh domfsfreeze only works on the hypervisor managing the domain.
The freeze command is currently generated in a form similar to:
virsh domfsfreeze <domain> && export FROZEN="TRUE"
trap '...' EXIT
Because the failing command is part of an && list, Bash set -e does not terminate the script. The following trap succeeds, masking the failure and allowing an unquiesced backup to continue.
Progress Status
Description
Live backups of VMs using a Ceph RBD datastore execute the complete backup preparation script on the host selected from the backup datastore's
BRIDGE_LIST(LAST_BRIDGE).When the VM is running on a different host, filesystem quiescing operations such as
virsh domfsfreeze,domfsthaw,suspend, andresumeare executed on the bridge instead of the VM hypervisor. Since the libvirt domain does not exist on the bridge, the guest filesystem is not frozen.Additionally, failures from
domfsfreezeandsuspendcan be masked by their use inside an&&list followed by a successfultrapcommand. The backup may therefore complete successfully without filesystem quiescing.To Reproduce
BRIDGE_LIST.FS_FREEZE = "AGENT".virsh domfsfreezeis executed onLAST_BRIDGE, where the domain does not exist.Expected behavior
Filesystem quiescing operations must be executed on the hypervisor where the VM is running.
The backup must not create a Ceph snapshot until the guest filesystem has been successfully frozen. If freeze, thaw, suspend, or resume fails, the error must be propagated and the backup must not be reported as successful.
The interaction between the VM hypervisor and
BRIDGE_LISTshould be explicitly defined. Possible approaches include running the complete live backup on the VM hypervisor or coordinating freeze/thaw on the hypervisor while keeping export and upload operations on the selected bridge.Details
Additional context
The Ceph
prebackup_livescript selects its execution host as follows:It then runs the complete generated backup script on rhost. That script includes both Ceph operations and local libvirt operations.
Ceph RBD commands work from the bridge because the storage is shared, but virsh domfsfreeze only works on the hypervisor managing the domain.
The freeze command is currently generated in a form similar to:
Because the failing command is part of an && list, Bash set -e does not terminate the script. The following trap succeeds, masking the failure and allowing an unquiesced backup to continue.
Progress Status