Incremental size way bigger than expected after fstrim

[Zoltan-LS]

Version used
0.99

Describe the bug
I am unsure if it is a normal behavior or if this is specific to kvm. The problem that I am seeing is that the incremental backups become 2-3 times the size of the disk after fstrim is executed.

-rw-r--r-- 1 root root 13G Oct 3 01:12 sda.full.data
-rw-r--r-- 1 root root 168M Oct 3 05:15 sda.inc.virtnbdbackup.1.data
-rw-r--r-- 1 root root 131M Oct 3 09:15 sda.inc.virtnbdbackup.2.data
-rw-r--r-- 1 root root 75M Oct 3 13:15 sda.inc.virtnbdbackup.3.data
-rw-r--r-- 1 root root 72M Oct 3 17:15 sda.inc.virtnbdbackup.4.data
-rw-r--r-- 1 root root 72M Oct 3 21:15 sda.inc.virtnbdbackup.5.data
-rw-r--r-- 1 root root 38G Oct 4 01:21 sda.inc.virtnbdbackup.6.data
-rw-r--r-- 1 root root 174M Oct 4 05:21 sda.inc.virtnbdbackup.7.data

Expected behavior
I'd expect/like to see this smaller in size. Is there any work around for this?

Hypervisor information:

• OS: Alma
• HV type plain libvirt

Logfiles:
Dont see any relevant logs to provide.

Workaround:
Work around is to execute the fstrim before the full backup

[abbbi]

The Backup operates on blocklevel and i think fstrim results in changed blocks. So qemu marks these blocks as dirty and they end up in the Backup (as they are Part of the qcow bitmap as in marked „dirty“). I dont know if there is a way to Tell qemu to „discard“ changes done by fstrim. I dont think i can change anything in virtnbdbackup to behave different.

skipping blocks marked as dirty is Not an option.. results in unusable disks After restore.

Maybe libvirt/qemu projects have some documentation on that.

[Zoltan-LS]

Fair enough. I thought that may be the case here, it was just a slight hope that maybe you encountered this and found a fix for such cases.

[abbbi]

I dont have a Solution (Other than timing fstrim with full backups) .. Or check libvirt/qemu docs for options that might help
I see you still use verison 0.99: you should consider updating, the project has been seen various updates since this version.

[abbbi]

other solutions Based on dirty bitmaps have the same „Issue“:

https://forum.proxmox.com/threads/huge-dirty-bitmap-after-sunday.110233/

So id say it works as designed. Maybe running fstrim on the host makes more Sense.

[dlunev]

This is definitely a bug and the problem is in the way how the software is obtaining changed bitmap from the QEMU.

The problem of this software is that it MUST query 2 metacontexts from NBD:

• bitmap
• base allocation

In this case each request for block status will provide 2 arrays of extents - one for CBT and one for the actual block status. The software should request data block for reading only if it is marked as non-zero and changed. Zero changed blocks should not be requested.

Volya.

Changed zeroed blocks are legal. Backup software should handle this correctly.

[dlunev]

They will be represented in the bitmap. That is a requirement. But we should detected them efficiently, avoid reading them from the NBD and store these ranges efficiently. The protocol allows all these things.

[abbbi]

They will be represented in the bitmap. That is a requirement. But we should detected them efficiently, avoid reading them from the NBD and store these ranges efficiently. The protocol allows all these things.

the extend handler that is currently used then has to be enhanced to compare the changed block state of the bitmap against the base allocation and skip if it is zero. I quite understand but i currently dont have the time and/or intention to implement this + the testcase that should cover such stituations in a reproducible manner.

Ive moved this into:

#250

[dlunev]

They will be represented in the bitmap. That is a requirement. But we should detected them efficiently, avoid reading them from the NBD and store these ranges efficiently. The protocol allows all these things.

the extend handler that is currently used then has to be enhanced to compare the changed block state of the bitmap against the base allocation and skip if it is zero. I quite understand but i currently dont have the time and/or intention to implement this + the testcase that should cover such stituations in a reproducible manner.

a little bit more complex. List of extents for block allocation and list of extents for bitmaps could be different, f.e.

• allocation: [0-1G] - zero, present; [1G-2G] - data, present
• dirty: [0-512M] - clean, [512M-2G] - dirty
  though this is not principal.

[abbbi]

the extend handler that is currently used then has to be enhanced to compare the changed block state of the bitmap against the base allocation and skip if it is zero. I quite understand but i currently dont have the time and/or intention to implement this + the testcase that should cover such stituations in a reproducible manner.

a little bit more complex. List of extents for block allocation and list of extents for bitmaps could be different, f.e.

• allocation: [0-1G] - zero, present; [1G-2G] - data, present
• dirty: [0-512M] - clean, [512M-2G] - dirty
  though this is not principal.

this is where things get tricky.
Currently the extent handler is implemented in a way it queries only one specific context. Besides adding the base context during nbd client init, one must also enhance the part of the code that is querying the block status as in:

• store the block statuses for the individual available contexts in different lists (use block_status_filter, probably)
• Unify the extent alignments in the lists
• Compare the lists and adjust the block boundaries, offsets and lengths as mentioned above to only get extents that are actually holding data that needs to be backed up

To get this going first it must made sure we have proper testing environment for automated tests. The current tests only use self constructed qcow2 images for real basic testing (github runners not beeing actually that "thick"). I could imagine a very basic alpine image with additional disk spun up where qemu-agent is contained. Using the qemu agent to issue required commands to get a reproducible test case (create data, delete data, call fstrim).

[slu-010101]

@dlunev

This is definitely a bug and the problem is in the way how the software is obtaining changed bitmap from the QEMU.

The problem of this software is that it MUST query 2 metacontexts from NBD:

* bitmap

* base allocation

do you use this virtnbdbackup version with the new zeros detection on big .qcow2 machines?

We running into a time issue, this need on some machines very long time:
#260 (comment)

I would be interested to know if you have this too?

[abbbi]

See issue #250

[dlunev]

On 3/27/25 17:37, Samuel wrote: @dlunev <https://github.com/dlunev> This is definitely a bug and the problem is in the way how the software is obtaining changed bitmap from the QEMU. The problem of this software is that it MUST query 2 metacontexts from NBD: |* bitmap * base allocation | do you use this virtnbdbackup version with the new zeros detection on big .qcow2 machines? We running into a time issue, this need on some machines very long time: #260 (comment) <#260 (comment)> I would be interested to know if you have this too? — Reply to this email directly, view it on GitHub <#139 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC4HBFPOWYNC6IZP2ASQ67D2WQLKVAVCNFSM6AAAAABXAXYEKWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTENRUGM2TMNQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

that looks strange. The real question is where the time is spent. CPU time should not be a problem. Any verbose logs from the tool?

[slu-010101]

No verbose logs so far, since this is a production system my testing is limited.

[slu-010101]

The two affected machines are both ext4 Debian server who write/move/delete a lot of small files over the day (file server / IMAP server). For some reason that is a performance problem.

[gitthangbaby]

I'm getting such huge incremental backups of ZFS OS daily, and to save space, after an NVME broke with too much writes, i'm running full backup. Checked for fstrim, iostat, snapshot increments on the host, and just don't see where it was coming from.