Can bookmarks and holds cause excessive writes?

[filip-paczynski]

Hi, I am using zrepl for backups.
I have noticed that when zrepl is used in a mode which uses bookmarks, excessive write I/O occurs at the scale of 40-60MB/s. This happens while zrepl performs zfs send.

I have read docs regarding bookmarks, and I cannot find the reason as to why making a bookmark and subsequently zfs send-ing it should result in writes.

Questions:

• Why bookmarks cause write I/O?
• Is there any way to keep using bookmarks without excessive write I/O?

I do not think that zrepl does anything other than zfs send/bookmark/hold.

My config:
Pool:

NAME                        STATE     READ WRITE CKSUM
belt                        ONLINE       0     0     0
  mirror-0                  ONLINE       0     0     0
    scsi-35000cca25d54c184  ONLINE       0     0     0
    scsi-35000cca25d551040  ONLINE       0     0     0
  mirror-1                  ONLINE       0     0     0
    scsi-35000cca24310c774  ONLINE       0     0     0
    scsi-35000cca25d54c164  ONLINE       0     0     0
  mirror-2                  ONLINE       0     0     0
    scsi-35000cca25d542350  ONLINE       0     0     0
    scsi-35000cca25d54c130  ONLINE       0     0     0

Zpool props status:

NAME  PROPERTY                       VALUE                          SOURCE
belt  size                           10.9T                          -    
belt  capacity                       29%                            -    
belt  altroot                        -                              default
belt  health                         ONLINE                         -    
belt  guid                           6669544528628890987            -
belt  version                        -                              default
belt  bootfs                         -                              default
belt  delegation                     on                             default
belt  autoreplace                    off                            default
belt  cachefile                      -                              default
belt  failmode                       wait                           default
belt  listsnapshots                  off                            default
belt  autoexpand                     on                             local
belt  dedupratio                     1.00x                          -
belt  free                           7.69T                          -
belt  allocated                      3.19T                          -
belt  readonly                       off                            -
belt  ashift                         0                              default
belt  comment                        -                              default
belt  expandsize                     -                              -
belt  freeing                        0                              -
belt  fragmentation                  42%                            -
belt  leaked                         0                              -
belt  multihost                      off                            default
belt  checkpoint                     -                              -
belt  load_guid                      6630180993196285211            -
belt  autotrim                       off                            default
belt  compatibility                  openzfs-2.1-linux              local
belt  bcloneused                     0                              -
belt  bclonesaved                    0                              -
belt  bcloneratio                    1.00x                          -
belt  feature@async_destroy          enabled                        local
belt  feature@empty_bpobj            active                         local
belt  feature@lz4_compress           active                         local
belt  feature@multi_vdev_crash_dump  enabled                        local
belt  feature@spacemap_histogram     active                         local
belt  feature@enabled_txg            active                         local
belt  feature@hole_birth             active                         local
belt  feature@extensible_dataset     active                         local
belt  feature@embedded_data          active                         local
belt  feature@bookmarks              active                         local
belt  feature@filesystem_limits      enabled                        local
belt  feature@large_blocks           enabled                        local
belt  feature@large_dnode            enabled                        local
belt  feature@sha512                 enabled                        local
belt  feature@skein                  active                         local
belt  feature@edonr                  active                         local
belt  feature@userobj_accounting     active                         local
belt  feature@encryption             enabled                        local
belt  feature@project_quota          active                         local
belt  feature@device_removal         enabled                        local
belt  feature@obsolete_counts        enabled                        local
belt  feature@zpool_checkpoint       enabled                        local
belt  feature@spacemap_v2            active                         local
belt  feature@allocation_classes     enabled                        local
belt  feature@resilver_defer         enabled                        local
belt  feature@bookmark_v2            active                         local
belt  feature@redaction_bookmarks    enabled                        local
belt  feature@redacted_datasets      enabled                        local
belt  feature@bookmark_written       active                         local
belt  feature@log_spacemap           active                         local
belt  feature@livelist               active                         local
belt  feature@device_rebuild         enabled                        local
belt  feature@zstd_compress          active                         local
belt  feature@draid                  disabled                       local
belt  feature@zilsaxattr             disabled                       local
belt  feature@head_errlog            disabled                       local
belt  feature@blake3                 disabled                       local
belt  feature@block_cloning          disabled                       local
belt  feature@vdev_zaps_v2           disabled                       local

OS:

# uname -a
Linux dom0-ispb2b 6.1.0-35-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.137-1 (2025-05-07) x86_64 GNU/Linux
# cat /etc/debian_version 
12.11

ZFS:

# zfs -V
zfs-2.2.7-1~bpo12+1
zfs-kmod-2.2.7-1~bpo12+1

This is a Xen dom0 so (mainly) zvols are used for storage.