special_small_blocks not working as intended

[FFourtyTwo]

Hello! I have a strange behavior using special device and special_small_blocks property on my storage pool.

OS: Debian Bullseye (Proxmox 7.3-3)
ZFS: 2.1.6

zpool status dpool:

dpool
	mirror-0
		wwn-0x5000cca223e7bcba
		wwn-0x5000cca223e7a52e
	mirror-3
		wwn-0x5000cca223e77079
		wwn-0x5000cca224e781e4
special
	mirror-1
		nvme-eui.0025385801ae6fcd-part4
		nvme-eui.002538592190c400-part4
logs
	mirror-2
		nvme-eui.0025385801ae6fcd-part5
		nvme-eui.002538592190c400-part5

zfs get special_small_blocks:

NAME                           PROPERTY              VALUE                 SOURCE
dpool                          special_small_blocks  64K                   local
dpool/data                     special_small_blocks  64K                   inherited from dpool
dpool/store                    special_small_blocks  64K                   inherited from dpool

zpool list -v:

NAME                                  SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
special                                  -      -      -        -         -      -      -      -  -
  mirror-1                            149G  17.8G   131G        -         -     0%  11.9%      -    ONLINE
    nvme-eui.0025385801ae6fcd-part4   150G      -      -        -         -      -      -      -    ONLINE
    nvme-eui.002538592190c400-part4   150G      -      -        -         -      -      -      -    ONLINE

zdb -Lbbb dpool:

Block Size Histogram
  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    16K:   201K  4.57G  6.02G  36.2K   688M   954M   202K  4.58G  6.18G
    32K:   265K  10.8G  16.8G  9.94K   449M  1.37G   266K  10.8G  17.0G
    64K:   231K  21.4G  38.2G  8.99K   834M  2.18G   231K  21.4G  38.3G
   128K:  8.66M  1.08T  1.12T  9.52M  1.19T  1.19T  8.66M  1.08T  1.12T

Docs says:
Blocks smaller than or equal to this value will be assigned to the special allocation class

As you can see i certanly have 17.8G (32K blocks+metadata) allocated to special device mirror, but according to histogram there must be at least 38.3G+metadata allocated. This pool was freshly created with special_small_blocks property set to 64K. After this i run send | recv snapshot from old pool and there was no special device in old pool so property was not set.

To be sure that old pool props will not affect new pool i also overrided them with "-o" option, full command to move data was zfs send -R odpool@clone | zfs recv -Fv -o special_small_blocks=64K -o xattr=sa dpool.

What i did wrong? Or it's a bug?
Thank you!

[rincebrain]

My immediate thought would be to ask what zfs get type,recordsize dpool/data dpool/store and what, in full, zfs version says.

The former because zvol data blocks are ineligible for landing on special vdevs ever, and the latter because until a fix went in a while ago, special_small_blocks was, IIRC, doing < recordsize not <= recordsize, though that should be well before 2.1.6.

[rincebrain]

Well, yeah, changing recordsize on the destination isn't going to rewrite things from your send stream.

[FFourtyTwo]

Do you think this information is enougth to open issue?

[rincebrain]

No?

It seems to work as expected when I test it quickly:

$ sudo zpool create mypool /poolfile1 special /poolfile2
$ sudo zfs create mypool/source1
$ sudo dd if=/dev/urandom of=/mypool/source1/file1 bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.15092 s, 445 MB/s
$ sudo zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
mypool          39G  64.2M  38.9G        -         -     0%     0%  1.00x    ONLINE  -
  /poolfile1    20G    64M  19.4G        -         -     0%  0.32%      -    ONLINE
special           -      -      -        -         -      -      -      -         -
  /poolfile2    20G   250K  19.5G        -         -     0%  0.00%      -    ONLINE
$ sudo zfs snapshot mypool/source1@testme
$ sudo zfs send mypool/source1@testme | sudo zfs recv -o special_small_blocks=128k mypool/dest1
$ sudo zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
mypool          39G   128M  38.9G        -         -     0%     0%  1.00x    ONLINE  -
  /poolfile1    20G    64M  19.4G        -         -     0%  0.32%      -    ONLINE
special           -      -      -        -         -      -      -      -         -
  /poolfile2    20G  64.4M  19.4G        -         -     0%  0.32%      -    ONLINE
$ sudo zfs create mypool/source2
$ sudo zfs set recordsize=64k mypool/source2
$ sudo dd if=/dev/urandom of=/mypool/source2/file1 bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 0.147555 s, 4
55 MB/s
$ sudo zfs snapshot mypool/source2@testme
$ sudo zfs send mypool/source2@testme | sudo zfs recv -o special_small_blocks=128k mypool/dest2
$ sudo zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
mypool          39G   257M  38.7G        -         -     0%     0%  1.00x    ONLINE  -
  /poolfile1    20G   128M  19.4G        -         -     0%  0.64%      -    ONLINE
special           -      -      -        -         -      -      -      -         -
  /poolfile2    20G   129M  19.4G        -         -     0%  0.64%      -    ONLINE
$ sudo zfs send mypool/source2@testme | sudo zfs recv -o special_small_blocks=64k mypool/dest3
$ sudo zpool list -v
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
mypool          39G   321M  38.7G        -         -     0%     0%  1.00x    ONLINE  -
  /poolfile1    20G   128M  19.4G        -         -     0%  0.64%      -    ONLINE
special           -      -      -        -         -      -      -      -         -
  /poolfile2    20G   193M  19.3G        -         -     0%  0.96%      -    ONLINE

Can you try repeating what I did there and confirm it behaves that way for you as well, or if you're seeing something different happen?

[FFourtyTwo]

I tried what you suggested and i see the same results as you, blocks successfully allocated to special device. I tried to make the test with more data with different block sizes:

root@ftbmpve:~# zpool destroy ssbsource
root@ftbmpve:~# zpool destroy ssbdest
root@ftbmpve:~# zpool create ssbsource /dev/nvme0n1p6
root@ftbmpve:~# zpool create ssbdest /dev/nvme1n1p6 special /dev/nvme1n1p7
root@ftbmpve:~# for i in {1..10000}; do dd if=/dev/urandom of=/ssbsource/file16K$i bs=16K count=1; done >/dev/null 2>&1
root@ftbmpve:~# for i in {1..10000}; do dd if=/dev/urandom of=/ssbsource/file32K$i bs=32K count=1; done >/dev/null 2>&1
root@ftbmpve:~# for i in {1..10000}; do dd if=/dev/urandom of=/ssbsource/file64K$i bs=64K count=1; done >/dev/null 2>&1
root@ftbmpve:~# for i in {1..10000}; do dd if=/dev/urandom of=/ssbsource/file128K$i bs=128K count=1; done >/dev/null 2>&1
root@ftbmpve:~# zdb -bbb ssbsource
Block Size Histogram
  block   psize                lsize                asize
   size   Count   Size   Cum.  Count   Size   Cum.  Count   Size   Cum.
    16K:  9.77K   156M   160M  11.3K   180M   180M  9.77K   156M   165M
    32K:  9.77K   313M   473M  9.77K   312M   493M  9.77K   312M   477M
    64K:  9.77K   625M  1.07G  9.77K   625M  1.09G  9.77K   625M  1.08G
   128K:  9.77K  1.22G  2.29G  9.79K  1.22G  2.32G  9.77K  1.22G  2.30G
   256K:      0      0  2.29G      0      0  2.32G      0      0  2.30G
root@ftbmpve:~# zfs snap ssbsource@clone
root@ftbmpve:~# zfs send -R ssbsource@clone | zfs recv -Fv -o special_small_blocks=64K ssbdest
root@ftbmpve:~# zpool list -v ssbdest
NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ssbdest        19G  2.30G  16.7G        -         -     0%    12%  1.00x    ONLINE  -
  nvme1n1p6    10G  1.22G  8.28G        -         -     0%  12.8%      -    ONLINE
special          -      -      -        -         -      -      -      -  -
  nvme1n1p7    10G  1.08G  8.42G        -         -     0%  11.3%      -    ONLINE

And it also works as expected but i totally don't understand why it didn't do the same on my main pool, and with real files from previous tries. Anyway much thanks for help, i think need to go learn something more about blocks allocation. Thank you!

[rincebrain]

My suggestion, looking over your outputs again, would be to assume a couple of MB of data is not enough to show something put anything on the special device, really, other than metadata.

Also that zpool list -v may take a few seconds after zfs receiving to finish writing whatever it's doing, so it may not show all space allocations or frees yet.

(My last remark would be, while you're correct that zdb won't show histograms smaller than the whole pool, it will report on the recordsizes of individual records in files, so you could use that to see whether the file you're shuffling around is the recordsize you expect, and if it ended up on the special vdev or not, directly...)