ZFS Expansion, second round didn't add any new storage?

[bobsdacool]

Hey all.

Hoping someone can help me out here. I know that there are inconsistencies with volume measures when using the attach command, however I seem to be missing an entire disks worth of data! I'd really appreciate any help on this.

I know that zfs, df, ls -s etc... will give inconsistent values after the expansion, I just am not confident that it should be off by 10Tb! as discussed below.

Since zfs 2.3.1 just released on debian backports I thought I'd have a good whirl at an expansion.

My previous array was 4x12Tb drives in a raidz2 configuration.

I've added two drives using the zpool attach command, both ran successfully, 0B repaired after each scrub.

However.

I'm at a loss as to the usable space, I'm getting some strange numbers, and missing storage space.

With my previous setup I had about 20Tb available, as I'd expect, however, having added two drives, the total space is now only 32Tb... I was expecting something in the range of 40+ because I'm adding two 12Tb (10.9) drives.

zpool list -v seems correct; however df -h, and zfs list show about 10Tb less than I expected to have been added. I'm not sure why, and it persists after a reboot.

I was expecting a %used value of approximately 30-40%, however its sitting at 50%.

do I need to rebalance the drives, would this account for the missing 10Tb?

Here is the outputs of some relevant commands.

@Home:~$ zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ars 65.5T 32.6T 32.9T - - 6% 49% 1.00x ONLINE -
raidz2-0 65.5T 32.6T 32.9T - - 6% 49.7% - ONLINE
wwn-0x5000c500b5bc158e 10.9T - - - - - - - ONLINE
wwn-0x5000c500b608d18b 10.9T - - - - - - - ONLINE
wwn-0x5000c500b676f422 10.9T - - - - - - - ONLINE
wwn-0x5000c500c3a8fb2e 10.9T - - - - - - - ONLINE
wwn-0x5000c500b33aa957 10.9T - - - - - - - ONLINE
wwn-0x5000c500b66af1eb 10.9T - - - - - - - ONLINE

@Home:~$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
ars 15.8T 15.8T 15.7T /ars

@Home:~$ zpool status
pool: ars
state: ONLINE
scan: scrub repaired 0B in 09:15:01 with 0 errors on Sat Mar 29 18:16:04 2025
expand: expanded raidz2-0 copied 32.6T in 21:59:38, on Sat Mar 29 09:01:03 2025
config:

    NAME                        STATE     READ WRITE CKSUM
    ars                         ONLINE       0     0     0
      raidz2-0                  ONLINE       0     0     0
        wwn-0x5000c500b5bc158e  ONLINE       0     0     0
        wwn-0x5000c500b608d18b  ONLINE       0     0     0
        wwn-0x5000c500b676f422  ONLINE       0     0     0
        wwn-0x5000c500c3a8fb2e  ONLINE       0     0     0
        wwn-0x5000c500b33aa957  ONLINE       0     0     0
        wwn-0x5000c500b66af1eb  ONLINE       0     0     0

errors: No known data errors

not sure where the missing storage is, or where I can go to from here.

Any help is much appreciated.

Cheers.

[IvanVolosyuk]

See #15232 (comment)