Understanding space accounting and recordsize

[frukto]

I have 8 disks raidz2 pool with ashift=13. I created a bunch of datasets with different recordsize and did write 64M of data parted in files of various sizes:

• 65536 1K files
• 32768 2K files
• ...
• 2 32M files

I checked the used property of the file system and captured the values in the table below.

For any combination of recordsize and the patterns of number_of_files and file_size the test was:

zfs create pool/fs -o recordsize=${recordsize} -o mountpoint=/mnt/fs
for n  in $(seq 1 ${number_of_files}) ; do
    dd if=/dev/urandom of=/mnt/fs/$n bs=${file_size} count=1
done
# store the value of: zfs get used pool/fs
zfs destroy pool/fs

I am having trouble to make sense of the values. Why is there such a high overhead for writing (small) files, in particular for small recordsize?

recordsize	65536@1K	32768@2K	16384@4K	8192@8K	4096@16K	2048@32K	1024@64K	512@128K
4K	1103.85M	552.27M	276.44M	522.23M	389.30M	322.68M	289.21M	272.76M
8K	1103.79M	552.30M	276.44M	138.55M	261.49M	194.84M	161.40M	144.95M
16K	1103.82M	552.30M	276.35M	138.58M	133.61M	194.93M	161.68M	144.88M
32K	1103.82M	552.24M	276.41M	138.52M	133.55M	67.02M	97.49M	81.10M
64K	1103.85M	552.36M	276.54M	138.61M	133.52M	67.09M	65.46M	81.04M
128K	1103.88M	552.27M	276.44M	138.58M	133.55M	67.02M	65.46M	64.93M
256K	1103.82M	552.24M	276.47M	138.58M	133.61M	67.21M	65.59M	64.96M

recordsize	256@256K	128@512K	64@1M	32@2M	16@4M	8@8M	4@16M	2@32M
4K	264.49M	260.37M	260.30M	258.37M	257.74M	257.96M	257.81M	257.71M
8K	136.86M	132.52M	130.40M	130.40M	129.40M	129.12M	129.21M	129.09M
16K	136.64M	132.71M	130.43M	129.46M	129.34M	128.87M	128.68M	128.71M
32K	72.74M	68.58M	66.52M	65.56M	64.96M	64.93M	64.65M	64.53M
64K	72.74M	68.58M	66.52M	65.49M	64.93M	64.68M	64.65M	64.46M
128K	72.67M	68.58M	66.49M	65.49M	64.93M	64.68M	64.49M	64.46M
256K	60.87M	64.62M	62.50M	61.43M	60.97M	60.72M	60.50M	60.40M

[gmelikov]

• you can't write a block in less than 2^ashift bytes sector size (embedded_block feature aside), sector size is a minimal block size to write
• in raidz you have to have a padding block per every block, which can't be less than minimal sector size too

Useful table https://docs.google.com/spreadsheets/d/1tf4qx1aMJp8Lo_R6gpT689wTjHv6CGVElrPqTA0w_ZY/edit?pli=1#gid=2126998674

[gmelikov]

I've created the issue to document it better openzfs/openzfs-docs#263

[rincebrain]

Ah, you're looking at the value of zfs get used, I overlooked that in your initial post. You may find it more informative to look at zpool get alloc,free before and after, as well, and compare the two, probably with -p to get exact values.

[JBigglesworth]

I'm coming in here somewhat naively and I'd welcome any correction if my following input is wrong or misguided.

My crude understanding is that you're talking in term of numbers of sectors but, even if you're only writing just one byte of data you're incurring a write of an 8KB data block (ashift=13) plus one 8KB block of parity (RaidiZ1) before you go anywhere. (ie: 1 byte data => 16KB disk space).

It leads me to think that block size becomes problematic for RAIDZx as target file size reduces and also that recordsize is immaterial.

Putting aside redundacy and write/read performance concerns, wouldn't this mean that for small files sizes, where block size seems to be more significant that 2 way mirrors are more space efficient than RAIDZ for small file sizes?

[rincebrain]

raidz and small records are not the ideal mix, which is one reason you can do things like shove {metadata, small records} onto a {mirrored} special vdev.

[JBigglesworth]

@rincebrain
Thanks for mentioning special vdevs, I'd heard of them but haven't experimented with them yet. They seem a much better alternative than partioning data via multiple datasets to optimise data storage!

[frukto]

@JBigglesworth

a write of an 8KB data block (ashift=13) plus one 8KB block of parity (RaidiZ1) before you go anywhere. (ie: 1 byte data => 16KB disk space).

Exactly, even worse with raidz2, that would be 24K. That is how I am understanding this as well.

Maybe the confusion is just about zfs get used vs zpool get alloc? Comparing values of the latter, I do see roughly 1.5G used for 65536 1K files. That is sound. But zfs get used shows me 1103.85M.

My understanding of the man page is that allocated does include everything (parity, padding, metadata,...) but what about used then?

Reading about "raidz deflation" in lib/libzfs/libzfs_dataset.c#L5285:

In an 8 disk wide raidz2 with ashift=13 a 128K block would be written as 192K = 24 x 8K = 16 x 8K data + 6 x 8K parity + 2 x 8K skip blocks, so raidz_deflation = 128K / 192K = 2/3. Is used just scaled by this factor. So the real 1.5G are accounted as 2/3 * ~1.5G = 1103.85M? That would at least roughly match.

Also everything above does not consider any metadata, so probably its even more complicated.