Poor streaming performance for any recordsize > 1M

[dbavatar]

Configuration is raidz2 with 8 hard drives, ZFS 2.1.2 on Ubuntu 22.04/Linux 5.15, new empty pool with encryption.

Expectation: reading/writing to array with 1MB recordsize produces 1MB/6~=170kB chunks on disk. Using a recordsize of 16MB produces 16MB/6~=2730kB chunks on disk. Hard drives consistently perform better with larger size reads, up to several MB contiguous, so every increase above 1MB recordsize should produce faster throughput, approaching 6x spindle rate, or ~1300MBps, assuming enough cpu/controller bandwidth.

Actual result: Any increase over 1MB slows down streaming reads, with 1MB being the fastest size, and 16MB being the slowest, and 4MB having an odd bump, but still not as fast as 1MB

Theory: streaming readahead is non-functional when going above 1MB, a 16MB recordsize involves the disks idling while userspace reads that last record out of arc, before being able to issue more physical reads. Low disk utilization with large recordsize seen. Although the read speed out of arc would seem to work against that theory, it's quite fast. Don't know where all the idle time is coming from.

Examples:

dd if=1MB of=/dev/null bs=$((16*1024*1024)) count=300
5033164800 bytes (5.0 GB, 4.7 GiB) copied, 5.71819 s, 880 MB/s

dd if=2MB of=/dev/null bs=$((16*1024*1024)) count=300
5033164800 bytes (5.0 GB, 4.7 GiB) copied, 9.05627 s, 556 MB/s

dd if=4MB of=/dev/null bs=$((16*1024*1024)) count=300
5033164800 bytes (5.0 GB, 4.7 GiB) copied, 6.83308 s, 737 MB/s

dd if=8MB of=/dev/null bs=$((16*1024*1024)) count=300
5033164800 bytes (5.0 GB, 4.7 GiB) copied, 7.93915 s, 634 MB/s

dd if=16MB of=/dev/null bs=$((16*1024*1024)) count=300
5033164800 bytes (5.0 GB, 4.7 GiB) copied, 12.8435 s, 392 MB/s

Raw disk performance at top of drive:

dd if=/dev/sde of=/dev/null bs=$((16*1024*1024)) count=300
5033164800 bytes (5.0 GB, 4.7 GiB) copied, 22.6404 s, 222 MB/s

Low drive utilization during 16MB reads, mostly in the 20%'s:

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb             50.70     45.07     0.00   0.00   17.06   910.23    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.86  46.08
sdc             50.40     44.80     0.00   0.00    8.95   910.20    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.45  26.60
sdd             50.70     45.07     0.00   0.00    8.96   910.22    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.45  27.24
sde             50.60     44.97     0.00   0.00    8.46   910.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.43  25.36
sdf             50.90     45.23     0.00   0.00    9.09   910.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.46  26.80
sdg             50.00     44.43     0.00   0.00    8.93   909.98    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.45  26.28
sdh             49.50     44.00     0.00   0.00    8.92   910.23    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.44  26.08
sdj             50.10     44.53     0.00   0.00    8.89   910.23    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.45  26.20

arc cache throughput test (100% pre-cached):

dd if=16MB of=/dev/null bs=$((16*1024*1024))  count=100
1677721600 bytes (1.7 GB, 1.6 GiB) copied, 0.305357 s, 5.5 GB/s

scrub speed of array is ~1000MBps:

        421G scanned at 1.04G/s, 398G issued at 1000M/s, 421G total
        0B repaired, 94.34% done, 00:00:24 to go

Knobs tried:
zfs_vnops_read_chunk_size
zfs_vdev_aggregation_limit
zfs_vdev_async_read_max_active
zfs_vdev_async_read_min_active
zfs_vdev_cache_bshift
zfs_vdev_cache_max
zfs_vdev_cache_size
zfs_vdev_max_active
zfs_vdev_read_gap_limit
zfs_vdev_sync_read_max_active
zfs_vdev_sync_read_min_active
dmu_prefetch_max
zio_dva_throttle_enabled

io schedulers: mq-deadline,non,bfq,kyber

What's next?

[rincebrain]

How's it behave without encryption, and what CPU? Encryption can be a significant bottleneck even if you have various hardware support for it.

zfetch_array_rd_sz is the tunable for "do not prefetch above this size", and it defaults to 1M, along with various other things starting with zfetch.

That tunable and its friends at least date back to 2.0.X, so you should have them.

[dbavatar]

Behaves similarly without encryption, just faster (570MBps vs 392MBps at 16MB). Although the encryption cost does also seem to scale negatively with recordsize as well (it's the same amount of work!!!). Ryzen 5900X

zfetch_array_rd_sz does nothing
zfetch_max_distance makes a huge difference! Any value below 16MB has poor performance. Maxes out at 64MB:

dd if=16MB of=/dev/null bs=$((16*1024*1024)) count=300
5033164800 bytes (5.0 GB, 4.7 GiB) copied, 4.8242 s, 1.0 GB/s

Seems like, zfs_max_recordsize and zfetch_max_distance should get correlated settings some how? I couldn't find a single reference to it earlier.

[amotin]

Right now I am working on one more patch to the speculative prefetcher, where among other things I am going to increase maximum prefetch distance to 64MB, that on my tests with 48-wide striped HDD pool allows to reach ~1.5GB/s in one stream. Though I haven't tried that high recordsizes, only 32KB and 128KB.

[amotin]

#13452