ZFS kernel threads priority

[slavanap]

I'm looking for a setting that would assist in resolution of the following issue:

1. create zfs dataset with compression=gzip-9
2. start audio playback with pulseaudio & vlc (or similar)
3. copy 10GB worth of data to newly created dataset
4. observe as audio playback hangs until copying is completed

Is there any way to operate with huge amounts of data stored in ZFS and play audio without interruptions simultaneously?

I tried chrt vlc & pulseaudio -- it does not help. Probably USB driver is missing CPU cycles, because ZFS with compression=gzip-9 loads 100% of all CPU cores with kernel threads for compression. ZFS kernel threads nice value is -19 with SCHED_OTHER. They are spawned frequently, thus I can't alter their priority consistently. Maybe there's a way to drop priority change for them or spawn them in SCHED_IDLE queue temporarily?

OS: Ubuntu 22.04.1, CPU: Intel Core i7, 2 cores

[FFourtyTwo]

Hello!
First of all, why gzip-9? By my experience this is one of most ineffective and intensive algorithms for any workload. Also why so high ratio "-9"? For your 2c/4t CPU it looks like too much.

I suggest you to try zstd algorithm first, cause it's most effective we have starting from ZFS 2.0. If you really need to save every byte you can try zstd-19 but remember that every ratio step is less efficient than previous one. Depending on your data it may not worth to set high ratios. For example you will get +0.5% of space but it will take +50% of CPU time.

By my opinion default rate of compression=zstd-3 (that equals to compression=zstd) is best option for generic workloads. If you wanna save more than use zstd-5, zstd-7. After this (zstd-9+) it will not worth it.

Check this:
https://www.reddit.com/r/zfs/comments/svnycx/a_simple_real_world_zfs_compression_speed_an/
https://freebsdfoundation.org/wp-content/uploads/2021/05/Zstandard-Compression-in-OpenZFS.pdf
https://openzfs.org/w/images/4/4d/Compression-Saso_Kiselkov.pdf

Don't forget that any process scheduler (CFS in linux) works the worse the higher the load. In your case with 2c/4t and 100% load your scheduler spends much time to switch execution context. I suggest first to lower the overall load using better compression algorithm.

[slavanap]

Gzip-9 is because I use the dataset for archive purposes and gzip provides 60% compression ratio vs. 20-25% of zstd. The whole point of the question is not about it. Once compression threads start to load CPU because disk bandwidth is higher than compressor throughput the issue with the audio will occur with any compression algorithm.

Don't forget that any process scheduler (CFS in linux)

I use SCHED queues and load CPU with no issues at 100% while audio is playing and long running tasks are being running consuming remaining 100% of available CPU idle time. I experience issues only with ZFS now because it runs at high priority and steals CPU cycles from processes from which it should not to.

The topic question is that is there currently an option to alter ZFS threads priority so they won't interfere with other kernel threads operation as they are now at high throughput IO?

[IvanVolosyuk]

Are you confusing lz4 and zstd? I think zstd-9 supposed to have similar compression level as gzip-9.

[slavanap]

1229323

[IvanVolosyuk]

I had similar issues with sound in my qemu/kvm workload. I managed to fully mitigate it by using preemptive kernel (CONFIG_PREEMPT=y) and also setting spl.spl_taskq_thread_bind=0 zfs.spl_taskq_thread_bind=0 in kernel boot parameters. One of them is unused but I didn't spend time finding out which one.

My theory is system devices like sound or network device use cpu affinity mask to operate on a specific CPU thread, but ZFS will also schedule big chunk of work on those CPU thread which cannot be reallocated to another one. Also, compression in ZFS without preemptive kernel doesn't let higher priority tasks to be executed for a long time even if they are latency critical.

I only had this issue with qemu and I run it with realtime scheduling priority using chrt -r 5 qemu.... This command also allows to adjust scheduling for already running processes. You can try to bump up priority of pulseaudio and your sound app.

[Zocker1999NET]

Most probably pulseaudio / pipewire has already a higher, preferred priority (if not even being configured to be a realtime process) to avoid such problems as this should be the default for most graphical installations.

[Zocker1999NET]

I had a similar problem but with a much more capable setup and less performance dragging configuration: I only had a few audio stutters but my CPU (Ryzen 4 1600, 6 cores / 12 threads) was only about 50 % max due to ZFS compression. Normally, I could utilize my full CPU power (e.g. compiling Lineage OS with 12 threads) and I wouldn't hear any audio stutters due to working priorities and scheduling. ZFS compression seem to be excluded from that.

I could mitigate them for me without rebooting while restarting mpv with PIPEWIRE_LATENCY=1024/48000 mpv (increasing pipewire's buffers).
Next boot I may try the kernel boot parameters.

But still I would wish to be able to deprioritize ZFS's compression thread properly, best with an option possible to alter while the system is running.

[CaptaiNiveau]

How'd that go? I'm getting some audio stutters on encrypted zfs root with full preemption on a 5800x. LZ4 compression shouldn't be an issue, audio stutters are also quite rare. Haven't been able to find a simple way to reproduce the issue, but running fzf with fd in a big folder seems to trigger it.

[memeruiz]

I had a similar problem. Also with a much more capable system: Ryzen 9 5900X 24-threads 94GB RAM. Using zstd-10 copying from from zfs (default compression algo) to a newer zfs (zstd-10).

First problem I experimented was, slow copy speed. Checking I noticed a lot of swap activity. I manage to "control" this by changing a couple of zfs parameters concerning memory usage. The first was zfs_arc_max. I set it to much less than 1/2 of RAM. The other was zfs_arc_sys_free. First tried to set it to 10GB and it didn´t solve the problem, 40GB did it. My guess is that zstd in zfs using a separated RAM space which depends on the zstd chosen level, with higher requiring more RAM. Since zfs arc RAM usage has apparently nothing to do with zstd then zfs arc RAM was strangling zstd and sending it to swap.

Once this problem was solved then during copy, with some files CPU usage went really somewhat high, using basically all CPU threads and causing the same type of mpv audio problems. I played with ionice and renice, giving the rsync (copy) processes the lowest possible priority and mpv the highest possible priority but it didn't improve the audio problem. Also I tried the PIPEWIRE_LATENCY=1024/48000 suggested above and it didn't improve audio glitching. spl.spl_taskq_thread_bind=0 was already to 0.

There must be some zstd related kernel/module activity that has a really high priority that can't be changed.

Computer is completely unusable while zfs compress copy. Not only mpv audio is affected, browsing and everything else is sluggish. A Ryzen 9 brought to its knees with just this.

I think the best solution would be to have a more comprehensive memory control of zfs (not just arc) that includes memory control for all zfs activities, and also a priority setting for different zfs activities (hashing, encryption and compression).

Edit: The latency problem was solved changing preempt from voluntary to full:

echo "full" | sudo tee /sys/kernel/debug/sched/preempt

I thought full was the default with Debian Kernels

In short. To use zstd above 3 one needs to

• Reduce zfs arc memory usage (to avoid swap usage)
• Increase zfs arc sys free parameter (to avoid swap usage)
• Switch preempt to full (to avoid sluggishness)

Kernel 6.4.0-0-amd64
zfs version: 2.1.12-1

[memeruiz]

Already with zstd-5 it was horrible.

[memeruiz]

@IvanVolosyuk I just configured to preempt=full and now all is fine. I had it at "voluntary". No more sluggishness. Did it before seeing your comment, but thanks! Though of it a bit late! To be honest, I thought debian kernels came preempt=full by default, but no.

[slavanap]

@rincebrain, encryption and compression in ZFS is not I/O, but its priority is on I/O level.

[rincebrain]

I'm not really sure how you think "bytes being read/written to disk need to be {de,}{compressed,encrypted}" isn't in the IO pipeline, but you're wrong.

[Zocker1999NET]

So setting it to something lower would enable fun priority inversion problems where lower priority things can block -20 IO threads.

With that perspective, it makes sense to me why ZFS compression threads have a high priority as ZFS has no idea if its current tasks are essential to the system (being accesses to the system drive) or not (copying files sequentially to external hard drives).

Might it be possible to allow users selecting a priority for those ZFS compression threads per pool (maybe even per dataset would be nice) and without re-compiling the kernel or the ZFS module?
If setting kernel thread priorities dynamically is not really possible or feasible, it could be enough if there are two thread groups for compression/encryption/IO, one with -20 (still being the default) & one with a much lower priority. With an option per pool/dataset, admins can then select how important access to these pools/datasets are while docs hint them at the risks that come with deprioritizing compression/encryption.

An example use case is to have a system and an archival pool. The system pool is allowed to use the high priority threads and will also only have zstd compression as it is vital to the system. The archival pool should use the non-prioritized IO threads for best compression possible, as 1st, it is not vital to the system and its programs, 2nd, the higher compression rate is more valued than access latency and, 3rd, the system is expected to have enough free time for that lower priority IO thread, and if not, then there are other issues more important to the users (graphics, sound, input latency, …).