ZFS on slow 32bit ARM devices instable - 0.6 vs 2.0

[digi24]

Hello,

I know, that the usage of ZFS on 32bit devices is not encouraged. However, I have been using zfs 0.6x and 0.7x reliably on my device for a long time. I cannot judge the performance, but it was perfectly stable, the only downside were troubles with the debian dkms packages.

This was fixed in later versions and now the kernel modules compile perfectly without interaction. But my problem is, that zfs is not stable anymore. I tried reducing threads, tweaking the module parameters to a minimum... However just a simple scrub can result in a hanging system. Same applies to heavy writing and reading on the zfs volumes.

My question:
I would like to keep using zfs 2.0x to be feature-compatible with my servers. But I do not seem to be able to slow down zfs in such a way that it provides the same stability like it did on 0.6/0.7. Or is it related to the GPL-changes and should I just unpatch those changes and compile manually?

Did anyone find the perfect kernel parameters to maximise stability, even on weak hardware?

[rincebrain]

Could you possibly specify in more detail what hardware you're running on, what specific OS version, what pool layout and any non-default properties, and any messages that came out in dmesg from it?

Does /proc/spl/kstat/zfs/dbgmsg say anything interesting when this happens? What does /proc/spl/kstat/zfs/[poolname]/txgs say when it's doing this?

[digi24]

@rincebrain Thank you for your question

I am using an Asus Tinker Board (slightly better than Raspberry Pi 3), 2GB RAM and Armbian (just Debian buster with some adaptions for the device) as an OS.

Kernel: Linux tinkerboard 5.10.60-rockchip #21.08.1 SMP PREEMPT Wed Aug 25 19:43:43 UTC 2021 armv7l GNU/Linux
ZFS: zfs-2.0.3-9~bpo10+1 zfs-kmod-2.0.3-9~bpo10+1

With zfs 0.6/0.7 I was using an Armbian version based on Debian stretch

The pool layout is RAIDZ1, consisting of three external drives connected via USB2, the board does not support USB3. The drives are ok, no SMART-problems, no USB cable problems, everything powered properly, I have not changed the hardware setup since the 0.6/0.7 times but cross-checked to rule-out hardware problems. The ARM-board is stable under load or during long-term operations. Root is on ZFS and regular services (eg. light databse usage) and their disk access do not create any problems. Only heavy disk access (eg writing tens of GB, scrub) creates trouble.

No special properties, compression enabled, dedup off, some harmless perfomance optimisations.

Kernel boot line has: zfs.zvol_threads=4 added

and modprobe currently has (I have tested various settings properly before, but lost my values due to to unrelated reasons, these ones are not that systematic, except arc)

options zfs zfs_arc_max=67108865
options zfs zfs_arc_min=67108864

options zfs zfs_scan_vdev_limit=4096

options zfs zfs_vdev_scrub_min_active=1
options zfs zfs_vdev_scrub_max_active=1

options zfs zfs_vdev_sync_write_min_active=1
options zfs zfs_vdev_sync_write_max_active=2

options zfs zfs_vdev_sync_read_min_active=1
options zfs zfs_vdev_sync_read_max_active=4

options zfs zfs_vdev_async_read_min_active=1
options zfs zfs_vdev_async_read_max_active=4

options zfs zfs_vdev_async_write_min_active=1
options zfs zfs_vdev_async_write_max_active=2

options zfs zfs_vdev_queue_depth_pct=1

options zfs zfs_scrub_min_time_ms=100

options zfs zfs_vdev_nia_credit=1

I do not have any insight what happens when the board hangs. Root is on the zfs pool and the machine is headless. The system logs to RAM and ocasionally writes to disk. I do not want to unnecessarily stress-test this system, I just would like it to be slow and stable.

[rincebrain]

Curious. I've used ZFS on rather weak systems for a long time, and not had this particular problem.

I should point out that while the Pi 3 may be weaker in an absolute sense, it does have the benefit of being a 64-bit chip, which could make a difference here. I'll try firing up my Pi 3 with a 32-bit OS and see what I can make it do, though I don't have a set of USB hard drives I could turn into a raidz1 particularly easily...

[digi24]

I do not think it is related to 32 vs 64 bit-chip. My suspicion is the slow USB 2 connection of the disks in combination with the improvements in ZFS. I think I started seeing these problems with zfs 0.8x, but those were not isolated upgrades, so there could be many reasons.

[rincebrain]

Certainly one place I'd be suspicious. I believe on the Pi 3 it's still the case that the whole USB2 port set is behind a single actual port to the SoC and then an onboard hub, so if I can dig up enough disks, I can probably choke similarly...

[rincebrain]

So, I tried reproducing this on my Pi 3 running Debian buster armhf with an extremely slow (and, as it turns out, error-prone) set of disks in raidz1, and while I reproduced someone else's bug, I did not reproduce this one.

I'll see if my Pi 1 can make any more excitement...

[digi24]

Thanks for trying. Did you use any special kind of settings? Just the default zfs packages or build from source? I will try upgrading to bullseye, but do not expect much to change.

[rincebrain]

No, just the defaults and compression=lz4.

I haven't had a chance to throw my Pi 1 at it yet, a few other things came up...

[digi24]

It is not really an answer, but after an upgrade to 2.1.4 and and a 5.15 kernel, the system is stable and consistent again. Unfortunately I upgraded to many things at once to narrow it down.