Need help tracking down cause of random metadata reads in a raidz pool

[adammj]

Issue:
TL;DR: 1 drive in a 3 drive RAIDZ pool will be semi-periodically accessed by zfs, for metadata updates of some sort (probably), but only when there is memory pressure (probably). When the computer is "quiet", no activity ever occurs on that pool. There are never any accessed or open files, which is why the other 2 drives will still remain in standby.

I have a RAIDZ1 pool that is made up of 3 entire hard disks. The pool is not used very often at the moment, so hdparm is used to allow the drives to go into standby. This works, and the drives will stay in standby (even delaying all smartd updates) for days and weeks on end.

However, when I start training 1 or 2 large pytorch networks from scripts and data on a different pool (which involves caching GBs onto /dev/shm (limited to 64 GB) and also allowing zfs ARC to cache the rest (limited to 64 GB) - total ram is 192 GB) then sometime later zfs will start making semi-periodic reads (about every 15 to 20 minutes) to 1 of the hard disks in the large pool. It's always the same hard disk, and the sector, according to blktrace, seems to be 1 of 2 (sometimes 6085468648, and other times 7051678704). Using zpool iostat I can see that there are never any read or write operations. However, there is "68" (is that kB?) of "read bandwidth" that coincides with the blktrace activity.

Even extending my knowledge with two different LLMs, I can't see to exactly nail down what is causing this issue. I can't seem to find any other activity with respect to that pool. And I can't seem to find a way to map those sectors (using zdb?) to figure out what they are.

[amotin]

I'd probably try to use bpftrace to tap the ZIO requests. Either stack might be interesting, or zbookmark_phys_t inside the ZIO might point to what's been accessed.

[adammj]

Thanks. Next time it occurs, I'll try this out.