Slow resilvering of special device mirror

[heinowalther]

Hi there
I have a zpool (180TB) where I have a 500G mirrored special Sevice attached (of which only 110G is used).
The two devices mirrored was nvme.
Because I need to move this zpool onto another server, I need to move the special device on to disks, so I attached two 10k SAS disks to it... I know that nvme and SAS isn't the same speed, but is is only a temp. arrangement.
What took me by surprise was the time it took to mirror over the data... about 4-5 hours per disk.
Also, it starts by stating that it is resilvering, but in iostat I can see that it mostly reads from the existing disks... this takes about an hour until it gets to about 99.95% done, and a (resilvering) appears on the device attached... after this I can see that there are some activity, and it seems that this is slow because of the many IOPs... (about 500 IOPs) and about 15MB/sec. I guess this is because of the many small "files" on this device?
It managed to finish, and I am now going the other way from the SAS disks to NVME devices again...

Is there any way to speed up this process? And what is ZFS doing at the first part of the resilvering? (building a list of files to transfer manybe?)

[amotin]

I haven't tested special vdev resilvering, but next week I am going to present my scrub optimization work on OpenZFS Dev Summit. I was able to about double its performance earlier this year. So depending what version you are running, it may help you to update to the latest 2.1.6.

[heinowalther]

Yeah this on 2.1.4 (Ubuntu 22.04). Any idea if it is possible to upgrade zfs on this disto?
An alternative could be AlmaLinux which is also LTS as using a later ZFS version.. maybe even newer than 2.1.6 as far as I recall..

[rincebrain]

To answer your question about what ZFS Is doing (there's at least one talk on this, if you'd like someone both more authoritative and going into more detail):

• ZFS's sequential scrub/resilver code currently does two things - walks the metadata to figure out all the data it needs to go read/write and combine regions it wants to issue IO to because spinning disks are bad at lots of tiny IOs, and when that list gets full enough, it starts blasting them out
• "in theory", this should work fine for NVMe -> spinning rust transfer, since NVMe drives certainly handle sequential IO just as well if not better than spinning disks
• I thought I saw some work to not do the preprocessing as much if there's SSDs...I wonder if it only checks for any SSDs at all, and thus, it's trying to use SSD-optimal settings on spinning rust destination, and sadness?

Wild speculation, I don't have a test setup I could try that with quickly here right now, but it'd be interesting to see what zpool iostat -v said for a little while during the phase where it was writing to the spinning disks...

(You could also try making use of zpool attach -s, which is a different set of tradeoffs, amounting essentially to dd if=disk1 of=disk2 && zpool scrub pool, since it does no real checking while just copying in order from one disk to another...)