RAIDZ2 with 8 drives: Is it really that bad?

[mabod]

Hi,

I am planing to build a RAIDZ2 with eight 2 TB drives. I have read a few negative things about raidz2 with 8 drives and I am wondering if that information is still accurate.

I have read this blog from @ahrens :
https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz

It made me confident that performance wise RAIDZ2 on 8 drives is not too bad. But then I read things about wasted space: https://www.truenas.com/community/threads/raidz2-number-of-disks.16153/#post-82191

This post claims a wasted space of 1 TB for a RAIDZ2 with 8x 3 TB. Is that correct?

Are there any recommendations how to configure RAIDZ2 with 8 drives? My use case is a backup array with recordsize=1M and compression=on.

[rincebrain]

The first blogpost you linked has a spreadsheet that explains the overheads involved per record size as a function of sector-sized units and number of disks+parity involved.

For an 8-disk raidz2, the lowest the overhead gets to is 34%, which is what you'd expect, for 6 disks of data for 2 disks of parity, and the highest is 200%, which again, is what you'd expect if you wrote one sector's worth of data and then had to have 2 more sectors of parity.

[mabod]

I saw that excel table and I am confused. The table suggests that 8 discs (34 % for parity+padding) is better than 7 discs (41 %) and better than 6 discs (50 %). That tells me that 8 discs has not disadvantage over 6 discs.

But this post here says that 8 discs are worse than 6 discs:
https://lists.freebsd.org/pipermail/freebsd-fs/2013-May/017337.html

And when it comes to recordsize=1M, which is not considered in any document i have seen, I am lost.

[rincebrain]

The blogpost explains at length what happens when, but briefly, raidz overheads are generally minimized for a given configuration when you get sufficiently large records at once, but small records still suck more than large ones even in ideal configurations.

So 256 4k sectors (1M) has 34% overhead, which is basically just parity cost, but if you were writing, say, 6 4k sectors (24k), it goes up to 50%, because you're getting those 6 sectors, plus 2 parity, plus a filler sector because raidz enforces allocations being a multiple of P+1 to ensure you can never end up with free spaces so small nothing can use them (since if you didn't have allocations that were a multiple of the smallest allocation possible, e.g. one data sector and P parity sectors, with fragmentation and reallocation, you might end up with free space regions smaller than P+1 sectors, which would be impossible to ever allocate unless space around them was freed...)

Sure it is. 256 sectors at 4k a sector is 1M, and 256 is on that chart. If you're using 512b sectors, feel free to make a copy of the excel chart and add a row for 1024, and copy the formula down a row - it should still do the right thing, but I would expect it to produce the same results as 256 in most if not all cases.

[mabod]

The link to delphix does no longer exist:

https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz

Do you host a copy of this blog post somewhere?

[gmelikov]

I don't know what's with Delphix blog, but I've tried to summarize RAIDZ intro here https://openzfs.github.io/openzfs-docs/Basic%20Concepts/RAIDZ.html several months ago, you can find same spreadsheet here too.

[rincebrain]

https://www.delphix.com/blog/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz

[mabod]

Thanks for the quick reply. The new link from delphix works fine. But there are two delphix links at the end of the blog which do not work either:

https://www.delphix.com/http:/dtrace.org/blogs/ahl/2006/06/18/double-parity-raid-z
https://www.delphix.com/http:/dtrace.org/blogs/ahl/2009/12/21/acm_triple_parity_raid

[rincebrain]

I have, to be clear, no control over that blog, I just happened to have solved this problem when I went looking for that link a week or less ago and be on the CCs of this discussion.

http://dtrace.org/blogs/ahl/2006/06/18/double-parity-raid-z/ and http://dtrace.org/blogs/ahl/2009/12/21/acm_triple_parity_raid should DTRT.

[mabod]

Thank you, @rincebrain, I am sure other users will find the new links you have found very useful as well.