Gluster architecture help

[edrock200]

Hello all,
I'm fairly new to gluster and had some architecture/design questions I'm hoping someone can help me with. So far I'm loving how easy it is to get started with gluster, but have found in my testing that proper planning is key.

My intent is to layer gluster onto zfs bricks. The primary reason being ZFS can use my nvme drive as a cache disk to improve read/write performance. Each node has 10 HDD platter disks, these are configured in raidz2 with nvme as cache. I have 5 nodes, and I did some testing. First, I tried gluster dispersed 4+1 redundancy so I could lose 1 node without issue. This seems to perform very well. My delimma is in this configuration, to expand, I have to add 5 nodes at a time moving forward, correct?

Second question is, would it be better to just treat each of the 10 disks as a brick, eliminating the ZFS layer and NVME cache, and do something like 8+2? The performance impacts of increasing bricks and redundancy erasure code level isn't clear to me, so hoping someone with experience can chime in. With the setup above, I'm also losing a lot of space to parity data with the zfs+erasure code loss. So I'm considering tearing it down and rebuilding it 8+2 but curious on others thoughts.

[olegkrutov]

Hi,
answer to question 1 is yes, you'll need 5 more bricks to create distributed dispersed volume. About question 2: just try it ;) btw, do you really need raidz2, isn't fault tolerance of raidz1 + regular scrubbing enough?

[edrock200]

I've thought about this a lot. I was either going to to raidz1 + hot spare or raid z2. Reason being the sans are not local to me. If I could "touch" the drives in person I would be ok with z1. But without being able to swap out instantly my fear is if 2 drives die, the entire brick is lost, if 4 drives die (2+2) the entire cluster is lost. I realize that's probably not common but if that were to happen, and I looked back I would regret not having the extra parity.

That what made me consider ignoring zfs and the nvme cache benefit and just doing 10 bricks per node on each drive with 6+2 erasure code. On the one hand the sheer number of inodes per disk would be reduced by a factor of 10. On the other I don't know what the performance losses would be without the nvme cache and having to strip or read and rebuild on the fly across so many bricks. That's why I was hoping someone could chime in with some insight.

With regards to testing, the delimma is everything performs better when it has some data vs 60-70% full. I don't have enough sans to test it this full as I need to drain my existing sans to the first 5 node cluster, then repurpose the existing sans to expand the new cluster and copy the remaining data from an eol system. So don't want to get 80% of the way through and find out performance sucks. :) It's also timeline, it will take quite a while to migrate the data even with 10G interfaces, and I won't be able to copy full speed as the existing are still serving out data, so I have to throttle a bit to ensure they can copy the data off and serve out data at the same time.

[olegkrutov]

Hm, I did not dig so deep yet :) just a couple of thoughts... AFAIK zfs performance degrades heavily when free space is exhausting, so if you will not use zfs benefits and will use gluster volume with many bricks on single device each, maybe it's worth using some more simple fs like ext4 or so? Or maybe even thin-LVM + nvme-based lvmcache... And, btw, 6+2 config is not optimal by performance -- it is recommended that data bricks number be power of two (say, 2+p, 4+p, 8+p)... to prevent read-modify-write issues. And to be more sure about your drives health, it may be a sane idea to check drives' SMART status by smartctl and maybe run non-captive SMART tests periodically in relatively free time.

[edrock200]

Thank you for the reply. I will try 10 bricks (8+2). I do have a smartctl script running that greps and alerts on "FAILED" against all drives.

[edrock200]

I ended up not layering on zfs and just doing xfs on each drive as a brick. In my testing, it's actually performing better than when layered on zfs. I'm doing a mass copy over now so we will see how it does once it has some load and data usage.

[D3nis]

Hello Edrock200,
did You have some benchmarks tests? If "yes" can You share some results?

[edrock200]

Sorry I didn't see this earlier. In my testing, from one 10g node to gluster can hit 6-7gbs in a multi threaded copy. The bottleneck appears to be the 10g node drives itself. From an internet source direct to gluster, it can near max out the line teetering between 8-9gbs. I started a second 10gbs node copying and it took hit 8-9gbs, for a total throughput of ~18gbs writing to the drives. Note this is a cluster with 150 bricks over 15 nodes with 10gbs interfaces. I suspect, but don't know for sure, that with less nodes the performance may decrease.