[BUG] Persistent Hard Crashing with XFS Temp Drive in MDADM RAID0

[andrewseid]

Describe the bug
XFS is known to be the fastest format for plotting, and testing proves this out. However, it also seems to result in reliable hard crashes on Ubuntu, usually within the first day of beginning plotting. I have experienced this issue about 15 times, on a mix of Ubuntu GUI 20.04, Ubuntu GUI 21.04, and Ubuntu Server 21.04. I've experienced it on three different systems, two AMD builds (3960X and 3990X), and one Intel build (i7-11700K). All systems have been using between two and four Samsung 980 Pro NVMe drives in MDADM RAID0.

The issue seems to go away when I format the temp drive RAID0 array with ext4.

To Reproduce

1. Create an XFS MDADM RAID0 array on Ubuntu 20.04 or 21.04 (GUI or server, doesn't matter), using 2-4 NVMe drives (in my case, Samsung 980 Pro 2TB, running on PCIe Gen 4).
2. Start 10+ plotting queues with -n 5 -r, depending on system specs.
3. Let system plot for 24-48 hours.

Expected behavior
Observe eventual hard crash.

Screenshots
On Ubuntu GUI, the desktop just completely freezes wherever it is.
On Ubuntu Server, I got this:

Desktop:

• OS: Ubuntu GUI 20.04, Ubuntu GUI 21.04, Ubuntu Server 21.04
• CPU: AMD 3990X, AMD 3960X, Intel i7-11700K
• NVMe: 2-4 Samsung 980 Pro 2TB in MDADM RAID0

Additional context
Random theory that you can feel free to ignore: since this is an extremely high performance setup, maybe it's hitting some kind of performance threshold or race condition during plotting? Or maybe it's something else entirely XD Thank you!

[Bigman397]

Try EXT4 instead of xfs... I personally switched as XFS was performing oddly for me on mdraid.

edit: and of course you tried that and I missed it, sorry =)

[andrewseid]

Try EXT4 instead of xfs... I personally switched as XFS was performing oddly for me on mdraid.

edit: and of course you tried that and I missed it, sorry =)

No problem! I'm indeed now using ext4, and it's proven quite stable. Just a shame to miss out on the ~7% performance improvement of XFS.

Others (JM/Quindor) seem to be using XFS MDRAID0 without issue, so I'm not sure what's going on with our systems. Cheers!

[xorinox]

I know LVM raid-0 uses MD under the hood, yet I would give it a try as using LVM might configure it differently and xfs reading the raid config via LVM might also lead to a different configuration. Assume 2 drives

wipefs -a /dev/nvme0n1
wipefs -a /dev/nvme1n1
vgcreate nvme_scratch /dev/nvme0n1 /dev/nvme1n1
lvcreate --type raid0 --stripes 2 --stripesize 256 -l 100%free -n plots nvme_scratch
mkfs.xfs /dev/nvme_scratch/plots -L scratch01
mount -L scratch01 /mnt

More extreme option, try latest kernel.

[andrewseid]

@xorinox Thank you for the detailed instructions!

I will give this a try and let you know how it goes :D

[dyn4mic]

@andrewseid did you have success with lvm raid 0?
I run into the same issue that my madmax raid 0 with xfs run into kernel panic. With not syncing.
Same disks with hpool or ext4 no issues. I run Ubuntu server 21.04 as well.

[pwntr]

Same issue here with xfs after just a few minutes of plotting (full freeze, needed to hard-reset, but no entries in the logs/journal). A switch to ext4 seemed promising at first, but after ~23 hours on non-stop plotting with the madmax plotter, the same issue appeared. Same with the base chia plotter, albeit at a later stage.

What provided final salvation was disabling continuous trim on my raid 0 temp drives! I'm currently running the lvm based raid with xfs setup proposed by @xorinox, and after over 36 hours without a single crash/freeze, I'm somewhat confident to say that continuos trim might be the culprit rather than the filesystem (as his steps mount the array without continuous trim enabled, and I did not explicitly enable continuos trim in /etc/fstab via the discard option or in the lvm config).

To everyone who experiences the freezes/crashes with a raid 0 setup for the temp drives: did you try to run your plotting without continuous trim enabled on the array yet (e.g. no discard mount option in /etc/fstab)?

I'll try a vanilla mdadm raid 0 with xfs and without the lvm layer again soon just to cross-check and further isolate the issue. My best bet is still on disabling continuous trim though. There's a bunch of issues around this in the kernel maling lists and even in the source itself documented for different drives and configs, so it wouldn't surprise me for this to be the culprit of the iffyness we see with a raid 0 and hammering IO to those poor drives.

Also just for completeness: plotting on single drives without raid 0 was stable at all times, regardless if I was using xfs or ext4 as the fs, and also regardless if continuous trim was enabled on them or not. The single drive setup was just a fair bit slower than the raid 0 for my particular drives.

System config:

• Ubuntu 21.04
• Ryzen 5950x
• Asus Dark Hero VIII x570 MB
• 128 GB RAM (Crucial Ballistix @ 3600 MHz) -> now plotting with tmpfs ramdisk to get ~ 20 minutes/plot and reduced drive writes, highly recommended especially when having continuous trim disabled :)...
• 2 x NVMe Silicon Power A80 @ 2 TB each in LVM based raid 0, as per @xorinox' instructions, used as pure temp drive(s)
• 1 x NVMe Crucial P5 @ 500 GB for OS and tools

[DrShotsPHD]

@pwntr, I tried xfs on mdadm with discard unset and the system still crashed FYI.

[dyn4mic]

i can confirm that @xorinox solution worked fine, whatever ubuntus lvm does under the hood it allowed me to have a 8 disk raid 0 with xfs

[ramin-afshar]

I have exactly the same problem. Crashes usually happen after a couple of minutes into a plot.
I used to have the drives mounted without the discard option and my raid still crashed.
Might give ext4 a go. Or maybe btrfs and see how that goes.

[ramin-afshar]

I have exactly the same problem. Crashes usually happen after a couple of minutes into a plot.
I used to have the drives mounted without the discard option and my raid still crashed.
Might give ext4 a go. Or maybe btrfs and see how that goes.

update

I tried the commands provided by @xorinox to see if I could create a stable xfs filesystem this way, but unfortunately when using the mkfs.xfs command after creating the volumes with LVM it never completes the format, the process just hangs there,
I even waited an hour to be sure. I can format the volume as ext4 or btrfs without any issues. And I can format a single non-raid drive as xfs. So there must be some kind of bug i guess. I'm using ubuntu 20.04 with kernel 5.8.0-55-generic.

update 2
I managed to format the volume adding following options and I've already completed a couple of plots without any crashes.
sudo mkfs.xfs -f -d su=512k,sw=2 /dev/mp510_raid/raid0

[xorinox]

@ramin-afshar is it hanging in there discarding blocks? Run this in a screen/background until it's finished. Can take some time (up to hours).

[Spektre99]

How does one break apart a RAID created with vgcreate and lvcreate as given above?

Nothing shows when looking at cat /proc/mdstat despite having made a RAID0 array using xorinox's post as a guide above.

[ramin-afshar]

How does one break apart a RAID created with vgcreate and lvcreate as given above?

lvremove

https://linux.die.net/man/8/lvremove

[ramin-afshar]

@ramin-afshar is it hanging in there discarding blocks? Run this in a screen/background until it's finished. Can take some time (up to hours).

I see this sample output data when I execute the mkfs command in terminal, but it never finished. But when I used the options i mentioned in my previous message I don't have this problem.

meta-data=/dev/xvdf isize=512 agcount=4, agsize=26214400 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=0 data = bsize=4096 blocks=104857600, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=51200, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0