Cannot import zpool, disks ok, but mdadm claims them as linux raid members

[deajan]

I have two disks with multiple partitions, of which 5th was member of a mdraid.

So far so good, I removed that raid, and gave the 5th partition of both disks to zfs.
Everything worked for weeks until my last reboot.
Now my pool doesn't get imported no matter what (tried zpool import -a, zpool import -D, zpool import -d /dev/disk/by-id/[...]part5...).
I've checked that the ZFS filesystem is readable on those partitions

# zdb -l /dev/nvme1n1p5

    version: 5000
    name: 'private_vm'
    state: 0
    txg: 13738
    pool_guid: someguid
    errata: 0
    hostname: 'redacted.local'
    top_guid: anotherguid
    guid: someotherguid
    vdev_children: 1
    vdev_tree:
        type: 'mirror'
        id: 0
        guid: aguid
        metaslab_array: 256
        metaslab_shift: 33
        ashift: 12
        asize: 1592217042944
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: anogtherguid
            path: '/dev/disk/by-id/nvme-SAMSUNG_MZ1L21T9HCLS-00A07_1234567890-part5'
            whole_disk: 0
            DTL: 11537
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 16871778785733784102
            path: '/dev/disk/by-id/nvme-SAMSUNG_MZ1L21T9HCLS-00A07_1234567891-part5'
            whole_disk: 0
            DTL: 11536
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3

Both partitions have the same result.
But mdadm still thinks those disks are linux raid members:

blkid | grep "nvme0n1p5\|nvme1n1p5"
/dev/nvme0n1p5: UUID="ff56f4db-6bc9-c41b-cfb1-607d1f2cef71" UUID_SUB="76a33191-b482-8cf1-79e5-26e5634203c1" LABEL="localhost:PRIV_VM_DATA" TYPE="linux_raid_member" PARTUUID="1791eb9a-99db-499c-9bb3-4ed4a6ea6a14"
/dev/nvme1n1p5: UUID="ff56f4db-6bc9-c41b-cfb1-607d1f2cef71" UUID_SUB="4ec5bd4a-0db6-ad79-926f-c6db3a20cd6d" LABEL="localhost:PRIV_VM_DATA" TYPE="linux_raid_member" PARTUUID="a807533f-ac19-456c-9d93-b40694db1702"

I have absolutely no idea how to reimport that pool.
Any guidance please ?

[amotin]

blkid has a hardcoded order of labels it probes. If there is no RAID any more, you should wipe its labels. wipefs if used carefully should be able to help with that, hopefully not damaging ZFS in process.

[amotin]

That I won't help with. I hope it is not trying to "recover" the RAID right now, scrambling your pool.

[deajan]

Okay, got it, I missed mdadm --stop /dev/md124
From there, I used wipefs to remove the magic number, which sufficed to be able to reimport the pool.

@amotin THANK YOU, like a lot, for the guidance about wipefs !

I'm a seasoned sysadmin, but somehow I searched to change the label via gdisk or part id via fdisk instead of the magic numbers.
I use zfs for more than 10 years (knew TrueNAS 11, knew zfs-fuse...).
Didn't know that wipefs did anything else than "erasing labels".

Can I buy you one (or more) coffee / beers ? Do you have a sponsoring page / paypal / whatever ?

[amotin]

@deajan I don't really have any donations account, unless you send US Amazon gift card or something. But it was confirmed that OpenZFS donation URL is valid, and it was added into the docs: https://openzfs.github.io/openzfs-docs/Project%20and%20Community/Donate.html .

[deajan]

@amotin Can do an Amazon US gift card. Where do I send that ?

[amotin]

@deajan [email protected], as in commit messages.