After rebooted, zpool becomes unmountable and shows "invalid label"

[ctrlshiftm129]

Hi! After I rebooted my server, zpool becomes unmountable and shows "invalid label".

My zfs version is 2.3.1-1

zpool import:
 pool: zpool
     id: 17974986851045026868
  state: UNAVAIL
status: One or more devices contains corrupted data.
 action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
 config:

        zpool                    UNAVAIL  insufficient replicas
          raidz1-0               UNAVAIL  insufficient replicas
            sdc1                 ONLINE
            sdd1                 ONLINE
            sdi1                 ONLINE
            6057603923239297990  UNAVAIL  invalid label
            sdk1                 UNAVAIL  invalid label

It's strange. My pool name should be "zpool4t".

This is what I found from history which created my pool.

zpool create -f -o ashift=12 -O atime=off -o autotrim=on -O compression=on
-O normalization=formC -O utf8only=on zpool4t raidz
ata-ST4000VN008-2DR166_ZDH8HKB0 ata-ST4000VN008-2DR166_ZDH8J4R4
ata-ST4000VN008-2DR166_ZDH8J4Y4 ata-ST4000VN008-2DR166_ZDH9AX8E
ata-ST4000VN008-2DR166_ZDH9C7HR

Then I use "zdb -l /dev/sdx1" for my 5 drivers

# zdb -l /dev/sdd1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'zpool'
    state: 1
    txg: 3114017
    pool_guid: 17974986851045026868
    errata: 0
    hostname: 'Tower'
    top_guid: 10484232838912187944
    guid: 9613584519376571584
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 10484232838912187944
        nparity: 1
        metaslab_array: 135
        metaslab_shift: 34
        ashift: 12
        asize: 20003911106560
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 9780443222822920977
            path: '/dev/sdc1'
            whole_disk: 0
            DTL: 425
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 9613584519376571584
            path: '/dev/sdd1'
            whole_disk: 0
            DTL: 424
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 6499147292202988868
            path: '/dev/sdi1'
            whole_disk: 0
            DTL: 423
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 6057603923239297990
            path: '/dev/sdj1'
            whole_disk: 0
            not_present: 1
            DTL: 422
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 13162368864450388321
            path: '/dev/sdk1'
            whole_disk: 0
            DTL: 421
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3

# zdb -l /dev/sdc1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'zpool'
    state: 1
    txg: 3114017
    pool_guid: 17974986851045026868
    errata: 0
    hostname: 'Tower'
    top_guid: 10484232838912187944
    guid: 9780443222822920977
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 10484232838912187944
        nparity: 1
        metaslab_array: 135
        metaslab_shift: 34
        ashift: 12
        asize: 20003911106560
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 9780443222822920977
            path: '/dev/sdc1'
            whole_disk: 0
            DTL: 425
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 9613584519376571584
            path: '/dev/sdd1'
            whole_disk: 0
            DTL: 424
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 6499147292202988868
            path: '/dev/sdi1'
            whole_disk: 0
            DTL: 423
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 6057603923239297990
            path: '/dev/sdj1'
            whole_disk: 0
            not_present: 1
            DTL: 422
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 13162368864450388321
            path: '/dev/sdk1'
            whole_disk: 0
            DTL: 421
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3

zdb -l /dev/sdi1
------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'zpool'
    state: 1
    txg: 3114017
    pool_guid: 17974986851045026868
    errata: 0
    hostname: 'Tower'
    top_guid: 10484232838912187944
    guid: 6499147292202988868
    vdev_children: 1
    vdev_tree:
        type: 'raidz'
        id: 0
        guid: 10484232838912187944
        nparity: 1
        metaslab_array: 135
        metaslab_shift: 34
        ashift: 12
        asize: 20003911106560
        is_log: 0
        create_txg: 4
        children[0]:
            type: 'disk'
            id: 0
            guid: 9780443222822920977
            path: '/dev/sdc1'
            whole_disk: 0
            DTL: 425
            create_txg: 4
        children[1]:
            type: 'disk'
            id: 1
            guid: 9613584519376571584
            path: '/dev/sdd1'
            whole_disk: 0
            DTL: 424
            create_txg: 4
        children[2]:
            type: 'disk'
            id: 2
            guid: 6499147292202988868
            path: '/dev/sdi1'
            whole_disk: 0
            DTL: 423
            create_txg: 4
        children[3]:
            type: 'disk'
            id: 3
            guid: 6057603923239297990
            path: '/dev/sdj1'
            whole_disk: 0
            not_present: 1
            DTL: 422
            create_txg: 4
        children[4]:
            type: 'disk'
            id: 4
            guid: 13162368864450388321
            path: '/dev/sdk1'
            whole_disk: 0
            DTL: 421
            create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
        com.klarasystems:vdev_zaps_v2
    labels = 0 1 2 3

zdb -l /dev/sdk1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

zdb -l /dev/sdj1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

lsblk -bo NAME,LABEL,MAJ:MIN,TRAN,ROTA,ZONED,VENDOR,MODEL,SERIAL,PARTUUID,START,SIZE,PARTTYPENAME
NAME  LABEL  MAJ:MIN TRAN ROTA ZONED VENDOR   MODEL               SERIAL               PARTUUID                             START           SIZE PARTTYPENAME
loop0          7:0           1 none                                                                                                    582975488 
loop1          7:1           1 none                                                                                                     72441856 
loop2          7:2           1 none                                                                                                  42949672960 
loop3          7:3           1 none                                                                                                   1073741824 
sda            8:0   usb     1 none  SanDisk' Cruzer Fit          03023403120223145222                                               30784094208 
└─sda1
      UNRAID   8:1           1 none                                                    5c2c5657-01                             32    30784077824 W95 FAT32 (LBA)
sdb            8:16  sas     1 none  ATA      WDC WUH721818ALE6L4 6FG0P8AR                                                        18000207937536 
└─sdb1
               8:17          1 none                                                    017f2682-29ac-45e6-a59b-caa639956351    64 18000207887872 Linux filesystem
sdc            8:32  sas     1 none  ATA      ST4000VN008-2DR166  ZDH9AX8E                                                         4000787030016 
└─sdc1
      zpool    8:33          1 none                                                    f57f3e70-cd17-4eed-90e2-4ae83cc4fdc6    64  4000786980352 Linux filesystem
sdd            8:48  sas     1 none  ATA      ST4000VN008-2DR166  ZDH9C7HR                                                         4000787030016 
└─sdd1
      zpool    8:49          1 none                                                    993a41f9-49f9-430f-a966-af3c8d6a13c1    64  4000786980352 Linux filesystem
sde            8:64  sas     1 none  ATA      ST18000NM000J-2TV10 ZR5AF2M0                                                        18000207937536 
└─sde1
               8:65          1 none                                                    7768217a-f5b6-4875-8b42-30b246d9d560    64 18000207887872 Linux filesystem
sdf            8:80  sas     1 none  ATA      ST18000NM000J-2TV10 WR502KDM                                                        18000207937536 
└─sdf1
               8:81          1 none                                                                                            64 18000207887872 
sdg            8:96  sas     1 none  ATA      ST6000VM000-2BC186  ZCT3R9MQ                                                         6001175126016 
└─sdg1
               8:97          1 none                                                    ca4ba1df-168e-487b-9eb0-e2ac05692f4a    64  6001175076352 Linux filesystem
sdh            8:112 sas     1 none  ATA      WDC WUH721818ALE6L4 6FG0ZY8S                                                        18000207937536 
└─sdh1
               8:113         1 none                                                    13b0e797-c070-4c6c-a04b-71d43490d403    64 18000207887872 Linux filesystem
sdi            8:128 sas     1 none  ATA      ST4000VN008-2DR166  ZDH8J4Y4                                                         4000787030016 
└─sdi1
      zpool    8:129         1 none                                                    e812d7ce-59a5-4082-a6d7-efc1f4b4b891    64  4000786980352 Linux filesystem
sdj            8:144 sas     1 none  ATA      ST4000VN008-2DR166  ZDH8J4R4                                                         4000787030016 
└─sdj1
               8:145         1 none                                                    c979ca88-3b94-45fc-9ccb-a4c2c5aef6f2    64  4000786980352 Linux filesystem
sdk            8:160 sas     1 none  ATA      ST4000VN008-2DR166  ZDH8HKB0                                                         4000787030016 
└─sdk1
               8:161         1 none                                                    969fc824-447e-4b12-b60a-a611b015d790    64  4000786980352 Linux filesystem
sdl            8:176 sas     1 none  ATA      WDC WUH721818ALE6L4 6EG4JZVL                                                        18000207937536 
└─sdl1
               8:177         1 none                                                    66d4c810-0425-4d7f-abc4-4d1904494d65    64 18000207887872 Linux filesystem
md1p1          9:1           1 none                                                                                               18000207884288 
md2p1          9:2           1 none                                                                                               18000207884288 
md3p1          9:3           1 none                                                                                               18000207884288 
md4p1          9:4           1 none                                                                                               18000207884288 
md5p1          9:5           1 none      

The information of "invalid label" in the internet is so rare.
Please tell me how to fix my label? I just can't believe reboot server could make my data disappear.
My hard drive has not experienced physical damage (I have tested it).

[IvanVolosyuk]

There are horror stories of using sdX device names for pool which can change at boot in some configuration. Have you tried to import the pool using zpool import -d /dev/disk/by-id ? Where the name 'zpool' comes from? Did you try to create zpool in your previous attempts?

[ctrlshiftm129]

Yes. My previous zpool name is "zpool".
Still failed.

zpool import -d /dev/disk/by-id
  pool: zpool
    id: 17974986851045026868
 state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
config:

        zpool                             UNAVAIL  insufficient replicas
          raidz1-0                        UNAVAIL  insufficient replicas
            wwn-0x5000c500c803a10b-part1  ONLINE
            wwn-0x5000c500c802c84e-part1  ONLINE
            wwn-0x5000c500c41f7ec6-part1  ONLINE
            6057603923239297990           UNAVAIL  invalid label
            sdk1                          UNAVAIL  invalid label

What do you mean by "using sdX device"? I used this, not sdX

zpool create -f -o ashift=12 -O atime=off -o autotrim=on -O compression=on
-O normalization=formC -O utf8only=on zpool4t raidz
ata-ST4000VN008-2DR166_ZDH8HKB0 ata-ST4000VN008-2DR166_ZDH8J4R4
ata-ST4000VN008-2DR166_ZDH8J4Y4 ata-ST4000VN008-2DR166_ZDH9AX8E
ata-ST4000VN008-2DR166_ZDH9C7HR

Thanks.

[ctrlshiftm129]

I forget how I removed or formatted the "zpool". Maybe through webui? It seems like a bad idea...

[IvanVolosyuk]

It might be that labels from older pool somehow found by system before labels for a newer pool. Say, one pool is whole disk one, but another constructed from partitions. I don't know what to do in this situation, if you really think that the 'zpool' is not the correct one, you can use 'zpool labelclear [device]' https://openzfs.github.io/openzfs-docs/man/master/8/zpool-labelclear.8.html
This will destroy any signs of the pool and if you really know that there is another one - it may be found after that, but I will not bet on this my money.