Replacing disk in draid pool? #13990

heinowalther · 2022-10-03T12:59:38Z

heinowalther
Oct 3, 2022

HI there,

[Ubuntu 22.04, ZFS 2.1.4]
I am currently testing the draid pool with 12 disks.
I created it as draid2:9d:12c:1s and write about a TB to it...
I then pulled one disk, and the pool resilvered to the spare device without any problems.
After it had completed, I then re-inserted the disk I pulled, offined it and did a wipefs and dd to clear it...
But I am having issues replacing the disk in the pool... not sure what is going on?
Here are some more details..

This is how it looked after resilvering, and I have offlined the disk 0b-4...

`root@boxen:~# zpool status test
pool: test
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 52.3M in 00:00:01 with 0 errors on Mon Oct 3 13:26:54 2022
config:

    NAME                  STATE     READ WRITE CKSUM
    test                  DEGRADED     0     0     0
      draid2:9d:12c:1s-0  DEGRADED     0     0     0
        spare-0           DEGRADED     0     0     0
          0b-4            OFFLINE      4   159     0
          draid2-0-0      ONLINE       0     0     0
        0b-5              ONLINE       0     0     0
        0b-6              ONLINE       0     0     0
        0b-7              ONLINE       0     0     0
        0b-12             ONLINE       0     0     0
        0b-13             ONLINE       0     0     0
        0b-14             ONLINE       0     0     0
        0b-15             ONLINE       0     0     0
        0b-20             ONLINE       0     0     0
        0b-21             ONLINE       0     0     0
        0b-22             ONLINE       0     0     0
        0b-23             ONLINE       0     0     0
    spares
      draid2-0-0          INUSE     currently in use

errors: No known data errors`

Here I am wiping the disk... and showing that there is a device behind 0b-4...

`root@boxen:~# ls -l /dev/disk/by-vdev/0b-4
lrwxrwxrwx 1 root root 10 Oct 3 14:40 /dev/disk/by-vdev/0b-4 -> ../../sdac

root@boxen:~# wipefs -a /dev/disk/by-vdev/0b-4
/dev/disk/by-vdev/0b-4: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/disk/by-vdev/0b-4: 8 bytes were erased at offset 0x105efffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/disk/by-vdev/0b-4: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/disk/by-vdev/0b-4: calling ioctl to re-read partition table: Success

root@boxen:~# dd if=/dev/zero of=/dev/disk/by-vdev/0b-4
^C1855046+0 records in
1855046+0 records out
949783552 bytes (950 MB, 906 MiB) copied, 17.0863 s, 55.6 MB/s
`
Here I then try to online the disk again...

root@boxen:~# zpool online test /dev/disk/by-vdev/0b-4 warning: device '/dev/disk/by-vdev/0b-4' onlined, but remains in faulted state use 'zpool replace' to replace devices that are no longer present

Which gives this pool status:

` pool: test
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
scan: resilvered 52.3M in 00:00:01 with 0 errors on Mon Oct 3 13:26:54 2022
config:

    NAME                  STATE     READ WRITE CKSUM
    test                  DEGRADED     0     0     0
      draid2:9d:12c:1s-0  DEGRADED     0     0     0
        spare-0           DEGRADED     0     0     0
          0b-4            UNAVAIL      4   159     0
          draid2-0-0      ONLINE       0     0     0
        0b-5              ONLINE       0     0     0
        0b-6              ONLINE       0     0     0
        0b-7              ONLINE       0     0     0
        0b-12             ONLINE       0     0     0
        0b-13             ONLINE       0     0     0
        0b-14             ONLINE       0     0     0
        0b-15             ONLINE       0     0     0
        0b-20             ONLINE       0     0     0
        0b-21             ONLINE       0     0     0
        0b-22             ONLINE       0     0     0
        0b-23             ONLINE       0     0     0
    spares
      draid2-0-0          INUSE     currently in use

errors: No known data errors`

I then try to replace the disk...

root@boxen:~# zpool replace test 0b-4 cannot open '/dev/disk/by-vdev/0b-4-part1': Device or resource busy

And after that, the pool looks like this...

` pool: test
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 4.89M in 00:00:01 with 0 errors on Mon Oct 3 14:46:14 2022
config:

    NAME                  STATE     READ WRITE CKSUM
    test                  ONLINE       0     0     0
      draid2:9d:12c:1s-0  ONLINE       0     0     0
        0b-4              ONLINE       4   159     0
        0b-5              ONLINE       0     0     0
        0b-6              ONLINE       0     0     0
        0b-7              ONLINE       0     0     0
        0b-12             ONLINE       0     0     0
        0b-13             ONLINE       0     0     0
        0b-14             ONLINE       0     0     0
        0b-15             ONLINE       0     0     0
        0b-20             ONLINE       0     0     0
        0b-21             ONLINE       0     0     0
        0b-22             ONLINE       0     0     0
        0b-23             ONLINE       0     0     0
    spares
      draid2-0-0          AVAIL

errors: No known data errors`

Any suggestions?
I have also tried to do a "zpool replace test 0b-4 sdac" which is of cause the same disk, and it also didn't fix it...
The disks are installed in a disk shelf and the vdev_id.conf is used to give the disk vdevs a number that represents their location in the shelf. I will try to move the 0b-4 to another slot, but in the future there might not be a free slot to move it to...

Any help is very welcome...

Just an addon...

I did a "zpool offline test 0b-4" which gave this pool status:

` pool: test
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 4.89M in 00:00:01 with 0 errors on Mon Oct 3 14:46:14 2022
config:

    NAME                  STATE     READ WRITE CKSUM
    test                  DEGRADED     0     0     0
      draid2:9d:12c:1s-0  DEGRADED     0     0     0
        0b-4              OFFLINE      4   159     0
        0b-5              ONLINE       0     0     0
        0b-6              ONLINE       0     0     0
        0b-7              ONLINE       0     0     0
        0b-12             ONLINE       0     0     0
        0b-13             ONLINE       0     0     0
        0b-14             ONLINE       0     0     0
        0b-15             ONLINE       0     0     0
        0b-20             ONLINE       0     0     0
        0b-21             ONLINE       0     0     0
        0b-22             ONLINE       0     0     0
        0b-23             ONLINE       0     0     0
    spares
      draid2-0-0          AVAIL

errors: No known data errors`

I then pulled the disk, and the pool started to resilver?!? I don't understand this...
I didn't pool the wrong disks (then if should tell me that in the pool...)

` pool: test
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver (draid2:9d:12c:1s-0) in progress since Mon Oct 3 15:01:02 2022
111G scanned at 2.15G/s, 102G issued 1.98G/s, 1.20T total
9.23G resilvered, 9.02% done, 00:08:39 to go
config:

    NAME                  STATE     READ WRITE CKSUM
    test                  DEGRADED     0     0     0
      draid2:9d:12c:1s-0  DEGRADED     0     0     0
        spare-0           DEGRADED     0     0     0
          0b-4            OFFLINE      4   159     0
          draid2-0-0      ONLINE       0     0     0  (resilvering)
        0b-5              ONLINE       0     0     0  (resilvering)
        0b-6              ONLINE       0     0     0  (resilvering)
        0b-7              ONLINE       0     0     0  (resilvering)
        0b-12             ONLINE       0     0     0  (resilvering)
        0b-13             ONLINE       0     0     0  (resilvering)
        0b-14             ONLINE       0     0     0  (resilvering)
        0b-15             ONLINE       0     0     0  (resilvering)
        0b-20             ONLINE       0     0     0  (resilvering)
        0b-21             ONLINE       0     0     0  (resilvering)
        0b-22             ONLINE       0     0     0  (resilvering)
        0b-23             ONLINE       0     0     0  (resilvering)
    spares
      draid2-0-0          INUSE     currently in use

errors: No known data errors`

I don't quite understand this behaviour?

I have now inserted the pulled disk into another slot, and will wipe it, and try to replace 0b-4 once the resilvering is done...

Well, the resilver completed, and so did the scrub...

I then did a "zpool replace test 0b-4 0b-0" where 0b-0 is the same device, just moved to another slot in the shelf... and of cause wipefs -a and dd if=/dev/zero .... to make sure the disk was clean... yet I am unable to replace the failed disk in the pool?

root@boxen:~# zpool replace test 0b-4 0b-0 cannot replace 0b-4 with 0b-0: 0b-0 is busy, or device removal is in progress

Is this a known bug, or am I doing something wrong?

/H

Answered by heinowalther

Oct 3, 2022

So I ended up f*ucking up my pool... reading too many "bad" suggestions I found ;-)
I did a "zpool detach test ", which removed the failed device, but I was unable to add the new disk again.. (I was aware that detach properly wasn't the right way to go)...
I build the pool again, and found out that "zpool labelclear" doesn't work on devices like /dev/sda, but want the first partition like /dev/sda1... (which isn't stated anywhere) ? :-)

But once the pool was build I copied some data onto it, pulled a disk, which started the resilvering and subsequent scrub.
I then did a "zpool offline test 0b-0" (0b-0 was the disks I pulled)...
I inserted the disk again...
Then did a "zpool labelclear 0b-…

View full answer

heinowalther · 2022-10-03T14:39:24Z

heinowalther
Oct 3, 2022
Author

So I ended up f*ucking up my pool... reading too many "bad" suggestions I found ;-)
I did a "zpool detach test ", which removed the failed device, but I was unable to add the new disk again.. (I was aware that detach properly wasn't the right way to go)...
I build the pool again, and found out that "zpool labelclear" doesn't work on devices like /dev/sda, but want the first partition like /dev/sda1... (which isn't stated anywhere) ? :-)

But once the pool was build I copied some data onto it, pulled a disk, which started the resilvering and subsequent scrub.
I then did a "zpool offline test 0b-0" (0b-0 was the disks I pulled)...
I inserted the disk again...
Then did a "zpool labelclear 0b-0-part1".
And then "zpool replace test 0b-0" which resilvered the pool again back to an OK state...

I guess it is up to the moderator to decide if this should be left here for some someone else to enjoy? ;-)

One thing I will add is that the resilver process is much faster than my old raidz3 pool... it is more or less running at disk speed (about 200MB/sec) and I guess you cannot expect a faster rebuild...
Later on I will give it a try with a 24 disk pool... draid2:21d:24c:1s if that is slower I will resort to two vdevs with 12 disks each.

/H

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replacing disk in draid pool? #13990

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Replacing disk in draid pool? #13990

Uh oh!

Uh oh!

heinowalther Oct 3, 2022

Replies: 1 comment

Uh oh!

heinowalther Oct 3, 2022 Author

heinowalther
Oct 3, 2022

heinowalther
Oct 3, 2022
Author