raidz2 pool with

[cousins]

On a CentOS 7 (I know... we are migrating...) system with zfs-2.0.7-1 I have a pool made up of a number of 10-drive raidz2s. It is at a remote location and I noticed that raidz2-2 had two bad drives. The system didn't have any spare drives available and the local technician was not going to be available for a few days so I decided to take one drive offline from one raidz2 group and use that to replace a drive in the one that had two bad drives. What happened though, was that the drive ended up being listed in both raidz2s. The disk is called "Slot_30". Once I had more redundancy, I tried to fix it by offlining Slot_30 and replacing it with another drive: Slot_46. That's where it stands but the layout is even more strange, with a "replacing-9" section that has three devices, two of which are OFFLINE. resilvering has completed and I scrubbed and rebooted, hoping that this would clear things up. It hasn't.

The original bad drive that I think is now labeled as "old" was Slot_29. Here is what I see in these groups:

	 ...
 	  raidz2-2       DEGRADED     0     0     0
	    Slot_20      ONLINE       0     0     0
	    Slot_21      ONLINE       0     0     0
	    Slot_22      ONLINE       0     0     0
	    Slot_23      ONLINE       0     0     0
	    Slot_24      ONLINE       0     0     0
	    Slot_25      ONLINE       0     0     0
	    Slot_26      ONLINE       0     0     0
	    Slot_27      ONLINE       0     0     0
	    Slot_28      ONLINE       0     0     0
	    replacing-9  DEGRADED     0     0     0
	      old        OFFLINE      0     0     0
	      Slot_46    ONLINE       0     0     0
	      **Slot_30    OFFLINE      0     0     0**
	  raidz2-3       DEGRADED     0     0     0
	    **Slot_30      OFFLINE      0     0     0**
	    Slot_31      ONLINE       0     0     0
	    Slot_32      ONLINE       0     0     0
	    Slot_33      ONLINE       0     0     0
	    Slot_34      ONLINE       0     0     0
	    Slot_35      ONLINE       0     0     0
	    Slot_36      ONLINE       0     0     0
	    Slot_37      ONLINE       0     0     0
	    Slot_38      ONLINE       0     0     0
	    Slot_39      ONLINE       0     0     0
	 ...

The "**" marks are an attempt to highlight Slot_30 in both groups but it doesn't seem to work in a "code" section.

This is what I see in "zpool history"

2024-04-17.14:35:24 zpool offline pool20 Slot_30
2024-04-17.14:38:14 zpool replace pool20 Slot_29 Slot_30
2024-04-22.15:20:30 zpool import -c /etc/zfs/zpool.cache -aN
2024-04-22.16:31:25 zpool offline pool20 Slot_30
2024-04-22.16:43:14 zpool replace -f pool20 Slot_30 Slot_46
2024-04-22.16:53:49 zpool offline pool20 Slot_30
2024-04-22.16:54:57 zpool replace -f pool20 Slot_30 Slot_30
2024-04-22.16:56:18 zpool offline pool20 Slot_30
2024-04-25.11:24:12 zpool clear pool20
2024-04-25.11:51:25 zpool scrub pool20
2024-04-26.13:00:57 zpool import -c /etc/zfs/zpool.cache -aN

I obviously did something wrong. Right now, I'd like to try to get it back to where it should be but I'd also like to figure out what I did wrong so it doesn't happen again.

If anyone has any ideas, please let me know.

Thanks,

Steve

[cousins]

Any ideas on this?

@scineram I noticed the thumbs down. I'm curious why.

Thanks,

Steve

[tonyhutter]

Hmm... I'm not sure how this was possible:

2024-04-17.14:35:24 zpool offline pool20 Slot_30
2024-04-17.14:38:14 zpool replace pool20 Slot_29 Slot_30

When I do the same thing with zfs-2.1.15, I get:

$ zpool offline tank U30
$ zpool replace tank U29 U30
invalid vdev specification
use '-f' to override the following errors:
/dev/disk/by-vdev/U30 is part of active pool 'tank'

I can't even force it:

$ zpool replace -f tank U29 U30
invalid vdev specification
the following errors must be manually repaired:
/dev/disk/by-vdev/U30 is part of active pool 'tank'

My only thoughts:

1. If possible, try replacing Slot_46 with another free disk (other than Slot_30). That might get it out of it's weird state. If successful, then do a zpool online of Slot_30 or a zpool replace pool20 Slot_30.

2. If 1 is not successful, try force-faulting Slot_46: zpool offline -f pool20 Slot_46 and then replace it with itself zpool replace pool20 Slot_46. If that doesn't work, you can clear the force-faulted state with zpool clear pool20 Slot_46

[cousins]

I forgot about that issue with it not replacing the drive, even with -f. I ran into that issue too. I wrote a script to wipe the metadata from the disk prior to replacement:

dd if=/dev/zero of=$DEV bs=1M count=10
dd if=/dev/zero of=$DEV bs=512 count=2048 seek=$((`blockdev --getsz $DEV` - 2048)) 

Thanks for the ideas.

Best,

Steve

[cousins]

This has been resolved using "detach" a couple of times. I tried a number of things and finally tried:

zpool detach pool20 Slot_30

This got rid of the "old" entry in the "replacing-9" section so that there then were just Slot_30 and Slot_46 in it and Slot_30 was also still listed in raidz2-3. I then tried it again and it got rid of the "replacing-9" section all together and raidz2-2 was "ONLINE" with all 10 drives "ONLINE" and Slot_30 now listed just once as OFFLINE in raidz2-3.

Finally, I replaced Slot_30 with itself with:

zpool replace pool20 Slot_30 Slot_30

and it is resilvering. All looks to be back where it should be.

Steve