Restore pool configuration to prior to incorrectly added device.

[shousper]

Hello,

I've fallen victim to this mistake, and before researching more carefully how to go about rectifying it, I fell victim to this mistake and now have an unimportable pool with an unavailable device. Worse still, I had already erased the partition table in preparation to move data around to re-create the pool before realising my first mistake.

Is it at all possible to revert the configuration to it's state prior to the device being incorrectly added? Nothing was written to the new device, it was never used, so I'm hoping there might be a way to rollback the configuration or manually hack it by hand?

This is the current state of the pool:

$ sudo zpool import -F
   pool: tank
     id: 11002558381027971913
  state: UNAVAIL
 status: One or more devices are faulted.
 action: The pool cannot be imported due to damaged devices or data.
 config:

        tank                                          UNAVAIL  insufficient replicas
          raidz1-0                                    ONLINE
            ata-WDC_WD40EFPX-68C6CN0_WD-WX12A82EDEVN  ONLINE
            ata-WDC_WD40EFPX-68C6CN0_WD-WX32A826DLN2  ONLINE
            ata-WDC_WD40EFPX-68C6CN0_WD-WX52D62A2D0V  ONLINE
          ata-WDC_WD40EFPX-68C6CN0_WD-WX32A82FTTCJ    FAULTED  corrupted data

Any attempt to import it with any combination of flags yields the same:

cannot import 'tank': one or more devices is currently unavailable

From the 2 discussions I referenced above, I could only try changing zfs_max_missing_tvds with echo 1 > /sys/module/zfs/parameters/zfs_max_missing_tvds, but for some reason I cannot write to it 🤔

I'm currently using a standard install of [email protected] installed via apt on [email protected] -- which I see is significantly behind the current release. If there is anything to be gained in upgrading to help resolve this, please let me know.

Thanks in advance 🙏🏻

[rincebrain]

That parameter doesn't exist on a ZFS version that ancient. Try a newer one.

[rincebrain]

It's not avoidable at this point, really.

Even if you could hack the configuration so the vdev list no longer mentioned the absent disk, there's lots of spots on disk that mention the vdev that no longer is there.

It will, in general, try to warn you not to do that unless you use -f, but if you use -f, nothing's going to stop you.

[shousper]

Even if you could hack the configuration so the vdev list no longer mentioned the absent disk, there's lots of spots on disk that mention the vdev that no longer is there.

Sounds fun!

How many places on disk? Could updates potentially be scripted? How would one go about locating them all?

Where is the config specifying the vdev?

Sorry, that's a lot more questions 😅 I do really appreciate you taking the time though 🖖

[rincebrain]

Every block pointer on disk contains a reference to which vdev the data in it is from, and in turn, every metadata block has its checksum stored in the parent metadata block, and so on.

So you'd get to rewrite rather a lot of data to try and fudge this on-disk, and that's just off the top of my head.

[shousper]

Okay, could be a bit of a challenge, haha.

As I said, no data has been written to the accidentally added device -- at least not outside of zpool interactions.

When a device is added to the pool, are all the blocks and their metadata rewritten by the add (or attach?) subcommand to include the new vdev? Or is the vdev only added to blocks with data on the new device?

If it is rewritten when the command runs to add the new device, I could maybe reverse engineer that process from the source. Depending how complex it is.

If it's not rewritten, perhaps there are a lot less blocks, but I'd still need help identifying them..

Honestly, I may need help identifying a few things, like where in the source the block and metadata format is specified, and where the CLI command source links through to the code that actually writes to disk. I'm not familiar at all with how kernel modules work (or the current state of C development 😅), though I do have a bunch of time up my sleeve.

That being said, perhaps you're right and it's not worth the time versus recreating the pool. Especially with only 3 people potentially running into this scenario. I feel I've already taken too much of your time.

[Fatimah1986]

Yes I want to

[IvanVolosyuk]

You may have to have a look at ZFS on disk format. This is the first link in Google on this topic: http://www.giis.co.in/Zfs_ondiskformat.pdf You may also want to put sparse images of the disks of the broken pool on ZFS and snapshot them first before doing experiments on the data. You may need a lot of extra space for that. Have a look at zdb command if you didn't do that yet.

…

On Sun, 15 Jan 2023, 13:13 Corey McGregor, ***@***.***> wrote: Okay, could be a bit of a challenge, haha. As I said, no data has been written to the accidentally added device -- at least not outside of zpool interactions. When a device is added to the pool, are all the blocks and their metadata rewritten by the add (or attach?) subcommand to include the new vdev? Or is the vdev only added to blocks with data on the new device? If it is rewritten when the command runs to add the new device, I could maybe reverse engineer that process from the source. Depending how complex it is. If it's not rewritten, perhaps there are a lot less blocks, but I'd still need help identifying them.. Honestly, I may need help identifying a few things, like where in the source the block and metadata format is specified, and where the CLI command source links through to the code that actually writes to disk. I'm not familiar at all with how kernel modules work (or the current state of C development 😅), though I do have a bunch of time up my sleeve. That being said, perhaps you're right and it's not worth the time versus recreating the pool. Especially with only 3 people potentially running into this scenario. I feel I've already taken too much of your time. — Reply to this email directly, view it on GitHub <#14389 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HJXX6DHTQ3VO7HQISTWSNMMFANCNFSM6AAAAAAT2KBRMU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[shousper]

Hi @IvanVolosyuk, thanks for the direction 🙏🏻

I may be wrong, but it appears zdb unfortunately only works with imported/importable pools? 🤷🏻 I tried a few options (-C, -d, -S from the manpage examples) and they all responded with: "zdb: can't open 'tank': No such file or directory"

The on disk format document looks quite helpful though, thanks!

[rincebrain]

-e is for exported pools.

…

On Sun, Jan 15, 2023 at 6:33 PM Corey McGregor ***@***.***> wrote: Hi @IvanVolosyuk <https://github.com/IvanVolosyuk>, thanks for the direction 🙏🏻 I may be wrong, but it appears zdb unfortunately only works with imported/importable pools? 🤷🏻 I tried a few options (-C, -d, -S from the manpage examples) and they all responded with: "zdb: can't open 'tank': No such file or directory" The on disk format document looks quite helpful though, thanks! — Reply to this email directly, view it on GitHub <#14389 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7LGQMZHPOU34N4BETTWSSCMPANCNFSM6AAAAAAT2KBRMU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shousper]

Thanks, I'm currently waiting for zpool import -F -X to see if it can tell me anything useful.

[rincebrain]

You're aware zpool import -F -X is very destructive, right?

…

On Sun, Jan 15, 2023 at 8:43 PM Corey McGregor ***@***.***> wrote: Thanks, I'm currently waiting for zpool import -F -X to see if it can tell me anything useful. — Reply to this email directly, view it on GitHub <#14389 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7PZBWVMO262MYR7DRLWSSRVLANCNFSM6AAAAAAT2KBRMU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[shousper]

Yup 🙂 I've already pulled my data onto another disk so I could experiment with this problem. Looks like it's happily found a previous state it could import -- I'm not sure how to tell when it's from, but I can see it's from before I upgraded the features on the pool.

$ zpool status
  pool: tank
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
  scan: resilvered 684K in 00:00:01 with 0 errors on Thu Jan 12 22:48:15 2023
config:

	NAME                                          STATE     READ WRITE CKSUM
	tank                                          ONLINE       0     0     0
	  raidz1-0                                    ONLINE       0     0     0
	    ata-WDC_WD40EFPX-68C6CN0_WD-WX12A82EDEVN  ONLINE       0     0     0
	    ata-WDC_WD40EFPX-68C6CN0_WD-WX32A826DLN2  ONLINE       0     0     0
	    ata-WDC_WD40EFPX-68C6CN0_WD-WX52D62A2D0V  ONLINE       0     0     0

errors: No known data errors

I'm just running a checksum comparison of all the data (rsync -avcP --dry-run /mnt/recovery /mnt/tank) to see if it's rolled back far enough to lose more data, or if it's rolled back just far enough to correct the pool configuration by dropping the dud device.

[shousper]

It would appear zpool import -F -X has successfully rolled back the configuration without destroying any data. I was now able to successfully add the remaining device as a spare to the pool 👍🏻

I was surprised to learn one cannot add a device to an existing raidz configuration though? Not to worry.

Thanks for all the help 🙏🏻

[rincebrain]

Check out #12225 for that feature.