Damaged(?) pools get partially set up and cannot be recovered without booting into rd.rescue

[refi64]

So, sometimes I get kernel panics, for various reasons that look like "novelty hardware with RE'd drivers sometimes causes weird problems". Relatively often, when this happens, I cannot actually reboot back into the system.

I have this on my kernel command line:

root=/dev/stratis/fedora/root stratis.rootfs.pool_uuid=fccd3f78-2670-40b6-8e51-f1c326a4147b

And the system...starts booting normally. Notably, I can unlock the rootfs and have access to its contents (though writes seemingly always get discarded for ??? reasons). However, after the rootfs is mounted, nothing else in the fstab can be, resulting in boot failure:

(I'm using stratis-min here because dbus isn't running yet at this point in boot.)

The full service logs are:

(these screenshots overlap on their lower and upper portions, respectively)

Somehow it seems to "set up" a pool that needs repairing?

The only way for me to fix this is to boot with rd.rescue and do:

$ stratis-min pool start --name fedora --token-slot any --prompt
$ stratis-min pool stop --name fedora

The first command here takes a good 30 seconds to run but completes successfully, and then after running the second and rebooting, the pool is mounted normally, with all its contents intact. If I forget the second command, it still won't work.

[refi64]

(also apologies for all this being screenshots, I couldn't find my USB stick to try and offload log files)

[jbaublitz]

Hi @refi64 we have seen this kind of error before in the case where the metadata and the actual devicemapper devices on the system have diverged. I believe this may be a similar case given the kernel panics you mentioned. I think the first step for us is to determine why this is occurring. From a quick look at the logs, I believe that it's possible in a case where there's a repair action that something is causing the metadata to diverge from the tables actually loaded. What I'm guessing is happening here is that, as I believe based on previous communication you are using the dracut module, the dracut module may be running some kind of repair on the pool when it sets it up for the root filesystem but is failing to save the metadata. Then stratisd is started outside of the initrd and reloads the metadata which does not match the devices set up after the repair. Do you have the logs from the initrd to confirm that that is what is happening?

[jbaublitz]

@mulkieran I'm going to take a look at our repair code and see if I can find anything particularly interesting.

[mulkieran]

@refi64 If you still have the pool that came up, would you be able to post the output of the command stratis pool debug get-metadata --UUID=<your pool uuid> and attach that to the issue?

[mulkieran]

This is related to, but very different from, #3520 .

[refi64]

@mulkieran currently, the output of that is

{"name": "fedora", "backstore": {"data_tier": {"blockdev": {"allocs": [[{"parent": "bb5fbb38-2263-4cc2-bfba-ef6d3a961a30", "start": 8192, "length": 7791183872}]], "devs": [{"uuid": "bb5fbb38-2263-4cc2-bfba-ef6d3a961a30"}]}}, "cap": {"allocs": [[0, 7791183872]]}}, "flex_devs": {"meta_dev": [[1638400, 32768]], "thin_meta_dev": [[0, 32768], [1671168, 7733248]], "thin_data_dev": [[65536, 1572864], [17137664, 7774046208]], "thin_meta_dev_spare": [[32768, 32768], [9404416, 7733248]]}, "thinpool_dev": {"data_block_size": 2048, "feature_args": ["error_if_no_space", "no_discard_passdown", "skip_block_zeroing"], "fs_limit": 100, "enable_overprov": true}, "started": true}