Migration from old zfs to zfs 2.1.5, adding encryption in the meantime...what an adventure :-)

[ofthesun9]

Hello !

I wanted to share my experience with my recent modification on my server (hobbyist).

Step 0, long time ago: The pool "Pool1" was initially created with nas4free in 2014. (4 disk, 2x2mirrored)

Step 1, in 2020: I switched the OS to ubuntu 20.04, installed on its own SSD ((lvm ext4). zfs 0.8.3

Step 2 I decided to move the OS on the Pool1 itself, on an unencrypted dataset. Migration done with rsync. Changing the bootloader from grub to zfsbootmenu. This was working very nicely

Step 3: I decided to move the OS from the unencrypted dataset to an encrypted one (using syncoid to send/recv)
=> zfsbootmenu (using zfs 2.1.5) was able to import the Pool at the first boot, but not anymore after that. "Input/Output error", which is of course mandatory to find the kernel and the initramfs to boot the OS.
=> digging in all discussion and bug report, I finally found a workaround: After the dataset recv operation, I did the first mount/modification of the dataset with zfs 2.1.5. As soon as the first mount/modification of the encrypted dataset was done by zfs 0.8.3, zfsbootmenu (zfs 2.15) was unable to import the Pool anymore (Input/Output error)

Step 4: I noticed that zpool status was reporting some corruption error

errors: Permanent errors have been detected in the following files:

        <0x13fbf>:<0x0>

I cannot tell if the corruption happened before or after I started to use sanoid/syncoid to send backup of the encrypted dataset to another dataset

Step 5: I moved to jonathon's ppa to have ubuntu 20.04 using zfs 2.1.5. I ran scrub on the Pool that finally cleared the error

Step 6 : 16 days later, another corruption is detected (currently running scrub). I have removed all snapshots created with zfs 0.8.3

I guess my next step could be to send/recv again the encrypted dataset, as it would be done under zfs 2.1.5 this time ?