grub can boot from a single pool without disabling features, and it can include encrypted datasets with an additional patch

[digitalsignalperson]

(just posted this to the grub mailing list and thought I'd share it here to see if anyone has any thoughts)

I've found that grub can boot from a single pool without explicitly disabling any ZFS features (as long as unimplemented features aren't active). Furthermore, disabling [grub's] check_mos_features allows for booting from a pool where encrypted datasets may be present, but where grub only needs to read from unencrypted datasets. I'm wondering if a change can be made [to grub] to relax these restrictions around ZFS feature checking. This would simplify many use cases and reduce the need to use multiple pools with grub.
Here's an example (zfs 2.1.12) where grub has no problem booting with kernels and initramfs from rpool/sys/BOOT/default as shown below:

zpool create -f \
    -o ashift=12 \
    -O acltype=posixacl \
    -O canmount=off \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O atime=off \
    -O xattr=sa \
    -o autotrim=on \
    -O mountpoint=none \
    -o cachefile=none \
    -R ${ALTROOT} \
    ${RPOOLNAME} ${RPOOLDEV}

zfs create -o canmount=off -o mountpoint=none ${RPOOLNAME}/sys
zfs create -o canmount=noauto -o mountpoint=/ ${RPOOLNAME}/sys/ROOT/default
zfs create -o canmount=off -o mountpoint=none ${RPOOLNAME}/sys/BOOT
zfs create -o mountpoint=/boot ${RPOOLNAME}/sys/BOOT/default

I didn't think this was possible since all the ZFS-on-root guides follow the pattern of creating a bpool and rpool, where bpool will only enable specific features compatible with grub.

Let's change the dataset config so we encrypt everything except BOOT:

echo "12345678" | zfs create -o canmount=off -o mountpoint=none -o encryption=on -o keylocation=prompt -o keyformat=passphrase ${RPOOLNAME}/sys
zfs create -o canmount=off -o mountpoint=none ${RPOOLNAME}/sys/ROOT
zfs create -o canmount=noauto -o mountpoint=/ ${RPOOLNAME}/sys/ROOT/default
zfs create -o canmount=off -o mountpoint=none -o encryption=off ${RPOOLNAME}/sys/BOOT
zfs create -o canmount=noauto -o mountpoint=/boot ${RPOOLNAME}/sys/BOOT/default

Now grub will not be able to boot anything, regardless of the grub.cfg only referencing things in sys/BOOT and not the encrypted datasets. If I remember right it will print "unknown filesystem" when trying to load anything or do anything like an "ls (hd0,gpt4)" on the pool's filesystem. Also note grub-mkconfig will no longer include the insmod zfs and set root= parts, but adding them back manually or using the F2 console and looking at set debug=zfs will further confirm being stuck.

But a small patch fixes this. By ignoring check_mos_features() in zfs_mount(), grub-mkconfig creates a proper grub.cfg again, and I can successfully boot from this unencrypted rpool/sys/BOOT/default, meanwhile rpool/sys is the encryption root with rpool/sys/ROOT/default and anything else encrypted.

diff --git a/grub-core/fs/zfs/zfs.c b/grub-core/fs/zfs/zfs.c
index 0e195db97..c08b367bd 100644
--- a/grub-core/fs/zfs/zfs.c
+++ b/grub-core/fs/zfs/zfs.c
@@ -3705,10 +3705,11 @@ zfs_mount (grub_device_t dev)
  if (ub->ub_version >= SPA_VERSION_FEATURES &&
      check_mos_features(&osp->os_meta_dnode, ub_endian, data) != 0)
    {
-      grub_error (GRUB_ERR_BAD_FS, "Unsupported features in pool");
-      grub_free (osp);
-      zfs_unmount (data);
-      return NULL;
+      grub_dprintf ("zfs", "Ignoring unsupported features\n");
+      // grub_error (GRUB_ERR_BAD_FS, "Unsupported features in pool");
+      // grub_free (osp);
+      // zfs_unmount (data);
+      // return NULL;
    }
 
  /* Got the MOS. Save it at the memory addr MOS. */

Could such a change be accepted? Removing the check entirely seem to have no risk since everything is read-only. And as shown here, prematurely erroring out is wrong because grub could do all the reads it needed to without running into any of the unimplemented features. And other more specific errors are thrown otherwise if grub still fails.

Trying to understand in more detail the ZFS features required for reading data, I looked at what features were disabled on a bpool created with recommended settings for grub:

 feature@multi_vdev_crash_dump
 feature@large_dnode
 feature@sha512
 feature@skein
 feature@edonr
 feature@encryption
 feature@device_removal
 feature@obsolete_counts
 feature@bookmark_v2
 feature@redaction_bookmarks
 feature@redacted_datasets
 feature@bookmark_written
 feature@log_spacemap
 feature@livelist
 feature@device_rebuild
 feature@zstd_compress
 feature@draid

Narrowing these, I asked ChatGPT which of these are required for reading data:

• feature@large_dnode: Enables support for larger dnodes, which improves space efficiency and read performance. Required for reading data stored with large dnodes.
• feature@sha512: Provides support for SHA-512 hash algorithms, which are used for data integrity checks during reads and writes.
• feature@skein: Provides support for Skein hash algorithms, which are used for data integrity checks during reads and writes.
• feature@edonr: Provides support for Edon-R hash algorithms, which are used for data integrity checks during reads and writes.
• feature@encryption: Enables encryption features for data at rest. Required to read encrypted data.
• feature@log_spacemap: Optimizes the handling of log spacemaps, which can improve metadata read performance.

Out of these, the only read-related feature I think I'd ever run into is encryption, and as shown above we can just point grub to read from a non-encrypted dataset and it's happy.

I was also curious about the presence of grub-core/fs/zfs/zfscrypt.c and grub's zfs code having calls to this in various places to seemingly be able to load a key and decrypt data. It has history back to 2011, but I couldn't figure out how it works or if it's dead at this point. Hacking around a bit beyond ignoring the check_mos_features, if I try to read from an encrypted dataset the first error I think comes up is "compression algorithm inherit not supported". I also tried using zfskey and insmod zfscrypt, but didn't get past "no key for txg " amongst other errors.

[IvanVolosyuk]

FYI: Instead of asking chatgpt you can 'man zpool-features' and see which ones are read-only compatible.

…

On Sun, Aug 13, 2023 at 9:03 PM andy ***@***.***> wrote: (just posted this to the grub mailing list and thought I'd share it here to see if anyone has any thoughts) I've found that grub can boot from a single pool without explicitly disabling any ZFS features (as long as unimplemented features aren't active). Furthermore, disabling [grub's] check_mos_features allows for booting from a pool where encrypted datasets may be present, but where grub only needs to read from unencrypted datasets. I'm wondering if a change can be made [to grub] to relax these restrictions around ZFS feature checking. This would simplify many use cases and reduce the need to use multiple pools with grub. Here's an example (zfs 2.1.12) where grub has no problem booting with kernels and initramfs from rpool/sys/BOOT/default as shown below: zpool create -f \ -o ashift=12 \ -O acltype=posixacl \ -O canmount=off \ -O compression=lz4 \ -O normalization=formD \ -O relatime=on \ -O atime=off \ -O xattr=sa \ -o autotrim=on \ -O mountpoint=none \ -o cachefile=none \ -R ${ALTROOT} \ ${RPOOLNAME} ${RPOOLDEV} zfs create -o canmount=off -o mountpoint=none ${RPOOLNAME}/sys zfs create -o canmount=noauto -o mountpoint=/ ${RPOOLNAME}/sys/ROOT/default zfs create -o canmount=off -o mountpoint=none ${RPOOLNAME}/sys/BOOT zfs create -o mountpoint=/boot ${RPOOLNAME}/sys/BOOT/default I didn't think this was possible since all the ZFS-on-root guides follow the pattern of creating a bpool and rpool, where bpool will only enable specific features compatible with grub. Let's change the dataset config so we encrypt everything except BOOT: echo "12345678" | zfs create -o canmount=off -o mountpoint=none -o encryption=on -o keylocation=prompt -o keyformat=passphrase ${RPOOLNAME}/sys zfs create -o canmount=off -o mountpoint=none ${RPOOLNAME}/sys/ROOT zfs create -o canmount=noauto -o mountpoint=/ ${RPOOLNAME}/sys/ROOT/default zfs create -o canmount=off -o mountpoint=none -o encryption=off ${RPOOLNAME}/sys/BOOT zfs create -o canmount=noauto -o mountpoint=/boot ${RPOOLNAME}/sys/BOOT/default Now grub will not be able to boot anything, regardless of the grub.cfg only referencing things in sys/BOOT and not the encrypted datasets. If I remember right it will print "unknown filesystem" when trying to load anything or do anything like an "ls (hd0,gpt4)" on the pool's filesystem. Also note grub-mkconfig will no longer include the insmod zfs and set root= parts, but adding them back manually or using the F2 console and looking at set debug=zfs will further confirm being stuck. But a small patch fixes this. By ignoring check_mos_features() in zfs_mount(), grub-mkconfig creates a proper grub.cfg again, and I can successfully boot from this unencrypted rpool/sys/BOOT/default, meanwhile rpool/sys is the encryption root with rpool/sys/ROOT/default and anything else encrypted. diff --git a/grub-core/fs/zfs/zfs.c b/grub-core/fs/zfs/zfs.c index 0e195db97..c08b367bd 100644--- a/grub-core/fs/zfs/zfs.c+++ b/grub-core/fs/zfs/zfs.c@@ -3705,10 +3705,11 @@ zfs_mount (grub_device_t dev) if (ub->ub_version >= SPA_VERSION_FEATURES && check_mos_features(&osp->os_meta_dnode, ub_endian, data) != 0) {- grub_error (GRUB_ERR_BAD_FS, "Unsupported features in pool");- grub_free (osp);- zfs_unmount (data);- return NULL;+ grub_dprintf ("zfs", "Ignoring unsupported features\n");+ // grub_error (GRUB_ERR_BAD_FS, "Unsupported features in pool");+ // grub_free (osp);+ // zfs_unmount (data);+ // return NULL; } /* Got the MOS. Save it at the memory addr MOS. */ Could such a change be accepted? Removing the check entirely seem to have no risk since everything is read-only. And as shown here, prematurely erroring out is wrong because grub could do all the reads it needed to without running into any of the unimplemented features. And other more specific errors are thrown otherwise if grub still fails. Trying to understand in more detail the ZFS features required for reading data, I looked at what features were *disabled* on a bpool created with recommended settings for grub: ***@***.***_vdev_crash_dump ***@***.***_dnode ***@***.*** ***@***.*** ***@***.*** ***@***.*** ***@***.***_removal ***@***.***_counts ***@***.***_v2 ***@***.***_bookmarks ***@***.***_datasets ***@***.***_written ***@***.***_spacemap ***@***.*** ***@***.***_rebuild ***@***.***_compress ***@***.*** Narrowing these, I asked ChatGPT which of these are required for reading data: - ***@***.***_dnode: Enables support for larger dnodes, which improves space efficiency and read performance. Required for reading data stored with large dnodes. - ***@***.***: Provides support for SHA-512 hash algorithms, which are used for data integrity checks during reads and writes. - ***@***.***: Provides support for Skein hash algorithms, which are used for data integrity checks during reads and writes. - ***@***.***: Provides support for Edon-R hash algorithms, which are used for data integrity checks during reads and writes. - ***@***.***: Enables encryption features for data at rest. Required to read encrypted data. - ***@***.***_spacemap: Optimizes the handling of log spacemaps, which can improve metadata read performance. Out of these, the only read-related feature I think I'd ever run into is encryption, and as shown above we can just point grub to read from a non-encrypted dataset and it's happy. I was also curious about the presence of grub-core/fs/zfs/zfscrypt.c and grub's zfs code having calls to this in various places to seemingly be able to load a key and decrypt data. It has history back to 2011, but I couldn't figure out how it works or if it's dead at this point. Hacking around a bit beyond ignoring the check_mos_features, if I try to read from an encrypted dataset the first error I think comes up is "compression algorithm inherit not supported". I also tried using zfskey and insmod zfscrypt, but didn't get past "no key for txg " amongst other errors. — Reply to this email directly, view it on GitHub <#15169>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HLN4Y5HGFYCYNZRITLXVCX7LANCNFSM6AAAAAA3OSGR5Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[rincebrain]

The list is also, naturally, wrong.

feature@zstd_compress, for example, would absolutely be needed for read if active.

This is expected behavior. If the feature isn't marked active, you don't need to support it to use the pool, but since any feature marked enabled can become active with very little deliberate effort on anyone's part, it'd probably be better to at least print a warning that this can happen if you're going to change the behavior.

(For example, setting compression=zstd-3 on a dataset will fail if the feature@zstd_compress feature is disabled, but will succeed if it's enabled, and then the next time you write something compressible to the dataset, it'll become active, potentially blowing up your ability to read the dataset.)

AFA zfscrypt.c, IIRC Oracle shoved code upstream that can do their native encryption implementation, but that's not, AFAIK, compatible on-disk with OpenZFS's native encryption. You could do the work to make it so, of course, but you couldn't reuse any of the existing CDDL code in the OpenZFS codebase, since GRUB's going to want GPL code.