How to use clevis for ZFS native encryption passphrase?

[rdmitry0911]

Hi guys, ZFSBootMenu is OS-agnostic, has small footprint and is maintenance easy. Set it and forget it kind of things. To boot a system from an encrypted zfs root ZFSBootMenu at the moment can get a passphrase remotely via dropbear and localy from a console input. So all we need to make it full featured is to attach clevis to it to get passphrase automatically in trusted environment. What is the best way to make it?

[ahesford]

I don't know how Clevis works, but the best approach would probably be using it to unlock keys on another partition that you could then copy to the path specified in the filesystems' keylocation property (or just mount the unlocked key filesystem where ZBM can find the keys). This could be accomplished in an early-setup hook that runs before ZBM attempts to import pools and scan filesystems.

Deeper integration with ZBM and Clevis is unlikely to happen because it would require special handling of specific, nonstandard unlocking methods.

[rdmitry0911]

From the clevis landing page in github: "Clevis is a pluggable framework for automated decryption" As far as clevis is a set of bash scripts and depends only on a broadly used packages: libc6, libjansson4, libjose0, libssl3, cracklib-runtime, curl, jose, libpwquality-tools, luksmeta, I think it is possible to embed clevis in custom build ZFSBootMenu. Unfortunately I didn't find clevis package in void OS repository to try it out. May be there are other distributions that support clevis where I can make a custom built ZFSBootMenu?

[zdykstra]

ZFSBootMenu can be built on Debian, Ubuntu, Alpine, openSuSE and Fedora. Though it's probably possible to get Clevis working on Void. Either way, a custom early hook that populates a known directory with key files will be the best option - then the normal ZBM key handling mechanism will 'just work'.

[rdmitry0911]

ZFSBootMenu can be built on Debian, Ubuntu, Alpine, openSuSE and Fedora. Though it's probably possible to get Clevis working on Void. Either way, a custom early hook that populates a known directory with key files will be the best option - then the normal ZBM key handling mechanism will 'just work'.

Thanks for the tips. Does that mean, that if I build ZFSBootMenu in Ubuntu, than the resulting ZFSBootMenu will be Ubuntu-based and I can embed there some Ubuntu packages during the build process?

[zdykstra]

That's correct, if you don't use the containerized build process. Refer to https://docs.zfsbootmenu.org/en/v2.2.x/guides/ubuntu/uefi.html#install-zfsbootmenu for the list of packages / dependencies that should be installed if you're going to build from source.

[rdmitry0911]

As far as I would like to make zfsbootmenu to automaticaly boot a system from a trused environment and ask a passphrase via remote access or from a local console if the environment is not trusted, clevis framework seems to be the optimal solution for this. The only system I found that has support for

• custom building zfsbootmenu
• clevis package
• initramfs remote access package

is Fedora.
While building zfsbootmenu with remote access in Fedora I've found some inconsistences in the documentation:

• While installing zfs this command should be used, as the version of zfs was changed: dnf --releasever=${VERSION_ID} install -y https://zfsonlinux.org/fedora/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
• While in chroot this command has to be run to avoid errors while building zfsbootmenu: dnf install -y systemd-boot-unsigned
• This line: add_dracutmodules+=" network-legacy " should be added to /etc/zfsbootmenu/dracut.conf.d/dropbear.conf to avoid errors while building zfsbootmenu with remote accesss

Now I'm in a point where I have to add clevis to initramfs but it depends on systemd, which is excluded from dracut modules. How safe is it to drop this restriction and let dracut to install systemd to initramfs?

[zdykstra]

Unless things have changed recently, the Dracut systemd modules will break the ZFSBootMenu initramfs image. #81 . You can attempt to add them, but I fully expect the generated image to be incorrect / non-functional.

Regarding your three notes:

1. I've updated the URL to the RPM that adds the package repository
2. Is this under Fedora 37? If so, what specific error(s) were you getting?
3. What's the URL to the documentation that you were following?

[rdmitry0911]

Unless things have changed recently, the Dracut systemd modules will break the ZFSBootMenu initramfs image. #81 . You can attempt to add them, but I fully expect the generated image to be incorrect / non-functional.

Ok, got it, thank you

Regarding your three notes:

1. I've updated the URL to the RPM that adds the package repository

2. Is this under Fedora 37? If so, what specific error(s) were you getting?

No, it is Fedora 38
Errors without network-legacy:

[root@fedora ~]# generate-zbm 
No initramfs generator specified; using dracut
Creating ZFSBootMenu 2.2.0 from kernel /boot/vmlinuz-6.3.11-200.fc38.x86_64
dracut: dracut module 'dbus-broker' depends on 'systemd', which can't be installed
dracut: dracut module 'dbus' depends on 'dbus-broker', which can't be installed
dracut: dracut module 'network-manager' depends on 'dbus', which can't be installed
dracut: dracut module 'network' depends on 'network-manager', which can't be installed
dracut: dracut module 'ifcfg' depends on 'network', which can't be installed
dracut: dracut module 'memstrack' depends on 'systemd', which can't be installed
dracut: dracut module 'crypt-ssh' cannot be found or installed.
Failed to create /tmp/3JAteQiTQP/zfsbootmenu.img
[root@fedora ~]# 

Errors without systemd-boot-unsigned:
Unable to find UEFI stub loader at default locations:

3. What's the URL to the documentation that you were following?

This one

[zdykstra]

The steps in the Fedora 37 guide are only valid/tested for that exact release. The guide will need to be re-validated and updated for Fedora 38.

How was dracut-crypt-ssh installed? Did you install it from source, or use a Fedora package? Were you following https://docs.zfsbootmenu.org/en/v2.2.x/guides/general/remote-access.html#dracut or did you use some other documentation? Basically, I'm trying to figure out what exactly needs to be updated / where to make that process easier going forward.

[rdmitry0911]

The steps in the Fedora 37 guide are only valid/tested for that exact release. The guide will need to be re-validated and updated for Fedora 38.

How was dracut-crypt-ssh installed? Did you install it from source, or use a Fedora package?

I used Fedora package as it is written here

dnf copr enable uriesk/dracut-crypt-ssh
dnf install dracut-crypt-ssh

Were you following https://docs.zfsbootmenu.org/en/v2.2.x/guides/general/remote-access.html#dracut or did you use some other documentation?

Yes, I used the one you mentioned.

[rdmitry0911]

I managed to make zfsbootmenu booter with clevis support. It is arch linux based. Here is how it works now

From inside zfsbootmenu during first boot I bind zfs pool password to tpm2. I bind it to several pcr_ids including pcr_ids 9. This pcr_id is used to measure initramfs. In our case it measures zfsbootmenu itself. I used this command:
echo $POOL_PASSWORD|clevis encrypt tpm2 '{"pcr_ids":"1,4,5,7,9","pcr_bank":"sha256"}' > zfs.jwe
and then I store zfs.jwe in zfs pool user attributes

zfs set latchset.clevis:decrypt=yes $ZFS_DATASET
zfs set latchset.clevis:jwe=$(cat zfs.jwe) $ZFS_DATASET

Next boot I can manually unlock the pool from inside zfsbootmenu using this command:

JWE="$(zfs get -H -p -o value latchset.clevis:jwe -s local $ZFS_DATASET)"
echo $JWE | clevis decrypt | zfs load-key -L prompt $ZFS_DATASET

The pool will be unlocked ONLY if all pcr_ids remain the same as they were when we bound our pool password to tpm2 device. As far as I didn't make any changes nither to hardware nor to zfsbootmenu booter, I have unlocked pool and can boot the system.

Now I'd like to make all this automatically. The sequence should be like this:

• In zfsbootmenu before asking the password I'm trying to unlock the pool using clevis
• On failure
  -> Print a warning that the pool can not be unlocked automatically
  -> Ask the pool password
  -> Ask the permission to use this password for automatic pool unlocking from now on
  -> If yes, use clevis to bind this password to tpm2 for automated pool unlocking next time
• boot the system as far as at this poing we have pool unlocked.

With this little automation the system will boot up automatically unless somebody is trying to intrude into the boot process. In this case we will know about that and do whatever is good.

So my question is where is the place in zfsbootmenu to put this piece of code?

[ahesford]

You can create an early-setup hook that will run before ZFSBootMenu attempts to import any pools. This script can do whatever you like to get the pool to an importable state.

[rdmitry0911]

You can create an early-setup hook that will run before ZFSBootMenu attempts to import any pools. This script can do whatever you like to get the pool to an importable state.

In this case this hook will import and decrypt all the encrypted pools with automatic decryption flag set to yes. Would it be fine for rurther processing in zfsbootmenu?

[ahesford]

ZBM will probably break if zbm.prefer is set on the command line to a pool that you import in an early-setup hook, because it will attempt to import the desired pool and then fail when it finds that the pool is already imported. Just leave off this option and you should be fine, but you should really read through the initialization script and also do some experiments before you attempt to hijack ZBM import behavior.

[rdmitry0911]

ZBM will probably break if zbm.prefer is set on the command line to a pool that you import in an early-setup hook, because it will attempt to import the desired pool and then fail when it finds that the pool is already imported. Just leave off this option and you should be fine, but you should really read through the initialization script and also do some experiments before you attempt to hijack ZBM import behavior.

Ok, I made it, Now it is fully automated and independent of the system it is booting. No maintenance needed unless something's changed in hardware configuration or there is a need to update zfsbootmenu itself. However, even though early_setup_hook is a working solution, it is not too much convenient for setting up. First, for the new installation of the booter It needs user interaction to provide a passphrase to seal it with clevis and tpm2. And if user does that via ssh he has to manually start the hook as it runs before zfsbootmenu. Second, If the hook is already running on the console and wait for user response while user is working over ssh, he needs to find this console running hook from ssh session, kill it, export all imported pools and then start the hook over from ssh. This makes the logic of the hook extra heavy. The optimal solution would be to call a special user defined hook just before asking a passphrase for a datset. It can be done based on the special user propery flag of this dataset, say latchset.clevis:decrypt. The hook is executed only if it is set to yes. The only argument for this hook is dataset name. In this case the logic of the hook would be very simple and setting it up will be convenient. Any help on how to make it easy would be very much appreciated.

[ahesford]

There isn't a convenient way around this because we only run hooks at three points: right before everything will be imported, at the start of the zfsbootmenu run loop, and right before a kernel is about to be booted via kexec.

you shouldn't try to work out the logic of one-shot initialization in your hook; you should do this once, at the console, and be done with it. Otherwise, if you need to guarantee that only a single instance of the hook is running, you have no choice but to implement your own locking and process management in the hook.

[rdmitry0911]

There isn't a convenient way around this because we only run hooks at three points: right before everything will be imported, at the start of the zfsbootmenu run loop, and right before a kernel is about to be booted via kexec.

you shouldn't try to work out the logic of one-shot initialization in your hook; you should do this once, at the console, and be done with it. Otherwise, if you need to guarantee that only a single instance of the hook is running, you have no choice but to implement your own locking and process management in the hook.

I made a dirty hack in load_key() module in zfsbootmenu-core.sh It is very simple and just works. Just before

  # Default to 0 when unset
  [ -n "${CLEAR_SCREEN}" ] || CLEAR_SCREEN=0
  [ -n "${NO_CACHE}" ] || NO_CACHE=0

I put a call to a small function
load_key_clevis "${encroot}"
This function provides automatic decryption of the dataset using clevis and tpm2. Everything works now as expected. I checked it via console and via ssh. I forked ZBM and put modified zfsbootmenu-core.sh there. If anybody is interested in such a functionality, I prepared a pr for it