FDE for Proxmox VE ZFS Raid 1 on NVME

[chris-neely]

Hello, after following the steps, i have everything configured as detailed but run into the following error:

root@prox:~# update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-6.8.12-8-pve
cryptsetup: ERROR: Couldn't resolve device rpool/ROOT/pve-1
cryptsetup: WARNING: Couldn't determine root device
Running hook script 'zz-proxmox-boot'..
Re-executing '/etc/kernel/postinst.d/zz-proxmox-boot' in new private mount namespace..
WARN: /dev/disk/by-uuid/156E-2EC4 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
WARN: /dev/disk/by-uuid/156E-8444 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping

root@prox:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios
WARN: /dev/disk/by-uuid/156E-2EC4 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping
WARN: /dev/disk/by-uuid/156E-8444 does not exist - clean '/etc/kernel/proxmox-boot-uuids'! - skipping

Here are the values in the config files:

root@prox:~# blkid | grep "/dev/[a-z0-9]*1" | grep -v "/dev/zd"
/dev/nvme0n1: UUID="85dc593c-d726-4c3c-bc18-8b6d1eb78ac0" TYPE="crypto_LUKS"
/dev/nvme1n1: UUID="59201688-04e1-4327-b28d-0fcf582cf723" TYPE="crypto_LUKS"

root@prox:~# cat /etc/crypttab
# <target name> <source device>         <key file>      <options>
luks-nvme0n1 UUID=85dc593c-d726-4c3c-bc18-8b6d1eb78ac0 none luks,discard,initramfs
luks-nvme1n1 UUID=59201688-04e1-4327-b28d-0fcf582cf723 none luks,discard,initramfs

root@prox:~# cat /etc/kernel/cmdline
cryptdevice=/dev/nvme0n1:luks-nvme0n1 cryptdevice=/dev/nvme1n1:luks-nvme1n1 root=ZFS=rpool/ROOT/pve-1 resume=/dev/mapper/luks-nvme0n1 resume=/dev/mapper/luks-nvme1n1 boot=zfs

root@prox:~# cat /etc/initramfs-tools/modules 
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
dmcrypt

[mr-manuel]

Check the UUID's of the partition(s)

lsblk -o NAME,PATH,UUID,WWN,MOUNTPOINTS,FSTYPE,LABEL,MODEL,SERIAL | grep -v "/dev/zd"

and remove all other from /etc/kernel/proxmox-boot-uuids. Then run update-initramfs -u -k all and proxmox-boot-tool status again.

[chris-neely]

So maybe that's the issue. After going through the ZFS steps and enabling LUKS I no longer see the partitions:

nvme0n1        /dev/nvme0n1             85dc593c-d726-4c3c-bc18-8b6d1eb78ac0 nvme.1987-3745463630373231304536383030303838373835-53616272656e74-00000001             crypto_LUKS       Sabrent              7EF607210E6800088785
└─luks-nvme0n1 /dev/mapper/luks-nvme0n1 13299945771131656196                                                                                                        zfs_member  rpool       

nvme1n1        /dev/nvme1n1             59201688-04e1-4327-b28d-0fcf582cf723 nvme.1987-3745393630373231304542453030313233343236-53616272656e74-00000001             crypto_LUKS       Sabrent              7E9607210EBE00123426
└─luks-nvme1n1 /dev/mapper/luks-nvme1n1 13299945771131656196                                                                                                        zfs_member  rpool   

root@prox:~# zpool status
  pool: rpool
 state: ONLINE
  scan: resilvered 1.88G in 00:00:03 with 0 errors on Thu Feb 27 13:03:14 2025
config:

        NAME              STATE     READ WRITE CKSUM
        rpool             ONLINE       0     0     0
          mirror-0        ONLINE       0     0     0
            luks-nvme0n1  ONLINE       0     0     0
            luks-nvme1n1  ONLINE       0     0     0

[chris-neely]

My original pool looked like this:

root@prox:~# zpool status
  pool: rpool
 state: ONLINE
config:

        NAME                                                                                       STATE     READ WRITE CKSUM
        rpool                                                                                      ONLINE       0     0     0
          mirror-0                                                                                 ONLINE       0     0     0
            nvme-nvme.1987-3745463630373231304536383030303838373835-53616272656e74-00000001-part3  ONLINE       0     0     0
            nvme-nvme.1987-3745393630373231304542453030313233343236-53616272656e74-00000001-part3  ONLINE       0     0     0

errors: No known data errors

Should the LUKS and the zpool commands be run against the parition 3 only?

[mr-manuel]

You did not delete only the third partition but the whole disk.

nvme0n1     259:0    0   1.8T  0 disk
├─nvme0n1p1 259:1    0  1007K  0 part
├─nvme0n1p2 259:2    0     1G  0 part
└─nvme0n1p3 259:3    0   1.8T  0 part

Instead of

cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --hash sha512 --use-random /dev/nvme0n1p3

you did

cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --hash sha512 --use-random /dev/nvme0n1

Do you have any data on the disks? If not it would be the easiest to make a reinstall.

[chris-neely]

Yup, i suspect that's the issue. Just as a heads up your commands do not contain paritions:

zpool offline rpool sdb3

cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --hash sha512 --use-random /dev/sdb3

[chris-neely]

Should this mapping be the partition 3 as well specifically?

cryptsetup luksOpen /dev/sdb3 luks-sdb3

[mr-manuel]

No, I found some missleading informations in the how to and I'm reworking it right now.

The partitions should already be present in all commands?

[chris-neely]

It's only the disk showing in every commands:

zpool offline rpool sdb3
cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 512 --hash sha512 --use-random /dev/sdb3
cryptsetup luksOpen /dev/sdb3 luks-sdb3
zpool replace rpool sdb3 /dev/mapper/luks-sdb3

luks-sda3 UUID="<disk UUID>" none luks,discard,initramfs
luks-sdb3 UUID="<disk UUID>" none luks,discard,initramfs
... eventually further disks of the root partition/rpool (root ZFS partition)

# For ZFS (RAID1), ZFS (RAID10), ZFS (RAIDZ-1), ZFS (RAIDZ-2), ZFS (RAIDZ-3)
# You need to add all disks of the rpool (root ZFS pool)
echo 'cryptdevice=/dev/sda3:luks-sda3 cryptdevice=/dev/sdb3:luks-sdb3 root=ZFS=rpool/ROOT/pve-1 resume=/dev/mapper/luks-sda3 resume=/dev/mapper/luks-sdb3 boot=zfs' > /etc/kernel/cmdline

[mr-manuel]

sdb is the disk and sdb3 is the 3rd partition of the disk sdb. So in all this commands partitions are mentioned and not disks. The documentation was updated to hopefully provide more clarity.

[chris-neely]

yeah, for some reason i was thinking of the NVME structure where it has the p before the partition! Thanks for the patience!

[mr-manuel]

No problem, you're welcome.

[SWORDIntel]

IM ttrying to automate the damn thing,iits killing me inside
https://github.com/SWORDIntel/LUKS2-ON-PROXMOX

[mr-manuel]

What are you trying to automate? The whole process from non encrypted disk to encrypted disk?