feat: added additional zfs services to support encrypted volumes

[runningman84]

This should work fine but I am not sure if the dependencies work that way. Please review...

With this config zfs filesystems with encrypted volumes and local keys (for example stored in /var) will be auto mounted.

This is an example from my test system:

root@worker2:~# df
Filesystem                      1K-blocks     Used Available Use% Mounted on
overlay                         975037276 13135376 961901900   2% /
tmpfs                               65536        0     65536   0% /dev
overlay                               256      256         0 100% /host
rootfs                           16242936   104588  16138348   1% /host/.extra
devtmpfs                         16242936        0  16242936   0% /host/dev
tmpfs                            16295300        0  16295300   0% /dev/shm
tmpfs                            16295300        0  16295300   0% /host/proc/acpi
tmpfs                            16295300        0  16295300   0% /host/proc/scsi
efivarfs                              192       92        96  50% /host/sys/firmware/efi/efivars
tmpfs                            16295300     2628  16292672   1% /host/run
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/system/kubelet/rootfs
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/shm
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/shm
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/shm
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/shm
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/6b249ed4e115676c6ac82de699264f07701de5d26398e78d1e4b2b7a184e099f/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a3afd6d213e959957cf5edf2b9db2ae42ddf660121c32399bcbc5ff9bc8886db/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/84ee344d0c6af8b4d9a4be0f0cecb3667229043c61e98003f8f2edf7bcb73271/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/dce879378a9651f24fd5e9c72ae50bed34946d67ed57f0226ec4e8f832275488/rootfs
tmpfs                            16295300      240  16295060   1% /host/system
overlay                               256      256         0 100% /host/system/libexec/apid/apid
/dev/mapper/nvme0n1p3-encrypted     80544     5140     75404   7% /host/system/state
tmpfs                               65536        0     65536   0% /host/tmp
overlay                          16295300      240  16295060   1% /host/usr/etc/udev
/dev/mapper/nvme0n1p4-encrypted 975037276 13135376 961901900   2% /host/var
tmpfs                            32291592        4  32291588   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~secret/memberlist
tmpfs                            32291592        0  32291592   0% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~secret/longhorn-grpc-tls
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/0d0689d2-e263-42b1-82a9-22ad3ade5ccc/volumes/kubernetes.io~projected/kube-api-access-qmg8n
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/eb279ec9-bf74-4102-ab89-3fe44cac45e8/volumes/kubernetes.io~projected/kube-api-access-m2v2n
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/b21d24d3-f21b-4dfe-ae43-4108b9241327/volumes/kubernetes.io~projected/kube-api-access-kwtmz
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~projected/kube-api-access-h5nvx
tmpfs                            32291592       12  32291580   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/784bf64e-0f81-4e15-ae98-d98f055a5748/volumes/kubernetes.io~projected/kube-api-access-lhwhw
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~projected/kube-api-access-s6lfj
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/fbbe174c-8888-4302-a8be-3ffaa7f8fc9b/volumes/kubernetes.io~projected/kube-api-access-jpsv5
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/55c9398f-7fa1-4a01-99fa-7ad13a04e506/volumes/kubernetes.io~projected/kube-api-access-hflnz
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/11bc1b90-1509-4084-bb11-85acd8276f0f/volumes/kubernetes.io~projected/kube-api-access-8t4zh
overlay                         975037276 13135376 961901900   2% /host/etc/cni
overlay                         975037276 13135376 961901900   2% /host/etc/kubernetes
overlay                         975037276 13135376 961901900   2% /host/usr/libexec/kubernetes
overlay                         975037276 13135376 961901900   2% /host/opt
overlay                          16295300      240  16295060   1% /host/usr/local/lib/containers/tgtd
overlay                          16295300      240  16295060   1% /host/usr/local/lib/containers/zpool-importer
overlay                          16295300      240  16295060   1% /host/usr/local/lib/containers/iscsid
root@worker2:~# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank          1.57T  9.21T    96K  /var/hddstorage
tank/private   195G  9.21T   195G  /var/hddstorage/private
tank/public   1.38T  9.21T  1.38T  /var/hddstorage/public
root@worker2:~# zfs load-key -a         
2 / 2 key(s) successfully loaded
root@worker2:~# zfs mount -a
root@worker2:~# df
Filesystem                        1K-blocks       Used  Available Use% Mounted on
overlay                           975037276   13135432  961901844   2% /
tmpfs                                 65536          0      65536   0% /dev
overlay                                 256        256          0 100% /host
rootfs                             16242936     104588   16138348   1% /host/.extra
devtmpfs                           16242936          0   16242936   0% /host/dev
tmpfs                              16295300          0   16295300   0% /dev/shm
tmpfs                              16295300          0   16295300   0% /host/proc/acpi
tmpfs                              16295300          0   16295300   0% /host/proc/scsi
efivarfs                                192         92         96  50% /host/sys/firmware/efi/efivars
tmpfs                              16295300       2628   16292672   1% /host/run
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/system/kubelet/rootfs
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/shm
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/shm
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/shm
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/shm
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/6b249ed4e115676c6ac82de699264f07701de5d26398e78d1e4b2b7a184e099f/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a3afd6d213e959957cf5edf2b9db2ae42ddf660121c32399bcbc5ff9bc8886db/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/84ee344d0c6af8b4d9a4be0f0cecb3667229043c61e98003f8f2edf7bcb73271/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/dce879378a9651f24fd5e9c72ae50bed34946d67ed57f0226ec4e8f832275488/rootfs
tmpfs                              16295300        240   16295060   1% /host/system
overlay                                 256        256          0 100% /host/system/libexec/apid/apid
/dev/mapper/nvme0n1p3-encrypted       80544       5140      75404   7% /host/system/state
tmpfs                                 65536          8      65528   1% /host/tmp
overlay                            16295300        240   16295060   1% /host/usr/etc/udev
/dev/mapper/nvme0n1p4-encrypted   975037276   13135432  961901844   2% /host/var
tmpfs                              32291592          4   32291588   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~secret/memberlist
tmpfs                              32291592          0   32291592   0% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~secret/longhorn-grpc-tls
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/0d0689d2-e263-42b1-82a9-22ad3ade5ccc/volumes/kubernetes.io~projected/kube-api-access-qmg8n
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/eb279ec9-bf74-4102-ab89-3fe44cac45e8/volumes/kubernetes.io~projected/kube-api-access-m2v2n
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/b21d24d3-f21b-4dfe-ae43-4108b9241327/volumes/kubernetes.io~projected/kube-api-access-kwtmz
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~projected/kube-api-access-h5nvx
tmpfs                              32291592         12   32291580   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/784bf64e-0f81-4e15-ae98-d98f055a5748/volumes/kubernetes.io~projected/kube-api-access-lhwhw
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~projected/kube-api-access-s6lfj
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/fbbe174c-8888-4302-a8be-3ffaa7f8fc9b/volumes/kubernetes.io~projected/kube-api-access-jpsv5
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/55c9398f-7fa1-4a01-99fa-7ad13a04e506/volumes/kubernetes.io~projected/kube-api-access-hflnz
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/11bc1b90-1509-4084-bb11-85acd8276f0f/volumes/kubernetes.io~projected/kube-api-access-8t4zh
overlay                           975037276   13135432  961901844   2% /host/etc/cni
overlay                           975037276   13135432  961901844   2% /host/etc/kubernetes
overlay                           975037276   13135432  961901844   2% /host/usr/libexec/kubernetes
overlay                           975037276   13135432  961901844   2% /host/opt
overlay                            16295300        240   16295060   1% /host/usr/local/lib/containers/tgtd
overlay                            16295300        240   16295060   1% /host/usr/local/lib/containers/zpool-importer
overlay                            16295300        240   16295060   1% /host/usr/local/lib/containers/iscsid
tank                             9889162240        128 9889162112   1% /host/var/hddstorage
tank/private                    10093786496  204624384 9889162112   3% /host/var/hddstorage/private
tank/public                     11371600512 1482438400 9889162112  14% /host/var/hddstorage/public

[smira]

I would really love to land some support for mounting/volumes in Talos 1.7 to avoid such workaround if possible.

[runningman84]

Okay if you bring native support for that in 1.7 I am totally fine… if not please consider merging this workaround.

[runningman84]

@smira I played arround with zfs a bit. Zfs encryption in talos is tricky at the moment:

• if you mount an encrypted volume (zfs load-key -a; zfs mount -a) using a debug pod the files are not accessible in another pod
• having a mix of encrypted and unencrypted zfs datasets results in none of them being automounted anymore
• normally all datasets are automated if you use zfs but sometimes this does not happen, so you have to call zfs mount --a , this is also problematic because pods might write data to some local fs instead of the real zfs fs. This can result in data loos.

I think there should be some tests and documentation for zfs usage to make this usable without these issues,

[rdegez]

Hi there!

I wonder if we could provide some help or human bandwidth to make progress and clear the path forward as much as possible on that matter ?
We really miss having at least basic at-rest ZFSencryption working on Talos nodes 😢

@smira following on what you said in june :

I would really love to land some support for mounting/volumes in Talos 1.7 to avoid such workaround if possible.

So I suppose in a post Talos 1.9 world, this means extending/enriching [VolumeConfig] (https://www.talos.dev/v1.9/reference/configuration/block/volumeconfig) but I guess this is no small task.. 😅

Maybe the solution proposed in this PR would not be such a bad temporary workaround until you got this figured out in a generic way in the volume management system ?

Or even a simpler variation of it, only issuing a zfs load-key -a immediately after zpool import and leaving everything else to the CSI (like openebs-zfs-localpv) that should be able to transparently create/mount zfs volumes within the pool without even having to know if the root pool is encrypted or not (since volumes created in an encrypted root pool inherit encryption config).

Also, I think this "only deal with the root pool" strategy should make all the issue raised by @runningman84 in #400 (comment) void.

This key-loading step could be conditioned to the detection of encrypted zpool (easy to do with zfs and zpool binary).
Or even simply with a zfs load-key -n (performing a dry-run), see doc

Maybe not be such a bad temporary workaround until you got this figured in a generic way in the volume management system ?

We could work on a new PR and stress-test this a bit if that make sense ?

[jfroy]

I contributed a service for the zfs extension (#513) which is included in Talos 1.9. This service runs zpool import -fal to import pools. So if you set keylocation on a pool to a file path stored on a Talos volume, you should get automatic encrypted pool import at boot. I store my keys in /var/zfs and use Talos disk encryption to secure that.

[smira]

We don't have any bandwidth at the moment to work on ZFS yet, the Volume Management work is still ongoing.

[rdegez]

@jfroy woah I totally missed that! Super cool thanks a lot! :-)

[github-actions]

This PR is stale because it has been open 45 days with no activity.