Node encryption

[schlichtanders]

I would like to encrypt all nodes with a single master key using LUKS (including auto-scaling groups)

Is this possible? How to do it?

[mysticaltech]

@schlichtanders Do try gpt-4, you may be surprised. Especially that is just got way better browsing capability now to fetch the latest on this matter.

[mysticaltech]

Given your interest in Python, Kubernetes, and software engineering, I assume you have a comprehensive understanding of container orchestration and secure software development practices. Let's get into the nuances of encrypting ephemeral volumes in a Kubernetes environment, specifically considering the Hetzner Cloud environment as per the discussion thread you've posted.

Hetzner CSI and Its Limitations

The Hetzner Cloud's CSI driver allows for LUKS-based encryption, but it's crucial to note that it's specifically designed for Hetzner Cloud Volumes, not for the local storage tied to the compute instances. It won't encrypt the default local storage (ephemeral storage), which might be used for Kubernetes emptyDir volumes.

OpenEBS and Kasten K10

As JustinGuese pointed out, OpenEBS can manage local folder encryption. OpenEBS does have support for encrypted volumes, but the documentation seems to be scattered and somewhat outdated, as you've noted. Kasten K10's focus is more on data management and backup rather than encryption. OpenEBS has its own suite of CSI drivers, and theoretically, you can employ an encryption key when provisioning the volume, though I don't have a concrete example at hand.

Security Concerns with Multiple Actors

If your nodes are running code from multiple actors, encryption at the volume level may not be sufficient for the security requirement to isolate data. Kubernetes RBAC and Network Policies can limit access to resources and network communication, but they won't necessarily prevent a sophisticated user from attempting to read another user's data directly from the node's storage. For this, OS-level encryption might be more appropriate.

Full Node Encryption with LUKS

One comprehensive approach, which you seem to be considering, is full node encryption using Linux Unified Key Setup (LUKS). In an auto-scaling scenario, you could potentially use cloud-init or similar to run a script when a new node is instantiated, which would handle the LUKS setup and use a master key for decryption. This way, the entire node's storage is encrypted, offering a more robust level of security.

Docker Writable Layer

Encrypting the Docker/Containerd writable layer is another level of complexity. Typically, Docker relies on the underlying OS for encryption and doesn't provide its own encryption mechanisms for individual containers. You could potentially achieve this with some deep customization, but this would definitely fall under the category of "adventurous" and likely be high-maintenance.

OverlayFS and Direct Block Storage Access

Your last point about OverlayFS is crucial. OverlayFS itself won't provide direct access to block storage. However, it does share the underlying ext4 or xfs filesystem with the host and other containers. So while OverlayFS provides a measure of isolation, it's not necessarily a secure isolation. For security-sensitive applications, OverlayFS's isolation might not be sufficient.

Summary

Encrypting ephemeral volumes in Kubernetes is not a straightforward endeavor, especially when multiple actors with different levels of trust are sharing the same physical node. While Kubernetes itself provides some level of isolation between pods, a fully secure environment would ideally combine Kubernetes-level restrictions with OS-level and possibly even hardware-level security measures.

Given your proficiency in the field, it might be worth looking into custom solutions that integrate OS-level encryption mechanisms with Kubernetes storage provisioning. You could potentially contribute to this area, given that it's an open problem with significant implications for multi-tenant Kubernetes environments.

[schlichtanders]

Thank you for your help, however I find that 80% of this is actually rather noise.
I create this discussion because I streamlined my thoughts. As stated, I am interested in full node encryption, so the only relevant paragraph from your gpt4 response is.

Full Node Encryption with LUKS

One comprehensive approach, which you seem to be considering, is full node encryption using Linux Unified Key Setup (LUKS). In an auto-scaling scenario, you could potentially use cloud-init or similar to run a script when a new node is instantiated, which would handle the LUKS setup and use a master key for decryption. This way, the entire node's storage is encrypted, offering a more robust level of security.

Which unfortunately is really not what I was hoping for.
I am rather looking for help how to setup full-node encryption with kube-hetzner, because I hope someone already did it.

If someone else can help with information about how to setup full-node encryption with kube-hetzner Please don't answer here as a comment, but create a new answer

[schlichtanders]

@JustinGuese @aleksasiriski maybe you can help me on this - have you experience with setting up full-node encryption using LUKS?

[aleksasiriski]

I haven't done this with my cluster, but I'm using LUKS on my workstation (before it was on Arch manually, now it's Fedora Silverblue guided with the installer)

My suggestion would be: have a separate /var partition and encrypt only that with LUKS, it'll be much easier to achieve using terraform and cloud-init, and should be enough for what you're trying to achieve. After that, make sure that all of your volumes are stored inside newly encrypted /var (longhorn and openebs do this by default, and it also seems to be the case with emtryDir)

[schlichtanders]

Thank you so much for the hint!