Add LUKS Encryption Support in Ceph NVMe-oF Gateway

[Greenpepper15]

Add LUKS Encryption Support in Ceph NVMe-oF Gateway

1. Summary

This proposal aims to add support for Data-At-Rest Encryption to the Ceph NVMe-oF Gateway by exposing the native encryption capabilities of librbd. This will allow the Gateway to serve LUKS-encrypted RBD images to NVMe initiators transparently.

2. Motivation

Currently, the Ceph NVMe-oF Gateway serves RBD images as plain block devices. If a user requires Data-At-Rest encryption due to compliance or security concerns, they must handle encryption within the client (Initiator). Should this not be possible due to capacity constrains outsourcing it in the gateway would be ideal.

The Solution:
Offload the encryption/decryption task to the NVMe-oF Gateway. The Gateway unlocks the RBD image using librbd and presents a standard, unencrypted NVMe namespace to the client. libRBD itself already natively supports LUKS encryption using AES-XTS.

3. Proposed Design

Preferred Approach: librbd Native Encryption (LUKS)

We propose extending the SPDK bdev_rbd module to support the built-in encryption API provided by librbd.

• Why this path? librbd supports the LUKS standard. An image encrypted via the Gateway remains compatible with other Ceph RBD clients (QEMU with libRBD, krbd+DmCrypt, rbd-nbd).
• Why not SPDK Virtual Crypto Bdev? While SPDK has a bdev_crypto module, it creates a layering of encryption that is not natively understood by standard Ceph tools. Using librbd ensures the image is "just a LUKS image" to the rest of the Ceph ecosystem.

Proposed Workflow

1. Gateway receives NamespaceAdd() RPC with all the nessary information to bind an encrypted RBD image to gateway
2. Gateway leverages SPDK via bdev_rbd_create() to use libRBD to bind RBD image to gateway and configure encryption

4. Implementation Details

We need to modify the interaction between the Gateway's control plane and the underlying SPDK bdev_rbd module.

A. SPDK Extension

The bdev_rbd_create method in SPDK needs to accept encryption context. To mitigate any breaking changes we
should maybe create a new function bdev_encrypted_rbd_create in SPDK.

Current Signature:

bdev_rbd_create(name, pool_name, rbd_name, ...)

Proposed Signature Extension:

bdev_encrypted_rbd_create(
    name, 
    pool_name, 
    rbd_name, 
    ...
    encryption_format="luks",    # Type of LUKS: luks1, luks2
    encryption_passphrase="...", # The key/passphrase
    encryption_algorithm = 1 # implies AES-256 in XTS mode see rbd_encryption_algorithm_t enum in librbd.h
    # maybe other parameters
)

An implementation of using the libRBD encryption feature would be similar to the
QEMU integration of the libRBD encryption feature.

To simplify the logic we could impose that the Ceph NVMe-oF Gateway can only use valid LUKS formatted RBD images. Then we only need to integrate the libRBD load encryption key operation and not worry about formatting the RBD image to be LUKS conform. This would work fairly well in the context of ceph-csi. When the rbd image is created the ceph-csi driver could also LUKS format the image (if specified) and then attach it to the Ceph NVMe-oF Gateway.

B. Gateway API Extension

The Ceph NVMe-oF Gateway (ceph-nvmeof) needs to expose these parameters in the namespace_add:

def namespace_add(self, request, context=None):
    """Adds a namespace to a subsystem."""
    # ...

    # Pass passphrase and other parameter from 'request' down to SPDK
    bdev_name = self.spdk_rpc_client.bdev_encrypted_rbd_create(
        ...,
        encryption_format=request.luks,
        encryption_passphrase=request.passphrase,
        encryption_algorithm = request.algorithm
        # maybe other parameters
    )
    
    # ...

I.e. this only covers the LUKS formatted RBD image attach case.

6. Conclusion

Implementing this feature aligns the NVMe-oF Gateway with the rest of the Ceph ecosystem, allowing for a seamless encrypted storage experience that is interoperable and standards-compliant.

Related Issues:

#1402

[caroav]

@Greenpepper15 in your vision, the GW will save the passphrases?

[Greenpepper15]

The passphrase storing would be done by the client and it needs to be provided to successfully mount the rbd image to the GW. During the mounting process the GW uses the passphrase to unlock the LUKS partition.

If the rbd image is created by the GW, the GW can also use librbd to format the image into the LUKS format using the client provided passphrase. For example, the formatting could be implemented like this on the GW using the encryption_format() function (). Otherwise if the image already exists and a passphrase is provided we must assume it was already LUKS formatted.

# control/cephutils.py
    def create_image(self, pool_name, data_pool_name, image_name, size,
                     rados_namespace_name=None) -> bool:
       # ...
                try:
                    rbd_inst.create(ioctx, image_name, size, data_pool=data_pool_name)
                    # now if the passphrase was provided we can format the image!
                    rbd.encryption_format(...)
                except rbd.ImageExists:
                    # ...

For both cases (image is already LUKS formatted before we mount it to the GW & GW creates and LUKS formats the rbd image) the client has to provide the passphrase and other crypto parameters to call thenamespace_add() function to use the librbd LUKS encryption/decryption feature.

[caroav]

Yes, but in the case that the GW restarts for example, it needs to reopen the RBD image right? client is not involved in this operation anymore. How does the GW know what is the passphrase in this scenario, unless it saved it somewhere?

[Greenpepper15]

That is true but those semantics are also true for a computer that mounts a LUKS disk via DmCrypt. When the computer restarts a client needs to provide the passphrase.

I am not familiar enough with the NVMEof protocol. When the GW restarts it is transparent to the clients? If so the encryption_load() function without a provided passphrase will fail. Client then will have to figure out a way to re-mount the encrypted rbd image to the GW.

If this behavior is undesired then yes we have to store the passphrase on the GW (or a least a reference). Could we use the keyring feature of SPDK for this (internally the keyring lib of SPDK can store the passphrase on the FS of the GW)?
I guess the GW also stores the CephX user credentials on the GW. So storing secrets on the FS of the GW is not completely unprecedented.

[bill-scales]

The challenge with the gateways is that they need to be able to restart without user intervention - that means there needs to be an automatic way for them to acquire to passphrases to unlock the images. I don't think we want the gateway to persistently store the passphrase, it might be able to obfuscate this by storing it in an encrypted store but ultimately it will always be retrievable. Using something like TPM hardware on the server that the gateway is running on would give a bit more security. This type of solution also has to deal with the difficultly of adding/removing/moving gateways - probably with some bespoke method of securely passing keys from gateway to gateway.

The alternative is for the gateway to fetch the passphrase from somewhere else. That could be via an out-of-band protocol such as KMIP to an external key server or it could use an in-band protocol such as TCG.OPAL or NVMe's KPIO to have the NVMe initiator pass the key to the gateway.

Support for the in-band protocols is pretty patchy - for example with the right kernel and user space tools you can get Linux working, but you might struggle with an off-the-shelf Linux distro and something like an ESXi host isn't going to support these protocols.