ReadOnlyMany persistant volumes

[schlichtanders]

Description

Current longhorn does not support ReadOnlyMany Volumes. It may arrive in their next milestone, but who knows.

It would be great to have documentation about which options there are for ReadOnlyMany Volumes on kube-hetzner.
As I understood, these are not supported at all at this stage (I hope I am wrong on this).

Use case: building an efficient file caching system on kubernetes. I haven't found a way to do this on kube-hetzner yet.

[mysticaltech]

@schlichtanders This is probably already possible through enable_csi_driver_smb, see kube.tf.example. Or you could install rook as a third party helm package.

---

See my conversation below with GPT-4:

ChatGPT:

There are a few ways you could approach ReadOnlyMany volume support on kube-hetzner. Here are a few possible solutions:

1. Use NFS (Network File System): NFS is a popular option for ReadOnlyMany volumes. It's worth noting that it may not offer the same performance as other solutions, but it should be satisfactory for most use cases. You can deploy an NFS server in your cluster and use it as a ReadOnlyMany volume. It requires some additional setup and management, but this could be a potential solution.

2. Use GlusterFS: GlusterFS is another shared storage system that allows you to set up a ReadOnlyMany volume. However, it also has a complex setup process and may not be suitable for all use cases.

3. Wait for Longhorn support: If you think Longhorn might support ReadOnlyMany volumes in the near future, it might be worth waiting. Longhorn is a popular choice due to its simplicity and ease of use.

4. Rook with CephFS or Ceph RBD: Rook, a cloud-native storage orchestrator, can manage a Ceph cluster which provides shared file storage with CephFS or block storage with Ceph RBD. Either of these can be used to create ReadOnlyMany persistent volumes.

5. Use an external storage provider: If none of the above options are suitable, you could look at using an external storage provider that supports ReadOnlyMany volumes. There would be additional costs associated with this, but it might be the most straightforward solution.

For each option, you can document the steps involved in setting it up on kube-hetzner, including any specific configurations that might be necessary. Note that you might need to run some tests to ensure the solutions work as expected on kube-hetzner and provide adequate performance. Remember to provide a warning about the potential risks and trade-offs involved with each option.

You:

Would smb work?

ChatGPT:

Yes, SMB (Server Message Block), also known as Common Internet File System (CIFS), can work for ReadOnlyMany volumes. This protocol is often used for providing shared access to files, printers, and serial ports over a network.

Here's a brief example of how this might work:

1. Set up an SMB server: This can be a separate server or an existing one, as long as it's reachable from the Kubernetes nodes. You'll need to configure the share to be read-only.

2. Use the CSI driver for SMB: There's a CSI (Container Storage Interface) driver for SMB that allows Kubernetes to use an SMB share as a volume. You'll need to install this in your Kubernetes cluster.

3. Create a PersistentVolume (PV) and PersistentVolumeClaim (PVC) for the SMB share: In the PV and PVC, you'll specify the details of the SMB share and set accessModes to ReadOnlyMany. The CSI driver will handle the connection to the SMB share.

This setup allows multiple pods to mount the same SMB share in read-only mode. Be aware that SMB performance may not be as high as some other options, but it's generally easy to set up and widely supported. As always, testing is crucial to ensure it meets your needs.

You can document this process for your users, highlighting any specifics related to kube-hetzner.

[schlichtanders]

I realized that the smb answer actually already refers to hetzner volumes.

csi-driver-smb

Will check that now

[schlichtanders]

Reading in another discussion, it seems the csi-driver-smb for hetzner storage box has problems with opening the firewall to permissively. So not a real option unfortunately as security is important for me.

[schlichtanders]

I now understood that the security concerns are only for outbound connections. And it is actually fine for me to open the cluster outbound to the internet.

Plus the price for the storage box is amazingly cheap. So this really a great option for caching of nonsensitive information. It happens that this is all I need for now.

(as mentioned in the other discussion there are some ways to maybe add encryption in rest, but it seemed like it is not easily done)

[mysticaltech]

@schlichtanders Encryption was solved, normally the info should be in kube.tf.example.

[mysticaltech]

Also, if encryption does not work for smb yet (not sure about that specific part), just use longhorn with disk storage with encryption if needed, it will be a lot more efficient then volumes, unless you really need many terrabytes of data. Maybe longhorn has modes of operation that will approximate readonlymany. Read the longhorn section in kube.tf.example for details and configuration options.