Failing to mount StorageBox using samba since today (15.11.2023)

[martinatwainobicom]

Hello,

first, thank you very much for this awesome project!

Second, I am not entirely sure if this is the right place to ask for help as it might also be related to MicroOS and cifs-utils but I do not yet know where to ask else.

I have two projects setup with kube-hetzner.

1. In the first project, which used an early version of kube-hetzner (there was no enable_csi_driver_smb = true), I setup samba mounts on the agents(!) to connect to a Hetzner StorageBox. I then mounted volumes of the mounted host directory into the pods. That worked until today where the mount.cifs mounting command fails with the following error:
   mount error(126): Required key not available

dmesg shows: CIFS: VFS: cifs_mount failed w/return code = -126

The mounting command used looks like this one:
mount.cifs -o user=<masked>,pass=<masked> //<masked>.your-storagebox.de/<masked> <targetdirectory>
as it is described by Hetzner here: https://docs.hetzner.com/de/robot/storage-box/access/access-samba-cifs

To narrow down the problem I've tested to mount the share on Windows and on a newly setup Hetzner Ubuntu server with cifs-utils added. In both cases it worked flawlessly, so there must be something off with MicroOS and the packages used, I guess.

2. The second project was setup later and enable_csi_driver_smb flag was available. I managed to connect pods to the StorageBox using the Samba CSI driver and that worked too. But today that stopped as well.
   The related pods are stuck in "ContainerCreating" state and if I "kubectl describe" them, the eventslist shows the following:
   Warning FailedMount 9m49s (x31 over 77m) kubelet Unable to attach or mount volumes: unmounted volumes=[docker-registry-storage], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition Warning FailedMount 4m27s (x42 over 79m) kubelet MountVolume.MountDevice failed for volume "docker-registry-smb-pv" : rpc error: code = Internal desc = volume(docker-registry-smb-volume) mount "//<masked>.your-storagebox.de/<masked>" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/<masked>/globalmount" failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t cifs -o <masked> //<masked>.your-storagebox.de/<masked> /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/<masked>/globalmount Output: mount error(4): Interrupted system call Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

dmesg (from within the agent node) shows: CIFS: VFS: cifs_mount failed w/return code = -4

I already contacted HETZNER support and am waiting for response but I would really appreciate any help here, as I am trying to get these systems up and running again but can't figure out what's wrong.

Edit 1: I enabled the the cifs logs (https://wiki.samba.org/index.php/LinuxCIFS_troubleshooting) and saw that it fails to resolve the IP address. I finally was able to mount the StorageBox again when using its IP address instead of its domain name. However, as the IP addresses might change this does not seem to be a long lasting fix.

Edit 2: For anyone who might have a similar issue, I've found the following bug-ticket here: https://bugzilla.opensuse.org/show_bug.cgi?id=1217194

[DerLinne]

Hi!
I have exact the same problem - but even the workaround with the IP does not want to work for me.

Could you share some detail about your working setup?

• StorageBox Settings for the used account
• StorageClass Definition
• PVC/PV Example

I tried several approaches - but nothing worked. The IP brought me a step more - but it does not work at all.
Thanks!

[DerLinne]

Update from my side:
With IP it is running now - I had not opened the firewall.

[martinatwainobicom]

Sorry, I did not have time to respond before. I only fixed the bug in the project described first as that had priority. Good to hear that it worked for you as well.