CSI driver-smb mount the directory and immediately unmount despite no errors.

[lanzmlt]

Hello,

I can't get csi smb driver to work properly in managed k8s cluster running on CentOS Linux 7 (KERNEL VERSION 3.10.0-1160.92.1.el7.x86_64).
I read a similar discussionCSI driver-smb not mounting, but it's not my case.
I hope someone can help me solve this problem.
I have deployed StorageClass and StatefulSet with following manifests from documentation.

StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: smb
provisioner: smb.csi.k8s.io
parameters:
  source: //10.0.0.10/Backup
  # if csi.storage.k8s.io/provisioner-secret is provided, will create a sub directory
  # with PV name under source
  csi.storage.k8s.io/provisioner-secret-name: smbcreds
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-stage-secret-name: smbcreds
  csi.storage.k8s.io/node-stage-secret-namespace: default
reclaimPolicy: Delete  # available values: Delete, Retain
volumeBindingMode: Immediate
mountOptions:
  - seal
  - vers=3.0
  - nosuid
  - noexec
  - dir_mode=0777
  - file_mode=0777
  - uid=1001
  - gid=1001
  - noperm
  - mfsymlinks
  - cache=strict
  - noserverino

StatefulSet:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: statefulset-smb
  namespace: default
  labels:
    app: nginx
spec:
  serviceName: statefulset-smb
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      containers:
        - name: statefulset-smb
          image: mcr.microsoft.com/oss/nginx/nginx:1.19.5
          command:
            - "/bin/bash"
            - "-c"
            - set -euo pipefail; while true; do echo $(date) >> /mnt/smb/outfile; sleep 1; done
          volumeMounts:
            - name: persistent-storage
              mountPath: /mnt/smb
              readOnly: false
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: nginx
  volumeClaimTemplates:
    - metadata:
        name: persistent-storage
      spec:
        storageClassName: smb
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

State of pod:

NAME                READY   STATUS    RESTARTS          AGE    IP             NODE             NOMINATED NODE   READINESS GATES
statefulset-smb-0   1/1     Running   0                 15m    192.168.2.214   192.168.242.78    <none>           <none>

After pod in up state, PVC was created automatically and Bound:

NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
persistent-storage-statefulset-smb-0   Bound    pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3   20Gi       RWO            smb                 2m55s

But directory is not mounted in pod:

kubectl exec -it statefulset-smb-0 -- df -h                                                                                                    
Filesystem                     Size  Used Avail Use% Mounted on
overlay                         89G   53G   32G  63% /
tmpfs                           64M     0   64M   0% /dev
tmpfs                          3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/mapper/vgpaas-kubernetes  9.8G   55M  9.2G   1% /mnt/smb
/dev/mapper/vgpaas-dockersys    89G   53G   32G  63% /etc/hostname
shm                             64M     0   64M   0% /dev/shm
tmpfs                          7.3G   12K  7.3G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                          3.9G     0  3.9G   0% /proc/acpi
tmpfs                          3.9G     0  3.9G   0% /proc/scsi
tmpfs                          3.9G     0  3.9G   0% /sys/firmware

kubectl logs csi-smb-controller-69f9ccfbb6-mpsnc -c smb -n kube-system

I1205 12:49:19.617813       1 utils.go:83] GRPC response: {}
I1205 12:49:56.471186       1 utils.go:76] GRPC call: /csi.v1.Controller/CreateVolume
I1205 12:49:56.471202       1 utils.go:77] GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3","parameters":{"csi.storage.k8s.io/pv/name":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-smb-0","csi.storage.k8s.io/pvc/namespace":"default","source":"//10.0.0.10/Backup"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["seal","vers=3.0","nosuid","noexec","dir_mode=0777","file_mode=0777","uid=1001","gid=1001","noperm","mfsymlinks","cache=strict","noserverino"]}},"access_mode":{"mode":7}}]}
I1205 12:49:56.471327       1 controllerserver.go:314] internally mounting //10.0.0.10/Backup at /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3
I1205 12:49:56.471413       1 nodeserver.go:209] NodeStageVolume: targetPath(/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3) volumeID(10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##) context(map[source://10.0.0.10/Backup]) mountflags([seal vers=3.0 nosuid noexec dir_mode=0777 file_mode=0777 uid=1001 gid=1001 noperm mfsymlinks cache=strict noserverino]) mountOptions([seal vers=3.0 nosuid noexec dir_mode=0777 file_mode=0777 uid=1001 gid=1001 noperm mfsymlinks cache=strict noserverino])
I1205 12:49:56.471620       1 mount_linux.go:218] Mounting cmd (mount) with arguments (-t cifs -o seal,vers=3.0,nosuid,noexec,dir_mode=0777,file_mode=0777,uid=1001,gid=1001,noperm,mfsymlinks,cache=strict,noserverino,<masked> //10.0.0.10/Backup /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3)
I1205 12:49:56.565836       1 nodeserver.go:241] volume(10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##) mount "//10.0.0.10/Backup" on "/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3" succeeded
I1205 12:49:56.593727       1 controllerserver.go:332] internally unmounting /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3
I1205 12:49:56.593749       1 nodeserver.go:264] NodeUnstageVolume: CleanupMountPoint on /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3 with volume 10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##
I1205 12:49:56.605737       1 mount_helper_common.go:93] unmounting "/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I1205 12:49:56.605761       1 mount_linux.go:360] Unmounting /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3
I1205 12:49:56.632903       1 mount_helper_common.go:150] Deleting path "/tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3"
I1205 12:49:56.632991       1 nodeserver.go:273] NodeUnstageVolume: unmount volume 10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3## on /tmp/pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3 successfully
I1205 12:49:56.633005       1 utils.go:83] GRPC response: {"volume":{"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3","csi.storage.k8s.io/pvc/name":"persistent-storage-statefulset-smb-0","csi.storage.k8s.io/pvc/namespace":"default","source":"//10.0.0.10/Backup","subdir":"pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3"},"volume_id":"10.0.0.10/Backup#pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3##"}}

Login and password is correct, and i can connect to smb from the debug pod on node:

# smbclient //10.0.0.10.local/Backup -U $USER%$PASSWORD -W $WORK_GROUP

Try "help" to get a list of possible commands.
smb: \> 

Also i can mount smb directory on node 192.168.242.78 where pod is created after apply statefulset:

# mount -v -t cifs //10.0.0.10/Backup /tmp/test -o vers=3.0,username=$USER,password=$PASSWORD,uid=1001,gid=1001,dir_mode=0777,file_mode=0777,cache=strict,actimeo=30
# ls 
pvc-8c9943b6-bdcc-4f10-bdf9-6bcc614857d3  test

# mount | grep cifs 
//10.0.0.10/Backup on /tmp/test type cifs (rw,relatime,vers=3.0,cache=strict,username=$USER,domain=$DOMAIN_NAME,uid=1001,forceuid,gid=1001,forcegid,addr=10.0.0.10,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=30)

Username, password and domain have been changed for security purposes.
Any help will be greatly appreciated... Thanks

[andyzhangx]

@lanzmlt what is your kubelet path? if it's not /var/lib/kubelet, follow guide here: https://github.com/kubernetes-csi/csi-driver-smb/tree/master/charts#tips

[lanzmlt]

@andyzhangx i can't run smb-controller on control plane node, because i have a managed k8s.

In my case kubelet plugins located in:

# ls -l /var/lib/kubelet/plugins/
total 36
drwxr-x--- 2 root root 4096 Nov  9 12:14 disk.csi.everest.io
drwxr-x--- 3 root root 4096 Jun  3  2024 kubernetes.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 local.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 nas.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 obs.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 proxy.csi.everest.io
drwxr-x--- 2 root root 4096 Nov  9 12:14 sfsturbo.csi.everest.io
drwxr-xr-x 2 root root 4096 Nov 14 13:20 smb.csi.io
drwxr-xr-x 2 root root 4096 Dec  5 14:12 smb.csi.k8s.io

So, unfortunately, these tips not my case and can't help me.

[andyzhangx]

can you provide csi driver daemonset pod logs on the node?

[lanzmlt]

kubectl get po -o wide                                                                                    
NAME                READY   STATUS    RESTARTS          AGE    IP             NODE             NOMINATED NODE   READINESS GATES
statefulset-smb-0   1/1     Running   0                 2m3s   192.168.2.214  192.168.242.78    <none>           <none>

kubectl --namespace=kube-system get pods --selector="app.kubernetes.io/name=csi-driver-smb" -o wide | grep 192.168.242.78
csi-smb-node-qjnsw                    3/3     Running   0              174m   192.168.242.78    192.168.242.78    <none>           <none>

kubectl -n kube-system logs csi-smb-node-qjnsw
Defaulted container "liveness-probe" out of: liveness-probe, node-driver-registrar, smb
I1209 08:56:25.799643       1 main.go:135] "Calling CSI driver to discover driver name"
I1209 08:56:25.804429       1 main.go:143] "CSI driver name" driver="smb.csi.k8s.io"
I1209 08:56:25.804481       1 main.go:172] "ServeMux listening" address="localhost:29643"

kubectl -n kube-system exec csi-smb-node-qjnsw -it -c smb -- mount | grep cifs
//10.0.0.10/Backup/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e on /mnt/paas/kubernetes/kubelet/plugins/kubernetes.io/csi/pv/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e/globalmount type cifs (rw,nosuid,noexec,relatime,vers=3.0,cache=strict,username=backup,domain=smb,uid=1001,forceuid,gid=1001,forcegid,addr=10.0.0.10,file_mode=0777,dir_mode=0777,seal,soft,nounix,mapposix,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)
//10.0.0.10/Backup/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e on /mnt/paas/kubernetes/kubelet/pods/a7c5766f-83f2-47d4-8d30-541c73a82d9a/volumes/kubernetes.io~csi/pvc-56e51337-255c-435a-98fa-8f83dcf9b43e/mount type cifs (rw,relatime,vers=3.0,cache=strict,username=backup,domain=smb,uid=1001,forceuid,gid=1001,forcegid,addr=10.0.0.10,file_mode=0777,dir_mode=0777,seal,soft,nounix,mapposix,mfsymlinks,noperm,rsize=1048576,wsize=1048576,echo_interval=60,actimeo=1)

[andyzhangx]

have you --set linux.kubelet="/mnt/paas/kubernetes/kubelet" in the driver install?

[lanzmlt]

No, i use the default path that set in chart values ​​​​for kubelet - /var/lib/kubelet

[andyzhangx]

pls use --set linux.kubelet="/mnt/paas/kubernetes/kubelet" in driver install

[lanzmlt]

This works for me
Thank you @andyzhangx !

[RobJohnson-RoJoPro]

I have this problem too with MicroK8s - home lab.
I can confirm I installed the driver with the correct kublet path as per the docs. The SC, PV and PVC all create correctly and become bound. I start a simple nginx pod with PVC. It runs successfully. However, files are not appearing on my local NAS.
I can mount from the node itself successfully. I can see the files appearing in the node /var/snap/microk8s/common/var/lib/kubelet/pods/
from the pod.
kube exec -it csi-smb-node-95z9k -n kube-system -c smb -- mount | grep cifs
Shows the correct paths and directories.
It is very frustrating.

[andyzhangx]

have you --set linux.kubelet="..." in the driver install? follow guide here: https://github.com/kubernetes-csi/csi-driver-smb/tree/master/charts#tips

[RobJohnson-RoJoPro]

Hi Andy,
This is what I used:
--set kubeletDir=/var/snap/microk8s/common/var/lib/kubelet

As I'm using MicroK8s as per the docs for install. Is this correct?

kube -n kube-system exec csi-smb-node-95z9k -it -c smb -- mount | grep cifs

//192.168.0.210/pv on /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/2a97516c354b68848cdbd8f54a226a0a55b21ed138e207ad6c5cbb9c00aa5aea/globalmount type cifs (rw,relatime,vers=2.0,cache=strict,username=*****,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.0.210,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,reparse=nfs,rsize=65536,wsize=65536,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
//192.168.0.210/pv on /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08/globalmount type cifs (rw,relatime,vers=2.0,cache=strict,username=*****,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.0.210,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,noperm,reparse=nfs,rsize=65536,wsize=65536,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)
//192.168.0.210/pv on /var/snap/microk8s/common/var/lib/kubelet/pods/1db63ac6-473a-4495-99f1-5a10cd6771f7/volumes/kubernetes.io~csi/pv-test/mount type cifs (rw,relatime,vers=2.0,cache=strict,username=*****,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.0.210,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,noperm,reparse=nfs,rsize=65536,wsize=65536,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1) 

[andyzhangx]

it's --set linux.kubelet="..."

[andyzhangx]

it should work per #674 (comment)

[RobJohnson-RoJoPro]

Oh, I can't believe I completely missed that.
Let me go and re-install 🤦

I got that initial command from the MicroK8s docs and never noticed the slight difference.
https://microk8s.io/docs/how-to-nfs#install-the-csi-driver-for-nfs

Thank you @andyzhangx !