moosefs-csi not starting on k3s

[aleahy1187]

This has been transferred from this discussion and reflects updates since the original post.

Overview: I have recently installed a MooseFS filesystem on a cluster of servers and am having difficult starting the CSI client.

Details:

Some software versions:

• kubernetes k3s v1.28.4+k3s2
• mfsmaster version: 4.57.5-1 ; build: 1937 (now from moosefs.com ubuntu 22.04 repos)
• moosefs-csi image registry.moosefs.com:443/moosefs-csi-plugin:latest

The following csi-moosefs-config.yaml has been applied:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: csi-moosefs-config
data:
  master_host: "server.mydomain.com"           # domain name is recommended, this IP is only an example
  master_port: "9421"            # Fixed 
  k8s_root_dir: "/kubexports"                    
  driver_working_dir: "pv_data"       # Created
  mount_count: "1" 
  mfs_logging: "true" 

pv_data exists in the appropriate spot on the mfs filesystem:

root@fermat:/home/knoxuser# ls -axl /mnt/mfs/kubexports/pv_data/
total 3907
drwxr-xr-x  2 root root       1 Mar 30 14:45 .
drwxr-xr-x 18 root root 4000216 Mar  2 14:10 ..

The k3s server is started with allow-privileged=true:

ExecStart=/usr/local/bin/k3s \
    server --write-kubeconfig-mode=644  --kube-apiserver-arg=allow-privileged=true

(I've tried this through the command line as above and by changing /etc/rancher/k3s/config.yaml. But I don't know how to make sure this is enabled.)

kubectl apply -f csi-moosefs.yaml leads to all pods crashing:

MooseFS-CSI# kubectl get pods | grep moosefs
csi-moosefs-controller-0                  3/4     CrashLoopBackOff   4 (71s ago)    2m43s
csi-moosefs-node-wsmb9                    1/2     CrashLoopBackOff   4 (74s ago)    2m43s
csi-moosefs-node-hr2b5                    1/2     CrashLoopBackOff   4 (72s ago)    2m43s
csi-moosefs-node-qmrd2                    1/2     CrashLoopBackOff   4 (71s ago)    2m43s
csi-moosefs-node-vr2l8                    1/2     CrashLoopBackOff   4 (66s ago)    2m43s
csi-moosefs-node-pk9rw                    1/2     CrashLoopBackOff   4 (67s ago)    2m43s
csi-moosefs-node-f8s2s                    1/2     CrashLoopBackOff   4 (63s ago)    2m43s

Here are the details for the controller pod:

MooseFS-CSI# kubectl describe pod csi-moosefs-controller-0
Name:                 csi-moosefs-controller-0
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      csi-moosefs-controller-sa
Node:                 epyc1/10.90.1.25
Start Time:           Mon, 31 Mar 2025 08:53:04 -0500
Labels:               app=csi-moosefs-controller
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=csi-moosefs-controller-6cf74f979b
                      role=csi-moosefs
                      statefulset.kubernetes.io/pod-name=csi-moosefs-controller-0
Annotations:          <none>
Status:               Running
IP:                   10.90.1.25
IPs:
  IP:           10.90.1.25
Controlled By:  StatefulSet/csi-moosefs-controller
Containers:
  csi-provisioner:
    Container ID:  containerd://7bb5d47923e112969624996cd851984978c91d6639ff1d293cae07a2eaf1c523
    Image:         quay.io/k8scsi/csi-provisioner:v1.4.0
    Image ID:      sha256:327e23baedbb939641bf5d2fb8d8ff9d8c41c2b2369cfe49a48eeef39a6e17fa
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
    State:          Running
      Started:      Mon, 31 Mar 2025 08:53:05 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fgql6 (ro)
  csi-attacher:
    Container ID:  containerd://bcd53cfe20b2a61fbf1e0388e95a5d4c453e49f7e0e6df4ae86251642e8a1d1a
    Image:         quay.io/k8scsi/csi-attacher:v3.1.0
    Image ID:      quay.io/k8scsi/csi-attacher@sha256:e3334d505bebc44afba610e2c218ca7b4993cac61fc0951d75033ccc971a1347
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
    State:          Running
      Started:      Mon, 31 Mar 2025 08:53:06 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fgql6 (ro)
  csi-resizer:
    Container ID:  containerd://f6619dd19e9599afff6b63d1632cf2ae6877e5d6135770fdcb8b2d13dccf1ee1
    Image:         quay.io/k8scsi/csi-resizer:v0.5.0
    Image ID:      quay.io/k8scsi/csi-resizer@sha256:6c6a0332693a7c456378f6abd2bb40611826c1e1a733cadbdae2daab3125b71c
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
    State:          Running
      Started:      Mon, 31 Mar 2025 08:53:07 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fgql6 (ro)
  csi-moosefs-plugin:
    Container ID:  containerd://8269bdcd8a880cbe870f2ad460161cadd6c5d7b2118c4e639dde55746885f52d
    Image:         registry.moosefs.com:443/moosefs-csi-plugin:latest
    Image ID:      registry.moosefs.com:443/moosefs-csi-plugin@sha256:457bdf58227c88437c341d7ba2163dcd788da09f2e698ac88a2779f6c6dd2a05
    Port:          <none>
    Host Port:     <none>
    Args:
      --mode=controller
      --csi-endpoint=$(CSI_ENDPOINT)
      --master-host=$(MASTER_HOST)
      --master-port=$(MASTER_PORT)
      --root-dir=$(ROOT_DIR)
      --plugin-data-dir=$(WORKING_DIR)
      --mfs-logging=$(MFS_LOGGING)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 31 Mar 2025 09:04:10 -0500
      Finished:     Mon, 31 Mar 2025 09:04:10 -0500
    Ready:          False
    Restart Count:  7
    Environment:
      CSI_ENDPOINT:  unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      MASTER_HOST:   <set to the key 'master_host' of config map 'csi-moosefs-config'>         Optional: false
      MASTER_PORT:   <set to the key 'master_port' of config map 'csi-moosefs-config'>         Optional: false
      ROOT_DIR:      <set to the key 'k8s_root_dir' of config map 'csi-moosefs-config'>        Optional: false
      WORKING_DIR:   <set to the key 'driver_working_dir' of config map 'csi-moosefs-config'>  Optional: false
      MFS_LOGGING:   <set to the key 'mfs_logging' of config map 'csi-moosefs-config'>         Optional: false
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fgql6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-fgql6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  11m                 default-scheduler  Successfully assigned kube-system/csi-moosefs-controller-0 to epyc1
  Normal   Pulling    11m                 kubelet            Pulling image "quay.io/k8scsi/csi-provisioner:v1.4.0"
  Normal   Pulled     11m                 kubelet            Successfully pulled image "quay.io/k8scsi/csi-provisioner:v1.4.0" in 802ms (802ms including waiting)
  Normal   Created    11m                 kubelet            Created container csi-provisioner
  Normal   Started    11m                 kubelet            Started container csi-provisioner
  Normal   Pulling    11m                 kubelet            Pulling image "quay.io/k8scsi/csi-attacher:v3.1.0"
  Normal   Pulled     11m                 kubelet            Successfully pulled image "quay.io/k8scsi/csi-attacher:v3.1.0" in 397ms (397ms including waiting)
  Normal   Created    11m                 kubelet            Created container csi-attacher
  Normal   Started    11m                 kubelet            Started container csi-attacher
  Normal   Pulling    11m                 kubelet            Pulling image "quay.io/k8scsi/csi-resizer:v0.5.0"
  Normal   Pulled     11m                 kubelet            Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.5.0" in 825ms (825ms including waiting)
  Normal   Created    11m                 kubelet            Created container csi-resizer
  Normal   Started    11m                 kubelet            Started container csi-resizer
  Normal   Pulled     11m                 kubelet            Successfully pulled image "registry.moosefs.com:443/moosefs-csi-plugin:latest" in 718ms (718ms including waiting)
  Normal   Pulled     11m                 kubelet            Successfully pulled image "registry.moosefs.com:443/moosefs-csi-plugin:latest" in 613ms (613ms including waiting)
  Normal   Started    11m (x2 over 11m)   kubelet            Started container csi-moosefs-plugin
  Normal   Pulling    11m (x3 over 11m)   kubelet            Pulling image "registry.moosefs.com:443/moosefs-csi-plugin:latest"
  Normal   Pulled     11m                 kubelet            Successfully pulled image "registry.moosefs.com:443/moosefs-csi-plugin:latest" in 662ms (662ms including waiting)
  Normal   Created    11m (x3 over 11m)   kubelet            Created container csi-moosefs-plugin
  Warning  BackOff    84s (x47 over 11m)  kubelet            Back-off restarting failed container csi-moosefs-plugin in pod csi-moosefs-controller-0_kube-system(1b8dfb7c-7d08-45d9-8c87-65481a07cb78)

Here are the logs on the controller pod:

MooseFS-CSI# kubectl logs  csi-moosefs-controller-0
Defaulted container "csi-provisioner" out of: csi-provisioner, csi-attacher, csi-resizer, csi-moosefs-plugin
I0331 13:53:05.988702       1 feature_gate.go:216] feature gates: &{map[]}
I0331 13:53:05.988797       1 csi-provisioner.go:98] Version: v1.4.0-0-g1d9bad3
I0331 13:53:05.988825       1 csi-provisioner.go:112] Building kube configs for running in cluster...
I0331 13:53:06.020886       1 connection.go:151] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0331 13:53:16.021166       1 connection.go:170] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0331 13:53:26.021149       1 connection.go:170] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock

The logs on the nodes:

MooseFS-CSI# kubectl logs csi-moosefs-node-f8s2s
Defaulted container "driver-registrar" out of: driver-registrar, csi-moosefs-plugin
I0331 13:53:04.862935       1 main.go:110] Version: v1.1.0-0-g80a94421
I0331 13:53:04.862973       1 main.go:120] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0331 13:53:04.862999       1 connection.go:151] Connecting to unix:///csi/csi.sock
W0331 13:53:14.863248       1 connection.go:170] Still connecting to unix:///csi/csi.sock
W0331 13:53:24.863221       1 connection.go:170] Still connecting to unix:///csi/csi.sock

[xandrus]

Hi,
I've made a few small changes in the deploy section (bc8eff0), so could you run some tests now?
Just switch to the dev branch to check the latest updates.

You can try these steps:

git clone https://github.com/moosefs/moosefs-csi.git
cd moosefs-csi
git checkout dev

Than update deploy/csi-moosefs-config.yaml file:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: csi-moosefs-config
data:
  master_host: "server.mydomain.com"
  master_port: "9421"
  k8s_root_dir: "/kubexports"                    
  driver_working_dir: "pv_data"
  mount_count: "1" 
  mfs_logging: "true" 
  mfs_mount_options: "mfscachemode=DIRECT"

And deploy config and MooseFS CSI driver.

I'd also like to mention that the latest drivers with the latest MooseFS client are now available on the Github container registry: https://github.com/moosefs/moosefs-csi/pkgs/container/moosefs-csi

I'm waiting for your feedback!

[aleahy1187]

I followed all of your steps and it worked!

kubectl get pods | grep moose

csi-moosefs-node-d6s9k                    2/2     Running       0              22m
csi-moosefs-node-bnqkc                    2/2     Running       0              22m
csi-moosefs-node-smxmk                    2/2     Running       0              22m
csi-moosefs-node-xmm2h                    2/2     Running       0              22m
csi-moosefs-node-jljqn                    2/2     Running       0              22m
csi-moosefs-controller-0                  4/4     Running       0              22m
csi-moosefs-node-k7tfq                    2/2     Running       0              22m
my-moosefs-pod                            1/1     Running       0              13m

I went all the way through the Verification steps, and everything seemed fine. Any idea when this can come off the dev branch? Thanks very much for your help!

kubectl describe pod csi-moosefs-controller-0

Name:                 csi-moosefs-controller-0
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      csi-moosefs-controller-sa
Node:                 epyc1/10.90.1.25
Start Time:           Sat, 05 Apr 2025 18:53:45 -0500
Labels:               app=csi-moosefs-controller
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=csi-moosefs-controller-6898c64577
                      role=csi-moosefs
                      statefulset.kubernetes.io/pod-name=csi-moosefs-controller-0
Annotations:          <none>
Status:               Running
IP:                   10.90.1.25
IPs:
  IP:           10.90.1.25
Controlled By:  StatefulSet/csi-moosefs-controller
Containers:
  csi-provisioner:
    Container ID:  containerd://32104b6d8a523e837b50969a489aa0a40cbae019dd2d1a92612728427b58e717
    Image:         quay.io/k8scsi/csi-provisioner:v2.1.0
    Image ID:      quay.io/k8scsi/csi-provisioner@sha256:e34cfa9582b17b701729cac575b6d8bb41bf55e6a35872a6462d30212cc24843
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
    State:          Running
      Started:      Sat, 05 Apr 2025 18:53:51 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sxhv2 (ro)
  csi-attacher:
    Container ID:  containerd://b45adb1608b5dc41313773ee3f75053b6bf8921dd2d37ae946f4d5600465cce1
    Image:         quay.io/k8scsi/csi-attacher:v3.1.0
    Image ID:      quay.io/k8scsi/csi-attacher@sha256:e3334d505bebc44afba610e2c218ca7b4993cac61fc0951d75033ccc971a1347
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
    State:          Running
      Started:      Sat, 05 Apr 2025 18:53:52 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sxhv2 (ro)
  csi-resizer:
    Container ID:  containerd://727453fc80028a1ddbb06125a9a35e485bd714473dfda9b643f8006629b5e2ba
    Image:         quay.io/k8scsi/csi-resizer:v0.5.0
    Image ID:      quay.io/k8scsi/csi-resizer@sha256:6c6a0332693a7c456378f6abd2bb40611826c1e1a733cadbdae2daab3125b71c
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
    State:          Running
      Started:      Sat, 05 Apr 2025 18:53:52 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sxhv2 (ro)
  csi-moosefs-plugin:
    Container ID:  containerd://cef0b4c93fe4fcd141f4b593a8f84afae6f866a45be833b8810632c35dbeac32
    Image:         ghcr.io/moosefs/moosefs-csi:dev
    Image ID:      ghcr.io/moosefs/moosefs-csi@sha256:88a697911be1a3fcd30dea14cde2c71ee0ad8cc84072de13967e8dfad86f7d36
    Port:          <none>
    Host Port:     <none>
    Args:
      --mode=controller
      --csi-endpoint=$(CSI_ENDPOINT)
      --master-host=$(MASTER_HOST)
      --master-port=$(MASTER_PORT)
      --root-dir=$(ROOT_DIR)
      --plugin-data-dir=$(WORKING_DIR)
      --mfs-logging=$(MFS_LOGGING)
      --mfs-mount-options=$(MFS_MOUNT_OPTIONS)
    State:          Running
      Started:      Sat, 05 Apr 2025 18:53:58 -0500
    Ready:          True
    Restart Count:  0
    Environment:
      CSI_ENDPOINT:       unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      MASTER_HOST:        <set to the key 'master_host' of config map 'csi-moosefs-config'>         Optional: false
      MASTER_PORT:        <set to the key 'master_port' of config map 'csi-moosefs-config'>         Optional: false
      ROOT_DIR:           <set to the key 'k8s_root_dir' of config map 'csi-moosefs-config'>        Optional: false
      WORKING_DIR:        <set to the key 'driver_working_dir' of config map 'csi-moosefs-config'>  Optional: false
      MFS_LOGGING:        <set to the key 'mfs_logging' of config map 'csi-moosefs-config'>         Optional: false
      MFS_MOUNT_OPTIONS:  <set to the key 'mfs_mount_options' of config map 'csi-moosefs-config'>   Optional: false
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sxhv2 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-sxhv2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  25m   default-scheduler  Successfully assigned kube-system/csi-moosefs-controller-0 to epyc1
  Normal  Pulling    25m   kubelet            Pulling image "quay.io/k8scsi/csi-provisioner:v2.1.0"
  Normal  Pulled     25m   kubelet            Successfully pulled image "quay.io/k8scsi/csi-provisioner:v2.1.0" in 4.841s (4.841s including waiting)
  Normal  Created    25m   kubelet            Created container csi-provisioner
  Normal  Started    25m   kubelet            Started container csi-provisioner
  Normal  Pulling    25m   kubelet            Pulling image "quay.io/k8scsi/csi-attacher:v3.1.0"
  Normal  Pulled     25m   kubelet            Successfully pulled image "quay.io/k8scsi/csi-attacher:v3.1.0" in 493ms (493ms including waiting)
  Normal  Created    25m   kubelet            Created container csi-attacher
  Normal  Started    25m   kubelet            Started container csi-attacher
  Normal  Pulling    25m   kubelet            Pulling image "quay.io/k8scsi/csi-resizer:v0.5.0"
  Normal  Pulled     25m   kubelet            Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.5.0" in 413ms (413ms including waiting)
  Normal  Created    25m   kubelet            Created container csi-resizer
  Normal  Started    25m   kubelet            Started container csi-resizer
  Normal  Pulling    25m   kubelet            Pulling image "ghcr.io/moosefs/moosefs-csi:dev"
  Normal  Pulled     25m   kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:dev" in 5.33s (5.33s including waiting)
  Normal  Created    25m   kubelet            Created container csi-moosefs-plugin
  Normal  Started    25m   kubelet            Started container csi-moosefs-plugin

[cyrax13]

Hello there)

The same issue, but on k8s (

Software versions:

• Kubernetes: k8s v1.32.5
• MooseFS Community Edition: 4.57.7 all packages from official repo
• OS: UbuntuServer 24.04 LTS on all 5x nodes
• Cluster nodes: 1x master server, 1x metalogger server, 3x chunk servers, total 5x
• moosefs-csi image: ghcr.io/moosefs/moosefs-csi:xxxxx, tested on tags 0.9.8-4.57.6, 0.9.8-4.57.7, dev

Config: ~/moosefs-csi/deploy/csi-moosefs-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-system
  name: csi-moosefs-config
data:
  master_host: "vmk40.vms.local" # local dns domain with master server
  master_port: "9421"            # ports opened
  k8s_root_dir: "/kube"
  driver_working_dir: "pv_data"
  mount_count: "1"
  mfs_logging: "true"
  mfs_mount_options: "mfsmd5pass=[MD5 hash]"

Patch /kube/pv_data exist on mfs filesystem.

Csi file: ~/moosefs-csi/deploy/csi-moosefs.yaml

For moosefs-controller and moosefs-node change tag version on image strings, test three times like this:
image: ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.6
image: ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7
image: ghcr.io/moosefs/moosefs-csi:dev

Deploy:

kubectl apply -f ~/moosefs-csi/deploy/csi-moosefs-config.yaml
kubectl apply -f ~/moosefs-csi/deploy/csi-moosefs.yaml

After the same problem, pods:

~# kubectl -n kube-system get pods | grep moosefs
csi-moosefs-controller-0                      3/4     CrashLoopBackOff   1 (4s ago)     11s
csi-moosefs-node-4p2sx                        1/2     CrashLoopBackOff   1 (8s ago)     11s
csi-moosefs-node-6wjnb                        1/2     CrashLoopBackOff   1 (7s ago)     11s
csi-moosefs-node-ff4m2                        1/2     CrashLoopBackOff   1 (7s ago)     11s
csi-moosefs-node-l6cw9                        1/2     CrashLoopBackOff   1 (7s ago)     11s
csi-moosefs-node-lm2cp                        1/2     CrashLoopBackOff   1 (7s ago)     11s
csi-moosefs-node-w26xg                        1/2     CrashLoopBackOff   1 (7s ago)     11s

Controller pod:

~# kubectl -n kube-system describe pod csi-moosefs-controller-0
Name:                 csi-moosefs-controller-0
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      csi-moosefs-controller-sa
Node:                 node16/10.0.2.16
Start Time:           Thu, 24 Jul 2025 19:05:49 +0200
Labels:               app=csi-moosefs-controller
                      apps.kubernetes.io/pod-index=0
                      controller-revision-hash=csi-moosefs-controller-855c846769
                      role=csi-moosefs
                      statefulset.kubernetes.io/pod-name=csi-moosefs-controller-0
Annotations:          <none>
Status:               Running
IP:                   10.0.2.16
IPs:
  IP:           10.0.2.16
Controlled By:  StatefulSet/csi-moosefs-controller
Containers:
  csi-provisioner:
    Container ID:  containerd://a922dd9ea1260b7c625a7c12530387afe35c99fe5be3de4ba1488399b2ec85da
    Image:         quay.io/k8scsi/csi-provisioner:v2.1.0
    Image ID:      quay.io/k8scsi/csi-provisioner@sha256:e34cfa9582b17b701729cac575b6d8bb41bf55e6a35872a6462d30212cc24843
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=$(ADDRESS)
      --v=5
    State:          Running
      Started:      Thu, 24 Jul 2025 19:05:51 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z4b79 (ro)
  csi-attacher:
    Container ID:  containerd://8b4c3c320cd0c660c061fa7bf62595234269cb1d6b1f9e9455d8c868104ee66b
    Image:         quay.io/k8scsi/csi-attacher:v3.1.0
    Image ID:      quay.io/k8scsi/csi-attacher@sha256:e3334d505bebc44afba610e2c218ca7b4993cac61fc0951d75033ccc971a1347
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
    State:          Running
      Started:      Thu, 24 Jul 2025 19:05:52 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z4b79 (ro)
  csi-resizer:
    Container ID:  containerd://2e2ca3650c3080cdebe05fce87955483483ea76ff5f7296db344d2f6805106d6
    Image:         quay.io/k8scsi/csi-resizer:v0.5.0
    Image ID:      quay.io/k8scsi/csi-resizer@sha256:6c6a0332693a7c456378f6abd2bb40611826c1e1a733cadbdae2daab3125b71c
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
    State:          Running
      Started:      Thu, 24 Jul 2025 19:05:53 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:  /var/lib/csi/sockets/pluginproxy/csi.sock
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z4b79 (ro)
  csi-moosefs-plugin:
    Container ID:  containerd://439cb152cbe9b083560fd94ef78c19a876800fe396b3b1327ebeb230fde7b858
    Image:         ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7
    Image ID:      ghcr.io/moosefs/moosefs-csi@sha256:8412a1c9c8df80c30ac63e9c9fe9427108a4354f027dd5c34670f531ee5bbe33
    Port:          <none>
    Host Port:     <none>
    Args:
      --mode=controller
      --csi-endpoint=$(CSI_ENDPOINT)
      --master-host=$(MASTER_HOST)
      --master-port=$(MASTER_PORT)
      --root-dir=$(ROOT_DIR)
      --plugin-data-dir=$(WORKING_DIR)
      --mfs-logging=$(MFS_LOGGING)
      --mfs-mount-options=$(MFS_MOUNT_OPTIONS)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 24 Jul 2025 19:06:14 +0200
      Finished:     Thu, 24 Jul 2025 19:06:14 +0200
    Ready:          False
    Restart Count:  2
    Environment:
      CSI_ENDPOINT:       unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      MASTER_HOST:        <set to the key 'master_host' of config map 'csi-moosefs-config'>         Optional: false
      MASTER_PORT:        <set to the key 'master_port' of config map 'csi-moosefs-config'>         Optional: false
      ROOT_DIR:           <set to the key 'k8s_root_dir' of config map 'csi-moosefs-config'>        Optional: false
      WORKING_DIR:        <set to the key 'driver_working_dir' of config map 'csi-moosefs-config'>  Optional: false
      MFS_LOGGING:        <set to the key 'mfs_logging' of config map 'csi-moosefs-config'>         Optional: false
      MFS_MOUNT_OPTIONS:  <set to the key 'mfs_mount_options' of config map 'csi-moosefs-config'>   Optional: false
    Mounts:
      /var/lib/csi/sockets/pluginproxy/ from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z4b79 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  socket-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-z4b79:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  43s                default-scheduler  Successfully assigned kube-system/csi-moosefs-controller-0 to node16
  Normal   Pulling    43s                kubelet            Pulling image "quay.io/k8scsi/csi-provisioner:v2.1.0"
  Normal   Pulled     43s                kubelet            Successfully pulled image "quay.io/k8scsi/csi-provisioner:v2.1.0" in 682ms (682ms including waiting). Image size: 21272437 bytes.
  Normal   Created    43s                kubelet            Created container: csi-provisioner
  Normal   Started    42s                kubelet            Started container csi-provisioner
  Normal   Pulling    42s                kubelet            Pulling image "quay.io/k8scsi/csi-attacher:v3.1.0"
  Normal   Pulled     41s                kubelet            Successfully pulled image "quay.io/k8scsi/csi-resizer:v0.5.0" in 545ms (545ms including waiting). Image size: 18412631 bytes.
  Normal   Started    41s                kubelet            Started container csi-attacher
  Normal   Pulling    41s                kubelet            Pulling image "quay.io/k8scsi/csi-resizer:v0.5.0"
  Normal   Pulled     41s                kubelet            Successfully pulled image "quay.io/k8scsi/csi-attacher:v3.1.0" in 462ms (925ms including waiting). Image size: 20167435 bytes.
  Normal   Created    41s                kubelet            Created container: csi-resizer
  Normal   Created    41s                kubelet            Created container: csi-attacher
  Normal   Started    40s                kubelet            Started container csi-resizer
  Normal   Pulled     39s                kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 726ms (1.074s including waiting). Image size: 73596247 bytes.
  Normal   Pulled     37s                kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 1.682s (1.682s including waiting). Image size: 73596247 bytes.
  Normal   Pulling    20s (x3 over 40s)  kubelet            Pulling image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7"
  Normal   Started    19s (x3 over 39s)  kubelet            Started container csi-moosefs-plugin
  Normal   Created    19s (x3 over 39s)  kubelet            Created container: csi-moosefs-plugin
  Normal   Pulled     19s                kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 733ms (733ms including waiting). Image size: 73596247 bytes.
  Warning  BackOff    6s (x5 over 37s)   kubelet            Back-off restarting failed container csi-moosefs-plugin in pod csi-moosefs-controller-0_kube-system(73c8648f-18b6-446e-9f5d-0b4d004f3fae)

Controller pod logs:

~# kubectl -n kube-system logs csi-moosefs-controller-0
Defaulted container "csi-provisioner" out of: csi-provisioner, csi-attacher, csi-resizer, csi-moosefs-plugin
I0724 17:05:51.179586       1 feature_gate.go:243] feature gates: &{map[]}
I0724 17:05:51.179725       1 csi-provisioner.go:132] Version: v2.1.0-0-gb7a8fe9fe
I0724 17:05:51.179746       1 csi-provisioner.go:155] Building kube configs for running in cluster...
I0724 17:05:51.200239       1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0724 17:06:01.200746       1 connection.go:172] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0724 17:06:11.201310       1 connection.go:172] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0724 17:06:21.200891       1 connection.go:172] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0724 17:06:31.200684       1 connection.go:172] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0724 17:06:41.201913       1 connection.go:172] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
W0724 17:06:51.201015       1 connection.go:172] Still connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock

Node pod:

~# kubectl -n kube-system describe pod csi-moosefs-node-w26xg
Name:                 csi-moosefs-node-w26xg
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      csi-moosefs-node-sa
Node:                 node19/10.0.2.19
Start Time:           Thu, 24 Jul 2025 19:05:49 +0200
Labels:               app=csi-moosefs-node
                      controller-revision-hash=547897bfcc
                      pod-template-generation=1
                      role=csi-moosefs
Annotations:          <none>
Status:               Running
IP:                   10.0.2.19
IPs:
  IP:           10.0.2.19
Controlled By:  DaemonSet/csi-moosefs-node
Containers:
  driver-registrar:
    Container ID:  containerd://27f5442fc78caf2ea601b915482d7c21baffc929369529ad890879fa8da3e83d
    Image:         quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
    Image ID:      quay.io/k8scsi/csi-node-driver-registrar@sha256:fb4f139b51ae5b449a132949d7f3d839fa9ea5c57df1c1cd61f781389fdacefb
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    State:          Running
      Started:      Thu, 24 Jul 2025 19:05:50 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:               /csi/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/csi.moosefs.com/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
    Mounts:
      /csi/ from plugin-dir (rw)
      /registration/ from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c7ptp (ro)
  csi-moosefs-plugin:
    Container ID:  containerd://afcc4404729f92a5286776faf5b0e1630dccfa0e3a5dd07625549994f847a1ae
    Image:         ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7
    Image ID:      ghcr.io/moosefs/moosefs-csi@sha256:8412a1c9c8df80c30ac63e9c9fe9427108a4354f027dd5c34670f531ee5bbe33
    Port:          <none>
    Host Port:     <none>
    Args:
      --mode=node
      --csi-endpoint=$(CSI_ENDPOINT)
      --master-host=$(MASTER_HOST)
      --master-port=$(MASTER_PORT)
      --node-id=$(NODE_ID)
      --root-dir=$(ROOT_DIR)
      --plugin-data-dir=$(WORKING_DIR)
      --mount-points-count=$(MOUNT_COUNT)
      --mfs-logging=$(MFS_LOGGING)
      --mfs-mount-options=$(MFS_MOUNT_OPTIONS)
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 24 Jul 2025 19:07:14 +0200
      Finished:     Thu, 24 Jul 2025 19:07:15 +0200
    Ready:          False
    Restart Count:  4
    Environment:
      CSI_ENDPOINT:       unix:///csi/csi.sock
      MASTER_HOST:        <set to the key 'master_host' of config map 'csi-moosefs-config'>  Optional: false
      MASTER_PORT:        <set to the key 'master_port' of config map 'csi-moosefs-config'>  Optional: false
      NODE_ID:             (v1:spec.nodeName)
      ROOT_DIR:           <set to the key 'k8s_root_dir' of config map 'csi-moosefs-config'>        Optional: false
      WORKING_DIR:        <set to the key 'driver_working_dir' of config map 'csi-moosefs-config'>  Optional: false
      MOUNT_COUNT:        <set to the key 'mount_count' of config map 'csi-moosefs-config'>         Optional: false
      MFS_LOGGING:        <set to the key 'mfs_logging' of config map 'csi-moosefs-config'>         Optional: false
      MFS_MOUNT_OPTIONS:  <set to the key 'mfs_mount_options' of config map 'csi-moosefs-config'>   Optional: false
    Mounts:
      /csi from plugin-dir (rw)
      /dev from device-dir (rw)
      /var/lib/kubelet from pods-mount-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c7ptp (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  Directory
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/csi.moosefs.com
    HostPathType:  DirectoryOrCreate
  pods-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet
    HostPathType:  Directory
  device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:
  kube-api-access-c7ptp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  111s                default-scheduler  Successfully assigned kube-system/csi-moosefs-node-w26xg to node19
  Normal   Pulled     111s                kubelet            Container image "quay.io/k8scsi/csi-node-driver-registrar:v2.1.0" already present on machine
  Normal   Created    111s                kubelet            Created container: driver-registrar
  Normal   Started    111s                kubelet            Started container driver-registrar
  Normal   Pulled     110s                kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 633ms (633ms including waiting). Image size: 73596247 bytes.
  Normal   Pulled     109s                kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 952ms (952ms including waiting). Image size: 73596247 bytes.
  Normal   Pulled     93s                 kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 766ms (766ms including waiting). Image size: 73596247 bytes.
  Normal   Pulled     69s                 kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 597ms (597ms including waiting). Image size: 73596247 bytes.
  Normal   Pulling    28s (x5 over 111s)  kubelet            Pulling image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7"
  Normal   Created    27s (x5 over 110s)  kubelet            Created container: csi-moosefs-plugin
  Normal   Started    27s (x5 over 110s)  kubelet            Started container csi-moosefs-plugin
  Normal   Pulled     27s                 kubelet            Successfully pulled image "ghcr.io/moosefs/moosefs-csi:0.9.8-4.57.7" in 1.002s (1.002s including waiting). Image size: 73596247 bytes.
  Warning  BackOff    2s (x10 over 107s)  kubelet            Back-off restarting failed container csi-moosefs-plugin in pod csi-moosefs-node-w26xg_kube-system(3c2de134-acf9-4956-a7d5-115b935a85f2)

Node pod log:

~# kubectl -n kube-system logs csi-moosefs-node-w26xg
Defaulted container "driver-registrar" out of: driver-registrar, csi-moosefs-plugin
I0724 17:05:50.646652       1 main.go:113] Version: v2.1.0-0-g80d42f24
I0724 17:05:50.648143       1 main.go:137] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0724 17:05:50.648207       1 connection.go:153] Connecting to unix:///csi/csi.sock
W0724 17:06:00.648729       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:06:10.648747       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:06:20.649148       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:06:30.649321       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:06:40.648600       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:06:50.648708       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:07:00.648367       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:07:10.648968       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:07:20.649862       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:07:30.649198       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:07:40.648852       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:07:50.649103       1 connection.go:172] Still connecting to unix:///csi/csi.sock
W0724 17:08:00.648596       1 connection.go:172] Still connecting to unix:///csi/csi.sock

Switch to dev branch did not help.

Any ideas?)