Talos with cinder-csi-config from openstack does not work

[suse-coder]

I have a openstack cluster where talos kubenretes is running in. Now i want to use the rbd csi to have from my storageclass (similar to ceph) a new rbd provisioned.

But I get always the error: Warning Failed 3s kubelet Error: failed to generate container "e47d19122acd778fd46400cc72b0157aa2d6aaed9a5ba4a6257041737aad8968" spec: failed to apply OCI options: failed to mkdir "/etc/cacert": mkdir /etc/cacert: read-only file system

How can I solve that?
test-pod.yaml.txt
secrets.yaml.txt
runtime.cue.txt
namespace.yaml.txt
configmaps.yaml.txt
cinder.cue.txt

Name:             openstack-cinder-csi-nodeplugin-6kffz
Namespace:        openstack
Priority:         0
Service Account:  csi-cinder-node-sa
Node:             talos-jsm-624/192.168.178.77
Start Time:       Fri, 28 Feb 2025 10:26:04 +0000
Labels:           app=openstack-cinder-csi
                  chart=openstack-cinder-csi-2.32.0
                  component=nodeplugin
                  controller-revision-hash=55455d9588
                  heritage=Helm
                  pod-template-generation=1
                  release=openstack-csi-cinder
Annotations:      <none>
Status:           Pending
IP:               192.168.178.77
IPs:
  IP:           192.168.178.77
Controlled By:  DaemonSet/openstack-cinder-csi-nodeplugin
Containers:
  node-driver-registrar:
    Container ID:  containerd://1422bdfd6ed847ef33c9ee4b3aca279794286e06e2ffaf31fcdc342c64fae87c
    Image:         registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.12.0
    Image ID:      registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:0d23a6fd60c421054deec5e6d0405dc3498095a5a597e175236c0692f4adee0f
    Port:          <none>
    Host Port:     <none>
    Args:
      -v=2
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    State:          Running
      Started:      Fri, 28 Feb 2025 10:26:06 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:               /csi/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/cinder.csi.openstack.org/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
    Mounts:
      /csi from socket-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzr7z (ro)
  liveness-probe:
    Container ID:  containerd://cd88d8a85b14c9a8e8d1c5083514d603bdc8ec7129f09b0d5e9804f1b1f1d7f8
    Image:         registry.k8s.io/sig-storage/livenessprobe:v2.14.0
    Image ID:      registry.k8s.io/sig-storage/livenessprobe@sha256:33692aed26aaf105b4d6e66280cceca9e0463f500c81b5d8c955428a75438f32
    Port:          <none>
    Host Port:     <none>
    Args:
      -v=2
      --csi-address=/csi/csi.sock
    State:          Running
      Started:      Fri, 28 Feb 2025 10:26:07 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzr7z (ro)
  cinder-csi-plugin:
    Container ID:  
    Image:         registry.k8s.io/provider-os/cinder-csi-plugin:v1.32.0
    Image ID:      
    Port:          9808/TCP
    Host Port:     9808/TCP
    Args:
      /bin/cinder-csi-plugin
      -v=2
      --endpoint=$(CSI_ENDPOINT)
      --provide-controller-service=false
      --cloud-config=$(CLOUD_CONFIG)
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:healthz/healthz delay=10s timeout=10s period=60s #success=1 #failure=5
    Environment:
      CSI_ENDPOINT:  unix://csi/csi.sock
      CLOUD_CONFIG:  /etc/kubernetes/cloud.conf
    Mounts:
      /csi from socket-dir (rw)
      /dev from pods-probe-dir (rw)
      /etc/cacert from cacert (ro)
      /etc/kubernetes from cloud-config (ro)
      /var/lib/kubelet from kubelet-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzr7z (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  socket-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/cinder.csi.openstack.org
    HostPathType:  DirectoryOrCreate
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  Directory
  kubelet-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet
    HostPathType:  Directory
  pods-probe-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
  cloud-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cinder-csi-secret
    Optional:    false
  cacert:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cacert
    HostPathType:  
  kube-api-access-qzr7z:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  8s               default-scheduler  Successfully assigned openstack/openstack-cinder-csi-nodeplugin-6kffz to talos-jsm-624
  Normal   Pulled     7s               kubelet            Container image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.12.0" already present on machine
  Normal   Created    6s               kubelet            Created container: node-driver-registrar
  Normal   Started    6s               kubelet            Started container node-driver-registrar
  Normal   Pulled     6s               kubelet            Container image "registry.k8s.io/sig-storage/livenessprobe:v2.14.0" already present on machine
  Normal   Created    5s               kubelet            Created container: liveness-probe
  Normal   Started    5s               kubelet            Started container liveness-probe
  Warning  Failed     5s               kubelet            Error: failed to generate container "f7f7ba2d290859d0f73dff7a1852dd368b032ec332be3cddaa534e055c04e2a2" spec: failed to apply OCI options: failed to mkdir "/etc/cacert": mkdir /etc/cacert: read-only file system
  Warning  Failed     4s               kubelet            Error: failed to generate container "67e6b7180eb52fe2bcaf3619754a404d31a00c1e3dfb2e368484a48d1d4d4357" spec: failed to apply OCI options: failed to mkdir "/etc/cacert": mkdir /etc/cacert: read-only file system
  Normal   Pulled     3s (x3 over 5s)  kubelet            Container image "registry.k8s.io/provider-os/cinder-csi-plugin:v1.32.0" already present on machine
  Warning  Failed     3s               kubelet            Error: failed to generate container "e47d19122acd778fd46400cc72b0157aa2d6aaed9a5ba4a6257041737aad8968" spec: failed to apply OCI options: failed to mkdir "/etc/cacert": mkdir /etc/cacert: read-only file system

[smira]

Fix your deployment to stop trying to mount non-existing /etc/cacert from the host? I guess it should be that simple.

I don't know why the workload you're trying to run does that, on Talos this path is /etc/ssl/ca-certificates.

There's no path /etc/cacert on Ubuntu for example.

[suse-coder]

thanks

[suse-coder]

Got it almost working with (controlplugin pods work): But the nodeplugin pod gives me an error:

/I0228 14:07:19.760539 1 connection.go:253] "Still connecting" address="unix:///var/lib/csi/sockets/pluginproxy/csi.sock"
I0228 14:07:29.760426 1 connection.go:253] "Still connecting" address="unix:///var/lib/csi/sockets/pluginproxy/csi.sock"
I0228 14:07:39.759965 1 connection.go:253] "Still connecting" address="unix:///var/lib/csi/sockets/pluginproxy/csi.sock"
E0228 14:07:39.760731 1 main.go:149] "Failed to connect to the CSI driver" err="context deadline exceeded" csiAddress="/var/lib/csi/sockets/pluginproxy/csi.sock"

where in talos should that point to?

cinder.cue (2).txt

[smira]

I don't think it's anywhere in Talos (?). Not sure what this is what it tries to connect to.

[suse-coder]

Its now working fully:
configmap.yaml.txt
namespace.yaml.txt
runtime.cue.txt
secrets.yaml.txt
test-pod.yaml.txt
cinder.cue.txt