Strimzi + Longhorn not able to mount/attach volumes

[simonebenati]

As the title suggests I'm not able to attach volumes to zookeeper pods.
The 3 pvc get created correctly:

k get pvc
NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-kafka-cluster-test-zookeeper-0   Bound    pvc-7c0a9002-2ac6-4b13-85a4-c4f04b6ed74a   10Gi       RWO            longhorn       2d15h
data-kafka-cluster-test-zookeeper-1   Bound    pvc-27de59f7-0d10-4060-a40b-32191d4ce25e   10Gi       RWO            longhorn       2d15h
data-kafka-cluster-test-zookeeper-2   Bound    pvc-24b5b8ea-1861-4a1b-a4d3-89fdf63b2045   10Gi       RWO            longhorn       2d15h

And they say they're bound but then when I describe the pod that supposedly this pvc is linked to I get this:

kubelet  MountVolume.MountDevice failed for volume "pvc-27de59f7-0d10-4060-a40b-32191d4ce25e" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-27de59f7-0d10-4060-a40b-32191d4ce25e" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/f0356a83725c06bcc22fac6d91d49f862518e56913cf0e402104f9c23f88adca/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021)
/dev/longhorn/pvc-27de59f7-0d10-4060-a40b-32191d4ce25e is apparently in use by the system; will not make a filesystem here!
)
  Warning  FailedMount  9m7s (x251 over 2d14h)   kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[zookeeper-metrics-and-logging zookeeper-nodes cluster-ca-certs kube-api-access-h668j strimzi-tmp data]: timed out waiting for the condition
  Warning  FailedMount  4m35s (x372 over 2d15h)  kubelet  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[strimzi-tmp data zookeeper-metrics-and-logging zookeeper-nodes cluster-ca-certs kube-api-access-h668j]: timed out waiting for the condition

Do you have any idea/suggestion on how to make this work? I wouldn't know how to proceed on this..
Thank you

[simonebenati]

Switching zookeeper storage type from persistent-claim to ephemeral makes the cluster run.
But doesn't solve the issue, I would like to have persistent-storage

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster-test
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.4"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 12Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

[scholzj]

Have you asked Longhorn about this issue? The error does not seem to suggest it has anything to do with Strimzi. Strimzi does not mount the volume or format it. That is done by Kubernetes ad the storage drivers.

[simonebenati]

Hello @scholzj , no I haven't asked longhorn just yet. I thought it had to due to some wrong config in my Kind: Kafka but thank you for pointing me in the right direction.

[scholzj]

To be honest, I do not know for sure what is going wrong here ... it's just that the messages you shared are basically all about the things Kubernetes does.