No space left on device issue while installing

[amarmy]

Hello,

I'm installing latest verion of strimzi-kafka-operator (0.31.1) on AWS EKS with kubernetes version (v1.22.13-eks-).

I have four nodes tainted for kafka&Zookeeper (32 GB & 16 cores) with 1 TB EBS. Getting below error while installing the kafka:

2022-10-27 21:22:57 WARN  AbstractOperator:516 - Reconciliation #4(timer) Kafka(team-7-kafka/team-7-kafka-cluster): Failed to reconcile
java.lang.RuntimeException: java.io.IOException: No space left on device
	at io.strimzi.operator.cluster.model.Ca.generateCaKeyAndCert(Ca.java:1031) ~[io.strimzi.operator-common-0.31.1.jar:0.31.1]
	at io.strimzi.operator.cluster.model.Ca.createRenewOrReplace(Ca.java:542) ~[io.strimzi.operator-common-0.31.1.jar:0.31.1]
	at io.strimzi.operator.cluster.operator.assembly.CaReconciler.lambda$reconcileCas$7(CaReconciler.java:266) ~[io.strimzi.cluster-operator-0.31.1.jar:0.31.1]
	at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62) ~[?:?]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113) ~[?:?]
	at sun.nio.ch.IOUtil.write(IOUtil.java:79) ~[?:?]
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:280) ~[?:?]
	at java.nio.channels.Channels.writeFullyImpl(Channels.java:74) ~[?:?]
	at java.nio.channels.Channels.writeFully(Channels.java:97) ~[?:?]
	at java.nio.channels.Channels$1.write(Channels.java:172) ~[?:?]
	at java.io.InputStream.transferTo(InputStream.java:705) ~[?:?]
	at java.nio.file.Files.copy(Files.java:3078) ~[?:?]
	at io.strimzi.certs.OpenSslCertManager.createDefaultConfig(OpenSslCertManager.java:105) ~[io.strimzi.certificate-manager-0.31.1.jar:0.31.1]
	at io.strimzi.certs.OpenSslCertManager.buildConfigFile(OpenSslCertManager.java:118) ~[io.strimzi.certificate-manager-0.31.1.jar:0.31.1]
	at io.strimzi.certs.OpenSslCertManager.generateCaCert(OpenSslCertManager.java:259) ~[io.strimzi.certificate-manager-0.31.1.jar:0.31.1]
	at io.strimzi.certs.OpenSslCertManager.generateRootCaCert(OpenSslCertManager.java:168) ~[io.strimzi.certificate-manager-0.31.1.jar:0.31.1]
	at io.strimzi.certs.OpenSslCertManager.generateSelfSignedCert(OpenSslCertManager.java:147) ~[io.strimzi.certificate-manager-0.31.1.jar:0.31.1]
	at io.strimzi.operator.cluster.model.Ca.generateCaKeyAndCert(Ca.java:1009) ~[io.strimzi.operator-common-0.31.1.jar:0.31.1]

Below is the kafka configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: team-7-kafka-cluster
spec:
  kafka:
    version: 3.2.3
    replicas: 3
    resources:
      requests:
        memory: "24Gi"
        cpu: "8"
      limits:
        memory: "24Gi"
        cpu: "12"
    template:
      pod:
        tolerations:
          - key: "dedicated"
            operator: "Equal"
            value: "is-kafka"
            effect: "NoSchedule"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: purpose
                      operator: In
                      values:
                        - "is-kafka"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    jvmOptions:
      "-Xms": "8192m"
      "-Xmx": "8192m"
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      # - name: tls
      #   port: 9093
      #   type: internal
      #   tls: true
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
    config:
      auto.create.topics.enable: false
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.2"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 1000Gi
        deleteClaim: false
        class: gp2
    rack:
      topologyKey: "topology.kubernetes.io/zone"
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
  zookeeper:
    replicas: 3
    template:
      pod:
        tolerations:
          - key: "dedicated"
            operator: "Equal"
            value: "is-kafka"
            effect: "NoSchedule"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: purpose
                      operator: In
                      values:
                        - "is-kafka"
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    storage:
      type: persistent-claim
      size: 320Gi
      deleteClaim: false
      class: gp2
    resources:
      requests:
        memory: "8Gi"
        cpu: "2"
      limits:
        memory: "8Gi"
        cpu: "4"
    jvmOptions:
      "-Xms": "4096m"
      "-Xmx": "4096m"
    logging:
      type: inline
      loggers:
        zookeeper.root.logger: "INFO"
  entityOperator:
    topicOperator: {}
    userOperator: {}

Not sure where should I look to debug the issue. Any help will be appreciated.

Thank you!

[scholzj]

That sounds like you have no space in the Cluster Operator pod in the /tmp directory. So you should probably check what is there inside. You can increase the emptyDir volume used for /tmp dir in the Cluster Operator Deployment. But it would be good to know why is it full.

[amarmy]

Below is the content from the pod and the /tmp dir:

[strimzi@strimzi-cluster-operator-647f7577cf-l7qc4 strimzi]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         1.0T  5.2G 1019G   1% /
tmpfs            64M     0   64M   0% /dev
tmpfs            16G     0   16G   0% /sys/fs/cgroup
tmpfs           1.0M  1.0M     0 100% /tmp
/dev/nvme0n1p1  1.0T  5.2G 1019G   1% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           384M   12K  384M   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            16G     0   16G   0% /proc/acpi
tmpfs            16G     0   16G   0% /sys/firmware
[strimzi@strimzi-cluster-operator-647f7577cf-l7qc4 strimzi]$ cd /tmp/
[strimzi@strimzi-cluster-operator-647f7577cf-l7qc4 tmp]$ ls -l
total 0
-rw-r--r-- 1 strimzi root  0 Oct 27 21:34 10826261110237062736.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:32 11090107692079372282.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:40 12777363623365540164.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:22 15537130604871443427.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:36 18374906629298407009.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:22 2179504821565969910.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:38 2975222718723258840.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:30 3469462711872467553.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:26 3558713481931695092.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:24 3619980543736504271.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:22 4270762029160988773.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:22 479182517982228027.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:44 5687356401317423578.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:28 7091532006610429797.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:46 7281571643558792853.tmp
-rw-r--r-- 1 strimzi root  0 Oct 27 21:42 7417068356354658890.tmp
drwxr-xr-x 2 strimzi root 60 Oct 27 21:08 hsperfdata_strimzi
drwxr-xr-x 2 strimzi root 60 Oct 27 21:09 px-java-symbolization-artifacts-27352-21499112-8211461f58b7148496531a00949f7485
drwx------ 2 strimzi root 40 Oct 27 21:08 vertx-cache-5ed15277-f559-49c8-ac5f-e795f103ec79

the size of /tmp is less. Do you see any other problem here?

[scholzj]

The usual context looks like this:

ls -l /tmp/
total 0
drwxr-xr-x 2 strimzi root 60 Oct 26 21:50 hsperfdata_strimzi
drwx------ 2 strimzi root 40 Oct 26 21:50 vertx-cache-8de25968-8c34-4e40-9213-038fc8d99ada

You have these files, as well. Not sure what are those other files. The NNN.tmp files might be result of some bug. Hard to say without seeing the whole log from the operator. But the remaining file is weird. Maybe you have something in your cluster what injects these?

When you delete the pod with kubectl delete pod, you should start with empty /tmp. As I said, you can also increase the size if needed by editing the Deployment.

[amarmy]

Yes, I've increased the tmpDirSizeLimit: 1Mi to 192Mi. And it works. I think those files are injected by newrelic, pixie & vertx pod.

Thanks a lot for the quick help. Really appreciated.

Also, I'm using the above kafka configuration for production purpose, do you suggest any changes which will be helpful for the prod environment.

Thanks again!

[scholzj]

192Mi is quite a lot and I think it is taken from the Pod memory. With such a bif emptyDir, you might want to remove the medium: Memory so that it uses actual disk space: https://github.com/strimzi/strimzi-kafka-operator/blob/main/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml#L22-L24

This is a bit tricky as every user has different requirements. Someone wants things to be as lean as possible. Someone uses all kind of tools and needs it big like you. So TBH, if we increase it, someone else will come tomorrow that it is too big and unused. I think right now, most users do fine with 1Mi because they do not use such tools. So we should stick with that I think as the default.