No space left on one pod causing it go into crashlookbackoff

[shobha-my]

Hello,

I'm using JBOD for persistent storage, my initial volume of 100 GB (got full), one of the pod is in crash state.

Things I've tried:

1. Added another volume of 200 Gb and did kubectl apply (even run the annotate rollback) - but no change.
2. Delete all topics as it's a test cluster, was hoping the disk space will be freed on the crashing pod but no change.
3. Finally deleted the pod to see if while coming up it'll fix itself - instead the pod is not coming up only.

I have 3 zookeeper and 3 kafka pods setup, right now only 2 pods are online. I tried again to apply the kafka.yaml through kubectl but no change.

Require help to understand how can I fix the space issue in case of no more space left (Host machine has space no issue over there) & how can I bring the deleted pod up.

Thank you.

[scholzj]

Kafka does not handle running out of disk space well. So you should in the first place make sure you do not run out of disk space. If you already run out of disk space and you broker is crashlooping, then you have in general two options:

• Increase the size of the current volumes (careful -> not reversible action, you cannot easily shrink it later on most platforms) which should get the brokers running again
• Mount the volumes into some helper pods and delete some data from them. Once there is enough space, the brokers can start again.

Only then you can think about the next step:

• Delete unused topics
• Reconfigure your topics to make sure they have the proper retention settings
• Add additional volumes to the JbOD storage array and rebalance the Kafka cluster

[shobha-my]

Storage was not resized, not sure if my storage supports resizing (using OpenEBS).

[scholzj]

Then I guess you would need to go the other way and mount the volume in some helper pod and delete some data manually.

[shobha-my]

Okay. I'm trying to recreate the kafka cluster as it's a dev env. KAFKA kind is not starting now.

2022-08-09 21:10:02 WARN AbstractOperator:532 - Reconciliation #23(timer) Kafka(kafka/kafka-cluster): Failed to reconcile
java.lang.RuntimeException: java.io.IOException: No space left on device
at io.strimzi.operator.cluster.model.Ca.generateCaKeyAndCert(Ca.java:1029) ~[io.strimzi.operator-common-0.30.0.jar:0.30.0]
at io.strimzi.operator.cluster.model.Ca.createRenewOrReplace(Ca.java:539) ~[io.strimzi.operator-common-0.30.0.jar:0.30.0]
at io.strimzi.operator.cluster.operator.assembly.CaReconciler.lambda$reconcileCas$7(CaReconciler.java:266) ~[io.strimzi.cluster-operator-0.30.0.jar:0.30.0]
at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76) ~[io.vertx.vertx-core-4.2.4.jar:4.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.77.Final.jar:4.1.77.Final]

I've dedicated a new node for kafka. I think something is wrong with the strimzi operator. Also, I've all nodes, they have space in it.

[shobha-my]

I tried tainting the node and installed the strimzi operator also in the new node, still no space issue. Any tips what else I can do to debug this situation?

[scholzj]

You probably need to delete the storage first before recreating the cluster?