Kafka pod keep CrashLoopBackOff - Could not read file 00000000000000000002.index

[jackchuong]

Hi all,
I have internal k8s cluster , storage provider is NFS , a QNAP device as NFS server.
I have setup kafka cluster 3 node

kubectl -n kafka get kafka ebms-kafka-new -o yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"Kafka","metadata":{"annotations":{},"name":"ebms-kafka-new","namespace":"kafka"},"spec":{"entityOperator":{"topicOperator":{},"userOperator":{}},"kafka":{"config":{"default.replication.factor":3,"inter.broker.protocol.version":"3.4.0","log.cleanup.policy":"compact,delete","min.insync.replicas":2,"num.partitions":3,"offsets.topic.replication.factor":3,"transaction.state.log.min.isr":2,"transaction.state.log.replication.factor":3},"listeners":[{"name":"plain","port":9092,"tls":false,"type":"internal"},{"name":"tls","port":9093,"tls":true,"type":"internal"}],"replicas":3,"resources":{"limits":{"cpu":"2","memory":"4Gi"},"requests":{"cpu":"0.5","memory":"1Gi"}},"storage":{"type":"jbod","volumes":[{"class":"nfs-idcnas002","deleteClaim":true,"id":0,"size":"10Gi","type":"persistent-claim"}]},"version":"3.4.0"},"zookeeper":{"replicas":3,"storage":{"class":"nfs-idcnas002","deleteClaim":true,"size":"10Gi","type":"persistent-claim"}}}}
  creationTimestamp: "2023-11-22T09:46:39Z"
  generation: 8
  name: ebms-kafka-new
  namespace: kafka
  resourceVersion: "19301995"
  uid: c2cc3083-d8f7-46ec-9481-402c6501fadd
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      default.replication.factor: 3
      inter.broker.protocol.version: 3.4.0
      log.cleanup.policy: compact,delete
      min.insync.replicas: 2
      num.partitions: 3
      offsets.topic.replication.factor: 3
      transaction.state.log.min.isr: 2
      transaction.state.log.replication.factor: 3
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    replicas: 3
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "0.5"
        memory: 1Gi
    storage:
      type: jbod
      volumes:
      - class: nfs-idcnas002
        deleteClaim: true
        id: 0
        size: 10Gi
        type: persistent-claim
    version: 3.4.0
  zookeeper:
    replicas: 3
    storage:
      class: nfs-idcnas002
      deleteClaim: true
      size: 10Gi
      type: persistent-claim
status:
  clusterId: lAk-Im-nQSWXrzktFeurFg
  conditions:
  - lastTransitionTime: "2024-01-22T09:27:56.678505793Z"
    message: Pod is unschedulable or is not starting
    reason: FatalProblem
    status: "True"
    type: NotReady
  observedGeneration: 8

It worked fine.
Several days ago, I put 1 worker node in maintenance to add more memory

kubectl cordon node01
kubectl drain --force --ignore-daemonsets --delete-emptydir-data node01
# add memory, start node again
kubectl uncordon node01

Since then, pod ebms-kafka-new-kafka-0 keep CrashLoopBackOff , this is error log when I tried to delete pod ebms-kafka-new-kafka-0 to force recreate it.

2024-01-24 03:12:11,812 ERROR Error while loading log dir /var/lib/kafka/data-0/kafka-log0 (kafka.server.LogDirFailureChannel) [main]
java.io.IOException: Could not read file /var/lib/kafka/data-0/kafka-log0/eBMSStaging.eBMSCatalogue.dbo.catVehicleType-1/00000000000000000002.index
        at kafka.log.LogLoader.$anonfun$removeTempFilesAndCollectSwapFiles$2(LogLoader.scala:230)
        at scala.collection.ArrayOps$WithFilter.foreach(ArrayOps.scala:73)
        at kafka.log.LogLoader.removeTempFilesAndCollectSwapFiles(LogLoader.scala:228)
        at kafka.log.LogLoader.load(LogLoader.scala:103)
        at kafka.log.UnifiedLog$.apply(UnifiedLog.scala:1853)
        at kafka.log.LogManager.loadLog(LogManager.scala:287)
        at kafka.log.LogManager.$anonfun$loadLogs$14(LogManager.scala:403)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

2 other pods ebms-kafka-new-kafka-2 , ebms-kafka-new-kafka-1 are still running normally.
This is screen shot about file status in NAS device.

Please give me some advice, thank you very much.

[jackchuong]

I deleted file 00000000000000000002.index then pod ebms-kafka-new-kafka-0 recreated it automatically, it works fine now.