Kafka cluster could not create pvc while different namespace with kafka operator

[chary1112004]

Hi,

We have tried install kafka operator in namespace kafka. And kafka cluster in namespace kafka-dev.

1. Kafka operator: is installed with helm chart https://artifacthub.io/packages/helm/strimzi/strimzi-kafka-operator

helm install my-strimzi-kafka-operator strimzi/strimzi-kafka-operator --version 0.34.0 -n kafka

with value file

watchNamespaces: [kafka-dev]

2. Kafka cluster: We could see that kafka zookeeper is provision however there is message that no pvc is created.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 3.4.0
    replicas: 3
    listeners:
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
    authorization:
      type: simple
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      num.io.threads: 8
      num.network.threads: 5
      num.partitions: 1
      num.replica.fetchers: 2
      message.max.bytes: 15728640
      replica.fetch.max.bytes: 15728640
      replica.fetch.response.max.bytes: 15728640
      replica.lag.time.max.ms: 30000
      socket.receive.buffer.bytes: 102400
      socket.request.max.bytes: 104857600
      socket.send.buffer.bytes: 102400
      unclean.leader.election.enable: true
      zookeeper.session.timeout.ms: 18000
      inter.broker.protocol.version: "3.4"
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 20Gi
          class: kafka-sc-gp2
          deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 30Gi
      class: kafka-sc-gp2
      deleteClaim: false
  entityOperator:
    topicOperator:
      watchedNamespace: tenant1
    userOperator:
      watchedNamespace: tenant1

$ kubectl get pods -n kafka-dev

NAME                        READY   STATUS    RESTARTS   AGE
kafka-cluster-zookeeper-0   0/1     Pending   0          27m
kafka-cluster-zookeeper-1   0/1     Pending   0          27m
kafka-cluster-zookeeper-2   0/1     Pending   0          27m

$ kubectl get pvc -n kafka-dev
No resources found in kafka-dev namespace.

Any solution to resolve this issue would be very much appreciated.

Thanks!

[scholzj]

You would probably need to share the full operator log. Also, it is not really clear how you deployed the operator. You should also doublecheck you don't have any other tooling such as ArgoCD etc. deleting the PVCs while the operator is creating them.

[chary1112004]

Hi @scholzj,

Yes, I have tested by create resource by manual then it works expect.

For the issue, I have deployed kafka operator and kafka cluster by argocd. Reason could be from how I organize project, application in argocd.
I have separated independently between kafka operator and kafka cluster then it works now.

Thanks!

[scholzj]

No, I think the issue is that ArgoCD decides to delete the PVCs in between the operator creating them and the pods starting. I think in the past, the options to tell Argo to not do it which I saw mentioned were:

• Setting some custom labels / annotations on the PVCs to tell Argo to not delete them (You can set custom labels / anotations in Kafka resource in .spec.kafka.template)
• Switch argoCD to use for example annotation based tracking

There are some other discussions on this Argo topic here with some more details.

[chary1112004]

Hi @scholzj,

Thank you. I will check other relate threads with the point you have mentioned.

Thanks!