auto.create.topics.enable: "false" ignored

[Cave-Johnson]

I'm testing strimzi with a basic app that uses faust-streaming. Faust likes to auto create topics if they do not exist which I currently want to disable.
In my kafka deployment yaml I am setting auto.create.topics.enable: "false" under the config section however this is being ignored and faust is still auto creating the topics

Full kafka-deployment.yaml

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-test
spec:
  kafka:
    version: 3.1.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: nodeport
        tls: false
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.1"
      auto.create.topics.enable: "false"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: false
  entityOperator:
    template:
      topicOperatorContainer:
        env:
          - name: STRIMZI_USE_ZOOKEEPER_TOPIC_STORE
            value: "true"
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

In addition, if i create a topic through a yaml file as outlined in the quick start guide, this applies and I can see it within rancher. However none of my faust producers and consumers can see or use this topic. What am i missing here?

The error i get is [2022-04-05 23:15:47,104] [1] [ERROR] Topic test-topic not found in cluster metadata

[scholzj]

In my kafka deployment yaml I am setting auto.create.topics.enable: "false" under the config section however this is being ignored and faust is still auto creating the topics

How do you know the topic was auto-created and not intentionally created by your application?

In addition, if i create a topic through a yaml file as outlined in the quick start guide, this applies and I can see it within rancher. However none of my faust producers and consumers can see or use this topic. What am i missing here?

The error i get is [2022-04-05 23:15:47,104] [1] [ERROR] Topic test-topic not found in cluster metadata

It is not clear where do you get the error. You also didn't shared any logs or full custom resources (including the .status section), so it is hard to say whather they were seen, whether there is some error etc.

[Cave-Johnson]

How do you know the topic was auto-created and not intentionally created by your application?

I believe this is the case because its a random ID added after the topic name such as ---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a, I was under the impression that setting auto.create.topics.enable: "false" would stop the application from being able to create any topics, is this not the case? I would rather define all my topics through yaml so I have control over what topics exist.

It is not clear where do you get the error. You also didn't shared any logs or full custom resources (including the .status section), so it is hard to say whather they were seen, whether there is some error etc.

Apologies, this error is from the application. Were do i find the output from .status?

Full custom resource is

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: test-topic
  namespace: my-kafka-cluster
  labels:
    strimzi.io/cluster: "my-cluster"
spec:
  partitions: 3
  replicas: 3
  config:
    retention.ms: 720000000
    segment.bytes: 1073741824

[scholzj]

I believe this is the case because its a random ID added after the topic name such as ---84e7a678d08f4bd226872e5cdd4eb527fadc1c6a, I was under the impression that setting auto.create.topics.enable: "false" would stop the application from being able to create any topics, is this not the case? I would rather define all my topics through yaml so I have control over what topics exist.

No, that is not the case. The topic auto-creation means that the broker creates the topic when some application tries to send or receive messages from it. But it does not prevent the application from creating the topic programatically. I do not really know Faust - but AFAIK it is kind-of an Python alternative to Kafka Streams API - and Kafka Streams API definitely creates the topics on its own programatically. So maybe Faust does the same?

Apologies, this error is from the application. Were do i find the output from .status?

You need to get the resource from Kubernetes. E.g. kubectl get kafkatopic my-topic -o yaml ... that will show it including the status which for example shows whether the operator saw it and what does it think about it.

[Cave-Johnson]

When I get the resource using your command above there is no .status section on the topics created using the yaml file i listed above. If i run it for one of the auto created topics i get

kubectl get kafkatopic autocreatedtopic-assignor-leader---af60747ed9b09f245af1c75ee40c0e2819677270 -o yaml -n my-kafka-cluster
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  creationTimestamp: "2022-04-05T23:26:36Z"
  generation: 1
  labels:
    strimzi.io/cluster: my-kafka-cluster
  name: autocreatedtopic-assignor-leader---af60747ed9b09f245af1c75ee40c0e2819677270
  namespace: my-kafka-cluster
  resourceVersion: "42089768"
  uid: cee9dc50-4a4b-494f-af11-c703f78c43db
spec:
  config: {}
  partitions: 1
  replicas: 1
  topicName: autocreatedtopic-__assignor-__leader
status:
  conditions:
  - lastTransitionTime: "2022-04-05T23:27:36.775455Z"
    reason: InvalidStateException
    status: "True"
    type: NotReady
  observedGeneration: 1
  topicName: autocreatedtopic-__assignor-__leader

[scholzj]

No status suggests that the operator didn't see it. Comparing this topic with what you shared originally, I guess you have the strimzi.io/cluster label wrong? In any case, the status suggests there might be some issues with the TO in your cluster, so you might want to check its logs I guess.

[Cave-Johnson]

Mis rename on my part. They are both the same in my actual config (different internal names). I'm going to try and redploy the topic operator.
Also Is there an easy way to pull out everything strimzi and start from scratch? I couldn't see that it in the documentation

[scholzj]

You can normally do something like kubectl delete $(kubectl get strimzi -o name) to delete all the Strimzi custom resources.