Kafka Connect unable to create offset/config/status topics under the kafka cluster

[williamyeungforsure]

Hi there,
I am deploying a Kafka Connector using the following configuration by Helm

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: {{ .Values.kafkaConnectCluster.name }}
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  version: 2.8.0
  replicas: 1
  image: {{ include "containerImageWithTagOrDigest" (list $ . $imageParams) }}
  authentication:
    type: tls
    certificateAndKey:
      secretName: {{ .Values.kafkaUserProducer.name }}
      certificate: user.crt
      key: user.key
  tls:
    trustedCertificates:
      - secretName: {{ include "common.tplvalues.render" ( dict "value" .Values.kafka.clusterName "context" . ) }}-cluster-ca-cert
        certificate: ca.crt
  bootstrapServers: {{ include "common.tplvalues.render" ( dict "value" .Values.kafka.bootstrapServers "context" . ) | quote }}
  config:
    group.id: {{ .Values.kafkaConnectCluster.groupId }}
    key.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: false
    value.converter.schemas.enable: false
    offset.storage.topic: {{ .Values.kafkaConnectCluster.internalTopics.offset }}
    config.storage.topic: {{ .Values.kafkaConnectCluster.internalTopics.config }}
    status.storage.topic: {{ .Values.kafkaConnectCluster.internalTopics.status }}
    config.storage.replication.factor: 3
    offset.storage.replication.factor: 3
    status.storage.replication.factor: 3
    config.providers: directory
    config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
  resources:
    requests:
      cpu: "1"
      memory: 2Gi
    limits:
      cpu: "2"
      memory: 2Gi
  jvmOptions:
    "-Xmx": "1g"
    "-Xms": "1g"
  template:
    pod:
      imagePullSecrets:
        - name: {{ .Values.imagePullSecrets.name }}
  externalConfiguration:
    volumes:
      - name: connector-config-postgresql-username
        secret:
          secretName: {{ .Values.postgres.usernameSecret }}
      - name: connector-config-postgresql-password
        secret:
          secretName: {{ .Values.postgres.passwordSecret }}
  metricsConfig:
    type: jmxPrometheusExporter
    valueFrom:
      configMapKeyRef:
        name: jmx-exporter-config-map
        key: kafka-connect

Earlier than that, I could those 3 topics, then I delete all kafka topics, clean up all the resources in my kubernetes cluster and redeploy the Kafkaconnect. However, after deploying the KafkaConnect, I could the Kafka Connect is working, but there's no offset/config/status topics under the kafka cluster when I enter kubectl get kafkatopic. I am wondering should I create those 3 topics manually? I thought they are created by KC automatically. Appreciate it if anyone could help.

[scholzj]

Kafka Connect creates its topics on its own. You do not need to create them. (you can create them if you want, but you need to make sure to keep the exact settings for the topics needed by Kafka Connect)

You do not provide exact step-by-step buide of what you are doing. But my guess is that by what you are doing (for example through I delete all kafka topics) you break the Topic Operator. So you should check in Kafka itself if the Connect topics exist or not.

[williamyeungforsure]

@scholzj ok I see, I tried entering the kafka pod and listing all the topics it has, but there's no topics returned in response. However, when I showed the logs in the kafka pod by using kubectl logs, I can see there's some logs related to the internal offset topic, so I am confused what could possibly go wrong here? Thanks in advance.

BTW, the exact steps I was doing in the previous comment is that:

1. Before I clean up the resources in my kubernetes cluster, KC is not running, but I could see the offset/status/config topic by entering kubectl get kafatopic actually mainly I deployed KC a long time ago. Then I clean up all resources by using kubectl delete.
2. Redeploy a Kafka Connect, a Kafka Connector(Debezium) and a Kafka user, then here comes this discussion.

[scholzj]

Sorry, but you need to start from the beginning. How do you setup Kafka cluster, what exactly does clean up all resources mean, how do you check the topics. Just share the configurations, commands, logs from the commands etc. Without being clear what are you doing, nobody will be able to help you.

[williamyeungforsure]

Sure, sorry for not being clear, will update that later.