KafkaConnect Internal topics partitions (offset, config, status) best practices on partitions

[cstmgl]

TL/DR what is the correct value/guidelines for config.storage.topic partitions?

Hi all,

I have a question I'm using the strimzi version 0.24 and I'm defining my kind: KafkaConnect

There I have the option that I need to define the internal topics:

    offset.storage.topic: kafkaconnect.offset
    config.storage.topic: kafkaconnect.configs
    status.storage.topic: kafkaconnect.status

This resources I will create with the kind: KafkaTopic.

I still sometimes have a problem that it seems the topics do not get created before the application starts, I had explain here something similar #5156 (comment).
Anyway I have a work-around for that.

Now my question is if there is any best practices for this internal topics.

Say for example I have it like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: kafkaconnect.configs
  labels:
    strimzi.io/cluster: cluster
spec:
  topicName: kafkaconnect.configs
  partitions: 3
  replicas: 3
  config:
    cleanup.policy: compact

My question is are my partitions and replicas wrong? Because our connect-cluster is behaving a bit weird and we wonder if this could be the cause.
Should they be 1? At least for the number of partitions?
Just for reference the number of replicas (workers) defined in the KafkaConnect is 3 or 5

I'm asking because I found this 2 examples that lead me to believe that it should be 1 but maybe this is just an old problem and my issue relies elsewhere?

https://stackoverflow.com/questions/46208498/worker-not-recovered-current-config-state-offset-5-is-behind-group-assignment
https://groups.google.com/g/confluent-platform/c/VWGet3c6HTU

Thank you for any feedback and sorry for the long post.

[scholzj]

TBH, I never customized this and always followed the default - at least for partition counts. The only thing I ever changed was the replication factor when I had less than 3 brokers.

The config storage does not have configurable partitions and the KIP-154 suggests at the end of the Proposed Changes section that this is intentional and the partition count should be always 1. Not sure it didn't changed in some later KIP, but I'm not aware of any. So I would stick with it.

The others should be configurable also directly in the Connect configuration and IMHO you should have the freedom to change them. I can imagine that if you have connectors using the offset topic heavily some tuning might help. But not sure where would the limit be.