Recommended process for raising RF of existing __consumer_offsets Topic?

[foet-cw]

Hi Strimzi,
We are running a Strimzi Kafka cluster which has needs a higher level of resilience than normal. As such, one of the configurations we'd like to have is running the __consumer_offsets topics at RF=5 (currently at 3). Taking the cluster offline to make the change is not an option.

What is the recommended way to do this in a safe/controlled manner, minimizing the risk of human error or inconsistency?

I tested one approach already, which works, but is tedious:

1. Create a json file with the suggested replica distribution
2. Exec into the Utility pod, run:

./kafka-reassign-partitions.sh --bootstrap-server $BOOTSTRAP_SERVER --command-config /tmp/utility-user.properties --reassignment-json-file /tmp/foo.json --execute

This worked in a test env.

Other ideas we had:

• Setting the offsets.topic.replication.factor setting. However, we have tested this several times, controllers and brokers rebuild via the operator, but at the end, the Topic doesn't update with the new RF.
• Add the __consumer_offsets Topic to the resource config so the operator can manage it. (Not sure if that would work, but it seems like if it did it would let us manage it the same way as the other Topics).

Logs from trying to set offsets.topics.replication.factor:

2024-12-13 17:52:51 INFO  KafkaBrokerConfigurationDiff:94 - Reconciliation #74(watch) Kafka(data-kafka-local/kafka-local): Configuration can't be updated dynamically due to: AlterConfigOp{opType=SET, configEntry=ConfigEntry(name=offsets.topic.replication.factor, value=5, source=UNKNOWN, isSensitive=false, isReadOnly=false, synonyms=[], type=UNKNOWN, documentation=null)}

We saw these type of lines for all the brokers:

2024-12-13 17:53:08 INFO  KafkaRoller:473 - Reconciliation #74(watch) Kafka(data-kafka-local/kafka-local): Rolling Pod kafka-local-broker-2/2 due to [Pod needs to be restarted, because reconfiguration cannot be done dynamically]
2024-12-13 17:53:08 INFO  PodOperator:54 - Reconciliation #74(watch) Kafka(data-kafka-local/kafka-local): Rolling pod kafka-local-broker-2

Thank you for your time.

[scholzj]

offsets.topic.replication.factor is used only when the topic is being created. So unless you delete it and have Kafka recreate it (but I guess you would not want to do that), it will have no impact once the topic exists.

[foet-cw]

Thanks, that aligns with what we found experimentally. We did think about how ideally we'd define that higher when we build a new cluster.

Any other ways you'd recommend adjusting it after the fact, or is the json file the only/best way?

[scholzj]

I think you summed the options pretty well. I'm not aware of any other option.

[foet-cw]

Thank you very much Jakub! As always, we appreciate your help! Have a nice rest of your day!