Bestpractice kafka cluster scaling

[chanvoyage]

hello everyone

i would like to scale up kafka cluster pod (statfulset) from replicas 3 to 4 , Could you please suggestion how to do that and
prerequisite or anything else should be careful. , if you have any document please tell me

thank you

[scholzj]

If you are scaling up, you normally just need to increase the replicas count in the Kafka CR. Once the new broker(s) is added, you probably want to balance your cluster to move some partition replicas to it - either using the built in Cruise Control support or manually using the Kafka reassignment tool. Without this, the new broker will be idle until some new topics are created and their partitions scheduled on the new node as well.

For a scale down, it is slightly more complicated. You should use the Kafka reassignment tool first to move all partition replicas out of the nodes which will be removed (the nodes are removed always from the highest index number in their pod name / highest broker.id). And only one all the replicas are reassigned to the remaining nodes, you should edit the Kafka CR and scale down the replicas.

What you should never do is to change the replicas in the StatefulSet. The operator will revert it back so it will not help.

[chanvoyage]

Dear scholzj

thank you very much

i have some question

1. before scaling up i was created KafkaRebalance after that i scaling up kafka to 4 replicas
   and Check some data movement between kafka cluster on chanza-dev-kafka-cluster-kafka-3 not found anythink
   Am I missing something?

2. i want to create simple functional test of kafka cruise control . Do you have any suggest me to Test ?

[scholzj]

Well, first of all, you need to rebalance after the scaling up, not before. Second, it depends on the goals which you configure in the rebalance. IIRC the default goals care mainly about capacity, but as long as the broker is not over the capacity, it is fine. But there are other goals you can specify to balance the distribution even when you are not over the capacity. Please check the Cruise Control related parts of our docs to see how to set the goals etc.