Performance investigations for slow cruise control rebalancing

[peterschmitzer]

We are currently practicing a scale out of our kafka cluster in our preprod and for unknown reason the rebalancing performs poorly and takes a lot of time. Unfortunately, we have not found a lot of documentation about performance considerations.

We are running strimzi/kafka:0.28.0-kafka-3.0.0

The cluster holds 4500 partitions with 30 replicas per topic and replication factor of 3 on 3 brokers when I scaled up to 4 brokers.
After applying the kafka rebalance with mode "add-brokers" it proposed to move 1000 replicas and 82MB. The whole process took more than one hour to complete. In our other cluster where we wanted to move only a little bit more data it seems to take forever (there we stoppe the rebalancing).

Does someone have experience with bad performance when applying a rebalance proposal? What may be possible root causes for the issues we see? In the past (with older strimzi and kafka versions) we have done a rebalance in a poc and moved 80GB of data which was completed in 15 minutes, so we are a little bit confused.

[scholzj]

Maybe @kyguy or @ppatierno might have some ideas. But keep in mind that performance is heavily dependent on your infrastructure and might not be easily reproducible.

[kyguy]

Hi @peterschmitzer, happy to take a closer look! Could you share your Kafka and KafkaRebalance custom resource and Cruise Control pod logs during the rebalancing?

[peterschmitzer]

cruise-control (1).txt
events.txt
kafka-rcd.txt
kafkarebalance.txt

Thanks a lot for the offer! Please find the requested information attached and let me know if I can supply further information.

[peterschmitzer]

hi @kyguy did you already have some time to look at our config? I am wondering if I could use the kafka-reassign-partition as a mitigation to move the partitions manually. Is that a valid approach shortterm?

[kyguy]

Hey @peterschmitzer apologies for the delay! I notice that the KafkaRebalance custom resource has the following configuration settings:

kind: KafkaRebalance
metadata:
...
spec:
  concurrentIntraBrokerPartitionMovements: 0
  concurrentLeaderMovements: 0
  concurrentPartitionMovementsPerBroker: 0
  replicationThrottle: 0
  skipHardGoalCheck: false

Is there any particular reason that concurrentLeaderMovements, concurrentPartitionMovementsPerBroker and replicationThrottle are set to 0? These configurations can slow down the rebalancing process and likely are the reason why the rebalancing is taking so long! Have you tried rebalancing without specifying these configurations so that the operator will use the larger default values?

[peterschmitzer]

It may be worth mentioning that the reassignment of partitions "manually" via the kafka-reassign-partitions.sh in kafka onboard tools worked flawlessly and was very fast. So we doubt it has something to do with our underlying infrastructure.

[scholzj]

Well, I'm not sure that is the case. It can impact how the reassignment qoutas might be set etc.