Cruise Control, can it be made more aggressive?

[craiglservin]

I've been trying to get Cruise Control to balance the disk use on an 8 broker cluster. Each broker has 2000G of storage, and 7 of the brokers were relatively close in disk use having 1.5 TB free, one of the brokers only has 320G free. Looking at the topics/partitions/replicas there does not seem to be any reason why things could not be balanced.

I've tried pealing away most of the goals that might interfere with replica movement:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: full-rebalance
  labels:
    strimzi.io/cluster: app
spec:
  skipHardGoalCheck: true
  goals:
    - RackAwareGoal
    - DiskUsageDistributionGoal
    - DiskCapacityGoal
  replicaMovementStrategies:
    - PrioritizeSmallReplicaMovementStrategy

When it comes up with a proposal, it happily improves things a little bit. But, it makes almost no difference. I'm just wondering if I missed something obvious that is causing CC to not be as aggressive as I would like.

[scholzj]

@kyguy Any idea?

[kyguy]

Hi @craiglservin, some Cruise Control configurations [1] to consider when balancing disks are :

• disk.balance.threshold - The maximum allowed extent of unbalance for disk utilization. For example, a value of 1.10 means the highest disk usage of a broker should not be above 1.10x of average disk utilization of all the brokers.
• disk.capacity.threshold- The maximum percentage of the total broker.disk.capacity that is allowed to be used on a broker. The analyzer will enforce a hard goal that the disk usage of a broker cannot be higher than (broker.disk.capacity * disk.capacity.threshold).
• disk.low.utilization.threshold - The threshold to define the utilization of disk is low enough that rebalance is not worthwhile. The cluster will only be in a low utilization state when all the brokers are below the low utilization threshold. Such a cluster is overprovisioned in terms of its disk utilization. The threshold is in percentage.

According to the numbers provided above, the average disk utilization of all brokers is 0.32

( ((7 brokers  * 500Gb) + (1 broker * 1680Gb))  / 2000Gb) / 8 brokers total = 0.32

That being said, unless the default values of the configurations listed above have been altered, they shouldn't be a problem since the offending broker's disk usage is far above the disk utilization average. However, setting skipHardGoalCheck = true prevents these settings from being enforced, so I would set it to false.

Could you post the Kafka resource as well?

[1] https://github.com/linkedin/cruise-control/wiki/Configurations#analyzer-configurations

[craiglservin]

I'll try out changing skipHardGoalCheck and pull the Kafka resource when back at a computer. Thanks for the feedback. I thought that skipping the hard goal check would force it to use only the goals that I was specifying, not that it would potentially alter the interpretation of those goals!!!