KafkaRebalance proposal always returning the same result

[bfarayev]

I'm using Strimzi 0.22.1 with Kafka brokers at version 2.7. I've scaled up my cluster (i.e increased the replica count) and I want to move some partitions to new brokers evenly in a way that they'll all have the similar disk usage. In summary, I'm doin the same steps that's been described in answer that was given by @scholzj on another discussion

Right now, Kafka brokers different disk usage (some are at 80Gb, some at 90Gb and the new ones are at 99Gb usage out of total 100Gb disk capacity). I'm using DiskCapacityGoal in .spec.goals. Like so:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: <name-of-kafka-rebalance>
  namespace: <my-namespace>
  labels:
    strimzi.io/cluster: <name-of-my-kafka-custom-resource>
spec:
  skipHardGoalCheck: true
  goals:
    - DiskCapacityGoal

CruiseControl's config is default i.e it expects all 6 hard-goals, hence I'm using skipHardGoalCheck: true here.

Problem is that each proposal is getting such results where it decides not to move anything:

spec:
  goals:
  - DiskCapacityGoal
  skipHardGoalCheck: true
status:
  conditions:
  - lastTransitionTime: "2021-11-24T03:45:16.824326Z"
    status: "True"
    type: ProposalReady
  observedGeneration: 1
  optimizationResult:
    dataToMoveMB: 0
    excludedBrokersForLeadership: []
    excludedBrokersForReplicaMove: []
    excludedTopics: []
    intraBrokerDataToMoveMB: 0
    monitoredPartitionsPercentage: 100
    numIntraBrokerReplicaMovements: 0
    numLeaderMovements: 0
    numReplicaMovements: 0
    onDemandBalancednessScoreAfter: 100
    onDemandBalancednessScoreBefore: 100
    provisionRecommendation: ""
    provisionStatus: UNDECIDED
    recentWindows: 1

which is wrong in my opinion. Also the proposal comes back within less than a second which makes me think that CruiseControl doesn't do any analysis before returning the result. Seems like a cached result? 🤔

I also tried to just call the CruiseControl as per the REST API docs and pass different query parameters like ignore_proposal_cache=true etc. but still no luck with any of those options either.

I was wondering if there's a way to force CruiseControl to calculate the capacities and proposals properly. I've seen it working in the previous versions of the Strimzi in my clusters (in Strimzi 0.19 with Kafka 2.5) but after recent upgrades into Strimzi 0.22 and Kafka 2.7 I couldn't get it working.

Any help / directions is much appreciated. Thanks

[scholzj]

This might be related to what goals you pic. For example, note the difference between these two:

• DiskCapacityGoal - Ensures that Disk space usage of each broker is below a given threshold.
• DiskUsageDistributionGoal - Attempts to keep the Disk space usage variance among brokers within a certain range relative to the average Disk utilization.

The first one will just make sure you are below the limit. The other one will actually try to balance them. (full list off goals is on the Cruise Control Github: https://github.com/linkedin/cruise-control#goals)

Also, keep in mind that depending on your partitions, their sizes etc., it might not be possible to make the disk utilization completely balanced between nodes.

Also the proposal comes back within less than a second which makes me think that CruiseControl doesn't do any analysis before returning the result. Seems like a cached result? 🤔

As far as I know, Cruise control does some calculations continuously. Not sure this really matters, it does not sound to me like the result here would be necessarily different few seconds earlier or few seconds later.

[bfarayev]

thanks @scholzj it was indeed the wrong goal that I was using 🤦 thanks for pointing it out.