Add support for Disk/Network PercentageUsageGoal

[imans777]

Hi
Currently it's possible to utilize DiskUsageDistributionGoal to balance distribution of disk across brokers.

Normally this is good, but sometimes we have a cluster that hardwares have different specs. for example, we have one row of brokers that have 10G network, and another row that has 1G network. What we want to achieve is that we ensure that %50 of network of each row is used, but using the current goals, we can only achieve this in a way that e.g. 1G brokers get 0.9G full, and then 10G brokers get the rest, which for example can be 5G. So we have several brokers that are overloaded whilst others have way more capacity.

What can improve this is having something like NetworkPctUsageGoal that ensures network is used e.g. %50 across all brokers. So 1G brokers get 0.5G and 10G brokers get 5G.

The same can be applied for Disk usage.

Currently it's possible to use a hack configuration to achieve this. Setting capacity of 1G network to 0.5G ensures that NetworkOutboundCapacityGoal prevents partitions from getting into such nodes. But it's not a good permanent method.

[kyguy]

Hi @imans777, Configuring resource capacity settings [1] is essential, especially in clusters where brokers have different hardware capabilities. It is not a hack. Without these settings, Cruise Control cannot accurately evaluate or compare resource utilization across brokers.

With the resource capacity settings properly configured we can ensure that resource usage is balanced evenly based on the individual resource capacities of each broker by leveraging the following goals:

• NetworkInboundUsageDistributionGoal
• NetworkOutboundUsageDistributionGoal
• DiskUsageDistributionGoal

and the following Cruise Control server configurations:

• network.inbound.balance.threshold
• network.outbound.balance.threshold
• disk.balance.threshold

[1] https://github.com/linkedin/cruise-control/blob/main/config/capacity.json

[imans777]

Thank you. I followed what you said and tested and it seemed OK.
However there's a case. Suppose in a broker we have 2 disks of 500GB (Total 1TB) and set capacity to 0.5 (So 500GB utilization allowed on this broker). There are two replicas of approximate 249GB. Cruise seems these two as eligible for moving them to this broker. However, they might come to the same disk, because there might be two replicas of another topic with size of e.g. 10MB exist on disk 2, and we see enough capacity on this broker, but cruise let's kafka decide for replica placement, and kafka puts the replicas on disks that have lower number of replicas. So they get put on the first disk, and disk becomes full!
I know that in a second phase we can run rebalance with rebalance_disk=true flag to fix this. But this is not automated (for example in self healing cases) and we should manually run it and this meanwhile might take a lot and cause disk to become full and unresponsive. (Our deployment is JBOD).
(Maybe this is another case from the original, but I wanted to say my concern)

[imans777]

@kyguy I'm motivated on working on this issue and send PR while working on it on weekends. I just want to make sure that I'm seeing the problem correctly:

However there's a case. Suppose in a broker we have 2 disks of 500GB (Total 1TB) and set capacity to 0.5 (So 500GB utilization allowed on this broker). There are two replicas of approximate 249GB. Cruise seems these two as eligible for moving them to this broker. However, they might come to the same disk, because there might be two replicas of another topic with size of e.g. 10MB exist on disk 2, and we see enough capacity on this broker, but cruise let's kafka decide for replica placement, and kafka puts the replicas on disks that have lower number of replicas. So they get put on the first disk, and disk becomes full! I know that in a second phase we can run rebalance with rebalance_disk=true flag to fix this. But this is not automated (for example in self healing cases) and we should manually run it and this meanwhile might take a lot and cause disk to become full and unresponsive. (Our deployment is JBOD).

What I have in my mind is that we shouldn't have a separate rebalancing process with rebalance_disk=true, because that would break automations like self-healings. And also clusters need a two-step rebalancing which can be very time and resource consuming in clusters with +200 brokers. I believe a single endpoint should rebalance the cluster in a way that it also handles the disks.

A second proposal is to add the disk rebalance scenario in self-healing automations, so any operations triggered by anomaly-detection should be followed-up with a separate disk rebalance (configurable by a flag, of course). Also this can be applied (through the same or another flag) on a regular rebalancing endpoint, i.e. whenever we trigger a rebalance, after the operation is done, it triggers a disk rebalance.

My personal opinion is the first option, because this new endpoint really seems dirty to me and also two-step rebalancing is not efficient as we expect in cruise-control. It doesn't seem right to me to have separate goals for Inter and Intra replica movements.

Thanks in advance.

(Please guide me if this needs a separate issue)
Edit: I see that I'd already created another issue #2234 for this :) Also there seems open PRs for this like #2288 and pull/6 (there's a bug I can't link it). It would be very appreciated if these are reviewed and merged.

[kyguy]

However there's a case. Suppose in a broker we have 2 disks of 500GB (Total 1TB) and set capacity to 0.5 (So 500GB utilization allowed on this broker). There are two replicas of approximate 249GB. Cruise seems these two as eligible for moving them to this broker. However, they might come to the same disk, because there might be two replicas of another topic with size of e.g. 10MB exist on disk 2, and we see enough capacity on this broker, but cruise let's kafka decide for replica placement, and kafka puts the replicas on disks that have lower number of replicas. So they get put on the first disk, and disk becomes full!

Wouldn't the DiskCapacityGoal prevent this from happening?

What I have in my mind is that we shouldn't have a separate rebalancing process with rebalance_disk=true, because that would break automations like self-healings. And also clusters need a two-step rebalancing which can be very time and resource consuming in clusters with +200 brokers. I believe a single endpoint should rebalance the cluster in a way that it also handles the disks.

I see that I'd already created another issue #2234 for this :)

For this other issue, let's follow up on the thread here #2234