CruiseControl metrics reporter topic can end up in a bad spread when both spec.kafka.cruiseControl and spec.kafka.rack are set

[devguyio]

Description

There's a behavior we're facing when we're deploying a Kafka cluster with Cruise Control in multi AZ setup using rack awareness. When setting spec.kafka.rack & spec.kafka.cruiseControl keys in a Kafka resource, the partition replicas of CC metrics reporter topic might not spread out across AZs properly because of a race between brokers getting ready and CC metrics reporter plugin creating the topic.

Expected behavior

Since the Kafka resource provides two api fields: spec.kafka.rack & spec.kafka.cruiseControl , users who set these two fields should end up with a Kafka & Cruise Control setup that satisfy the rack awareness expectations. Thus any topics that exist once Strimzi is finished with setup, should have the right AZ spread, including the cruise control metrics topic.

There are many challenges for reaching the expected behavior and this is a discussion around that.

Suggestions

1. Change the behavior in CC metrics reporter to create the topic only when the brokers are ready.
2. Disable auto creation of MR topic (you can see the config here ) and have the CO create the topic after deploying CC.
3. Instruct the users who want to set both fields to do it in a phased setup instead, first setting the spec.kafka.rack and once the cluster is ready, update the Kafka resource and setting the spec.kafka.cruiseControl.
4. CO initiates a KafkaRebalance with a RackAwareGoal goal. I'm not sure if it's possible to have an auto-approved KafkaRebalance though.

Related slack discussion: https://cloud-native.slack.com/archives/C018247K8T0/p1660905404139799

[k-wall]

Wouldn't 1 require that the broker is given knowledge of the number of desired brokers? There's also the possibility that the number of desired brokers could change, which would further complicate matters.

There is another alternative; that the CC metric reporter plugin is given logic to cause an alterPartitionReassignments on the topic if it detects that the rack assignment suboptimal has occurred. You could argue that CC owns this topic, so it should manage it itself without the user of CC needing to intervene manually. I wonder about raising a defect in the upstream project and discussing there.

[devguyio]

Wouldn't 1 require that the broker is given knowledge of the number of desired brokers? There's also the possibility that the number of desired brokers could change, which would further complicate matters.

I'm not sure, probably it can describe the cluster and waits on creating the topi while available brokers < topic replication factor.

There is another alternative; that the CC metric reporter plugin is given logic to cause an alterPartitionReassignments on the topic if it detects that the rack assignment suboptimal has occurred. You could argue that CC owns this topic, so it should manage it itself without the user of CC needing to intervene manually. I wonder about raising a defect in the upstream project and discussing there.

While creating the topic is a pretty typical and common behavior for Kafka producers, which in some cases is done explicitly in case the cluster has disabled auto topic creation, I'm not sure it's an expected behavior for the producer to closely manage the topic, which is considered to be more of an administrative task that requires more knowledge than what the producer can or should have. E.g. such a behavior might conflict with an ongoing manual partitions reassignment that's part of a broker removal or replacement task.

[scholzj]

I guess there is option 5 -> you can deploy it all at once and then you rebalance it with whatever reason you want. What are the use-cases which make the distribution of the Cruise Control topics important?

Also worth noting is the related discussion in https://cloud-native.slack.com/archives/C018247K8T0/p1660905404139799

[devguyio]

What are the use-cases which make the distribution of the Cruise Control topics important?

• Avoid losing the state upon which Cruise Control builds its proposals and starting all over. Specially that Cruise Control would be one of the tools relied on to maintain the cluster health when an AZ is down
• Avoid triggering alerts in place to detect topics with a bad spread
• Avoid blind-siding the users who would expect that the replicas of all topics are well spread, and would be confused why e.g. there's an AZ which usage skews from other AZs , or why when an AZ went down, the MR topic partition is starting empty.

[scholzj]

Avoid losing the state upon which Cruise Control builds its proposals and starting all over. Specially that Cruise Control would be one of the tools relied on to maintain the cluster health when an AZ is down

Stupid question, but how would it help? I would normally expect to wait for recovery of the AZ before touching any rebalancing.

[devguyio]

Stupid question, but how would it help? I would normally expect to wait for recovery of the AZ before touching any rebalancing.

I don't have extensive knowledge in Cruise Control but here're I would think n the case of a prolonged outage, it might be needed in case a disk skew occurs between the available brokers for some reason.

Another point I wondered about in case of a bad spread, wouldn't a disk failure in a broker with multiple replicas result in losing the partition data?

[katheris]

My understanding is that if we lose data from the metrics topic then Cruise Control could take some time to collect enough metrics to be able to be used again. With the default settings this is only 5 minutes or so I think, but depending on how the cluster is configured it could be longer. However I agree with Jakub that I don't think it would be good practice to run a rebalance during an outage. The cluster would already be under additional strain so running a rebalance could make matters worse.

I do think in an ideal world if the topic has been created with three replicas on a 6 broker cluster it should be spread out across the AZs to make the running of CC as smooth as possible. However this isn't a critical problem for this specific topic so implementing a very complicated solution also wouldn't be ideal. It definitely feels like a chicken and egg situation where the user can't create the topic until Kafka is running, but the cruise control plugin can't be started in Kafka until the topic is created. I don't know whether the plugin could be updated to wait for the topic to be there so the topic can be created by the user once Kafka is ready?

[scholzj]

Ok, so, something simple and relatively uninvasive what we can do for this issue:

• We make it configurable for users to disable the Cruise Control topics auto-creation
• Let users orchestrate them self to create the topic with their desired configuration.

For example, they can simply bundle Kafka and KafkaTopic CRs into a single YAML file and apply it together. The KafkaTopic will be created only once the TO is started. At that point, the brokers are ready and the topic should be create properly. At the same time, it is before Cruise Control server start. But in general, users can use any mechanism they want to create the topic.