Cruise Control Pod Not Start in a +80 Broker Kafka

[imans777]

Hi

Running cruise control in a kubernetes pod (built the image from 2.5.138 tag) in a kafka bare-metal cluster with +80 brokers passed as input parameter for bootstrap server, often fails and the pod doesn't start even after 30 minutes. Connection logs to Kafka were frequently being logged without any errors occurring. They sometimes started by repeatedly removing the pod or bootstrap metrics or recreating cc topics.

We found this number by dividing the brokers until it comes up. So we started with 400 and it didn't work. we went to 200, and so on, until we found that 80 works.

For short term, we passed only the first 80 brokers to cruise control and it starts fine. But this is not a clean solution. I wonder if anyone else encounter such problem.

[danielgospodinow]

That's interesting. I've heard that at LinkedIn they manage clusters with thousands of brokers.

• How does CC fail exactly?
• What's it doing for those 30+ minutes initially? What's it logging? Can you share some examples?
• What's the hardware on this K8s pod? Memory, CPU?
• CC is running on its dedicated pod, right? It's not co-located with a broker?

[imans777]

That's interesting for me as well :D
I'm not saying that it fails to manage the cluster, as it starts and manages cluster correctly when passed <80 brokers to it. (I think there's something wrong with the bootstrap.server parameter or memory management.)
The problem with failure is the pod doesn't become ready (startup probes are set to / http endpoint) and thus gets restarted. the 30 minutes is the grace time we gave to it by these k8s startup probes.
It logs everything as normal. The hardware is normal (we didn't pass request/limit to the pod). The pod is dedicated, single container with no colocation.

I will share some logs when I reach our environment. (Because it's air-gapped it's a bit complicated)