crashloopbackoff while doing horizontal pod autoscaler HPA

[vishk2203]

My pods are showing the status as "CrashLoopBackOff" after deploying HPA. Is there solution or precautionary action could be taken to resolve this?

[scholzj]

HPA rarely makes sense with most Kafka brokers. And it IMHO never makes sense with ZooKeeper. If you want to scale Kafka brokers or ZooKeepers, you should do it by changing the replicas fields in the Kafka custom resource. Not by scaling the StatefulSet. That will not work short-term and the operator will anyway revert it in the next reconciliation.

[scholzj]

But why do you think that adding more pods helps you with memory utilization?

With ZooKeeper, that will for sure not happen. Adding more nodes improves the availability. But it does not split the workload between the nodes. So I do not see why would memory consumption change significantly. From some number of noes, adding more nodes would actually hurt the performance because there will be lot more synchronization required

For Kafka, the default outcome would be again that you will have the old node unchanged and add some more free nodes. You would need to move the partition-replicas between them (which in many cases is lot of data which takes a lot of time to move). And even if you do it, Kafka uses most memory for disk cache, so over time the disk cache will fill in the memory again.

So I think this is a wrong metric and motivation for how the software is designed.

[vishk2203]

I'm working on a data ingestion so trying to automate all the features related to it. This features mainly focuses on avoiding manual intervention.

[tomncooper]

That is a nice idea but, as Jakub said, Kafka is not some stateless service. Adding new brokers will not relieve the load on the old ones without further manual intervention to rebalance the partitions between the old and new brokers.

[tomncooper]

We have items on the road map around using our Cruise Control support to automate this balancing for adding/removing brokers but that functionality is not there yet.

[vishk2203]

Oh great. What could be the time frame for this functionality?