Pods of kafka, zookeeper and cluster-entity-operator being restarted

[alternaivan]

Hello,

We have strange issue happening on our Kafka setup on EKS environment. Following is the information about our environment:

Environment:

• Strimzi version: 0.24.0
• Installation method: Helm charts
• Kubernetes cluster: Kubernetes 1.21
• Infrastructure: Amazon EKS

Summary

Kafka operator is deployed in the kafka namespace, while Kafka resources (kafka, zookeeper, cluster-entity-operator) are being deployed in a different namespace called application alongside our application. We noticed that there is a strange thing happening, for which we cannot determine the correct cause - pods of the kafka resources within application namespace and the cluster-operator within the kafka namespace are being restarted. The strange thing is that cluster-entity-operator and 2 out of 3 replicas of both kafka and zookeeper within application namespace are being restarted occasionally, which then causes our application to fail. When we checked for kubernetes state of the pods, those restarted ones report the status Completed. On the other hand, cluster operator which is running in the kafka namespace was restarted occasionally as well, a lot more number of restarts though, but the status of this pod when restarted is OOMKilled.

Question

Is it possible that cluster-operator restart can cause the other deployed resources to restart? If not, can you maybe point us to the direction in where to look why these issues happen. Currently we've increased the memory limit of the cluster-operator and will follow it in the next days to see how and if it correlates to the number of restarts of other resources.

We are using kafka persistent example with some modifications to deploy Kafka.

Let me know if you would need some more information.

Thanks in advance,
Marjan

[scholzj]

I think you would need to share the operator log (ideally on DEBUG level) - that should help us understand what the problem is.