[Bug]: strimzi cluster operator restarts often

[themkarimi]

Hi,

we have noticed that the strimzi cluster operator restarts often and randomly without any sort of reason. here are the logs before the shutdown hook is invoked.

2023-05-22 05:22:33 INFO LeaderElectionManager:119 - The new leader is 2023-05-22 05:22:33 INFO LeaderElectionManager:110 - Stopped being a leader 2023-05-22 05:22:33 INFO Main:242 - Stopped being a leader => exiting 2023-05-22 05:22:33 INFO ShutdownHook:51 - Shutdown started

Steps to reproduce

install strimzi cluster operator

Expected behavior

cluster operator pod is not restarted

Strimzi version

0.35.0

Kubernetes version

1.24.2

Installation method

helm chart

Infrastructure

BareMetal

Configuration files and logs

2023-05-22 05:22:33 INFO LeaderElectionManager:119 - The new leader is 2023-05-22 05:22:33 INFO LeaderElectionManager:110 - Stopped being a leader 2023-05-22 05:22:33 INFO Main:242 - Stopped being a leader => exiting 2023-05-22 05:22:33 INFO ShutdownHook:51 - Shutdown started

[ranyhb]

Hi
i was on my way to open an issue for that.
i have version 0.34 and i have the same problem.

[scholzj]

You should probably share your configuration of the Cluster Operator (the deployment or the full log from the start). You should also watch the Lease resource -> make sure you don't have some other tooling that is deleting it etc.

[themkarimi]

Hi,
Thanks for your response, we are using strimzi official helm chart for our cluster operator and here is the configuration:
operationTimeoutMs: "2400000"
resources:
limits:
memory: 2Gi
requests:
memory: 2Gi
watchAnyNamespace: true
these are the things we have customized and others are helm chart defaults.

I also check the lease for cluster operator and here is its status:
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
creationTimestamp: "2022-11-19T13:36:52Z"
name: strimzi-cluster-operator
namespace: kafka-operator
resourceVersion: "666536273"
uid: 8d426963-a1be-4627-8955-7b7596532fda
spec:
acquireTime: "2023-05-24T17:03:06.757569Z"
holderIdentity: strimzi-cluster-operator-cb47c4557-jz2jr
leaseDurationSeconds: 15
leaseTransitions: 322
renewTime: "2023-05-27T04:43:27.612022Z"
it seems that there is nothing wrong with this object

[themkarimi]

kafka-operator.log
here is the full log of the operator

[scholzj]

You should make sure to configure the CPU request and limit as well. This could be caused by CPU starvation.

[themkarimi]

Sorry for replying to this late. i have checked out configuration and found out that we have no cpu/memory limit configured, I have also checked our resources and there was no problem.

[scholzj]

Triaged on 1.6.2023: Does not seem like a generic bug in Strimzi but rather like a problem in single installation / environment. Should be converted to discussion.

[scholzj]

I guess sharing a full log would be a start.

[peter-hst]

Sorry, we do not save the log of the operator, it currently only sees the log of the current pod,

# here's an Exception in pod
 ERROR Util:154 - Reconciliation #4748(timer) Kafka(kafka/qa-broker): Exceeded timeout of 300000ms while waiting for Ingress resource qa-broker-kafka-bootstrap in namespace kafka to be addressable

see if it's useful
strimzi-cluster-operator-9ffd6787-twkpb.log

[peter-hst]

we using Vert.x asynchronous freameworks ?

[peter-hst]

Could it be that memory resources are allocated too small?

[scholzj]

That error suggests a problem with your Ingress Controler. But that would not lead to any operator restarts.