Third broker stops reporting metrics for 3-4 mins frequently

[abhi1git]

Following are the details of our current Setup and how we are using Strimzi Kafka in our production

Strimzi Kafka Operator : 0.22.1, Strimzi Kafka : 2.7.0, Kubernetes : 1.19.9
Kafka 3 brokers & 3 Zookeeper nodes
Listener of Ingress Type(using Nginx Ingress Controller with Internal Load balancer) is configured with SASL_SSL authentication

• So we suddenly started getting alerts that one of our Kafka brokers is down and the alert gets resolved in a span of 3-4 mins
• Checked the pod/container for restarts, but there was no restart on any of the brokers
• Following were the logs for the moment when broker-2 was down

  (kafka.controller.KafkaController) [controller-event-thread]
  2021-08-30 14:22:48,328 TRACE [Controller id=2] Checking need to trigger auto leader balancing 
  (kafka.controller.KafkaController) [controller-event-thread]
  2021-08-30 14:22:48,328 DEBUG [Controller id=2] Topics not in preferred replica for broker 2 Map() 
  (kafka.controller.KafkaController) [controller-event-thread]
  2021-08-30 14:22:48,328 TRACE [Controller id=2] Leader imbalance ratio for broker 2 is 0.0 
  (kafka.controller.KafkaController) [controller-event-thread]
  2021-08-30 14:22:48,328 DEBUG [Controller id=2] Topics not in preferred replica for broker 1 Map() 
  (kafka.controller.KafkaController) [controller-event-thread]
  2021-08-30 14:22:48,328 TRACE [Controller id=2] Leader imbalance ratio for broker 1 is 0.0 
  (kafka.controller.KafkaController) [controller-event-thread]
  2021-08-30 14:22:48,328 DEBUG [Controller id=2] Topics not in preferred replica for broker 0 Map() 
  (kafka.controller.KafkaController) [controller-event-thread]
  2021-08-30 14:22:48,328 TRACE [Controller id=2] Leader imbalance ratio for broker 0 is 0.0 
  (kafka.controller.KafkaController) [controller-event-thread] ```

• As you can see in the image within a span of approx two hours we got so many alerts and all the time the logs were the same

Could not accurately figure out why this is happening. Can you please help here. TIA

@scholzj

[scholzj]

I do not know what does this mean either. If the broker works it is probably either some issue with metrics scraping or with the metric. I think the first thing you will have to figure out is whether the broker is still part of the clusters, consumer and producers work, replicas on the broker are in sync etc. The should help you decide if it is monitoring or an actual issue.

I also wonder if the alert you use is correct - I think that kafka_server_replicamanager_leadercount means how replica leaders does the broker have. But it could be that the broker has no replica leaders without any errors or issues.

[abhi1git]

I checked the pointers which you asked me and the following are the findings :

• Broker is still a part of the cluster
  

• All Replicas on the broker are in sync
  

• Alert which I have used is created on the panel of Grafana Dashboards provided by Strimzi itself

• Also this issue started recently after I force renewed the certificates to extend their validity, but to add to that this certificate validity extension procedure was successfully carried out on all our environments and only on prod we are facing this issue. Before that the grafana Panel used to show count(kafka_server_replicamanager_leadercount) as 3 all time
  

• So this brings me to a conclusion that there is some issue within my broker itself

Can you please guide me in what scenarios does kafka brokers produce those logs which I mentioned in the question?

[scholzj]

I do not really know when does it produce them. I think that it is just a periodical check if the replica leaders are where they are supposed to be. But do not know for sure.

I think you should check the topics when it is happening. You said all replicas on the brokers are in sync. But that does to answer whether there are any leaders on the broker in that moment or not.