Kafka brokers getting timeout error

[Muddam234]

Hi All,
I am getting an error in my strimzi Kafka cluster like (error: Exceeded timeout of 300000ms while waiting for Service resource qa-xxx-kafka-gcp-kafka-externalint-bootstrap in namespace qa-xxxxx-kafka to be addressable)
And my zookeeper pod is up and running and Kafka broker pod is not running (even not showing Kafka pod inside the namespace) when i check the log showing the above error.
I have also deployed Kafka broker and deployment is successful but broker pods are not visible (edited)

[scholzj]

You should probably start by sharing your configuration. If you decided to use load balancer listener, it is likely that the loadbalancer is not being provisioned by your Kubernetes cluster.

[Muddam234]

please find the configuration for the Kafka broker

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: qa-xxxx-kafka-gcp
spec:
kafka:
version: 2.7.0
replicas: 3
authorization:
type: simple
superUsers:
- kafka-super-user
resources:
limits:
cpu: "2"
memory: xxxx
requests:
cpu: "0.75"
memory: xxxx
listeners:
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: scram-sha-512
- name: external
type: ingress
port: 9094
tls: true
authentication:
type: scram-sha-512
configuration:
class: nginx
bootstrap:
host: kafka-ext-bootstrap.qa-xxxx-kafka.gcp.mfs.xxxx.com
brokers:
- broker: 0
host: kafka-0.qa-xxxxx-kafka.gcp.mfs.xxxxx.com
- broker: 1
host: kafka-1.qa-xxxx-kafka.gcp.mfs.xxxx.com
- broker: 2
host: kafka-2.qa-xxxx-kafka.gcp.mfs.xxxx.com
- name: externalint
type: loadbalancer
port: 9095
tls: true
authentication:
type: scram-sha-512
configuration:
bootstrap:
loadBalancerIP: 172.xx.xxx.xxx
# alternativeNames:
# - 20.xx.xx.xx.bc.googleusercontent.com #GKE QA, loadbalancer
annotations:
networking.gke.io/load-balancer-type: "Internal"
brokers:
- broker: 0
loadBalancerIP: 172.xx.xx.xx
# advertisedHost: 21.xx.xx.xx.bc.googleusercontent.com #GKE QA, loadbalancer
annotations:
networking.gke.io/load-balancer-type: "Internal"
- broker: 1
loadBalancerIP: 172.xx.xx.xx
# advertisedHost: 22.xx.xx.xx.bc.googleusercontent.com #GKE QA, loadbalancer
annotations:
networking.gke.io/load-balancer-type: "Internal"
- broker: 2
loadBalancerIP: 172.xx.xx.xx
# advertisedHost: 23.xx.xx.xx.bc.googleusercontent.com #GKE QA, loadbalancer
annotations:
networking.gke.io/load-balancer-type: "Internal"
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.7"
request.timeout.ms: 120000
transaction.max.timeout.ms: 60000
transactional.id.expiration.ms: 1814400000
# Retention - keep data up to 28 days
log.retention.hours: 672

[scholzj]

If you format it as a code, it would be much easier to read it. It would also keep the whitespaces which are important for YAML. Without the white spaces, it is hard to say if the structure of the YMAL is valid or not.

But in general, you are using type: loadbalancer listener. So the error message means that Kubernetes did not provision the load balancers in time. If you do kubectl get service -o wide you should see the load balancer services and the external address should say something like <pending> because the load balancer was not provisioned.

Why was it not provisioned, depends on your environment. You might have reached the limit of allowed load balancers, you might have a wrong configuration (e.g. the IP address you specified there), etc. You would need to check your Kubernetes and cloud provider logs to find out what the exact issue is.

[Muddam234]

please find the below output and it was working earlier. it started throwing error from last couple of days. please suggest

[scholzj]

Well, as I said above ... this is about your infrastructure. You will need to look there into the Kubernetes logs, to the cloud provider etc. Strimzi just creates the load balancer type services and the rest is about Kubernetes and your cloud.

[Muddam234]

Thanks Scholzj and my issue is resolved.