Kafka cluster pod is not creating after zookeeper pod started

[subinmt]

Dear Strimzi Team,

I am trying to create a Kafka cluster with an ingress external listener. Everything is created except kafka broker pod. I am pasting all configurations and logs here. Could you please help me with this?

Note: The cluster is created in a DC and I added some annotations,initContainers and configmaps for the ingress controller.

nginx version: nginx/1.19.9
Strimzi Kafka Operator version: 0.27.1

Kafka Cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi
  namespace: kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 1
    listeners:
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          class: kafka
          bootstrap:
            host: dev-kafka-bootstrap.mydomain.com
            annotations:
              danm.k8s.io/interfaces: '[ {"clusterNetwork":"flannel-local", "ip": "dynamic"}, {"clusterNetwork":"mx-linknet", "ip": "30.xx.xx.xx"} ]'
          brokers:
          - broker: 0
            host: dev-kafka-brokers.mydomain.com
            annotations:
              danm.k8s.io/interfaces: '[ {"clusterNetwork":"flannel-local", "ip": "dynamic"}, {"clusterNetwork":"mx-linknet", "ip": "30.xx.xx.xx"} ]'
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      inter.broker.protocol.version: "3.0"
      log.retention.hours: 1
    storage:
      type: ephemeral
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - kafka
              topologyKey: "kubernetes.io/hostname"
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                  - key: app
                    operator: In
                    values:
                    - kafka
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: app.kubernetes.io/name
                  operator: In
                  values:
                  - zookeeper
              topologyKey: "kubernetes.io/hostname"
  entityOperator:
    topicOperator: {}
    userOperator: {}

Strimzi kafka cluster Operator logs:

kubectl logs strimzi-cluster-operator-d48446dd7-x6bd8 -n kafka
2022-02-14 04:08:25 WARN  AbstractOperator:532 - Reconciliation #953(watch) Kafka(kafka/strimzi): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Ingress resource strimzi-kafka-bootstrap in namespace kafka to be addressable
	at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:150) ~[io.strimzi.operator-common-0.27.1.jar:0.27.1]
	at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:69) ~[io.vertx.vertx-core-4.2.1.jar:4.2.1]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-02-14 04:10:09 INFO  ClusterOperator:121 - Triggering periodic reconciliation for namespace kafka
2022-02-14 04:10:09 INFO  AbstractOperator:226 - Reconciliation #957(timer) Kafka(kafka/strimzi): Kafka strimzi will be checked for creation or modification
2022-02-14 04:10:09 WARN  AbstractModel:253 - Reconciliation #957(timer) Kafka(kafka/strimzi): A ZooKeeper cluster with a single replica and ephemeral storage will be in a defective state after any restart or rolling update. It is recommended that a minimum of three replicas are used.

Ingress Controller log:

kubectl logs ingress-kafka-ingress-nginx-controller-5d4f44f5f4-4vmxf -n kafka
I0214 03:58:24.082847       7 main.go:101] "successfully validated configuration, accepting" ingress="kafka/strimzi-kafka-0"
I0214 03:58:24.083299       7 admission.go:149] processed ingress via admission controller {testedIngressLength:1 testedIngressTime:0.045s renderingIngressLength:1 renderingIngressTime:0s admissionTime:18.4kBs testedConfigurationSize:0.045}
I0214 03:58:24.083353       7 main.go:101] "successfully validated configuration, accepting" ingress="kafka/strimzi-kafka-bootstrap"
I0214 03:58:24.089604       7 store.go:424] "Found valid IngressClass" ingress="kafka/strimzi-kafka-bootstrap" ingressclass="kafka"
I0214 03:58:24.089770       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kafka", Name:"strimzi-kafka-bootstrap", UID:"8d42b582-c0a2-46af-a55e-24afc8f4bdfc", APIVersion:"networking.k8s.io/v1", ResourceVersion:"193842652", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0214 03:58:24.104355       7 store.go:424] "Found valid IngressClass" ingress="kafka/strimzi-kafka-0" ingressclass="kafka"
I0214 03:58:24.104569       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"kafka", Name:"strimzi-kafka-0", UID:"932dcee1-5411-4881-bb6d-102b85e18ffa", APIVersion:"networking.k8s.io/v1", ResourceVersion:"193842654", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0214 03:58:25.124530       7 controller.go:1083] Service "kafka/strimzi-kafka-external-bootstrap" does not have any active Endpoint.
W0214 03:58:25.124608       7 controller.go:1083] Service "kafka/strimzi-kafka-0" does not have any active Endpoint.
I0214 03:58:25.124776       7 controller.go:155] "Configuration changes detected, backend reload required"
I0214 03:58:25.216338       7 controller.go:172] "Backend successfully reloaded"
I0214 03:58:25.216670       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kafka", Name:"ingress-kafka-ingress-nginx-controller-5d4f44f5f4-4vmxf", UID:"b9175373-f8c5-4504-8023-4aab1c24c187", APIVersion:"v1", ResourceVersion:"193837525", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
W0214 03:58:28.458005       7 controller.go:1083] Service "kafka/strimzi-kafka-external-bootstrap" does not have any active Endpoint.
W0214 03:58:28.458075       7 controller.go:1083] Service "kafka/strimzi-kafka-0" does not have any active Endpoint.

Pods:

kubectl get po -n kafka
NAME                                                      READY   STATUS    RESTARTS   AGE
ingress-kafka-ingress-nginx-controller-5d4f44f5f4-4vmxf   1/1     Running   0          25m
strimzi-cluster-operator-d48446dd7-x6bd8                  1/1     Running   0          3d22h
strimzi-zookeeper-0                                       1/1     Running   0          20m

StatefulSets:

kubectl get statefulset -n kafka
NAME                READY   AGE
strimzi-zookeeper   1/1     2m46s

PVC:

kubectl get pvc -n kafka   
No resources found in kafka namespace.

Ingress:

kubectl get ing -n kafka      
NAME                           CLASS   HOSTS                         ADDRESS   PORTS     AGE
strimzi-kafka-0           kafka   dev-kafka-brokers.mydomain.com               80, 443   2m57s
strimzi-kafka-bootstrap   kafka   dev-kafka-bootstrap.mydomain.com             80, 443   2m57s

Ingress Resource yaml format:

kubectl get ing -n kafka -oyaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      danm.k8s.io/interfaces: '[ {"clusterNetwork":"flannel-local", "ip": "dynamic"},
        {"clusterNetwork":"mx-linknet", "ip": "30.xx.xx.xx"} ]'
      ingress.kubernetes.io/ssl-passthrough: "true"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    creationTimestamp: "2022-02-14T03:58:24Z"
    generation: 1
    labels:
      app.kubernetes.io/instance: strimzi
      app.kubernetes.io/managed-by: strimzi-cluster-operator
      app.kubernetes.io/name: kafka
      app.kubernetes.io/part-of: strimzi-strimzi
      strimzi.io/cluster: strimzi
      strimzi.io/kind: Kafka
      strimzi.io/name: strimzi-kafka
    name: strimzi-kafka-0
    namespace: kafka
    ownerReferences:
    - apiVersion: kafka.strimzi.io/v1beta2
      blockOwnerDeletion: false
      controller: false
      kind: Kafka
      name: strimzi
      uid: c3b9396c-17bf-4164-b7ad-cf8c602fb041
    resourceVersion: "193842654"
    uid: 932dcee1-5411-4881-bb6d-102b85e18ffa
  spec:
    ingressClassName: kafka
    rules:
    - host: dev-kafka-brokers.mydomain.com
      http:
        paths:
        - backend:
            service:
              name: strimzi-kafka-0
              port:
                number: 9094
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - dev-kafka-brokers.mydomain.com
  status:
    loadBalancer: {}
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      danm.k8s.io/interfaces: '[ {"clusterNetwork":"flannel-local", "ip": "dynamic"},
        {"clusterNetwork":"mx-linknet", "ip": "30.xx.xx.xx"} ]'
      ingress.kubernetes.io/ssl-passthrough: "true"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    creationTimestamp: "2022-02-14T03:58:24Z"
    generation: 1
    labels:
      app.kubernetes.io/instance: strimzi
      app.kubernetes.io/managed-by: strimzi-cluster-operator
      app.kubernetes.io/name: kafka
      app.kubernetes.io/part-of: strimzi-strimzi
      strimzi.io/cluster: strimzi
      strimzi.io/kind: Kafka
      strimzi.io/name: strimzi-kafka
    name: strimzi-kafka-bootstrap
    namespace: kafka
    ownerReferences:
    - apiVersion: kafka.strimzi.io/v1beta2
      blockOwnerDeletion: false
      controller: false
      kind: Kafka
      name: strimzi
      uid: c3b9396c-17bf-4164-b7ad-cf8c602fb041
    resourceVersion: "193842652"
    uid: 8d42b582-c0a2-46af-a55e-24afc8f4bdfc
  spec:
    ingressClassName: kafka
    rules:
    - host: dev-kafka-bootstrap.mydomain.com
      http:
        paths:
        - backend:
            service:
              name: strimzi-kafka-external-bootstrap
              port:
                number: 9094
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - dev-kafka-bootstrap.mydomain.com
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

[scholzj]

The operator is waiting for the Ingresses to become accepted by the Ingress controler and be ready => that is indicated in their status. This does not seem to happen in your case. That is why the deployment fails with Exceeded timeout of 300000ms while waiting for Ingress resource strimzi-kafka-bootstrap in namespace kafka to be addressable. This is also the empty ADDRESS collumn when you list the Ingresses.

[subinmt]

So is this completely an ingress controller issue? I am a bit confused by your queries. Can you help me to provide the steps to resolve this issue?

[scholzj]

You Ingress controller should update the status of the Ingress resources which it doesn't. So the operator does not know if they are ready or not. I do not know how to resolve it. Strimzi is developed and tested with the Kubernetes Nginx Ingress controller which does this normally out of the box. But depending on your controller, it might differ.

[subinmt]

Thanks @scholzj for your all help. I resolved the issue. I added my external IP address on the below line of my ingress controller and the kafka cluster pod is created now.
https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml#L458

externalIPs:
- 30.xx.xx.xx