Kafka PODs are not being created, only zookeeper pods are running.

[Mmiglio]

Hi all,

I am using strimzi 0.25.0. After having installed the operator with helm I tested different configurations (e.g. strimzi-0.25.0/examples/kafka/kafka-ephemeral.yaml ) of kafka cluster to be sure that everything is working.

Then I tried to expose kafka using an ingress, NGINX in this case. After making sure that the controller was running with the TSL passthrough enabled, I created the cluster from this yaml.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 2.8.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: bootstrap.X.X.X.X
          brokers:
            - broker: 0
              host: broker-0.X.X.X.X
            - broker: 1
              host: broker-1.X.X.X.X
            - broker: 2
              host: broker-2.X.X.X.X
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.8"
      inter.broker.protocol.version: "2.8"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

where X.X.X.X is the K8S master IP.
However, in this case only the zookeeper pod is being created. I can see that the ingresses for the brokers are created, but the pods are not. If I remove the external listener everything work again. How can I debug this?

A couple of months ago I followed a similar procedure on another cluster with strimzi 0.22.1 and everything worked (and it is still running).

Thank you in advance.

[scholzj]

The Cluster Operator log should show why is it failing. But from how you describe it, it sounds to me like there is some issue with your Ingress. Strimzi will wait for the Ingress controller to confirm that it accepted the Ingress resource by updating the address in its .status section. When this doesn't happen, the deployment will fail. So I guess you can look into the operator logs or do kubectl get ingress -o yaml to confirm it. And if it is the case, probably look into the Ingress controller logs for the reason why does it not like the resources.

The broker address in the format <broker-X>.<IP-address> sounds a bit weird to me. Is that routable in your network? Does it work with Ingress for other applications? (I'm not sure this is the issue - it might be something else. I just never saw this so it looks suspicious :-o).

[Mmiglio]

The broker address in the format . sounds a bit weird to me. Is that routable in your network? Does it work with Ingress for other applications? (I'm not sure this is the issue - it might be something else. I just never saw this so it looks suspicious :-o).

Actually it is <broker-X>.<IP-address>.< myip.my.institution.com>, i forgot to write the last but important part :) . In the same cluster I'm exposing other services (e.g a dashboard) with <service>.<IP-address>.< myip.my.institution.com> so I think it is working.

This is an example of how I can create a working ingress (adapted from here)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: test.<IP-address>.< myip.my.institution.com>
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080

About kafka: If I look at the controller logs I see this set of messages repeated continuously

2021-09-01 20:06:12 WARN  AbstractOperator:481 - Reconciliation #114(timer) Kafka(default/kafka-cluster): Failed to reconcile
io.strimzi.operator.common.operator.resource.TimeoutException: Exceeded timeout of 300000ms while waiting for Ingress resource kafka-cluster-kafka-bootstrap in namespace default to be addressable
        at io.strimzi.operator.common.Util$1.lambda$handle$1(Util.java:139) ~[io.strimzi.operator-common-0.25.0.jar:0.25.0]
        at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:128) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
        at io.vertx.core.impl.future.FutureBase.lambda$emitFailure$1(FutureBase.java:71) ~[io.vertx.vertx-core-4.1.2.jar:4.1.2]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [io.netty.netty-transport-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.66.Final.jar:4.1.66.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.66.Final.jar:4.1.66.Final]
        at java.lang.Thread.run(Thread.java:829) [?:?]
2021-09-01 20:07:11 INFO  ClusterOperator:118 - Triggering periodic reconciliation for namespace default
2021-09-01 20:07:11 INFO  AbstractOperator:219 - Reconciliation #117(timer) Kafka(default/kafka-cluster): Kafka kafka-cluster will be checked for creation or modification
2021-09-01 20:07:11 WARN  AbstractModel:221 - Reconciliation #117(timer) Kafka(default/kafka-cluster): A ZooKeeper cluster with a single replica and ephemeral storage will be in a defective state after any restart or rolling update. It is recommended that a minimum of three replicas are used.
2021-09-01 20:08:11 INFO  AbstractOperator:363 - Reconciliation #117(timer) Kafka(default/kafka-cluster): Reconciliation is in progress
2021-09-01 20:09:11 INFO  ClusterOperator:118 - Triggering periodic reconciliation for namespace default
2021-09-01 20:09:11 INFO  AbstractOperator:363 - Reconciliation #117(timer) Kafka(default/kafka-cluster): Reconciliation is in progress

[scholzj]

Ok, so that address is more what I would expect. So that should be fine. I think the error suggests that your Ingress ocntroller does nto set the status. When you do kubectl get ingress -o yaml, you should see there something like this:

apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    # ...
  spec:
    # ...
  status:
    loadBalancer:
      ingress:
      - ip: 192.168.1.246
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Your Ingress controller should set the status. But it looks like it didn't. Could it be because of missing ingress class or something like that?

[Mmiglio]

I see this output

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
      ingress.kubernetes.io/ssl-passthrough: "true"
      nginx.ingress.kubernetes.io/backend-protocol: HTTPS
      nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    # ...
    spec:
    rules:
    - host: broker-0.<IP>.<myip.my.institution.com>
      http:
        paths:
        - backend:
            serviceName: kafka-cluster-kafka-0
            servicePort: 9094
          path: /
          pathType: Prefix
    tls:
    - hosts:
      - broker-0.<IP>.<myip.my.institution.com>
  status:
    loadBalancer: {}

there is clearly something missing, but I don't know what. In my other cluster, where things are working, I can see something similar to what I am expecting

# ....
status:
    loadBalancer:
      ingress:
      - ip: 10.10.10.10

[Mmiglio]

Adding the line class: "nginx" seems to solve the problem

 - name: external
    port: 9094
    type: ingress
    tls: true
    authentication:
      type: tls
    configuration:
      class: "nginx"
      bootstrap:
        # ...