[SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication

[ginwakeup]

Hi!

I am using Strimzi 0.33.0

I setup authentication on my cluster using SCRAM 512, no TLS:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: messaging
spec:
  kafka:
    version: 3.2.3 #Apache Kafka version
    replicas: 4
    resources:
      requests:
        cpu: 1000m
        memory: 3Gi
      limits:
        cpu: 2000m
        memory: 6Gi
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
        authentication:
          type: scram-sha-512
      - name: tls
        port: 9093
        type: internal
        tls: false
      - name: external
        port: 9094
        type: loadbalancer
        tls: false
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap:
            loadBalancerIP: ***
          brokers:
          - broker: 0
            loadBalancerIP: ***
          - broker: 1
            loadBalancerIP: ***
          - broker: 2
            loadBalancerIP: ***
          - broker: 3
            loadBalancerIP: ***
    authorization:
      type: simple
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 4
      transaction.state.log.replication.factor: 4
      transaction.state.log.min.isr: 2
      default.replication.factor: 4
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.2.3"
      num.recovery.threads.per.data.dir: 12
      num.network.threads: 4
      num.io.threads: 12
    readinessProbe:
      initialDelaySeconds: 30
      timeoutSeconds: 30
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

Everything seemed to work fine, until I started hitting these errors which seems related to authentication in Zookeeper, but I can't find an answer to this:

2023-02-08 22:07:09,605 INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /***(channelId=***:9092-***:55752-11) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-0-ListenerName(PLAIN-9092)-SASL_PLAINTEXT-6]

The errors happen as soon as the cluster is started, with nothing consuming or connecting to it.

Ideas?

[ginwakeup]

Nothing stands out in zookeeper logs:

2023-02-08 22:09:55,243 INFO Processing ruok command from /127.0.0.1:51406 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]
2023-02-08 22:10:05,159 INFO Processing ruok command from /127.0.0.1:60442 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-3]
2023-02-08 22:10:05,218 INFO Processing ruok command from /127.0.0.1:60470 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-4]
2023-02-08 22:10:15,140 INFO Processing ruok command from /127.0.0.1:41036 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-5]
2023-02-08 22:10:15,231 INFO Processing ruok command from /127.0.0.1:41064 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-6]
2023-02-08 22:10:25,143 INFO Processing ruok command from /127.0.0.1:33204 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-7]
2023-02-08 22:10:25,232 INFO Processing ruok command from /127.0.0.1:33238 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-8]
2023-02-08 22:10:35,201 INFO Processing ruok command from /127.0.0.1:37680 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-1]
2023-02-08 22:10:35,239 INFO Processing ruok command from /127.0.0.1:37712 (org.apache.zookeeper.server.NettyServerCnxn) [nioEventLoopGroup-4-2]

[scholzj]

The error message has nothing to do with ZooKeeper. You have some client or application connecting to the listener on port 9092 which is probably misconfigured and doesn't do the SASL authentication (and instead expects an anonymous connection without any authentication).

[ginwakeup]

@scholzj Is there a way to block these kind of connections after certain amount of attempts so they do not flood the logs?

[scholzj]

No, you have to find the connection and fix the application. Depending on the listener type, the full unredacted message should also give you the IP address of the client. So it should be easy to find and if needed you can also block it through network policies etc. That normally works fine for internal listeners where it would be the Pod IP. But for external listeners, it is usually masked through some infrastructure in between.