User:ANONYMOUS is Denied Operation

[scottc-git]

Problem
I created a KafkaUser and configured the MM2 to use the user for mTLS but the broker logs showed below errors. How can I allow the user to access all those KafkaConnect internal topics used for MM2? I even added the user to the superUsers list but to no avail.

Broker Logs

2022-07-15 22:47:07,833 INFO Principal = User:ANONYMOUS is Denied Operation = DescribeConfigs from host = 192.168.32.167 on resource = Topic:LITERAL:strimzi-mm2-offset for request = DescribeConfigs with resourceRefCount =
1 (kafka.authorizer.logger) [data-plane-kafka-request-handler-2]
2022-07-15 22:48:59,734 INFO Principal = User:ANONYMOUS is Denied Operation = Create from host = 192.168.32.167 on resource = Topic:LITERAL:strimzi-mm2-offset for request = CreateTopics with resourceRefCount = 1 (kafka.aut
horizer.logger) [data-plane-kafka-request-handler-4]
2022-07-15 22:48:59,744 INFO Principal = User:ANONYMOUS is Denied Operation = DescribeConfigs from host = 192.168.32.167 on resource = Topic:LITERAL:strimzi-mm2-offset for request = DescribeConfigs with resourceRefCount =
1 (kafka.authorizer.logger) [data-plane-kafka-request-handler-6]
2022-07-15 22:45:51,931 INFO Principal = User:ANONYMOUS is Denied Operation = Describe from host = 192.168.32.167 on resource = Topic:LITERAL:strimzi-mm2-offset for request = Metadata with resourceRefCount = 1 (kafka.autho
rizer.logger) [data-plane-kafka-request-handler-1]

The MirrorMaker pod printed similar error message.

2022-07-15 23:14:49,821 ERROR [Worker clientId=connect-1, groupId=strimzi-mm2] Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect
-1-1]
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [strimzi-mm2-offset]

Here's snippet of my Kafka spec

    - alias: target-broker
      bootstrapServers: strimzi-kafka-bootstrap:9093
      tls:
        trustedCertificates:
          - secretName: strimzi-cluster-ca-cert
            certificate: ca.crt
      authentication:
        type: tls
        certificateAndKey:
          secretName: mm2-user
          key: user.key
          certificate: user.crt
      config:
        group.id: strimzi-mm2
        # Let Strimzi manage these internal topics.   See ${project.root}./topics-internal
        offset.storage.topic: **strimzi-mm2-offset^**
        config.storage.topic: **strimzi-mm2-config^**
        status.storage.topic: **strimzi-mm2-status^**
        offset.storage.replication.factor: -1
        config.storage.replication.factor: -1
        status.storage.replication.factor: -1

(^: I hoped Strimzi would create those internal storage topics but it didn't so I created them)

and here's the mm2-user spec:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: mm2-user
  labels:
    strimzi.io/cluster: strimzi
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      # Topic
      - operation: Create
        resource:
          type: topic
          name: "*"
      - operation: Write
        resource:
          type: topic
          name: "*"
      - operation: Read
        resource:
          type: topic
          name: "*"
      - operation: Describe
        resource:
          type: topic
          name: "*"
      # Consumer Group
      - operation: Read
        resource:
          type: group
          name: "*"
      - operation: Describe
        resource:
          type: group
          name: "*"

Any help will be appreciated.

[scholzj]

If you are authenticated as ANONYMOUS, that normally means that you do not have enabled authentication. So in your Kafka cluster you most likely enabled only authorization (hence the access denied). You will need to enable authentication as well.

[scottc-git]

But I enabled the authentication as shown above. The mTLS and ACLs worked perfectly for other users I created.

[scottc-git]

An 'admin-user' created a topic and a 'read-only' user could not even 'see' the topic, so that tells me the ACL at least works. Just that 'mm2-user' cannot see those internal topics to begin with. @scholzj

[scholzj]

But I enabled the authentication as shown above.

You need to have it enabled in the Kafka broker. None of the above shows your broker configuration.

[scottc-git]

You will need to enable authentication as well.

This was absolutely true. I've been focusing only on the external listener (so far) and forgot to enable TLS for that internal listener. Once this was done, I didn't need to manually create those internal topics anymore. Thank you @scholzj Strimzi rocks!