MTLS between Kafka and KafkaConnect

[BenSyntio]

Hello, I haven't found a question like this so I am posting it.

My goal is to understand how to set up MTLS between Kafka cluster and KafkaConnect, both deployed using Strimzi.
When I say set up MTLS I mean setting up MTLS authentication, not authorization.

First question would be is it even possible to set up MTLS authentication, without having any type of authorization on cluster?

Now I will describe my process and give you the code that I used.

From my understanding, I first need to generate the certificate and key for KafkaConnect which i did using the following command:
openssl req -new -newkey rsa:4096 -x509 -sha256 -days 365 -nodes -out MyCertificate.crt -keyout MyKey.key

I deployed the strimzi operator, and set up the registry for KafkaConnect to push images to. After that I created a kafka cluster using the following manifest:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
spec:
  kafka:
    version: 3.1.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.1"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 50Gi
        deleteClaim: false
  zookeeper:
    replicas: 1
    storage:
      type: persistent-claim
      size: 20Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

Then I created a KafkaConnect cluster using the following manifest:

cat <<EOF | kubectl -n kafka-cluster-test apply -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-connect
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  replicas: 1
  bootstrapServers: kafka-cluster-kafka-bootstrap:9093
  authentication:
    type: tls
    certificateAndKey:
      certificate: MyCertificate.crt
      key: MyKey.key
      secretName: kafka-connect-crt
  tls:
    trustedCertificates:
      - secretName: kafka-cluster-cluster-ca-cert
        certificate: ca.crt
  config:
    group.id: connect-cluster
    config.storage.replication.factor: 1
    offset.storage.replication.factor: 1
    status.storage.replication.factor: 1
  build:
    output:
      type: docker
      image: mydockerrepo.azurecr.io/tls-connect-test:latest
      pushSecret: acrsecret
    plugins:
      - name: jdbc-connector
        artifacts:
          - type: zip
            url: https://d1i4a15mxbxib1.cloudfront.net/api/plugins/confluentinc/kafka-connect-jdbc/versions/10.3.3/confluentinc-kafka-connect-jdbc-10.3.3.zip
      - name: mongodb-connector
        artifacts:
          - type: jar
            url: https://repo1.maven.org/maven2/org/mongodb/kafka/mongo-kafka-connect/1.7.0/mongo-kafka-connect-1.7.0-all.jar
  logging:
    type: inline
    loggers:
      log4j.rootLogger: "INFO"
  readinessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
  livenessProbe:
    initialDelaySeconds: 15
    timeoutSeconds: 5
EOF

This is the error that I get:

The problems could be the configuration and/or my misunderstanding of TLS and certificates.
Another possible problem is that there needs to be some authorization (maybe I need to create a KafkaUser, but im not sure).

So the questions would be:

1. is it even possible to set up MTLS authentication, without having any type of authorization on cluster?
2. Do I need a KafkaUser?
3. Did I misunderstand something with the TLS certificates?

[scholzj]

You cannot use any ran dom certificate which you generate. Then it would not be an authentication as it would let anyone in. The user certificate for authentication needs to be signed by the Clients CA. You can use the User OPerator and the KAfkaUser resource to create it. Check the secured examples on GitHub, it should give you an idea.

[BenSyntio]

Thank you for the answer. That helped me understand pretty much all of my problems.