Setup MTLS using Cert-Manager as CA

[mohitruhaan]

Hello Experts,

Hope you are doing well!

We are using Strimzi 0.27.1 with Kafka 2.8.0 and trying to setup MTLS between Kafka and Clients. We are using Cert-Manager to get the Certificates signed and are not using Strimzi's Internal CA. Below is Listener Configuration snippet for Kafka.yaml. The secret "combine-acme" contains Chain of Trust Certificates(root and issuing crts for Cert-Manager) and a Server Certificate which is Signed by Cert-Manager.

    authorization:                       #Server Authentication(one way TLS) works fine if this is removed
      type: simple                        #Server Authentication(one way TLS) works fine if this is removed
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:                    #Server Authentication(one way TLS) works fine if this is removed
          type: tls                              #Server Authentication(one way TLS) works fine if this is removed
        configuration:
          brokerCertChainAndKey:
            secretName: combine-acme
            certificate: combine.crt
            key: tls.key
      - name: external
        port: 9094
        type: nodeport
        tls: true
        authentication:                   #Server Authentication(one way TLS) works fine if this is removed
          type: tls                             #Server Authentication(one way TLS) works fine if this is removed
        configuration:
          brokerCertChainAndKey:
            secretName: combine-acme
            certificate: combine.crt
            key: tls.key
          bootstrap:
            nodePort: 32100
          {{- with .Values.brokerConfig }}
            {{- toYaml . | nindent 10 }}
          {{- end }}
    {{- with .Values.kafkaResources }}

For Client Authentication, we created below Certificate Resource and got it Signed by Cert-Manager.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: producer
  namespace: databus
spec:
  secretName: producer-acme
  commonName: producer
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  usages:
    - server auth
    - client auth
  dnsNames:
  - producer.local
  issuerRef:
    name: keyfactor-acme-issuer
    kind: ClusterIssuer
    group: cert-manager.io

Then we extracted its corresponding secret and key and dumped into PKCS#12 (keystore-client.p12) along with Cert-Manager's root and issuing certificates. We are using this as Client's Keystore (keystore-client.p12) in client-ssl-properties file.

Then we created a KafkaUser with below Yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: producer
  labels:
    strimzi.io/cluster: databus-cluster
spec:
  authentication:
    type: tls-external
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: test-topic
          patternType: literal
        operation: Read
        host: "*"
      - resource:
          type: topic
          name: test-topic
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: group
          name: my-group
          patternType: literal
        operation: Read
        host: "*"

Below is the Client-ssl-Properties:

bootstrap.servers=<bootstrap-address>
security.protocol=SSL
ssl.truststore.location=/tmp/truststore-client
ssl.truststore.password=changeit
ssl.keystore.location=/tmp/keystore-client.p12
ssl.keystore.password=changeit
security.protocol=SSL
ssl.keystore.type=PKCS12

truststore-client: Contains Server Certificates.
keystore-client.p12: As referred above.

We are not able to achieve the connectivity with below error:

failed authentication due to: Failed to process post-handshake messages

Can you please guide/suggest what are we missing here?
Thanks!

[scholzj]

I'm not sure I got completely right what you are trying to do ... but ...

• The brokerCertChainAndKey configures the server certificate in the Kafka broker (e.g. in the Kafka broker keystore) -> so it has absolutely no involvment in the client authentication
• The client trust (e.g. the Kafka broker truststore) is based on the Clients CA. So if you want to use Cert Manager for clients, you should configure your own Clients CA and configure the Cert Manager CA used to sign the user certificates as the Clients CA. Then you can create the KafkaUser with tls-external authentication and use the user certificate issued through Cert Manager.

[mohitruhaan]

Thanks @scholzj !

The Client Authentication works now but Authorization is still not working.
Are there some extra config required with respect to Authorization in below KafkaUser?
Or maybe while getting the Certificate Signed from Cert-Manager (though server auth and client auth are listed in usage section of Certificate Resource shared in post above)?

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: producer
  labels:
    strimzi.io/cluster: databus-cluster
spec:
  authentication:
    type: tls-external
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: mohit-test
          patternType: literal
        operation: Read
        host: "*"
      - resource:
          type: topic
          name: mohit-test
          patternType: literal
        operation: Write
        host: "*"
      - resource:
          type: group
          name: console-consumer
          patternType: literal
        operation: Read
        host: "*"
      - resource:
          type: topic
          name: mohit-test
          patternType: literal
        operation: Create
        host: "*"
      - resource:
          type: topic
          name: mohit-test
          patternType: literal
        operation: Describe
        host: "*"

I even tried to add this user in SuperUser (just to check Authorization works but no luck):
Kafka Snippet with SuperUser:

    authorization:
      type: simple
      superUsers:
        - CN=producer
        - producer
    listeners:
      - name: plain
        port: 9092
        type: internal

Below Error on Client and Server End:

[kafka@producer tmp]$ /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server boostrap.staging-2.local:32100  --topic mohit-test --consumer.config /tmp/two-way.p12  --from-beginning
[2022-03-23 11:36:32,222] WARN [Consumer clientId=consumer-console-consumer-39263-1, groupId=console-consumer-39263] Error while fetching metadata with correlation id 2 : {mohit-test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2022-03-23 11:36:32,223] ERROR [Consumer clientId=consumer-console-consumer-39263-1, groupId=console-consumer-39263] Topic authorization failed for topics [mohit-test] (org.apache.kafka.clients.Metadata)
[2022-03-23 11:36:32,224] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [mohit-test]



[databus-cluster-kafka-5 kafka] 2022-03-23 11:36:32,214 INFO Principal = User:CN=producer.local,O=###,C=JP is Denied Operation = Describe from host = 240b:c0e0:104:5400:b412:2:0:0 on resource = Topic:LITERAL:mohit-test for request = Metadata with resourceRefCount = 1 (kafka.authorizer.logger) [data-plane-kafka-request-handler-2]
[databus-cluster-kafka-5 kafka] 2022-03-23 11:36:32,216 INFO Principal = User:CN=producer.local,O=###,C=JP is Denied Operation = Describe from host = 240b:c0e0:104:5400:b412:2:0:0 on resource = Group:LITERAL:console-consumer-39263 for request = FindCoordinator with resourceRefCount = 1 (kafka.authorizer.logger) [data-plane-kafka-request-handler-1]

[scholzj]

Well, as you can see from the logs, your certificate does not have the right subject. It is authenticated as User:CN=producer.####.local,O=###,C=JP. If you want to use the User Operator to manage the ACL rights, the subject needs to contain the CN only.

[mohitruhaan]

Thank you @scholzj for the direction. It all worked perfectly fine as you suggested!
Cheers!

Just for my info, what is the process to apply traces/logs while debugging where i could have catched this error (that User Operator would consider just CN in the Subject?)

[fwindxox]

Hello @mohitruhaan

This is the RM Insider Threat team contacting you to alert that our company data has been exposed to public.
We see our domains are mentioned several times here, and we would like to request to delete your issue here.
We take the security of our data, information systems, and intellectual property seriously. We hope you agree and will help us maintain a strong information security culture at RM
Thank you

[gbabbar]

Hi @mohitruhaan
Just a query regarding the above, have you used self signed certs in cert-manager or some public authority like LetsEncrypt?

[fwindxox]

Hello @mohitruhaan

This is the RM Insider Threat team contacting you to alert that our company data has been exposed to public.
We see our domains are mentioned several times here, and we would like to request to delete this discussion or remove "######" from all the lines you have mentioned.
We take the security of our data, information systems, and intellectual property seriously. We hope you agree and will help us maintain a strong information security culture at RM
Thank you

[AndersBennedsgaard]

@scholzj per #6511 (comment) I think this discussion should be removed, or the comments #6511 (reply in thread) and #6511 (comment) should be edited to not include the domain in question (funny that a threat team includes a reference to their secret domain 😅 )

[scholzj]

To be honest, I'm not sure I should edit someone else posts based on more or less anonymous requests. Ideally, the author should do it. If that is not possible for some reason, the request should be made at least from some account that does not look completely anonymous.

[AndersBennedsgaard]

I completely agree, but as @mohitruhaan already has edited his own comments, it is only your comment and the "Threat team" employee @fwindxox that has a reference to the domain.

[scholzj]

Ok, I edited them.