Unable to authenticate using TLS after using let's encrypt

[Nimeh]

Hi There,

I followed this blog to use cert-manager with let's encrypt on Strimzi:

https://strimzi.io/blog/2021/05/07/deploying-kafka-with-lets-encrypt-certificates/

However the blog does not cover mTLS authentication, so I enabled mTLS on my Kafka cluster, however when I try to connect to it using the let's encrypt certificate i am getting this error:

certificate verify failed: broker certificate could not be verified

here's my kafka yaml manifest:

kind: Kafka
metadata:
  name: strimzi-kafka-test
  namespace: strimzi-operator-test
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      auto.create.topics.enable: true
      default.replication.factor: 1
      delete.topic.enable: true
      inter.broker.protocol.version: 3.2
      log.retention.hours: 72
      min.insync.replicas: 1
      num.recovery.threads.per.data.dir: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
      unclean.leader.election.enable: false
    jvmOptions:
      -Xms: 4G
      -Xmx: 4G
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    - configuration:
        bootstrap:
          annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          host: bootstrap.example.com
        brokerCertChainAndKey:
          certificate: tls.crt
          key: tls.key
          secretName: strimzi-operator-test-lets-encrypt
        brokers:
        - annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          broker: 0
          host: broker-0.example.com
        - annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          broker: 1
          host: broker-1.example.com
        - annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          broker: 2
          host: broker-2.example.com
      name: external
      port: 9094
      tls: true
      authentication:
        type: tls
      type: ingress
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    replicas: 3
    resources:
      limits:
        cpu: 2
        memory: 8Gi
      requests:
        cpu: 1
        memory: 6Gi
    storage:
      type: jbod
      volumes:
      - class: gp2
        deleteClaim: false
        id: 0
        size: 50Gi
        type: persistent-claim
    version: 3.2.0
  zookeeper:
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    replicas: 3
    resources:
      limits:
        cpu: 1000m
        memory: 3Gi
      requests:
        cpu: 500m
        memory: 2Gi
    storage:
      class: gp2
      deleteClaim: false
      size: 50Gi
      type: persistent-claim

The cluster is deployed without issues however when I get the cert, key and ca from the secret I am unable to connect.

I have been trying to add the tls common configuration like this as per the docs but with no luck:

tls:
  trustedCertificates:
    - secretName: my-cluster-cluster-cert
      certificate: ca.crt
    - secretName: my-cluster-cluster-cert
      certificate: ca2.crt

Don't know what I am missing or misunderstanding, My goal is to disable Strimzi of trusting the default java certificate I want to set a certificate that the client uses to connect to it that is managed by cert-manager so I don't worry about renewal. :)

Thanks

[scholzj]

I think you should start from the beginning and explain what exactly you want to do and what are the steps you did. Without that, it is not clear what exactly you are talking about.

Let's Encrypt has nothing to do with mTLS client authentication. That is why the blog post does not mention it. It provides server certificates for verified domain names. That is only useful for server authentication.

[scholzj]

First of all, you always have to worry about certificate renewals. With Let's Encrypt, they happen fairly often. You have to make sure the tooling always works to get the new certificate, it was properly passed to Strimzi and that Strimzi rolls it out to the brokers. I do not want to discourage you form using Let's Encrypt, but you should not expect it be something you do not have to worry about.

Second, I still think you are not really clear at what did you do. So you should probably take it step by step. In particular this part So we added the tls authentication filed as shown in the yaml manifest above, however when trying to connect to the cluster using the certificate generated by cert-manager we are getting this error. makes no sense. You enabled mTLS authentication there, yes. But that is about it. There is no indication of using a custom Clients CA or anything like that. So you use whatever is in brokerCertChainAndKey as server certificate and the usual Strimzi Client CA for the client authentication.

[Nimeh]

ok I will share a step by step everything:

1. We deployed cert-manager.
2. We created cluster issuer with cloudflare as we are using it as our DNS.
3. We created a certificate, here's the yaml:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: strimzi-operator-test-lets-encrypt
  namespace: strimzi-operator-test
spec:
  secretName: strimzi-operator-test-lets-encrypt
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
    group: cert-manager.io
  subject:
    organizations:
      - exmple.com
  dnsNames:
    - bootstrap.exmple.com
    - broker-0.exmple.com
    - broker-1.exmple.com
    - broker-2.exmple.com

4. Deployed the following Strimzi cluster:

kind: Kafka
metadata:
  name: strimzi-kafka-test
  namespace: strimzi-operator-test
spec:
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    config:
      auto.create.topics.enable: true
      default.replication.factor: 1
      delete.topic.enable: true
      inter.broker.protocol.version: 3.2
      log.retention.hours: 72
      min.insync.replicas: 1
      num.recovery.threads.per.data.dir: 1
      offsets.topic.replication.factor: 1
      transaction.state.log.min.isr: 1
      transaction.state.log.replication.factor: 1
      unclean.leader.election.enable: false
    jvmOptions:
      -Xms: 4G
      -Xmx: 4G
    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    - configuration:
        bootstrap:
          annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          host: bootstrap.example.com
        brokerCertChainAndKey:
          certificate: tls.crt
          key: tls.key
          secretName: strimzi-operator-test-lets-encrypt
        brokers:
        - annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          broker: 0
          host: broker-0.example.com
        - annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          broker: 1
          host: broker-1.example.com
        - annotations:
            kubernetes.io/ingress.class: nginx
            nginx.ingress.kubernetes.io/ssl-passthrough: true
          broker: 2
          host: broker-2.example.com
      name: external
      port: 9094
      tls: true
      type: ingress
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    replicas: 3
    resources:
      limits:
        cpu: 2
        memory: 8Gi
      requests:
        cpu: 1
        memory: 6Gi
    storage:
      type: jbod
      volumes:
      - class: gp2
        deleteClaim: false
        id: 0
        size: 50Gi
        type: persistent-claim
    version: 3.2.0
  zookeeper:
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 5
    replicas: 3
    resources:
      limits:
        cpu: 1000m
        memory: 3Gi
      requests:
        cpu: 500m
        memory: 2Gi
    storage:
      class: gp2
      deleteClaim: false
      size: 50Gi
      type: persistent-claim

5. Tested telnet to the cluster and all went well:

telnet bootstrap.example.com 443

6. Created a test producer and consumer:

kafka-console-producer --bootstrap-server bootstraptest.example.com:443  --topic testtopic --producer-property security.protocol=SSL

kafka-console-consumer --bootstrap-server bootstraptest.example:com443 --topic testtopic --from-beginning --consumer-property security.protocol=SSL

Until here all works as a charm
Then we found out that this does not use mTLS so we added the tls authentication to the Kafka manifest, after that the producer and consumer is step 6 stopped working which is expected and exactly what we want as they don't provide any certificate or key

we tried to use the certificate in the secret strimzi-operator-test-lets-encrypt that was created before but we are not able to connect, we don't want to provide custom CA we just want to use the generated certificate on the clients.

Do we need to use custom CA, If yes, I read on another discussion that we should not use Let's Encrypt as a Custom CA, Honestly I thought that we can use Let's encrypt CA with the certificate and key that are in the generated secret.

When it renews we already have External secrets operator that is chained with vault so we just update vault and it will be updated everywhere.

Also please share any recommendations for Strimzi certificate management automation if you have any.

Thank you very much for your help.

[scholzj]

You are mixing two different things. The steps you described are fine for using your Let's Encrypt certificate as the server certificate. But that has nothing to do with mTLS. That is just TLS.

If you want to do mTLS with a custom certificate, you have to provide a custom Clients CA. You can do that and there are docs for that if you want to. But that is still something you would do with some kind of private CA and not with Let's Encrypt (you could still use Let's Encrypt for the server certificate of course, but some custom private CA for the Client CA for mTLS authentication).

Note: This is how TLS / mTLS works in general - not some Strimzi thing.

[Nimeh]

Got you,

So we need to create a Private CA then use that in the kafka manifest and manage it ourselves, if that is the case let me try it and see.

I found this repo https://github.com/scholzj/strimzi-custom-ca-test/tree/master which is created by you that has examples for that :)

Thanks again.

[scholzj]

So we need to create a Private CA then use that in the kafka manifest and manage it ourselves, if that is the case let me try it and see.

For the TLS client authentication, yes. As I said, you can keep using LEt's Encrypt as the server cert of course as these are two separate independent things.