[Bug]: Client truststore has not declared certificates from trustedCertificates

[FlowsenAusMonotown]

Bug Description

I'm trying to set up a kafka cluster using strimzi operator. Therefore I want to use my own generated certificates. I have multiple CAs, which sign the certificates for the kafka usage. Kafka it self has an certificate, which is signed by a service CA. Other clients have signed certificates by other intermidiate CA. All intermediate CAs are signed by one root CA.

So we got the following kafka operator configuration:

  apiVersion: kafka.strimzi.io/v1beta2
  kind: Kafka
  metadata:
    name: ${local.kafka-cluster-name}
    namespace: ${kubernetes_namespace_v1.instance_namespace.metadata.0.name}
  spec:
    kafka:
      version: 3.4.0
      replicas: 1
      listeners:
        - name: external
          port: 9093
          type: loadbalancer
          tls: true
          authentication:
            type: tls
          configuration:
            brokerCertChainAndKey:
              secretName: ${local.external-kafka-cert-secret-name}
              certificate: tls.crt
              key: tls.key
        - name: internal
          port: 29093
          type: cluster-ip
          tls: true
          authentication:
            type: tls
          configuration:
            brokerCertChainAndKey:
              secretName: ${local.internal-kafka-cert-secret-name}
              certificate: tls.crt
              key: tls.key
      config:
        offsets.topic.replication.factor: 1
        transaction.state.log.replication.factor: 1
        transaction.state.log.min.isr: 1
        default.replication.factor: 1
        min.insync.replicas: 1
        inter.broker.protocol.version: "3.4"
        offsets.retention.minutes: 525600
        allow.everyone.if.no.acl.found: false
        auto.create.topics.enable: false
      storage:
        type: jbod
        volumes:
        - id: 0
          type: persistent-claim
          size: 100Gi
          deleteClaim: false
      authorization:
        type: simple
        superUsers:
          - User:CN=${local.platform-admin-app-id}
          - User:Services:onlinesuiteplus-kafka
      tls:
        trustedCertificates:
          - secretName: ${kubernetes_secret_v1.root-ca-certificate.metadata.0.name}
            certificate: tls.crt
          - secretName: ${kubernetes_secret_v1.platform-ca-certificate.metadata.0.name}
            certificate: tls.crt
    zookeeper:
      replicas: 1
      storage:
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
    entityOperator:
      topicOperator: {}
      userOperator: {}

I checked the created keystores. The one for custom internal and custom external look quiet good, having all needed certicate information. But looking at the clients.truststore.p12 file there a not the ones which are declared within 'trustedCertificates'. There only one certificate exist: CN=clients-ca v0,O=io.strimzi

Steps to reproduce

• Use config from description
• Start Kubernetes strimzi kafka operator
• Pods are running...
• check created key- and trustores from created strimzi.properties files

Expected behavior

Created clients.trustore.p12 should contain certificate information from the 'trustedCertificates' configuration

Strimzi version

0.35.0

Kubernetes version

1.25.9

Installation method

Terraform files

Infrastructure

Minikube and Azure Cloud

Configuration files and logs

No response

Additional context

No response

[scholzj]

I'm not sure I understand what exactly are you trying to do. Can you please make sure the YAML you shared is properly formatted as without whitespaces it is unreadable? Also, can you please explain what certificates are you talking about? User certificates of clients connecting using mTLS? Or the certificates of the server compoenents? Etc.

[FlowsenAusMonotown]

Hi @scholzj. I updated the formatting of the description for a better readability.
We use our own certificate engine for generating certificates, which should used by kafka for client communication. The cluster communication can be done by Strimzi itself. We need to declare the certificates for external and internal usage and also the truststore to use for both. This truststore should contain multiple certificates (each for every intermidiate, the root and a custom one). How can I handle this with Strimzi? I tried it with the configuration above, but it seems only to work for the listeners certificates, but not for the truststore for it.

[scholzj]

Thanks for fixing the formatting.

In general, this part of the Kafka resource is fine:

          configuration:
            brokerCertChainAndKey:
              secretName: ${local.internal-kafka-cert-secret-name}
              certificate: tls.crt
              key: tls.key

Just keep in mind that it configures only the server certificate used by the given interface. So it has no impact on any trust stores in the broker.

However this part:

      tls:
        trustedCertificates:
          - secretName: ${kubernetes_secret_v1.root-ca-certificate.metadata.0.name}
            certificate: tls.crt
          - secretName: ${kubernetes_secret_v1.platform-ca-certificate.metadata.0.name}
            certificate: tls.crt

Is completely made up. There is no support for such fields. This exists in the Kafka client-based components such as Kafka Connect. But not in the Kafka resource.

If you want to use your own TLS certificates for client authentication (ie. control the trust store), you should configure a custom Clients CA: https://strimzi.io/docs/operators/latest/full/deploying.html#installing-your-own-ca-certificates-str ... I think you should be able to provide multiple public keys in the <cluster_name>-clients-ca-cert secret if you want multiple of them. You might not be able to generate the user certificates with the User Operator and the KafkaUser resources, but with your own certificate management you likely don't want to do it anyway, so it should not be an issue for you. And if you do not plan to do that, the private key (<cluster_name>-clients-ca secret) can just contain some dummy value.

[rjones-duchancell]

Can I jump in here please? I just created a discussion regarding certs, but after further searching, this discussion is similar to what I'm looking to do. If I configure my external listener with brokerCertChainAndKey, do I need to override the clientsCa secrets/certs? In my case, I need to enable producers and consumers that are outside the Kubernetes cluster (there will be internal clients, too). I've chosen to use an OpenShift route to do this. Thanks!

[FlowsenAusMonotown]

Our working solution looks as follows:

# Kafka Truststore certificate chain
resource "kubernetes_secret_v1" "kafka_cluster_clients_ca_cert" {
  metadata {
    name = "${var.instance_name}-clients-ca-cert"
    namespace = kubernetes_namespace_v1.instance_namespace.metadata.0.name
    labels = {
      "strimzi.io/cluster" = var.instance_name,
      "strimzi.io/kind" = "Kafka"
    }
    annotations = {
      "strimzi.io/ca-cert-generation": 0
    }
  }

  data = {
    "ca-root.crt"      = var.root-ca-certificate-pem
    "ca-service.crt"   = var.service-ca-certificate-pem
  }
}

#Not used but required
resource "kubernetes_secret_v1" "kafka_cluster_clients_ca" {
  metadata {
    name = "${var.instance_name}-clients-ca"
    namespace = kubernetes_namespace_v1.instance_namespace.metadata.0.name
    labels = {
      "strimzi.io/cluster" = var.instance_name,
      "strimzi.io/kind" = "Kafka"
    }
    annotations = {
      "strimzi.io/ca-key-generation": 0
    }
  }

  data = {
    "ca.key"  = tls_private_key.dummy_not_used_private_key.private_key_pem
  }
}

resource "tls_private_key" "dummy_not_used_private_key" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P384"
}