[Bug]: tls KafkaUsers secrets contain a CA certificate that is different from the kafka cluster CA certificate

[mdnfiras]

Bug Description

the secret for a tls authenticated KafkaUsers contains the key ca.crt with a CA certificate that is different from the kafka cluster CA certificate, and doesn't work. But using the secret <kafka-name>-cluster-ca from the kafka's namespace works.

Steps to reproduce

1. create a kafka cluster with userOperator enabled
2. create a tls user
3. mount the user's secret (generated by the userOperator) into a pod with kafkacat cli installed
4. use the user certificate and key, and the CA file to list topics

Expected behavior

topics are listed with no problem

Strimzi version

0.33.2

Kubernetes version

1.23

Installation method

Helm chart

Infrastructure

GKE

Configuration files and logs

using the CA certificate from the user's secret (generated by the userOperator) produces this error:

kafkacat -L -b "my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094" -X security.protocol=ssl -X ssl.ca.location=/tmp/ca.crt -X ssl.certificate.location=/tmp/user.crt -X ssl.key.location=/tmp/user.key

%3|1680094201.558|FAIL|rdkafka#producer-1| [thrd:ssl://my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094/boo]: ssl://my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 30ms in state CONNECT)
%3|1680094202.536|FAIL|rdkafka#producer-1| [thrd:ssl://my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094/boo]: ssl://my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 8ms in state CONNECT, 1 identical error(s) suppressed)
% ERROR: Failed to acquire metadata: Local: Broker transport failure
%3|1680094206.529|FAIL|rdkafka#producer-1| [thrd:ssl://my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094/boo]: ssl://my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094/bootstrap: No further error information available (after 0ms in state CONNECT)

but by using the CA certificate from the my-kafka-cluster-ca secret in the kafka namespace (where I deployed the kafka cluster which i called my-kafka) works:

kafkacat -L -b "my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094" -X security.protocol=ssl -X ssl.ca.location=/tmp/ca.cluster.crt -X ssl.certificate.location=/tmp/user.crt -X ssl.key.location=/tmp/user.key

Metadata for all topics (from broker -1: ssl://my-kafka-kafka-bootstrap.kafka.svc.cluster.local:9094/bootstrap):
 3 brokers:
  broker 0 at my-kafka-kafka-0.my-kafka-kafka-brokers.kafka.svc:9094 (controller)
  broker 1 at my-kafka-kafka-1.my-kafka-kafka-brokers.kafka.svc:9094
  broker 2 at my-kafka-kafka-2.my-kafka-kafka-brokers.kafka.svc:9094
 3 topics:
  topic "mytopic2" with 1 partitions:
    partition 0, leader 1, replicas: 1,2, isrs: 1,2
  topic "TASKS_PERSISTENCE" with 1 partitions:
    partition 0, leader 2, replicas: 2, isrs: 2
  topic "my-topic" with 1 partitions:
    partition 0, leader 2, replicas: 2,1, isrs: 2,1

Additional context

No response

[scholzj]

That is by design. It is not supposed to be the Cluster CA certificate. It is the Clients CA certificate => the public key of the CA used to sign the user certificate. Some clients might need it to be able to use the user certificate.

To get the Cluster CA secret, you should get it from the cluster CA secret => <cluster-name>-cluster-ca-cert.

[mdnfiras]

alright, thanks for the quick response!

a proposal then: wouldn't it be practical to add a key ca.cluster.crt to the generated user secret that contains the Kafka cluster's CA certificate?

I think this would be useful as:

• it would remove the burden of having to copy that secret (<cluster-name>-cluster-ca-cert) in other namespaces (where the users are) and make sure it is synchronised.
• it makes things more GitOps friendly: I wouldn't have to manually copy this secret again if I happen to deploy a new similar Kubernetes cluster for testing or any other purpose.

[scholzj]

I (personally) would prefer to not do this because it would introduce yet another dependency and yet another thing to watch. So it increases the complexity of the software.

I do not really understand why this matters for GitOps. Both secrets are generated by the operators and not declared. So you would still need to copy it.

To make it easier to access the cluster from other namespaces, one of the ideas is to use the service bindings and the Kafka Access operator which will be able to copy both secrets in a declarative way (although unfortunately, nobody had much time to progress with it lately).