Can't access external tls kafka listener using ingress rules

[anamaymane]

I'm facing some issues using ingress with external tls listners. I'm using a single load balancer for my cluster, and playing on the dns names to redirect traffic to the appropriate internal services.

Here is my KafkaCluster configuration:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    version: 3.2.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
      - name: external
        port: 9094
        type: ingress
        tls: true
        configuration:
          class: nginx
          bootstrap:
            host: kafkabootstrap.test.intra
          brokers:
          - broker: 0
            host: kafkabroker0.test.intra
          brokerCertChainAndKey:
            secretName: kafka-external-listner-secret
            certificate: tls.crt
            key: tls.key
    storage:
      type: ephemeral
    config:
      auto.create.topics.enable: "true"
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral

After creating the cluster, I can see that ingress are being created correctly:

arning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME                         CLASS    HOSTS                                                                                                                   ADDRESS       PORTS     AGE  
my-cluster-kafka-0           nginx    kafkabroker0.test.intra                                                                                10.100.0.98   80, 443   31m
my-cluster-kafka-bootstrap   nginx    kafkabootstrap.test.intra                                                                              10.100.0.98   80, 443   31m

Same with the services:

NAME                                                   TYPE           CLUSTER-IP       EXTERNAL-IP                   PORT(S)                               AGE
my-cluster-kafka-0                                     ClusterIP      10.247.137.246   <none>                        9094/TCP                              42m
my-cluster-kafka-bootstrap                             ClusterIP      10.247.18.179    <none>                        9091/TCP,9092/TCP,9093/TCP            42m
my-cluster-kafka-brokers                               ClusterIP      None             <none>                        9090/TCP,9091/TCP,9092/TCP,9093/TCP   42m
my-cluster-kafka-external-bootstrap                    ClusterIP      10.247.162.14    <none>                        9094/TCP                              42m

When trying to test the access via the kafkacat tool, I got the following result:

kafkacat -X security.protocol=ssl -X ssl.ca.location=ca.crt -X ssl.endpoint.identification.algorithm=none -X enable.ssl.certificate.verification=false -L -b kafkabootstrap.test.intra

% ERROR: Failed to acquire metadata: Local: Broker transport failure

Noting that when I use the external service name directly (without specifying the domain name), the connection is made without problems:

kafkacat -X security.protocol=ssl -X ssl.ca.location=ca.crt -X ssl.endpoint.identification.algorithm=none -X enable.ssl.certificate.verification=false -L -b my-cluster-kafka-external-bootstrap:9094

Metadata for all topics (from broker -1: ssl://my-cluster-kafka-external-bootstrap:9094/bootstrap):
 1 brokers:

[scholzj]

I don't know what the errors means as I don't use kafkacat. A good start is to use openssl s_client to check that you can reach the broker through the Ingress (you should basically see there the broker certificate and not the ingress certificate).

[anamaymane]

Thanks for the reply, here is the message I get using openssl s_client:

openssl s_client -connect kafkabootstrap.test.intra  -CAfile ca.crt
140527042250048:error:0200206E:system library:connect:Connection timed out:../crypto/bio/b_sock2.c:110:
140527042250048:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:111:
connect:errno=110

Noting that when executing a request with curl, I can see logs on my nginx controller:

172.16.5.1 - - [09/Jun/2022:12:57:41 +0000] "GET / HTTP/1.1" 308 164 "-" "curl/7.68.0" 106 0.000 [aymane-test-area-my-cluster-kafka-external-bootstrap-9094] [] - - - - edefabb8e82170aca46166965f71b5cf

[scholzj]

It is not HTTP, so curl shows a little info. In the OpenSSL comment, you need to pass the server name to use TLS/SNI.

[anamaymane]

yes I've forgot to put the https in front of the domain name, here is what I get now with openssl s_client:

CONNECTED(00000003)
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
   i:O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDcDCCAligAwIBAgIRAPLcK7mZ8sR7NJa38WCclV0wDQYJKoZIhvcNAQELBQAw
SzEQMA4GA1UEChMHQWNtZSBDbzE3MDUGA1UEAxMuS3ViZXJuZXRlcyBJbmdyZXNz
IENvbnRyb2xsZXIgRmFrZSBDZXJ0aWZpY2F0ZTAeFw0yMjA2MDkxMjQ4NDZaFw0y
MzA2MDkxMjQ4NDZaMEsxEDAOBgNVBAoTB0FjbWUgQ28xNzA1BgNVBAMTLkt1YmVy
bmV0ZXMgSW5ncmVzcyBDb250cm9sbGVyIEZha2UgQ2VydGlmaWNhdGUwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDwgtJbt364DyWY5HQXhXAAGA8GU7ah
GAfStobD9cLk0E6CGUWtcsR3RFoaBdBBbGNRkpEid3C32Hm1m1+I1jvXojGCO5yb
MDuxHRw0W+MLgWFg0f7c+a5L+CLFehU181pqPV+M2Cm82125cWLCzyB5EM6bDvj5
Tuf2rIVPgXnMTpKDyNO3ffsFvj33zziiQGRfgDi2RZT+YpeFUpys1sywEYBv8Cm0
dl4C5lcIvyugqUYhvOIugcrtchw/azhBnsYk8Az9qFuQAhFObvIS6Q2bRmUdA5lX
hNtezJ0W5jSRlyAywhuUnSS1SBCI3UPtZ9PXbinLCVwyaRo/AVPe68iXAgMBAAGj
TzBNMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEAAAKBggrBgEFBQcDATAMBgNVHRMB
Af8EAjAAMBgGA1UdEQQRMA+CDWluZ3Jlc3MubG9jYWwwDQYJKoZIhvcNAQELBQAD
ggEBANkvABLr6ByuhoY5IQKmfMfB5SjygFtDVCOdnTYpmop+Kpa4z+52Ao6f4H8+
svsQche6z/4dMpIlJC/KK3wRPV6yn36v3cfbXabkuw0UBFtVQ5LTlqMhIc/UuZNW
r1H3Tx1cRJvpn1WqiTs62ZC2lBWjHnfHGk/4GKsIwNpLs+qYU7gKqQEyh0xkRMMZ
ApE2nxEYBoc1ml/taAABoXU9Hb9LmmWKT0GIOt5GsINUr/jhIaM0djpMKjOVkAqQ
m747YUSPj1IN68Oo2vYbA4roQuFdzXpxGjlrY5oCUEZGaNc1wTtCogKIlAHTgEe2
m1eWxYjfTFwoO1/1aI2nd6uTgp0=
-----END CERTIFICATE-----
subject=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

issuer=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1444 bytes and written 414 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: B7B02BAB9B5C2A9B9D9E8BB8D969A06585E588E1CB2AD9454E9030455B93C637
    Session-ID-ctx: 
    Resumption PSK: BE09EB8BBEEE3976954B6B452682D9F95B5FF3472570A08D83B778E82C00591CD13854578152AEB3DA7DEAF2257CEF8F
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 69 10 1d 50 6a 8c 31 22-8b 1c 0a 44 a0 02 a1 e4   i..Pj.1"...D....
    0010 - aa 21 ee 5e ee 4d b0 bb-e5 d9 1d 05 d4 f6 98 9d   .!.^.M..........

    Start Time: 1654779797
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 0E55AE53E87EA13E20C66E78B598C4D737682B0544ED80B36CF96A39AAD09892
    Session-ID-ctx: 
    Resumption PSK: 99DF620FA80F85483C0DAF60B9CD3CAEC741CF89E74D9990F0D026993F7D5321B40353ADFC1F96C0B926D9A1C0921CC8
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 600 (seconds)
    TLS session ticket:
    0000 - 9f 5a de 9a d8 57 21 35-ff 54 42 a7 a9 f9 72 53   .Z...W!5.TB...rS
    0010 - 05 7b 69 b2 9c 1d 84 f5-26 6b 01 2e 8a a7 c1 e9   .{i.....&k......

    Start Time: 1654779797
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

Apparently the server is returning the kubernetes fake certificates.

[scholzj]

That suggests that you did not enabled TLS passthrough properly.

[anamaymane]

Thanks a lot @scholzj , that was exactly what was missing. The nginx controller wasn't enabling the SSL passthrough.

Here is the config that I added in my helm values file to enable it:

nginx-ingress-controller:
  extraArgs:
    enable-ssl-passthrough: true
  service:
    annotations:
      kubernetes.io/elb.id: "xxxxxxxxxxxxx.xxxxx.xxxx...."