configuration of brokerCertChainAndKey: is ignored when listener type is intenal

[sreejesh-radhakrishnan-db]

We use ISTIO gateway for ingress : where we have a ingress service configured to route teh TLS/TCP connection to respective kafka service .

what my observation - is when kafka listener is type: internal the SSL handshake fails and when listener is type: loadbalancer the SSL pass and I am able to connect to kafka and do subsequent operation. has anyone seen this issue and if so how to get around this?

MY Setup

The below listener config fails and ssl trace shown below -

  - name: tls
    port: 9093
    type: internal
    tls: true
    configuration:
      useServiceDnsDomain: false
      brokerCertChainAndKey:
        secretName: kafka-listener 
        certificate: tls.crt 
        key: tls.key 

SSL trace :

javax.net.ssl|DEBUG|12|kafka-admin-client-thread | adminclient-1|2022-02-14 19:41:00.602 GMT|ClientHello.java:653|Produced ClientHello handshake message (
"ClientHello": {
}

javax.net.ssl|ALL|12|kafka-admin-client-thread | adminclient-1|2022-02-14 19:41:00.629 GMT|SSLEngineImpl.java:752|Closing outbound of SSLEngine
javax.net.ssl|WARNING|12|kafka-admin-client-thread | adminclient-1|2022-02-14 19:41:00.630 GMT|SSLEngineOutputRecord.java:168|outbound has closed, ignore outbound application data
2022-02-14 19:41:00.630 WARN 14460 --- [| adminclient-1] org.apache.kafka.clients.NetworkClient : [AdminClient clientId=adminclient-1] Connection to node -1 (/100.88.4.114:443) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

2. Setup which work with NO other change - other than changing the Type to loadbalancer it works.

The below listener config passes -

  - name: tls
    port: 9093
    type: loadbalancer
    tls: true
    configuration:
      useServiceDnsDomain: false
      brokerCertChainAndKey:
        secretName: kafka-listener 
        certificate: tls.crt 
        key: tls.key 

SSL handshake seems to be okay.

[scholzj]

I would assume that the loadbalancer listener bypasses Istio because it goes through the outside of your cluster. Where as the internal connection would be using Istio - and judging by the error, Istio might not like the Kafka TCP traffic.

[sreejesh-radhakrishnan-db]

Thanks for the reply, all theistio routing is happening internally to cluster . the connection is taken as TLS - PAssthrough by ISTIO gateway and directly routed to kafka listener service. My Q is more specific to these parameter - when we use type-internal will these setting be ignored or still used for cert tls authentication?

  brokerCertChainAndKey:
    secretName: kafka-listener 
    certificate: tls.crt 
    key: tls.key 

[scholzj]

I think this should be used fine with internal listeners. But just to be clear, this is a server certificate used for server authentication only. Not for client authentication / mTLS. In any case, the error you shared above suggests your connection did not go far enough to reach the broker.