SSL_do_handshake error on nginx ingress #8121

ismailyenigul · 2023-02-17T15:14:38Z

ismailyenigul
Feb 17, 2023

Deployed strimzi kafka cluster on EKS running nginx ingress controller.
When I try to access broker url over ingress, I got bad gateway error.
I see the following logs

2023/02/15 19:00:20 [error] 175#175: *2299726  () failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:SSL alert number 42)
while SSL handshaking to upstream, client: 10.0.10.78, 
server: broker-2.mydomain.com, request: "GET /favicon.ico HTTP/1.1", 
upstream: "https://10.0.8.230:9094/favicon.ico", host: "broker-2.mydomain.com", referrer: "https://broker-2.mydomain.com/"

any idea about the issue and fix?
Thanks

Here is my config.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cluster
  namespace: kafka
spec:
  cruiseControl: {}
  kafka:
    version: 3.2.3
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9094
        type: ingress
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            host: kafka-bootstrap.mydomain.com
            annotations:
              external-dns.alpha.kubernetes.io/hostname: kafka-bootstrap.mydomain.com.
              external-dns.alpha.kubernetes.io/ttl: "600"
          brokers:
            - broker: 0
              host: broker-0.mydomain.com
              annotations:
                external-dns.alpha.kubernetes.io/hostname: broker-0.mydomain.com.
                external-dns.alpha.kubernetes.io/ttl: "600"
            - broker: 1
              host: broker-1.mydomain.com
              annotations:
                external-dns.alpha.kubernetes.io/hostname: broker-1.mydomain.com.
                external-dns.alpha.kubernetes.io/ttl: "600"
            - broker: 2
              host: broker-2.mydomain.com
              annotations:
                external-dns.alpha.kubernetes.io/hostname: broker-2.mydomain.com.
                external-dns.alpha.kubernetes.io/ttl: "600"
          class: ingress-internal
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.2"
    resources:
      requests:
        memory: 4Gi
        cpu: "1"
      limits:
        memory: 8Gi
        cpu: "2"
    jvmOptions:
      "-Xmx": "3g"
      "-Xms": "2g"
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 100Gi
          class: gp3
          deleteClaim: false
    template:
      pod:
        tolerations:
          - key: "service"
            operator: "Equal"
            value: "critical"
            effect: "NoSchedule"
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: service
                      operator: In
                      values:
                        - critical
    rack:
      topologyKey: topology.kubernetes.io/zone
  kafkaExporter:
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
            prometheus.io/scrape: "true"
  zookeeper:
    template:
      pod:
        metadata:
          annotations:
            prometheus.io/path: /metrics
            prometheus.io/port: "9404"
            prometheus.io/scrape: "true"
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      class: gp3
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
    ```

Answered by ismailyenigul

Feb 23, 2023

@scholzj finally it starting working. After working on 4 days...

controller:
    extraArgs: {"enable-ssl-passthrough": ""}
    containerPort:
      https: 443
    ingressClass: ingress-kafka
    ingressClassResource:
      name: ingress-kafka
      enabled: true
      default: false
      controllerValue: "k8s.io/ingress-kafka"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-internal: true
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "100"
        service.beta.kubernetes.io/aws…

View full answer

scholzj · 2023-02-17T15:16:32Z

scholzj
Feb 17, 2023
Maintainer

How exactly do you try to access broker url over ingress? I think you will need to share the command/client configuration.

3 replies

ismailyenigul Feb 17, 2023
Author

$ curl  https://broker-0.mydomain.com                                                                   1021ms
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

scholzj Feb 17, 2023
Maintainer

Well, Kafka is not an HTTP protocol. So you cannot access it with tools such as curl. You need to use a Kafka client to access it. Because of that, you also need to enable TLS passthrough in your Ingress controller (I think the error suggests it is not enabled, but it would fail anyway with curl).

ismailyenigul Feb 19, 2023
Author

Hi @scholzj

Enabled SSL

/nginx-ingress-controller --publish-service=$(POD_NAMESPACE)/eks-ingress-internal-ingress-nginx-controller --election-id=ingress-controller-leader --controller-class=k8s.io/ingress-internal-nginx --ingress-class=ingress-internal 
--configmap=$(POD_NAMESPACE)/eks-ingress-internal-ingress-nginx-controller --validating-webhook=:8443 
--validating-webhook-certificate=/usr/local/certificates/cert --validating-webhook-key=/usr/local/certificates/key 
--enable-ssl-passthrough

I0219 17:40:20.305602       7 nginx.go:754] "Starting TLS proxy for SSL Passthrough"
I0219 17:40:20.305671       7 nginx.go:303] "Starting NGINX process"

but still got

2023/02/19 17:42:27 [error] 33#33: *3229 SSL_do_handshake() failed (SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 
alert bad certificate:SSL alert number 42) while SSL handshaking to upstream, client: 10.0.2.176, 
server: broker-0.mydomain.com, request: "GET / HTTP/1.1", upstream: "https://10.0.12.233:9094/", host: "broker-0.mydomain.com"

ingress annotations:

  annotations:
    external-dns.alpha.kubernetes.io/hostname: broker-1.mydomain.com.
    external-dns.alpha.kubernetes.io/ttl: '600'
    field.cattle.io/publicEndpoints: >-
      [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"kafka:cluster-kafka-tls-1","ingressName":"kafka:cluster-kafka-tls-1","hostname":"broker-1.mydomain.com","path":"/","allNodes":false}]
    ingress.kubernetes.io/ssl-passthrough: 'true'
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
    
    
      spec:
  ingressClassName: ingress-internal
  tls:
    - hosts:
        - broker-1.mydomain.com.
  rules:
    - host: broker-1.dmydomain.com.
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: cluster-kafka-tls-1
                port:
                  number: 9094

    ```

ismailyenigul · 2023-02-19T18:57:20Z

ismailyenigul
Feb 19, 2023
Author

Here is the test result with kafka-console-producer on Mac


$ kubectl -n kafka get secret cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt 

$ keytool -import -trustcacerts -alias root -file kafka-ca-cert  -keystore truststore.jks -storepass password -noprompt


$ kafka-console-producer --broker-list kafka-bootstrap.mydomain.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic iy2


>[2023-02-19 21:49:58,941] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.mydomain.com/10.0.0.247:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-02-19 21:49:58,941] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-02-19 21:49:59,366] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.mydomain.com/10.0.2.193:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-02-19 21:49:59,367] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-02-19 21:50:00,106] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.mydomain.com/10.0.4.192:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-02-19 21:50:00,107] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-02-19 21:50:01,547] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.mydomain.com/10.0.0.247:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-02-19 21:50:01,548] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

1 reply

scholzj Feb 19, 2023
Maintainer

Well, yo have TLS client authentication enabled. So you would need to also create a user and configure a keystore in the user. Or you have to disable the authentication.

ismailyenigul · 2023-02-20T14:27:08Z

ismailyenigul
Feb 20, 2023
Author

can we disable authentication when we use ingress?
and how can I do that? To be honest, I lost myself in strimzi documentation :)
I am planing to write a end to end tutorial if I can solve my issue.

1 reply

scholzj Feb 20, 2023
Maintainer

Well, there is only one authentication section in your custom resource ... This configures the authentication:

        authentication:
          type: tls

So if you remove it, there will be none.

ismailyenigul · 2023-02-20T15:16:46Z

ismailyenigul
Feb 20, 2023
Author

I did and here is the latest configmap

Do I have to create custom SSL cert on kafka side? I am using default ones.


##########
# Listener configuration: PLAIN-9092
##########

##########
# Listener configuration: TLS-9094
##########
listener.name.tls-9094.ssl.keystore.location=/tmp/kafka/cluster.keystore.p12
listener.name.tls-9094.ssl.keystore.password=${CERTS_STORE_PASSWORD}
listener.name.tls-9094.ssl.keystore.type=PKCS12

tried the following commands all fails. maybe I do something wrong

$ kafka-console-producer --broker-list kafka-bootstrap.mydomain.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic iy2
>[2023-02-20 17:35:08,670] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.mydomain.com/10.0.2.193:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-02-20 17:35:08,671] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-02-20 17:35:09,156] ERROR [Producer clientId=console-producer] Connection to node -1 (kafka-bootstrap.mydomain.com/10.0.0.247:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2023-02-20 17:35:09,156] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)






$ kafka-console-producer --broker-list kafka-bootstrap.mydomain.com:443 --topic iy           
>[2023-02-20 17:35:56,138] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-02-20 17:35:59,354] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2023-02-20 17:36:02,616] WARN [Producer clientId=console-producer] Bootstrap broker kafka-bootstrap.mydomain.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)


curl  https://kafka-bootstrap.mydomain.com                                                         
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

I also see the folllowing logs in kafka pods

	2023-02-20 15:11:46,424 WARN [SocketServer listenerType=ZK_BROKER, nodeId=1] Unexpected error from /10.0.0.8 (channelId=10.0.13.176:9094-10.0.0.8:52204-0); closing connection (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(TLS-9094)-SSL-7]
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at kafka.network.Processor.poll(SocketServer.scala:1144)
	at kafka.network.Processor.run(SocketServer.scala:1047)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023-02-20 15:12:13,685 INFO [Producer clientId=CruiseControlMetricsReporter] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | CruiseControlMetricsReporter]
2023-02-20 15:12:50,314 WARN [SocketServer listenerType=ZK_BROKER, nodeId=1] Unexpected error from /10.0.0.8 (channelId=10.0.13.176:9094-10.0.0.8:56210-0); closing connection (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-1-ListenerName(TLS-9094)-SSL-8]
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at kafka.network.Processor.poll(SocketServer.scala:1144)
	at kafka.network.Processor.run(SocketServer.scala:1047)
	at java.base/java.lang.Thread.run(Thread.java:833)

10.0.0.8 is IP address of nginx ingress controller pod.

1 reply

scholzj Feb 20, 2023
Maintainer

TBH, this suggests the TLS passthrough is not working. It seems like your ingress forwards a plain connection to the broker from the error. You can also run the client with the Java system property -Sjavax.net.debug=all. That should give you more details of the TLS handshake.

ismailyenigul · 2023-02-20T19:26:02Z

ismailyenigul
Feb 20, 2023
Author


[2023-02-20 22:24:57,477] ERROR Error when sending message to topic iy2 with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205)
	at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:435)
	at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:523)
	at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:373)
	at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:293)
	at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
	at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
	at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
	at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
	at java.base/sun.security.validator.Validator.validate(Validator.java:256)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
	... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
	```

0 replies

ismailyenigul · 2023-02-23T14:59:07Z

ismailyenigul
Feb 23, 2023
Author

Hi @scholzj
I managed to send TLS traffic to nginx-ingress controller by adding
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
was the magic annotation.
it was service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp initially.
It is working fine for my test pod running HTTPS service.

still kafka connection over broker ingress hostname does not work.
did
$ kubectl -n kafka get secret devbox-cluster-ca-cert -o jsonpath='{.data.ca.crt}' | base64 -d > ca.crt

$ keytool -import -trustcacerts -alias root -file ca.crt
-keystore truststore.jks -storepass password -noprompt

$ kafka-console-producer --broker-list kafka-bootstrap.mydomain.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./truststore.jks --topic iy2

I would like to clarify something. you said that kafka is not HTTP protocol. But nginx ingress only works on HTTPS ports.
so how nginx-ingress connects to kafka without HTTP protocol?

I got the following snippet from nginx ingress controller's nginx.conf


            server {
                server_name kafka-bootstrap.mydomain.com ;

                listen 80  ;                        
                listen 442 proxy_protocol  ssl http2 ;

                set $proxy_upstream_name "-";

                ssl_certificate_by_lua_block {
                        certificate.call()
                }            

                location / {  

                        set $namespace      "kafka";
                        set $ingress_name   "devbox-kafka-tls-bootstrap";
                        set $service_name   "devbox-kafka-tls-bootstrap";
                        set $service_port   "9094";
                        set $location_path  "/";
                         proxy_next_upstream                     error timeout;           
                        proxy_next_upstream_timeout             0;    
                        proxy_next_upstream_tries               3;                  
                        proxy_pass https://upstream_balancer;                                                                                                                                                                            
                        proxy_redirect                          off;

As you see nginx terminates HTTPS traffic and starts a SSL proxy to open a new connection to backend in HTTPS

Also from https://kubernetes.github.io/ingress-nginx/user-guide/tls/#ssl-passthrough.

This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and 
handing it over to a local TCP proxy. 
This bypasses NGINX completely and introduces a non-negligible performance penalty.

it means openssl s_client will return SSL certificate from AWS not from kafka backend.

openssl s_client -connect kafka-bootstrap.mydomain.com:443 -servername kafka-bootstrap.mydomain.com
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
---
No client certificate CA names sent
Server Temp Key: ECDH, P-256, 256 bits

Do you think that will works fine? in this case nginx-ingress does not forward TLS request as it is. It terminates (so it send it's AWS certificate to the client instead of Kafka TLS)

I always get

[2023-02-23 17:41:43,494] ERROR [Producer clientId=console-producer] Connection to node -1 (
kafka-bootstrap.mydomain.com/10.0.0.167:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

$ kafka-console-producer --broker-list kafka-bootstrap.mydomain.com:443  --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./user-truststore.jks --topic iy4


javax.net.ssl|DEBUG|F1|kafka-producer-network-thread | console-producer|2023-02-23 17:57:02.385 TRT|SSLEngineInputRecord.java:213|READ: TLSv1.2 application_data, length = 304
javax.net.ssl|DEBUG|F1|kafka-producer-network-thread | console-producer|2023-02-23 17:57:02.390 TRT|SSLCipher.java:1709|Plaintext after DECRYPTION (
  0000: 48 54 54 50 2F 31 2E 31   20 34 30 30 20 42 61 64  HTTP/1.1 400 Bad
  0010: 20 52 65 71 75 65 73 74   0D 0A 44 61 74 65 3A 20   Request..Date: 
  0020: 54 68 75 2C 20 32 33 20   46 65 62 20 32 30 32 33  Thu, 23 Feb 2023
  0030: 20 31 34 3A 35 37 3A 30   32 20 47 4D 54 0D 0A 43   14:57:02 GMT..C
  0040: 6F 6E 74 65 6E 74 2D 54   79 70 65 3A 20 74 65 78  ontent-Type: tex
  0050: 74 2F 68 74 6D 6C 0D 0A   43 6F 6E 74 65 6E 74 2D  t/html..Content-
  0060: 4C 65 6E 67 74 68 3A 20   31 35 30 0D 0A 43 6F 6E  Length: 150..Con
  0070: 6E 65 63 74 69 6F 6E 3A   20 63 6C 6F 73 65 0D 0A  nection: close..
  0080: 0D 0A 3C 68 74 6D 6C 3E   0D 0A 3C 68 65 61 64 3E  ..<html>..<head>
  0090: 3C 74 69 74 6C 65 3E 34   30 30 20 42 61 64 20 52  <title>400 Bad R
  00A0: 65 71 75 65 73 74 3C 2F   74 69 74 6C 65 3E 3C 2F  equest</title></
  00B0: 68 65 61 64 3E 0D 0A 3C   62 6F 64 79 3E 0D 0A 3C  head>..<body>..<
  00C0: 63 65 6E 74 65 72 3E 3C   68 31 3E 34 30 30 20 42  center><h1>400 B
  00D0: 61 64 20 52 65 71 75 65   73 74 3C 2F 68 31 3E 3C  ad Request</h1><
  00E0: 2F 63 65 6E 74 65 72 3E   0D 0A 3C 68 72 3E 3C 63  /center>..<hr><c
  00F0: 65 6E 74 65 72 3E 6E 67   69 6E 78 3C 2F 63 65 6E  enter>nginx</cen
  0100: 74 65 72 3E 0D 0A 3C 2F   62 6F 64 79 3E 0D 0A 3C  ter>..</body>..<
  0110: 2F 68 74 6D 6C 3E 0D 0A                            /html>..
)
javax.net.ssl|DEBUG|F1|kafka-producer-network-thread | console-producer|2023-02-23 17:57:02.390 TRT|SSLEngineInputRecord.java:176|Raw read (
  0000: 15 03 03 00 1A 00 00 00   00 00 00 00 02 9D 92 78  ...............x
  0010: 39 BC 1D 61 FD 80 A3 6A   A5 AA 89 07 09 9D 7E     9..a...j.......
)
javax.net.ssl|DEBUG|F1|kafka-producer-network-thread | console-producer|2023-02-23 17:57:02.390 TRT|SSLEngineInputRecord.java:213|READ: TLSv1.2 alert, length = 26
javax.net.ssl|DEBUG|F1|kafka-producer-network-thread | console-producer|2023-02-23 17:57:02.391 TRT|SSLCipher.java:1709|Plaintext after DECRYPTION (
  0000: 01 00                                              ..
)
javax.net.ssl|DEBUG|F1|kafka-producer-network-thread | console-producer|2023-02-23 17:57:02.392 TRT|Alert.java:231|Received alert message (
"Alert": {
  "level"      : "warning",
  "description": "close_notify"
}
)
[2023-02-23 17:57:02,420] ERROR Uncaught exception in thread 'kafka-producer-network-thread | console-producer': (org.apache.kafka.common.utils.KafkaThread)
java.lang.OutOfMemoryError: Java heap space
	at java.base/java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:71)
	at java.base/java.nio.ByteBuffer.allocate(ByteBuffer.java:373)
	at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:102)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)

5 replies

scholzj Feb 23, 2023
Maintainer

TBH, that is one of the first things I said at the beginning => that you need to have TLS passthrough enabled. Without it, it will not work. If that does not work, the Kafka connections will not work.

ismailyenigul Feb 23, 2023
Author

ok but https://developers.redhat.com/blog/2019/06/12/accessing-apache-kafka-in-strimzi-part-5-ingress it says it is tested with nginx-ingress controller and I am testing it with nginx too :)
any idea? how did you test it?

ismailyenigul Feb 23, 2023
Author

TBH, that is one of the first things I said at the beginning => that you need to have TLS passthrough enabled. Without it, it will not work. If that does not work, the Kafka connections will not work.

and do you mean that I have to see kafka certificate on openssl s_client response?

scholzj Feb 23, 2023
Maintainer

ok but https://developers.redhat.com/blog/2019/06/12/accessing-apache-kafka-in-strimzi-part-5-ingress it says it is tested with nginx-ingress controller and I am testing it with nginx too :)
any idea? how did you test it?

Just to be clear, we are talking about the Ingress-NGINX Controller for Kubernetes - not the one from the Nginx company which AFAIU is different. So make sure you have the right one. But still - even with the right on - you need to have the TLS passthrough enabled. There is no way to work around it since Kafka does not speak HTTP protocol.

and do you mean that I have to see kafka certificate on openssl s_client response?

Yes, the OpenSSL command should return the broker certificate => that confirms that TLS passthrough works basically.

ismailyenigul Feb 23, 2023
Author

yes I am using nginx ingress from https://github.com/kubernetes/ingress-nginx and created a ticket about it at kubernetes/ingress-nginx#9658
but it seems I always get the certificate from AWS NLB. I am trying to configure AWS NLB without any default SSL certification.
I will keep you updated.

ismailyenigul · 2023-02-23T16:24:38Z

ismailyenigul
Feb 23, 2023
Author

@scholzj finally it starting working. After working on 4 days...

controller:
    extraArgs: {"enable-ssl-passthrough": ""}
    containerPort:
      https: 443
    ingressClass: ingress-kafka
    ingressClassResource:
      name: ingress-kafka
      enabled: true
      default: false
      controllerValue: "k8s.io/ingress-kafka"
    service:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-scheme: internal
        service.beta.kubernetes.io/aws-load-balancer-internal: true
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
        service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "100"
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

previously I had

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:xyz:certificate

I removed it and re-deployed nginx ingress
and it works!

$ kubectl -n kafka get secret mycluster-cluster-ca-cert  -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

$ keytool -keystore user-truststore.jks -alias CARoot -import -file ca.crt -storepass password -noprompt

$ kafka-console-consumer --bootstrap-server  kafka-bootstrap.mydomain.com:443   --consumer-property security.protocol=SSL --consumer-property ssl.truststore.password=password --consumer-property ssl.truststore.location=./user-truststore.jks --topic mytopic

$ kafka-console-producer --broker-list kafka-bootstrap.mydomain.com:443   --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=./user-truststore.jks --topic mytopic

2 replies

scholzj Feb 23, 2023
Maintainer

Great, glad it works now.

ismailyenigul Feb 23, 2023
Author

Thanks for your help @scholzj
I will write a document about it to make life easier for others. It took 5 days to understand all.

ismailyenigul · 2023-02-28T20:22:54Z

ismailyenigul
Feb 28, 2023
Author

As I promise!

https://ismailyenigul.medium.com/ingress-enabled-strimzi-kafka-deployment-on-aws-eks-40f451eecd3c

1 reply

scholzj Feb 28, 2023
Maintainer

That is great, thanks.

Strimzi

SSL_do_handshake error on nginx ingress #8121

Uh oh!

ismailyenigul Feb 17, 2023

Replies: 8 comments · 14 replies

Uh oh!

scholzj Feb 17, 2023 Maintainer

Uh oh!

ismailyenigul Feb 17, 2023 Author

Uh oh!

scholzj Feb 17, 2023 Maintainer

Uh oh!

ismailyenigul Feb 19, 2023 Author

Uh oh!

ismailyenigul Feb 19, 2023 Author

Uh oh!

scholzj Feb 19, 2023 Maintainer

Uh oh!

ismailyenigul Feb 20, 2023 Author

Uh oh!

scholzj Feb 20, 2023 Maintainer

Uh oh!

Uh oh!

ismailyenigul Feb 20, 2023 Author

Uh oh!

scholzj Feb 20, 2023 Maintainer

Uh oh!

ismailyenigul Feb 20, 2023 Author

Uh oh!

ismailyenigul Feb 23, 2023 Author

Uh oh!

scholzj Feb 23, 2023 Maintainer

Uh oh!

ismailyenigul Feb 23, 2023 Author

Uh oh!

ismailyenigul Feb 23, 2023 Author

Uh oh!

scholzj Feb 23, 2023 Maintainer

Uh oh!

ismailyenigul Feb 23, 2023 Author

Uh oh!

Uh oh!

ismailyenigul Feb 23, 2023 Author

Uh oh!

scholzj Feb 23, 2023 Maintainer

Uh oh!

ismailyenigul Feb 23, 2023 Author

Uh oh!

ismailyenigul Feb 28, 2023 Author

Uh oh!

scholzj Feb 28, 2023 Maintainer

ismailyenigul
Feb 17, 2023

Replies: 8 comments 14 replies

scholzj
Feb 17, 2023
Maintainer

ismailyenigul Feb 17, 2023
Author

scholzj Feb 17, 2023
Maintainer

ismailyenigul Feb 19, 2023
Author

ismailyenigul
Feb 19, 2023
Author

scholzj Feb 19, 2023
Maintainer

ismailyenigul
Feb 20, 2023
Author

scholzj Feb 20, 2023
Maintainer

ismailyenigul
Feb 20, 2023
Author

scholzj Feb 20, 2023
Maintainer

ismailyenigul
Feb 20, 2023
Author

ismailyenigul
Feb 23, 2023
Author

scholzj Feb 23, 2023
Maintainer

ismailyenigul Feb 23, 2023
Author

ismailyenigul Feb 23, 2023
Author

scholzj Feb 23, 2023
Maintainer

ismailyenigul Feb 23, 2023
Author

ismailyenigul
Feb 23, 2023
Author

scholzj Feb 23, 2023
Maintainer

ismailyenigul Feb 23, 2023
Author

ismailyenigul
Feb 28, 2023
Author

scholzj Feb 28, 2023
Maintainer