Getting Kafka TLS timeout error . #11709

L040639 · 2025-08-01T13:12:13Z

L040639
Aug 1, 2025

"ERROR: Client network socket disconnected before secure TLS connection was established" message sending failed

"stack": "Error: Client network socket disconnected before secure TLS connection was established\n TLSSocket.onConnectEnd (node:_tls_wrap:1732:19)\n
at LSSocket.emit (node:events:530:35)\n at endReadableNT (node: internal/streams/readable: 1698:12)\n
at process|1cksAndReject1ons
(node: internal/process/task_queues:90:21)"}

Getting above error couple of times while producing the message, after 2 retries then next we can able to send the message , for initial 2 tries it is getting above errors , not sure why . DO i need to increase any timeout configurations in my broker .So we are running in the production, so these retiries getting some delay in the message process .

# namespace name: kafka-streams-dev
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: <XXX>
spec:
  kafka:
    authorization:
      type: opa
      url: https://<XXX>/v1/data/kafka/allow
      allowOnError: false
      enableMetrics: true
      tlsTrustedCertificates:
        - secretName: lk-ca-root-bundle
          certificate: ca.crt
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.9'
      connections.max.idle.ms: 1800000
    storage:
      type: persistent-claim
      size: 500Gi
      class: gp3
    listeners:
      - authentication:
          checkAccessTokenType: false
          clientId: <XX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: '<XX>'
          maxSecondsWithoutReauthentication: 3600
          enableMetrics: true
          type: oauth
          validIssuerUri: '<XXX>'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-prd-cert-secret
        name: oauth
        port: 9093
        tls: true
        type: internal
      - authentication:
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: '<XXX>'
          maxSecondsWithoutReauthentication: 3600
          enableMetrics: true
          type: oauth
          validIssuerUri: '<XXX>'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-prd-cert-secret
          bootstrap:
            alternativeNames:
            - <XX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XX>.
              external-dns.alpha.kubernetes.io/access: "private"
          brokers:
          - broker: 0
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 1
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 2
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
        name: external
        port: 9094
        tls: true
        type: loadbalancer
    version: 3.9.0
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-streams-prd-metrics
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=5000
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=5000
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XX>   ##dedicated nodegroup for kafka
                      operator: Exists
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - kafka
                topologyKey: kubernetes.io/hostname
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-cluster
                strimzi.io/component-type: kafka
  entityOperator:
    topicOperator: {}
    userOperator: {}
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XXX>
                      operator: Exists
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
  zookeeper:
    storage:
      type: persistent-claim
      class: gp3
      size: 50Gi
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-streams-prd-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XX>  ##dedicated nodegroup for kafka
                      operator: Exists
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-cluster
                strimzi.io/component-type: zookeeper
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: node-role.kubernetes.io/worker
                      operator: Exists
        tolerations:
          - key: "<XXX>"
            operator: "Exists"
            effect: "NoSchedule"

ppatierno · 2025-08-01T13:19:58Z

ppatierno
Aug 1, 2025
Maintainer

Please don't tag specific people but just post the question and wait for help from anyone.
Said that, which client are you using? It looks to me that could be a client issue, it's disconnecting before the TLS handshake is done.
Maybe you can tune some timeout on the client side, but also checking what's it current default value.

7 replies

L040639 Aug 1, 2025
Author

as per the doc it is 1000 milli seconds

ppatierno Aug 1, 2025
Maintainer

we can continue back and forth this way without progress :-) but ... did you try to increase it to some higher value? It's even possible that a TLS handshake takes longer and KafkaJS is cutting connection too soon.

L040639 Aug 3, 2025
Author

Yeah i increase it to 3000 milli seconds, still getting the error is there any timeout increase settings i can set at the broker level ??

im-konge Aug 3, 2025
Maintainer

You should check what you can actually configure on Kafka and the client level and understand what is going on. Maybe 3000 milli seconds is not enough 🤷
Also I guess nobody from us know what does kafka.js actually do or what you can set there.
So you should search for the some discussions there.

L040639 Aug 4, 2025
Author

So if i am trying to connect my client to the kafka bootstrao kubernetes internal dns address , this is working fine with same client config with zero issues , but if i switched to the nlb bootstrap then after some messages there was a delay and failure again for few messages it went well again same issue only if i connect to the nlb bootstrap address , there is definitely taking a lot more time through nlb .

"stack": "Error: Client network socket disconnected before secure TLS connection was established\n TLSSocket.onConnectEnd (node:_tls_wrap:1732:19)\n
at LSSocket.emit (node:events:530:35)\n at endReadableNT (node: internal/streams/readable: 1698:12)\n
at process|1cksAndReject1ons
(node: internal/process/task_queues:90:21)"}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Getting Kafka TLS timeout error . #11709

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 7 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Strimzi

Getting Kafka TLS timeout error . #11709

Uh oh!

Uh oh!

L040639 Aug 1, 2025

Replies: 1 comment · 7 replies

Uh oh!

ppatierno Aug 1, 2025 Maintainer

Uh oh!

L040639 Aug 1, 2025 Author

Uh oh!

ppatierno Aug 1, 2025 Maintainer

Uh oh!

L040639 Aug 3, 2025 Author

Uh oh!

im-konge Aug 3, 2025 Maintainer

Uh oh!

Uh oh!

L040639 Aug 4, 2025 Author

L040639
Aug 1, 2025

Replies: 1 comment 7 replies

ppatierno
Aug 1, 2025
Maintainer

L040639 Aug 1, 2025
Author

ppatierno Aug 1, 2025
Maintainer

L040639 Aug 3, 2025
Author

im-konge Aug 3, 2025
Maintainer

L040639 Aug 4, 2025
Author