Strimzi kafka external access (Unexpected error and closing client connection)

[hmops]

Hi,
I am using Strimzi Kafka in the production environment. After the k8s cluster certificates were renewed, Kafka services are getting errors from external access via the node port.

2023-03-13 15:07:45,440 WARN [SocketServer listenerType=ZK_BROKER, nodeId=2] Unexpected error from /x.x.x.x; closing connection (org.apache.kafka.common.network.Selector) [data-plane-kafka-network-thread-2-ListenerName(EXTERNAL-9094)-PLAINTEXT-12]
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:105)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
	at kafka.network.Processor.poll(SocketServer.scala:989)
	at kafka.network.Processor.run(SocketServer.scala:892)
	at java.base/java.lang.Thread.run(Thread.java:829)

[scholzj]

This kind of error normally suggests a mix-up or TLS / non-TLS connections. In your case, it seems that the broker expects unencrypted plaintext connections -> so you might have a client connecting ith TLS encryption enabled.

[hmops]

Thanks for replying,

let me explain another issue, when I run the first below command for each Kafka broker, (second command)the status output of Kafka from the second one is different and show another privous exist and used port

#### first command 
kubectl exec -n kafka kafka-kafka-0 -c kafka -it -- cat /tmp/strimzi.properties | grep advertised

advertised.listeners=CONTROLPLANE-9090://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9090,REPLICATION-9091://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9091,PLAIN-9092://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9092,TLS-9093://kafka-kafka-2.kafka-kafka-brokers.kafka.svc:9093,EXTERNAL-9094://x.x.x.x:31795

advertised.listeners=CONTROLPLANE-9090://kafka-kafka-1.kafka-kafka-brokers.kafka.svc:9090,REPLICATION-9091://kafka-kafka-1.kafka-kafka-brokers.kafka.svc:9091,PLAIN-9092://kafka-kafka-1.kafka-kafka-brokers.kafka.svc:9092,TLS-9093://kafka-kafka-1.kafka-kafka-brokers.kafka.svc:9093,EXTERNAL-9094://x.x.x.x:31557

advertised.listeners=CONTROLPLANE-9090://kafka-kafka-0.kafka-kafka-brokers.kafka.svc:9090,REPLICATION-9091://kafka-kafka-0.kafka-kafka-brokers.kafka.svc:9091,PLAIN-9092://kafka-kafka-0.kafka-kafka-brokers.kafka.svc:9092,TLS-9093://kafka-kafka-0.kafka-kafka-brokers.kafka.svc:9093,EXTERNAL-9094://x.x.x.x:31791


kubectl  get kafka -n kafka -o yaml kafka
  - addresses:
    - host: x.x.x.x
      port: 30415
    - host: x.x.x.x
      port: 30415
    - host: x.x.x.x
      port: 30415
    bootstrapServers: x.x.x.x:30415,x.x.x.x:30415,x.x.x.x:30415
    type: external

As I see above there is a mix-up between the status of Kafka and strimzi advertisers configuration, by the way once I scaled down the Kafka and zookeeper services manually and after I scaled up to the desired state.

[scholzj]

I don't think this is necessarily wrong. The node port in the status is the bootstrap port. The ports in the Kafka configuration file are the per-broker ports. If you do kubectl get service, it should confirm that.