Getting java.net.SocketException: Connection reset errors in KafkaConnector for MSK

[LittleWat]

Hello! We use Amazon MSK for Kafka deployment and want to use Strimzi's KafkaConnect in EKS.
The connection between the MSK and Kafka-client in EKS uses mTLS handled by Istio.

The KafkaConnect Pod seems to be working 🤔
(Here is the Kafka connect pod log: https://gist.github.com/LittleWat/4d8f63e76f40dcccce83daf86a05132a)

I could produce messages inside the KafkaConnect Pod to the MSK using Kafka CLI.

But I am getting the error in KafkaConnector.

Here is the kubectl get -oyaml ... result.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: kafka-connect-sample-cluster
  namespace: kafka-connect-sample
spec:
  bootstrapServers: <MskBootstrapServers>
  config:
    config.storage.topic: kafka-connect-sample.configs
    group.id: kafka-connect-sample.sample-cluster
    key.converter: org.apache.kafka.connect.json.JsonConverter
    key.converter.schemas.enable: true
    offset.storage.topic: kafka-connect-sample.offsets
    status.storage.topic: kafka-connect-sample.status
    value.converter: org.apache.kafka.connect.json.JsonConverter
    value.converter.schemas.enable: true
  image: <MyImage> 
  replicas: 1
status:
  conditions:
  - lastTransitionTime: "2023-03-28T06:40:02.339687495Z"
    message: Connection reset
    reason: SocketException
    status: "True"
    type: NotReady
  labelSelector: strimzi.io/cluster=kafka-connect-sample-cluster,strimzi.io/name=kafka-connect-sample-cluster-connect,strimzi.io/kind=KafkaConnect
  observedGeneration: 12
  replicas: 1
  url: http://kafka-connect-sample-cluster-connect-api.kafka-connect-sample.svc:8083

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: kafka-connect-sample-foo
  namespace: kafka-connect-sample
  labels:
    strimzi.io/cluster: kafka-connect-sample-cluster
spec:
  class:  <foo>
  tasksMax: 1
  config:
    name:  <foo>
    topic:  <foo>
    ....
status:
  conditions:
  - lastTransitionTime: "2023-03-28T06:51:24.556911546Z"
    message: Connection reset
    reason: SocketException
    status: "True"
    type: NotReady
  observedGeneration: 9
  tasksMax: 1
  topics: []

Here is the strimzi operator's log.

2023-03-28 06:51:24 ERROR ConnectionBase: - Connection reset
2023-03-28 06:51:24 ERROR ConnectionBase: - Connection reset
2023-03-28 06:51:24 ERROR AbstractOperator:260 - Reconciliation #1(watch) KafkaConnect(kafka-connect-sample/kafka-connect-sample-cluster): createOrUpdate failed
java.net.SocketException: Connection reset
	at sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426) ~[?:?]
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:259) ~[io.netty.netty-buffer-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[io.netty.netty-buffer-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty.netty-transport-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.1.86.Final.jar:4.1.86.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

How should I debug this issue? Any ideas are really appreciated 🙏

[scholzj]

Sounds like your Connect does not work. You would need to share full logs I guess.

[scholzj]

And the operator log?

I guess he error suggests some networking issue between the operator and the Connect cluster as the COnnect cluster seems to work fine otherwise.

[LittleWat]

I restarted the operator, and the log is https://gist.github.com/LittleWat/4d8f63e76f40dcccce83daf86a05132a#file-strimzi-operator-log.
(it seems not to provide much information.)

Ah, I see. The operator is deployed in a different namespace, where istio-injection is not enabled.
(The istio-injection is enabled in kafka-connect-sample namespace)
This can be a cause. I will investigate this more. Thanks!

[LittleWat]

After enabling the istio-injection in the namespace where the strimzi-operator is deployed, it worked!!! I really appreciate your help!

[scholzj]

Strimzi in general does not support Istio. So you should be careful with that. (But if it works for you then I guess you are fine for the time being 🤷)

[LittleWat]

Sure. Yes, I am fine for the time being ^^;