kafka-connect version 2.8.1 , with strimzi 0.27.1 is in NotReady State

[hari819]

Hello team,

all of a sudden i see the KC is in NotReady state , it was working as expected , now i see timeout in the operator (deployed in same namespace) and connectors are failing because of this , @scholzj

io.netty.channel.ConnectTimeoutException: connection timed out: kafka-connect-cluster-connect-api.demo-performance-1.svc.cluster.local/10.223.167.165:8083
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) ~[io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) ~[io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) [io.netty.netty-transport-4.1.71.Final.jar:4.1.71.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty.netty-common-4.1.71.Final.jar:4.1.71.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2022-08-18 12:17:10 DEBUG AbstractOperator:440 - Reconciliation #1(watch) KafkaConnect(demo-performance-1/kafka-connect-cluster): Lock lock::demo-performance-1::KafkaConnect::kafka-connect-cluster released```

both KC and operator are running in the same namespace , please could you help ,

[hari819]

describe command gives below info ,

Status:
  Conditions:
    Last Transition Time:  2022-08-18T12:12:42.007868Z
    Message:               connection timed out: kafka-connect-cluster-connect-api.demo-performance-1.svc.cluster.local/10.223.167.165:8083
    Reason:                ConnectTimeoutException
    Status:                True
    Type:                  NotReady
  Label Selector:          strimzi.io/cluster=kafka-connect-cluster,strimzi.io/name=kafka-connect-cluster-connect,strimzi.io/kind=KafkaConnect```

[hari819]

its only connecting from the KC pods , and not accessible from anywhere , tried curl -v telnet://service-ip:8083 ,
odd

[scholzj]

Did you tried from the Cluster Operator pod as well?

[hari819]

it times out from operator as well , works only from within the KC pods ,

[scholzj]

So I guess it is either some networking policies or the networking itself.

[hari819]

you know what , its very odd , it started working as expected when we did a OS update on the kubernetes node and rebooted it,
i saw some updates like kernel ,..etc were in the list of OS libraries that got updated. So finally its a OS issue ,

its not a kafka-connect issue , KC is working as expected, thanks to Strimzi and thankyou @scholzj