Watcher closed exception Strimzi operator

[sgv0007]

Hi Team,

I recently upgraded to strimzi 0.35.0 version and kafka 3.4.0 . We have noticed below error in operator logs and Kafka and kafka connect is not getting reconciled .

2023-08-16T12:57:50.029611186Z 2023-08-16 12:57:50 ERROR AbstractOperator:492 - Watcher closed with exception in namespace eesb-sit-itaag911-kafka
2023-08-16T12:57:50.029654487Z io.fabric8.kubernetes.client.WatcherException: Could not process websocket message
2023-08-16T12:57:50.029661887Z at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-client-6.6.1.jar:?]
2023-08-16T12:57:50.029683187Z at io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl$ListenerAdapter.onError(JdkWebSocketImpl.java:85) ~[io.fabric8.kubernetes-httpclient-jdk-6.6.1.jar:?]
2023-08-16T12:57:50.029688787Z at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.processError(WebSocketImpl.java:508) ~[java.net.http:?]
2023-08-16T12:57:50.029694187Z at jdk.internal.net.http.websocket.WebSocketImpl$ReceiveTask.run(WebSocketImpl.java:462) ~[java.net.http:?]
2023-08-16T12:57:50.029699787Z at jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:149) ~[java.net.http:?]
2023-08-16T12:57:50.029705387Z at jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:230) ~[java.net.http:?]
2023-08-16T12:57:50.029710587Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
2023-08-16T12:57:50.029715787Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
2023-08-16T12:57:50.029721487Z at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-08-16T12:57:50.029726787Z Caused by: java.net.SocketException: Connection reset

please help.

Thanks,
Shiva

[scholzj]

That suggests some issues in communicating with your Kubernetes API server. But it is not really possibly to give more details based on this.

[sgv0007]

Thanks jakub.

Can you let me know if you want me to run any command and share more details please?

Thanks,
Shiva

[scholzj]

I don't know I'm afraid. You can try to share the full log, you can try to enable DEBUG logging, you should check your Kubernetes logs ... maybe one of them will show something useful.

[sgv0007]

Hi jakub,

Debug logging showed additional logs where I see there was a attempt to Force closing the watch and then it fails.

2023-08-16 13:56:08 DEBUG Reflector:206 - Starting watcher for kafka.strimzi.io/v1beta2/namespaces/eesb-sit-kafka-connect-source/kafkas at v23005806
2023-08-16 13:56:08 DEBUG AbstractWatchManager:283 - Watching https://10.0.0.1:443/apis/kafka.strimzi.io/v1beta2/namespaces/eesb-sit-kafka-connect-source/kafkas?resourceVersion=23005806&timeoutSeconds=600&allowWatchBookmarks=true&watch=true...
2023-08-16 13:56:08 DEBUG WatcherWebSocketListener:41 - WebSocket successfully opened
2023-08-16 13:56:08 DEBUG Reflector:129 - Watch started for kafka.strimzi.io/v1beta2/namespaces/eesb-sit-kafka-connect-source/kafkas
2023-08-16 13:56:08 DEBUG WatchConnectionManager:60 - Closing websocket io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl@5b328cec
2023-08-16T13:56:08.176560911Z 2023-08-16 13:56:08 DEBUG WatchConnectionManager:63 - Websocket already closed io.fabric8.kubernetes.client.jdkhttp.JdkWebSocketImpl@5b328cec
2023-08-16T13:56:08.176592611Z 2023-08-16 13:56:08 DEBUG AbstractWatchManager:292 - Force closing the watch io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@8a8f558
2023-08-16T13:56:08.176637211Z 2023-08-16 13:56:08 DEBUG AbstractWatchManager:169 - Ignoring duplicate firing of onClose event
2023-08-16 13:56:08 ERROR InformerUtils:29 - Caught exception in the Kafka informer which is started
2023-08-16T13:56:08.177317616Z io.fabric8.kubernetes.client.WatcherException: Could not process websocket message
2023-08-16T13:56:08.177324916Z at io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onError(WatcherWebSocketListener.java:51) ~[io.fabric8.kubernetes-

Thanks,
Shiva

[scholzj]

I'm not sure that really helps much. Either the WebSocket was closed from the other side or it send data that the client does not understand I guess.