tls-sidecar ssl issues

[kvenu1458]

Hi Folks,

We are seeing below ssl erros so frequently on kafka tls-sidecar containers, sometimes tls-sidecar able to connect imeediately, but sometimes, it's wait for more than hours to establish connectivity

2022.06.28 04:40:41 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:38180
2022.06.28 04:40:41 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 04:40:41 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:46120
2022.06.28 04:40:41 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 04:40:41 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 05:23:30 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.28 05:23:30 LOG5[1:140292044564224]: Connection reset: 699821 byte(s) sent to SSL, 972472 byte(s) sent to socket
2022.06.28 05:23:31 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:46024
2022.06.28 05:23:31 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 05:23:31 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:53968
2022.06.28 05:23:32 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 05:23:32 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 06:55:30 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:34510
2022.06.28 06:55:30 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 06:55:30 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:42454
2022.06.28 06:55:30 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 06:55:30 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 06:55:31 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.28 06:55:31 LOG5[1:140292044564224]: Connection reset: 1226429 byte(s) sent to SSL, 1689796 byte(s) sent to socket

[scholzj]

You have 3 connections in over 2 hours. Not sure that counts as frequent. Brokern-pipe can mean anything from network issues to some ZooKeeper restarts. But that is not clear from only a single small log file snippet.

PS: The TLS sidecar is not used by the Kafka containers for a very long time. So you should probably upgrade to osme newer Strimzi version.

[kvenu1458]

Please find the below log, if you can find anything

[root@d2lg507612 ~]# kubectl logs -c tls-sidecar dev-demosight-kafka-7
Starting Stunnel with configuration:
pid = /usr/local/var/run/stunnel.pid
foreground = yes
debug = notice
sslVersion = TLSv1.2
[zookeeper-2181]
client = yes
CAfile = /tmp/cluster-ca.crt
cert = /etc/tls-sidecar/kafka-brokers/dev-demosight-kafka-7.crt
key = /etc/tls-sidecar/kafka-brokers/dev-demosight-kafka-7.key
accept = 127.0.0.1:2181
connect = dev-demosight-zookeeper-client:2181
delay = yes
verify = 2

2022.06.27 11:16:22 LOG5[1:140292044568640]: stunnel 4.56 on x86_64-redhat-linux-gnu platform
2022.06.27 11:16:22 LOG5[1:140292044568640]: Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013
2022.06.27 11:16:22 LOG5[1:140292044568640]: Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,FIPS Auth:LIBWRAP
2022.06.27 11:16:22 LOG5[1:140292044568640]: Reading configuration from file /tmp/stunnel.conf
2022.06.27 11:16:22 LOG5[1:140292044568640]: FIPS mode is enabled
2022.06.27 11:16:22 LOG4[1:140292044568640]: Insecure file permissions on /etc/tls-sidecar/kafka-brokers/dev-demosight-kafka-7.key
2022.06.27 11:16:22 LOG5[1:140292044568640]: Configuration successful
2022.06.27 11:16:27 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:45322
2022.06.27 11:16:27 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 11:16:27 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:53266
2022.06.27 11:16:27 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 11:16:27 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 11:20:58 LOG5[1:140292044564224]: Connection closed: 55391 byte(s) sent to SSL, 174217 byte(s) sent to socket
2022.06.27 11:21:00 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:46210
2022.06.27 11:21:00 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 11:21:00 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:54154
2022.06.27 11:21:00 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 11:21:00 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 15:01:27 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.27 15:01:27 LOG5[1:140292044564224]: Connection reset: 556063 byte(s) sent to SSL, 1432010 byte(s) sent to socket
2022.06.27 15:01:28 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:58034
2022.06.27 15:01:30 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 15:01:30 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37752
2022.06.27 15:01:31 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 15:01:31 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 15:01:35 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.27 15:01:35 LOG5[1:140292044564224]: Connection reset: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.27 15:01:36 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:58068
2022.06.27 15:01:36 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 15:01:36 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37776
2022.06.27 15:01:36 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 15:01:36 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 15:01:42 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.27 15:01:44 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:58096
2022.06.27 15:01:44 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 15:01:44 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37804
2022.06.27 15:01:44 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 15:01:44 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 15:01:44 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.27 15:01:44 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:58100
2022.06.27 15:01:44 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 15:01:44 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37808
2022.06.27 15:01:44 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 15:01:44 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 15:58:05 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:40234
2022.06.27 15:58:09 LOG5[1:140292044564224]: Connection closed: 222022 byte(s) sent to SSL, 359256 byte(s) sent to socket
2022.06.27 15:58:10 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 15:58:10 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:48208
2022.06.27 15:58:10 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 15:58:10 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 15:58:10 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.27 15:58:12 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:40276
2022.06.27 15:58:12 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 15:58:12 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:48216
2022.06.27 15:58:12 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 15:58:12 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 15:58:12 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.27 15:58:12 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:40280
2022.06.27 15:58:12 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 15:58:12 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:48220
2022.06.27 15:58:12 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 15:58:12 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 21:02:14 LOG5[1:140292044494592]: Read socket error: Broken pipe (32)
2022.06.27 21:02:14 LOG5[1:140292044494592]: Connection reset: 572780 byte(s) sent to SSL, 1433725 byte(s) sent to socket
2022.06.27 21:02:15 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:39012
2022.06.27 21:02:16 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 21:02:16 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:46958
2022.06.27 21:02:16 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 21:02:16 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 21:02:16 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.27 21:02:18 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:39032
2022.06.27 21:02:18 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 21:02:18 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:46972
2022.06.27 21:02:18 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 21:02:18 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 21:02:18 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.27 21:02:18 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:39036
2022.06.27 21:02:18 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 21:02:18 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:46976
2022.06.27 21:02:18 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 21:02:18 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 23:55:54 LOG5[1:140292044494592]: Read socket error: Broken pipe (32)
2022.06.27 23:55:54 LOG5[1:140292044494592]: Connection reset: 2661325 byte(s) sent to SSL, 3461621 byte(s) sent to socket
2022.06.27 23:55:56 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:42468
2022.06.27 23:55:56 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 23:55:56 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:50412
2022.06.27 23:55:56 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 23:55:56 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.27 23:55:56 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.27 23:55:58 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:42482
2022.06.27 23:55:58 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.27 23:55:58 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:50422
2022.06.27 23:55:58 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.27 23:55:58 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 02:11:32 LOG5[1:140292044494592]: Read socket error: Broken pipe (32)
2022.06.28 02:11:32 LOG5[1:140292044494592]: Connection reset: 1241047 byte(s) sent to SSL, 1903346 byte(s) sent to socket
2022.06.28 02:11:33 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:38892
2022.06.28 02:11:37 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 02:11:37 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:46850
2022.06.28 02:11:37 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 02:11:37 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:02:45 LOG5[1:140292044494592]: Read socket error: Broken pipe (32)
2022.06.28 03:02:45 LOG5[1:140292044494592]: Connection reset: 1171712 byte(s) sent to SSL, 1437870 byte(s) sent to socket
2022.06.28 03:02:46 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:48260
2022.06.28 03:02:47 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:02:47 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:56210
2022.06.28 03:02:47 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:02:47 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:02:48 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.28 03:02:49 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:48282
2022.06.28 03:02:57 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:48312
2022.06.28 03:02:57 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:02:57 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:56256
2022.06.28 03:02:57 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:02:57 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:02:57 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 03:02:57 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:48320
2022.06.28 03:02:57 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:02:57 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:56260
2022.06.28 03:02:57 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:02:57 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:02:59 LOG3[1:140292044494592]: connect_blocking: s_poll_wait 10.233.31.67:2181: TIMEOUTconnect exceeded
2022.06.28 03:02:59 LOG5[1:140292044494592]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.28 03:18:45 LOG5[1:140292044564224]: Connection closed: 93629 byte(s) sent to SSL, 99053 byte(s) sent to socket
2022.06.28 03:18:47 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:51274
2022.06.28 03:18:52 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:18:52 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:59250
2022.06.28 03:18:52 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:18:52 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:18:52 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 03:18:52 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:51314
2022.06.28 03:18:52 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:18:52 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:59254
2022.06.28 03:18:52 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:18:52 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:24:08 LOG5[1:140292044564224]: Connection closed: 62044 byte(s) sent to SSL, 41931 byte(s) sent to socket
2022.06.28 03:24:08 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:52348
2022.06.28 03:24:08 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:24:08 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:60292
2022.06.28 03:24:08 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:24:08 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:24:08 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.28 03:24:10 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:52358
2022.06.28 03:24:10 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:24:10 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:60298
2022.06.28 03:24:10 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:24:10 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 03:24:10 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 03:24:10 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:52362
2022.06.28 03:24:10 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 03:24:10 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:60302
2022.06.28 03:24:10 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 03:24:10 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 04:32:11 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:36454
2022.06.28 04:32:12 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.28 04:32:12 LOG5[1:140292044564224]: Connection reset: 127134 byte(s) sent to SSL, 316557 byte(s) sent to socket
2022.06.28 04:32:18 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:36506
2022.06.28 04:32:19 LOG3[1:140292044494592]: Error resolving 'dev-demosight-zookeeper-client': Neither nodename nor servname known (EAI_NONAME)
2022.06.28 04:32:19 LOG3[1:140292044494592]: No host resolved
2022.06.28 04:32:19 LOG5[1:140292044494592]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.28 04:32:19 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 04:32:19 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:44450
2022.06.28 04:32:19 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 04:32:19 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 04:32:19 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 04:32:19 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:36514
2022.06.28 04:32:19 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 04:32:19 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:44454
2022.06.28 04:32:19 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 04:32:19 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 04:32:57 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.28 04:32:57 LOG5[1:140292044564224]: Connection reset: 15735 byte(s) sent to SSL, 9875 byte(s) sent to socket
2022.06.28 04:32:58 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:36676
2022.06.28 04:33:00 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 04:33:00 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:44620
2022.06.28 04:33:00 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 04:33:00 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 04:40:39 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.28 04:40:39 LOG5[1:140292044564224]: Connection reset: 1549910 byte(s) sent to SSL, 1853257 byte(s) sent to socket
2022.06.28 04:40:41 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:38172
2022.06.28 04:40:41 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 04:40:41 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:46116
2022.06.28 04:40:41 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 04:40:41 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 04:40:41 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 04:40:41 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:38180
2022.06.28 04:40:41 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 04:40:41 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:46120
2022.06.28 04:40:41 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 04:40:41 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 05:23:30 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.28 05:23:30 LOG5[1:140292044564224]: Connection reset: 699821 byte(s) sent to SSL, 972472 byte(s) sent to socket
2022.06.28 05:23:31 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:46024
2022.06.28 05:23:31 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 05:23:31 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:53968
2022.06.28 05:23:32 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 05:23:32 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 06:55:30 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:34510
2022.06.28 06:55:30 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 06:55:30 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:42454
2022.06.28 06:55:30 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 06:55:30 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 06:55:31 LOG5[1:140292044564224]: Read socket error: Broken pipe (32)
2022.06.28 06:55:31 LOG5[1:140292044564224]: Connection reset: 1226429 byte(s) sent to SSL, 1689796 byte(s) sent to socket
2022.06.28 08:05:30 LOG5[1:140292044494592]: Connection closed: 824382 byte(s) sent to SSL, 1170698 byte(s) sent to socket
2022.06.28 08:05:31 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:47310
2022.06.28 08:05:34 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 08:05:34 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:55254
2022.06.28 08:05:34 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 08:05:34 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 08:05:34 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 08:05:35 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:47326
2022.06.28 08:05:35 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 08:05:35 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:55266
2022.06.28 08:05:35 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 08:05:35 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 08:12:12 LOG5[1:140292044494592]: Read socket error: Broken pipe (32)
2022.06.28 08:12:12 LOG5[1:140292044494592]: Connection reset: 45134 byte(s) sent to SSL, 55068 byte(s) sent to socket
2022.06.28 08:12:13 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:48572
2022.06.28 08:12:15 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 08:12:15 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:56526
2022.06.28 08:12:15 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 08:12:15 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 08:12:15 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.28 08:12:16 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:48598
2022.06.28 08:12:16 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 08:12:16 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:56538
2022.06.28 08:12:16 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 08:12:16 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 08:41:43 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:53974
2022.06.28 08:41:45 LOG5[1:140292044494592]: Connection closed: 115671 byte(s) sent to SSL, 158644 byte(s) sent to socket
2022.06.28 08:41:48 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 08:41:48 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:33714
2022.06.28 08:41:48 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 08:41:48 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 08:41:48 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 08:41:48 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:54014
2022.06.28 08:41:48 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 08:41:48 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:33722
2022.06.28 08:41:48 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 08:41:48 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 09:03:17 LOG5[1:140292044564224]: Connection closed: 41740 byte(s) sent to SSL, 108897 byte(s) sent to socket
2022.06.28 09:03:18 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:57936
2022.06.28 09:03:26 LOG5[1:140292044494592]: Service [zookeeper-2181] accepted connection from 127.0.0.1:57990
2022.06.28 09:03:28 LOG5[1:140292044494592]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 09:03:28 LOG5[1:140292044494592]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37712
2022.06.28 09:03:28 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 09:03:28 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37708
2022.06.28 09:03:29 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 09:03:29 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 09:03:33 LOG5[1:140292044400384]: Service [zookeeper-2181] accepted connection from 127.0.0.1:58020
2022.06.28 09:03:33 LOG5[1:140292044400384]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 09:03:33 LOG5[1:140292044400384]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37728
2022.06.28 09:03:33 LOG5[1:140292044400384]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 09:03:33 LOG5[1:140292044400384]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 09:03:33 LOG5[1:140292044400384]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.28 09:03:34 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 0 byte(s) sent to socket
2022.06.28 09:03:34 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:58030
2022.06.28 09:03:34 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 09:03:34 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37738
2022.06.28 09:03:34 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 09:03:34 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 09:03:34 LOG5[1:140292044564224]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
2022.06.28 09:03:34 LOG5[1:140292044564224]: Service [zookeeper-2181] accepted connection from 127.0.0.1:58034
2022.06.28 09:03:34 LOG5[1:140292044564224]: connect_blocking: connected 10.233.31.67:2181
2022.06.28 09:03:34 LOG5[1:140292044564224]: Service [zookeeper-2181] connected remote server from 10.233.111.27:37742
2022.06.28 09:03:34 LOG5[1:140292044564224]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 09:03:34 LOG5[1:140292044564224]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 09:03:35 LOG5[1:140292044494592]: Certificate accepted: depth=1, /O=io.strimzi/CN=cluster-ca v0
2022.06.28 09:03:35 LOG5[1:140292044494592]: Certificate accepted: depth=0, /O=io.strimzi/CN=dev-demosight-zookeeper
2022.06.28 09:03:35 LOG5[1:140292044494592]: Connection closed: 49 byte(s) sent to SSL, 41 byte(s) sent to socket
[root@d2lg507612 ~]# date
Tue Jun 28 09:32:56

[scholzj]

That is still only one log. You need to correlate it with your Kafka logs and ZooKeeper logs which are the applications connecting through the sidecar. And as I said, this could be related even to your networking ot other things in your environment. In any case - as I said - the simplest solution is to upgrade to a newer version which does not use the sidecars in Kafka anymore.

[kvenu1458]

Thanks