Zookeeper throws timeout after several restarts of nodes

[fiftode]

I have a problem with zookeeper deployed on kubernetes using Confluentinc Docker Images.

Cluster setup:

• 3 nodes in digital ocean ( 4 CPU, 8 GB or RAM) as kafka dedicated nodepool. ( Only zookeeper and kafka are running here, and some monitoring daemon set).

• If a zookeeper node/pod ( any node) restarts several times the zookeeper cluster is not accessible anymore, and to make it work I have to redeploy all the nodes.

• Once the node rejoined the cluster, all zookeeper nodes start to show this kind of error:
  Each node has one kafka pod and one zookeeper pod.

Errors one each node:

zookeeper-node-0 logs:
[2020-10-29 00:29:20,852] INFO Refusing session request for client /10.244.4.86:47054 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
FULL LOGS: node-0-log.log

zookeeper-node-1 logs:
244.3.185:36040 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer).
FULL LOGS: node-1-log.log

zookeeper-node-2 logs:
[2020-10-29 00:28:59,554] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
FULL LOGS: node-2-log.log

although the zookeeper cluster seems to be up, I'm not able to get anything from it. If I try to use cli to list a path I'm getting a timeout.
Any Ideas?

If this is not the place for this issue please guide me.
thanks

[OneCricketeer]

zookeeper deployed on kubernetes using Confluentinc Docker Images

You'd have better support moving your issue to https://github.com/confluentinc/cp-helm-charts