Strimzi on GKE

[sreejesh-radhakrishnan-db]

Hi we have a hardened GKE (GKE Benchmark hardening guidelines, which recommend Issues a client certificate to authenticate be disabled. (https://cloud.google.com/kubernetes-engine/docs/how-to/hardening-your-cluster#restrict_authn_methods))

In out ORG, GKE cluster - "Issues a client certificate" to authenticate to the cluster endpoint. To maximize the security of your cluster, leave this option disabled. By any chance do you know if this will affect the working of KAFKA ?

[scholzj]

I think it talks about the Kubernetes API endpoints. So it might affect the operators but not the Kafka clients. It seems to me like they really remove some of the older stuff but keep things relatively standard. So I would not expect any issues. But I haven't tried it myself, so it is not on 100%.

[sreejesh-radhakrishnan-db]

@scholzj we have two cluster one with the above parameter TRUE and one with FALSE, the cluster with FALSE we are seeing ZOOKEEPER issue having following issue: Is this jsut co-incidence or its related to GKE being hardened ?

WARN Cannot open channel to 3 at election address uwq-cluster-zookeeper-2.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.41:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [QuorumConnectionThread-[myid=1]-2]

java.net.SocketTimeoutException: connect timed out at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

connect timed out at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.base/java.net.Socket.connect(Socket.java:609) at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

[scholzj]

I don't know based on a single exception without context. These exception happen normally while the ZooKeeper cluster is bootstrapping and the nodes are searching each other. It can also indicate some networking issues.