Change default cluster DNS domain for services

[alexsteeel]

Hi!

When I create a simple ZooKeeper cluster I get an error because I have a different DNS domain set than the default cluster.

2023-09-22 07:31:30,965 [myid:2] - WARN  [WorkerSender[myid=2]:QuorumPeer$QuorumServer@199] - Failed to resolve address: zk-server-default-2.zk-server-default.hbase.svc.cluster.local
java.net.UnknownHostException: zk-server-default-2.zk-server-default.hbase.svc.cluster.local
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1386)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1307)
        at java.base/java.net.InetAddress.getByName(InetAddress.java:1257)
        at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:197)
        at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:746)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:715)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:676)
        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:495)
        at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:474)
        at java.base/java.lang.Thread.run(Thread.java:829)

I found row with default DNS name in source code of ZooKeeper operator here server_role_service_fqdn and here ZookeeperPodRef
So I can make new Docker image but maybe there is another option for change this parameter?

Thanks.
Best regards.

[lfrancke]

Thanks for raising this.
Unfortunately, it's a known...bug/missing feature.
We opened it here as a result of your post: stackabletech/issues#436

It's a valid use-case and I'm sorry it's failing now but we currently don't have the capacity to work on this short-term.

[alexsteeel]

Thank you for answer. I built custom images from source code with a changed prefix and everything worked. Unfortunately I don't know Rust to make a pull request.

[sbernauer]

Hi @alexsteeel would you be willing to share the custom dns prefix you use?

[alexsteeel]

Hi!
Any dns prefix that is not equal to cluster.local, such as cluster.test or my-cluster.local.

[sbernauer]

I was planning on creating a Draft PR with your custom prefix (all our PRs create and upload a 0.0.0-pr123 version for testing). This way you would have been unblocked until we fix the problem.
But according to https://github.com/orgs/stackabletech/discussions/35#discussioncomment-7081090 you managed to build the image on your own, very good! Is there anything we can improve to make building the images yourself easier? Did you use https://github.com/stackabletech/zookeeper-operator/blob/main/docker/Dockerfile?

[alexsteeel]

Yes, I used this Dockerfile and it was a quick workaround. When a new version appears, I will take part in testing.

[sbernauer]

Awesome, thanks! We will surely ping you once the new feature is available!