Bitbucket nodes can't join the cluster #718
Description
Hello everyone.
I decided to create an issue here after I saw this issue.
I have a bitbucket application with 2 nodes. I m trying to install on docker(rootless) and rocky 8 OS.
I have two separate servers for both nodes. I installed docker rootless on these two servers. While installing Docker, I gave it a subnet so that the bridge network could use it. All my installations are done with automation, so the configurations in all environments are the same.
I run the same commands to start the Bitbucket container on these two servers (except for a few parameter values) and when both containers stand up, they have the same IP and port (inside the container). I added some container parameters to change this externally. But despite all the changes I have made, if I do not set authentication to false, there is a problem when the nodes connect to the cluster. Lastly, I added the public ip value I found above, but this time the cluster does not start at all and I cannot access the application. I'll put all my configs below.
Container1:
docker run -e RUN_USER=root -e RUN_GROUP=root -e RUN_UID=0 -e RUN_GID=0 --name="bitbucket-test" -v /path/bitbucket/ca/cacerts:/opt/java/openjdk/lib/security/cacerts:Z -v /path/bitbucket/localhome:/var/atlassian/application-data/bitbucket:Z --mount type=bind,source=/path/autofs/bitbucketshared,target=/var/atlassian/application-data/bitbucket/shared,bind-propagation=slave --mount type=bind,source=/path/autofs/bitbucketshared/bitbucket-lib/ojdbc10.jar,target=/var/atlassian/application-data/bitbucket/lib/ojdbc10.jar,bind-propagation=slave --memory=2g -e SEARCH_ENABLED=false -e SERVER_PROXY_NAME="bitbucketurl" -e SERVER_PROXY_PORT=443 -e SERVER_SCHEME=https -e SERVER_SECURE=true -e JVM_MINIMUM_MEMORY=1024m -e JVM_MAXIMUM_MEMORY=2048m -e HAZELCAST_PORT=5701 -e HAZELCAST_GROUP_NAME=clustername -e HAZELCAST_GROUP_PASSWORD=clusterpassword -e SEARCH_ENABLED=false -e JVM_SUPPORT_RECOMMENDED_ARGS="-Dcluster.node.name=bitbucket-test-node-server1-hostname -Dhttp.nonProxyHosts=*.atlassian.com" --hostname server1-hostname --add-host=server1-hostname:server1-ip -e TZ=Europe/Amsterdam -p 7990:7990 -p 7999:7999 -p server1-ip:5701:5701 -d atlassian/bitbucket:7.21.4
Container2:
docker run -e RUN_USER=root -e RUN_GROUP=root -e RUN_UID=0 -e RUN_GID=0 --name="bitbucket-test" -v /path/bitbucket/ca/cacerts:/opt/java/openjdk/lib/security/cacerts:Z -v /path/bitbucket/localhome:/var/atlassian/application-data/bitbucket:Z --mount type=bind,source=/path/autofs/bitbucketshared,target=/var/atlassian/application-data/bitbucket/shared,bind-propagation=slave --mount type=bind,source=/mypath/autofs/bitbucketshared/bitbucket-lib/ojdbc10.jar,target=/var/atlassian/application-data/bitbucket/lib/ojdbc10.jar,bind-propagation=slave --memory=2g -e SEARCH_ENABLED=false -e SERVER_PROXY_NAME="devstash.tst.vanderlande.com" -e SERVER_PROXY_PORT=443 -e SERVER_SCHEME=https -e SERVER_SECURE=true -e JVM_MINIMUM_MEMORY=1024m -e JVM_MAXIMUM_MEMORY=2048m -e HAZELCAST_PORT=5701 -e HAZELCAST_GROUP_NAME=clustername -e HAZELCAST_GROUP_PASSWORD=clusterpassword -e SEARCH_ENABLED=false -e JVM_SUPPORT_RECOMMENDED_ARGS="-Dcluster.node.name=bitbucket-test-node-server2-hostname -Dhttp.nonProxyHosts=*.atlassian.com" --hostname server2-hostname --add-host=server2-hostname:server2-ip -e TZ=Europe/Amsterdam -p 7990:7990 -p 7999:7999 -p server2-ip:5701:5701 -d atlassian/bitbucket:7.21.4
bitbucket.properties
server.port=7990
server.proxy-port=443
server.scheme=https
server.secure=true
server.proxy-name=bitbucketurl
jdbc.driver=oracle.jdbc.driver.OracleDriver
jdbc.url=jdbcurl
jdbc.user=jdbcuser
jdbc.password=jdbcpassword
db.pool.size.idle=5
db.pool.size.max=200
#Custom features enabled/disabled
feature.file.editor=false
feature.public.access=false
feature.auth.captcha=false
feature.pull.request.suggestions=false
syntax.highlighter.application/xml.extensions=vcproj,vcxproj,csproj,xaml
plugin.search.config.baseurl=http://opensearchurl:9200
plugin.search.config.username=username
plugin.search.config.password=password
hazelcast.port=5701
#hazelcast.network.multicast=true
hazelcast.network.tcpip=true
hazelcast.network.tcpip.members=server1-hostname,server2-hostname
hazelcast.node.authentication.enabled=false
hazelcast.group.name=clustername
hazelcast.group.password=clusterpassword
If I run it this way, the hazelcast cluster logs will look like this:
2024-01-23 16:20:07,102 INFO [hz.hazelcast.event-2] c.a.s.i.c.HazelcastClusterService Node 'server2-hostname/server2-ip:5701 (bitbucket-server2-hostname)' was ADDED to the cluster. Updated cluster:
[server1-hostname/192.168.254.2:5701 master this name='bitbucket-server1-hostname' uuid='e638ed59-589f-4f07-a924-c454460e185a' vm-id='0e944289-5b2e-425c-9089-eddd5437bd5d'],
[server2-hostname/server2-ip:5701 name='bitbucket-server2-hostname' uuid='d01be7b4-bca2-4b38-9b0c-43246f8dfc07' vm-id='4440d2d3-e201-4146-b505-e3cab356ad73']
While it gets the container IP for the master node, it gets the host IP for the added node.
and if I set hazelcast.node.authentication.enabled=true, I get the following error when adding the second node.
c.a.s.i.c.DefaultClusterJoinManager CONNECT(192.168.254.2:56219 -** server1-ip:5701): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
2024-01-23 12:52:45,242 WARN [hz.hazelcast.cached.thread-6] c.a.s.i.c.DefaultClusterJoinManager ACCEPT(192.168.254.2:5701 **- 192.168.254.1:59488): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
2024-01-23 12:52:45,243 WARN [hz.hazelcast.cached.thread-6] com.hazelcast.nio.tcp.TcpIpAcceptor [server2-hostname]:5701 [clustered] [3.12.12] com.atlassian.stash.internal.cluster.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
com.atlassian.stash.internal.cluster.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
For this problem, I saw the public ip parameter and added the following to JVM_SUPPORT_RECOMMENDED_ARGS separately for each of my commands.
-Dhazelcast.local.publicAddress=server1-hostname:5701
-Dhazelcast.local.publicAddress=server2-hostname:5701
If I give the ip here, I get this error
com.hazelcast.instance.AddressPicker [LOCAL] [clustered] [3.12.12] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2024-01-24 14:36:16,169 WARN [spring-startup] com.hazelcast.instance.AddressPicker [LOCAL] [clustered] [3.12.12] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2024-01-24 14:41:17,230 ERROR [spring-startup] com.hazelcast.instance.Node [server1-ip]:5701 [clustered] [3.12.12] Could not join cluster. Shutting down now!
2024-01-24 14:41:17,233 WARN [spring-startup] com.hazelcast.instance.Node [server1-ip]:5701 [clustered] [3.12.12] Terminating forcefully...
2024-01-24 14:41:17,264 WARN [spring-startup] o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Node failed to start!
2024-01-24 14:41:17,265 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket/shared
2024-01-24 14:41:17,285 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket
2024-01-24 14:41:17,289 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:767)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.base/java.lang.Thread.run(Thread.java:829)
... 19 frames trimmed
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1804)
... 5 common frames omitted
Caused by: java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.init(HazelcastInstanceImpl.java:138)
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.newInstance(HazelcastFactoryBean.java:126)
at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:66)
at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:34)
But if I give the server name instead of the IP, the master is added but the other node is not added and an error occurs.
Even though I tried many ways to solve it, I could not get the result I wanted yet. Currently, nodes can be added to the cluster when authentication is false, but this does not provide a secure structure.
I hope we can find a solution. Thank you in advance.