Bitbucket nodes can't join the cluster #718
Comments
|
I suggest to solve warning first. For example this warning (in your provided logging) seems solveable by using FQDN's and not using host names as member addresses: This error is interesting: And might be a bug, but seems solveable with |
|
Update: Yes, We solved the IP issue. Now each nodes use host IP. So We believe its not about network issue. Because it was tested with the host ip and hazelcast port in both containers and there is no any network problem. Also I deleted hazelcast parameters on docker commands so now both containers use the same bitbucket properties file in the shared directory. So it is not possible for any value not to be different. But still we are getting same authentication errors. We upgraded Bitbucket to the latest version because we thought the problem was related to the version, but the same error still persists. Error: |
Hello everyone.
I decided to create an issue here after I saw this issue.
I have a bitbucket application with 2 nodes. I m trying to install on docker(rootless) and rocky 8 OS.
I have two separate servers for both nodes. I installed docker rootless on these two servers. While installing Docker, I gave it a subnet so that the bridge network could use it. All my installations are done with automation, so the configurations in all environments are the same.
I run the same commands to start the Bitbucket container on these two servers (except for a few parameter values) and when both containers stand up, they have the same IP and port (inside the container). I added some container parameters to change this externally. But despite all the changes I have made, if I do not set authentication to false, there is a problem when the nodes connect to the cluster. Lastly, I added the public ip value I found above, but this time the cluster does not start at all and I cannot access the application. I'll put all my configs below.
Container1:
docker run -e RUN_USER=root -e RUN_GROUP=root -e RUN_UID=0 -e RUN_GID=0 --name="bitbucket-test" -v /path/bitbucket/ca/cacerts:/opt/java/openjdk/lib/security/cacerts:Z -v /path/bitbucket/localhome:/var/atlassian/application-data/bitbucket:Z --mount type=bind,source=/path/autofs/bitbucketshared,target=/var/atlassian/application-data/bitbucket/shared,bind-propagation=slave --mount type=bind,source=/path/autofs/bitbucketshared/bitbucket-lib/ojdbc10.jar,target=/var/atlassian/application-data/bitbucket/lib/ojdbc10.jar,bind-propagation=slave --memory=2g -e SEARCH_ENABLED=false -e SERVER_PROXY_NAME="bitbucketurl" -e SERVER_PROXY_PORT=443 -e SERVER_SCHEME=https -e SERVER_SECURE=true -e JVM_MINIMUM_MEMORY=1024m -e JVM_MAXIMUM_MEMORY=2048m -e HAZELCAST_PORT=5701 -e HAZELCAST_GROUP_NAME=clustername -e HAZELCAST_GROUP_PASSWORD=clusterpassword -e SEARCH_ENABLED=false -e JVM_SUPPORT_RECOMMENDED_ARGS="-Dcluster.node.name=bitbucket-test-node-server1-hostname -Dhttp.nonProxyHosts=*.atlassian.com" --hostname server1-hostname --add-host=server1-hostname:server1-ip -e TZ=Europe/Amsterdam -p 7990:7990 -p 7999:7999 -p server1-ip:5701:5701 -d atlassian/bitbucket:7.21.4
Container2:
docker run -e RUN_USER=root -e RUN_GROUP=root -e RUN_UID=0 -e RUN_GID=0 --name="bitbucket-test" -v /path/bitbucket/ca/cacerts:/opt/java/openjdk/lib/security/cacerts:Z -v /path/bitbucket/localhome:/var/atlassian/application-data/bitbucket:Z --mount type=bind,source=/path/autofs/bitbucketshared,target=/var/atlassian/application-data/bitbucket/shared,bind-propagation=slave --mount type=bind,source=/mypath/autofs/bitbucketshared/bitbucket-lib/ojdbc10.jar,target=/var/atlassian/application-data/bitbucket/lib/ojdbc10.jar,bind-propagation=slave --memory=2g -e SEARCH_ENABLED=false -e SERVER_PROXY_NAME="devstash.tst.vanderlande.com" -e SERVER_PROXY_PORT=443 -e SERVER_SCHEME=https -e SERVER_SECURE=true -e JVM_MINIMUM_MEMORY=1024m -e JVM_MAXIMUM_MEMORY=2048m -e HAZELCAST_PORT=5701 -e HAZELCAST_GROUP_NAME=clustername -e HAZELCAST_GROUP_PASSWORD=clusterpassword -e SEARCH_ENABLED=false -e JVM_SUPPORT_RECOMMENDED_ARGS="-Dcluster.node.name=bitbucket-test-node-server2-hostname -Dhttp.nonProxyHosts=*.atlassian.com" --hostname server2-hostname --add-host=server2-hostname:server2-ip -e TZ=Europe/Amsterdam -p 7990:7990 -p 7999:7999 -p server2-ip:5701:5701 -d atlassian/bitbucket:7.21.4
bitbucket.properties
server.port=7990
server.proxy-port=443
server.scheme=https
server.secure=true
server.proxy-name=bitbucketurl
jdbc.driver=oracle.jdbc.driver.OracleDriver
jdbc.url=jdbcurl
jdbc.user=jdbcuser
jdbc.password=jdbcpassword
db.pool.size.idle=5
db.pool.size.max=200
#Custom features enabled/disabled
feature.file.editor=false
feature.public.access=false
feature.auth.captcha=false
feature.pull.request.suggestions=false
syntax.highlighter.application/xml.extensions=vcproj,vcxproj,csproj,xaml
plugin.search.config.baseurl=http://opensearchurl:9200
plugin.search.config.username=username
plugin.search.config.password=password
hazelcast.port=5701
#hazelcast.network.multicast=true
hazelcast.network.tcpip=true
hazelcast.network.tcpip.members=server1-hostname,server2-hostname
hazelcast.node.authentication.enabled=false
hazelcast.group.name=clustername
hazelcast.group.password=clusterpassword
If I run it this way, the hazelcast cluster logs will look like this:
2024-01-23 16:20:07,102 INFO [hz.hazelcast.event-2] c.a.s.i.c.HazelcastClusterService Node 'server2-hostname/server2-ip:5701 (bitbucket-server2-hostname)' was ADDED to the cluster. Updated cluster:
[server1-hostname/192.168.254.2:5701 master this name='bitbucket-server1-hostname' uuid='e638ed59-589f-4f07-a924-c454460e185a' vm-id='0e944289-5b2e-425c-9089-eddd5437bd5d'],
[server2-hostname/server2-ip:5701 name='bitbucket-server2-hostname' uuid='d01be7b4-bca2-4b38-9b0c-43246f8dfc07' vm-id='4440d2d3-e201-4146-b505-e3cab356ad73']
While it gets the container IP for the master node, it gets the host IP for the added node.
and if I set hazelcast.node.authentication.enabled=true, I get the following error when adding the second node.
c.a.s.i.c.DefaultClusterJoinManager CONNECT(192.168.254.2:56219 -** server1-ip:5701): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
2024-01-23 12:52:45,242 WARN [hz.hazelcast.cached.thread-6] c.a.s.i.c.DefaultClusterJoinManager ACCEPT(192.168.254.2:5701 **- 192.168.254.1:59488): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
2024-01-23 12:52:45,243 WARN [hz.hazelcast.cached.thread-6] com.hazelcast.nio.tcp.TcpIpAcceptor [server2-hostname]:5701 [clustered] [3.12.12] com.atlassian.stash.internal.cluster.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
com.atlassian.stash.internal.cluster.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'hazelcast.group.name' and 'hazelcast.group.password' in bitbucket.properties.
For this problem, I saw the public ip parameter and added the following to JVM_SUPPORT_RECOMMENDED_ARGS separately for each of my commands.
-Dhazelcast.local.publicAddress=server1-hostname:5701
-Dhazelcast.local.publicAddress=server2-hostname:5701
If I give the ip here, I get this error
com.hazelcast.instance.AddressPicker [LOCAL] [clustered] [3.12.12] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2024-01-24 14:36:16,169 WARN [spring-startup] com.hazelcast.instance.AddressPicker [LOCAL] [clustered] [3.12.12] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
2024-01-24 14:41:17,230 ERROR [spring-startup] com.hazelcast.instance.Node [server1-ip]:5701 [clustered] [3.12.12] Could not join cluster. Shutting down now!
2024-01-24 14:41:17,233 WARN [spring-startup] com.hazelcast.instance.Node [server1-ip]:5701 [clustered] [3.12.12] Terminating forcefully...
2024-01-24 14:41:17,264 WARN [spring-startup] o.s.w.c.s.XmlWebApplicationContext Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Node failed to start!
2024-01-24 14:41:17,265 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket/shared
2024-01-24 14:41:17,285 INFO [spring-startup] c.a.s.internal.home.HomeLockAcquirer Releasing lock on /var/atlassian/application-data/bitbucket
2024-01-24 14:41:17,289 ERROR [spring-startup] c.a.j.s.w.s.JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': Unsatisfied dependency expressed through method 'setSessionFactory' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.resolveMethodArguments(AutowiredAnnotationBeanPostProcessor.java:767)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at javax.servlet.GenericServlet.init(GenericServlet.java:158)
at java.base/java.lang.Thread.run(Thread.java:829)
... 19 frames trimmed
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hazelcast' defined in class path resource [stash-context.xml]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Node failed to start!
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1804)
... 5 common frames omitted
Caused by: java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.HazelcastInstanceImpl.init(HazelcastInstanceImpl.java:138)
at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182)
at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57)
at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.newInstance(HazelcastFactoryBean.java:126)
at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:66)
at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:34)
But if I give the server name instead of the IP, the master is added but the other node is not added and an error occurs.
Even though I tried many ways to solve it, I could not get the result I wanted yet. Currently, nodes can be added to the cluster when authentication is false, but this does not provide a secure structure.
I hope we can find a solution. Thank you in advance.
The text was updated successfully, but these errors were encountered: