problem setting up multi host cluster #10
Comments
|
I need to dig into this blog post and your gist, but before I do:
|
|
Hey thanks for the quick reply! To your questions:
|
|
There is a bit of customization to the Can you try spinning up the nodes again, but before attempting to join them into a cluster, ensure that the values for |
|
Oh yeah, I just showed the unchanged content of Now I see the following. The container of the first riak node has the IP Having both containers natively linked using the docker link option, link the containers using that IP's. Using the ambassador pattern the linked IP's are not the IP's of the riak node containers itself. That is may an issue, but I still don't understand why it should be. May it is because of missing knowledge of riak internals. Anyway, the container of the first riak node ( Should |
|
Ok, I guess I understand what the problem is. Riak node two initially communicates with riak node one using the IP given by the env var Update: |
|
I spent some time reading through a few strategies for connecting containers across multiple Docker hosts this weekend. I want to mess with a few locally to see which work best with Riak. As soon as I have something working, I'll update this issue. Thus far, I'm not convinced that the ambassador pattern will work well because of the number of ports inter-Erlang node communication requires ( |
|
Ok cool. I am always happy to hear your ideas. A friend of mine talked about erlang based node communication today, and I agree that the ambassador pattern may not is a good choice here. Anyway, in some way the multi host support of connected docker containers goes further and I am glad to see what you come up with. Thanks! |
|
I just wanted to update this issue with some progress (of lack of progress). I created a Vagrant project to test strategies for connecting containers across Docker hosts: https://github.com/hectcastro/vagrant-multi-docker-riak Unfortunately, I'm stuck on an issue with an approach that leverages Open vSwitch to bridge the network between Docker hosts. Connectivity between containers across Docker hosts works, but something is preventing the Below are some details around the Docker host:
|
|
I have been trying to solve this same issue with no success using Docker < 0.11.1. However, I was able to easily stand up a cluster across different hosts in EC2 using Docker 0.11.1 [1]. Docker 0.11.1 introduced a new feature called "Host mode" where the container shares the host network interface directly, instead of using a bridge. Since the container is sharing the host interface, it is faster (reportedly) and the nodes are able to join the cluster. This comes at a cost though. For example I am unable to create a shell in the container sandbox, so running riak-admin is out. I enabled Riak Control in all my nodes and was thus able to do the necessary admin work to cluster and manage the nodes, but I would prefer to be able to get access to riak-admin directly. [1] http://blog.docker.io/2014/05/docker-0-11-release-candidate-for-1-0 -- john kline /cc @trotter |
|
@pakfur Yes, it's a workaround thanks for sharing. But you are stuck with one riak container per host, isn't it? Could be problematic for coreos managment (fleetctl). @hectcastro Do you have any update on this? |
|
Using the following two branches (one from this project and one from vagrant-multi-docker-riak), I was able to wire up containers across two Docker hosts via ambassador containers:
Hopefully the two The main reason for leveraging Fig is that it helps lay things out a bit more elegantly than a series of |
|
@hectcastro I am having the exact same issue as you mention in #10 (comment) with riak nodes on a libnetwork provisioned Calico network spanning multiple hosts, using Docker 1.10.3 so am curious if you ever worked out what was causing that RPC error? |
|
Unfortunately, I did not. It is possible that someone has a better answer on the riak-users mailing list. |
|
Do you know if using docker-swarm would solve this issue ? We would like to deploy a riak cluster with as many as 200's nodes and this clearly requires spanning multiple hosts. |
Hey there,
I am playing around with the docker-riak repo and have some problems. I hope it is ok to address my issue here, even if it is not the correct platform (maybe the riak/coreos mailing list would be more appropriate). If so, sorry!
Anyway, I am trying to figure out how to set up a docker-riak cluster across multiple hosts. The idea is the following. Having two coreos [1] machines running, the cluster should be created using one riak node on the first coreos machine, and one riak node on the second coreos machine. Linking should be enabled using the ambassador pattern [2]. I have a gist [3] describing how it should work. Everything comes up properly, but the riak nodes don't get linked to one cluster. It is possible to access the riak nodes from inside the containers, but for some reason the
riak-admin cluster join <node>does not do what it should.I don't understand the problem, because the riak nodes can access the correct IP's and be able to communicate. For me there is no obvious reason why that scenario fails. Maybe somebody could take a look to get that to work. Would be really cool to be able to do this cluster thing across multiple hosts.
[1] https://coreos.com/
[2] http://coreos.com/blog/docker-dynamic-ambassador-powered-by-etcd/
[3] https://gist.github.com/zyndiecate/74e8df820ccee60f67ae
All the best, Tim
The text was updated successfully, but these errors were encountered: