Problem in deploying cadence frontend, history, worker, matching in different machines using docker

[subhash-veluru]

I am trying to run cadence services on different machines and getting this problem in the frontend service, it seems like a communication issue between services, ringpop is working fine. Can someone help me figure out what config I am missing...

error

"msg":"Uncategorized error","service":"cadence-frontend","wf-domain-name":"samples-domain","wf-id":"df79b06e-16a6-4594-a02d-a4e9615f0b2c","wf-run-id":"181acebf-7ae3-4f72-9235-e32cde921865","error":"code:unavailable message:connection error: desc = "transport: Error while dialing dial tcp 172.17.0.2:7834: connect: connection refused"","logging-call-at":"workflowHandler.go:4130"

Posting the docker commands I am running in respective machines

host addresses of machines :
frontend : 10.114.4.106
matching: 10.114.23.38
worker : 10.114.10.148
history: 10.114.4.127

frontend:

sudo docker run -p 7933:7933 -p 7833:7833 -e CASSANDRA_SEEDS=10.114.3.232
-e CASSANDRA_USER=cassandra
-e CASSANDRA_PASSWORD=cassandra
-e KEYSPACE=cadence
-e VISIBILITY_KEYSPACE=cadence_visibility
-e CASSANDRA_PROTO_VERSION=4
-e LOG_LEVEL=debug,info
-e SERVICES=frontend
-e RINGPOP_SEEDS=10.114.4.106:7933,10.114.4.127:7934,10.114.23.38:7935,10.114.10.148:7939
-e FRONTEND_SERVICE=10.114.4.106
-e PRIMARY_FRONTEND_SERVICE=10.114.4.106
-e SECONDARY_FRONTEND_SERVICE=10.114.4.106
ubercadence/server:master

matching:

sudo docker run -p 7935:7935 -p 7835:7835 -e CASSANDRA_SEEDS=10.114.3.232
-e CASSANDRA_USER=cassandra
-e CASSANDRA_PASSWORD=cassandra
-e KEYSPACE=cadence
-e VISIBILITY_KEYSPACE=cadence_visibility
-e CASSANDRA_PROTO_VERSION=4
-e LOG_LEVEL=debug,info
-e SERVICES=matching
-e RINGPOP_SEEDS=10.114.4.106:7933,10.114.4.127:7934,10.114.23.38:7935,10.114.10.148:7939
-e FRONTEND_SERVICE=10.114.4.106
-e PRIMARY_FRONTEND_SERVICE=10.114.4.106
-e SECONDARY_FRONTEND_SERVICE=10.114.4.106
ubercadence/server:master

worker:

sudo docker run -p 7939:7939 -e CASSANDRA_SEEDS=10.114.3.232
-e CASSANDRA_USER=cassandra
-e CASSANDRA_PASSWORD=cassandra
-e KEYSPACE=cadence
-e VISIBILITY_KEYSPACE=cadence_visibility
-e CASSANDRA_PROTO_VERSION=4
-e LOG_LEVEL=debug,info
-e SERVICES=worker
-e RINGPOP_SEEDS=10.114.4.106:7933,10.114.4.127:7934,10.114.23.38:7935,10.114.10.148:7939
-e FRONTEND_SERVICE=10.114.4.106
-e PRIMARY_FRONTEND_SERVICE=10.114.4.106
-e SECONDARY_FRONTEND_SERVICE=10.114.4.106
ubercadence/server:master

history:

sudo docker run -p 7934:7934 -p 7834:7834 -e CASSANDRA_SEEDS=10.114.3.232
-e CASSANDRA_USER=cassandra
-e CASSANDRA_PASSWORD=cassandra
-e KEYSPACE=cadence
-e VISIBILITY_KEYSPACE=cadence_visibility
-e CASSANDRA_PROTO_VERSION=4
-e LOG_LEVEL=debug,info
-e SERVICES=history
-e RINGPOP_SEEDS=10.114.4.106:7933,10.114.4.127:7934,10.114.23.38:7935,10.114.10.148:7939
-e NUM_HISTORY_SHARDS=2
-e FRONTEND_SERVICE=10.114.4.106
-e PRIMARY_FRONTEND_SERVICE=10.114.4.106
-e SECONDARY_FRONTEND_SERVICE=10.114.4.106
ubercadence/server:master

[mantas-sidlauskas]

Self-identification part is reporting ip address from container environment. As your services are running in separate containers, they are announcing self-address 172.17.0.2 via ringpop.

You can use docker run --network host to resolve machine IP, and/or try to hint using docker run with -e BIND_ON_IP

[subhash-veluru]

Thank you worked