Skip to content
This repository has been archived by the owner on Jul 29, 2024. It is now read-only.

what causes the delay after start or scale? #3

Open
kklepper opened this issue Apr 14, 2016 · 2 comments
Open

what causes the delay after start or scale? #3

kklepper opened this issue Apr 14, 2016 · 2 comments

Comments

@kklepper
Copy link

Hi Ilkka!

I spent quite some time with your most valuable work in order to learn how things work together. When trying to build my own image I ran into troubles until I found your updated version here.

This is based on consul-template 0.13.0, the newest version is 0.14.0, so I was tempted to change the Dockerfile in this respect.

I have defined a docker-compose.yml file which makes it easy to switch between the versions and scale services up and down:

rest:
  image: kklepper/haproxy:0.14.0
#  image: kklepper/haproxy:0.13.0
#  image: sirile/haproxy
  dns: $DOCKER_BASE_IP
  container_name: lb
  links:
    - consul
  ports:
    - "80:80"
    - "1936:1936"
  environment:
    SERVICE_NAME: rest

At first, I thought the new version 0.14.0 had a problem, so I had a look at the version history https://github.com/hashicorp/consul-template/blob/master/CHANGELOG.md and didn't find anything.

But then I found that I just wasn't patient enough. Both versions behave exactly the same.

$ curl localhost/hello/v1
curl: (56) Recv failure: Connection reset by peer
$ curl localhost/hello/v1
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
$ curl localhost/hello/v1
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
$ curl localhost/hello/v1
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
$ curl localhost/hello/v1
{"hostname":"85fd70452e8c","time":"2016-04-14T13:28:44Z","language":"Scala"}

Obviously, consul needs quite some time to pick up changes (around 10-15 s). Registrator should communicate changes immediately, if I understand this construction correctly, but consul obviously needs time to restart HAProxy or maybe HAProxy needs time to restart.

Do you know more about the mechanisms or do you have ideas of how to find out which is responsible for the delay? Or am I making mistakes, not understanding what's happening here?

One more detail: I have defined a constant (in my case 172.17.0.1) as it didn't work with your dns instruction, and it also works with the address of the consul container (172.17.0.2 -- which isn't known beforehand for sure, though)

export DOCKER_BASE_IP=$(ifconfig docker0 | grep 'inet addr'| awk -F':' '{print $2}' | awk '{print $1}')

With these questions I only can experiment because I don't understand the underlying mechanism. In this respect, your example has helped me quite a lot so that I am confident to be able to transform the whole construction to other setups as well.

Thank you very much.
Karl
@kklepper
Copy link
Author

More data with scaling:

$ docker-compose scale hello=3
Creating and starting 2 ... done
Creating and starting 3 ... done
$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:25Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:26Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:27Z","language":"Scala"}$ curl localhost/hello/v1
................... all the same
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:44Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:45Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:45Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"fd20fa1dc827","time":"2016-04-14T14:47:45Z","language":"Scala"}$ curl localhost/hello/v1
................... here he picks up the second after 20 seconds
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:45Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"fd20fa1dc827","time":"2016-04-14T14:47:46Z","language":"Scala"}$ curl localhost/hello/v1
................... here he picks up the third after 21 seconds
{"hostname":"8e34267ae74b","time":"2016-04-14T14:47:46Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:46Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"fd20fa1dc827","time":"2016-04-14T14:47:47Z","language":"Scala"}$ curl localhost/hello/v1
................... and now it is roundrobin
{"hostname":"8e34267ae74b","time":"2016-04-14T14:47:47Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:48Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"fd20fa1dc827","time":"2016-04-14T14:47:48Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"8e34267ae74b","time":"2016-04-14T14:47:49Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:49Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"fd20fa1dc827","time":"2016-04-14T14:47:49Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"8e34267ae74b","time":"2016-04-14T14:47:50Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"9e9d740ae616","time":"2016-04-14T14:47:50Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"fd20fa1dc827","time":"2016-04-14T14:47:50Z","language":"Scala"}$ curl localhost/hello/v1
{"hostname":"8e34267ae74b","time":"2016-04-14T14:47:51Z","language":"Scala"}

@kklepper
Copy link
Author

In the meantime, I have replaced the simple hello application with my very complex full-fledged setup, comprised of web application plus data, memcached, database and search engine containers, and scaling the web application seems to be immediate. What a surprise!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kklepper