v7 (#364)

* fix where providing a SERVICE_NAME for a container with multiple ports exposed would cause services to overwrite each other

* Synchornize etcd cluster in registrator on service registration

* note on docker hub tags

* link to boot2docker

* analytics

* Default to tcp for PortType if not provided

* Allow DEV_RUN_OPTS to be used when calling make dev

* Add new version checker

Checks for new versions with the "usage" service and automatically displays a
standard version message when the "--version" flag is passed.

* Adding retries to backend service in the startup

Signed-off-by: Marcelo Salazar R <[email protected]>

* Added retry parameters documentation

Signed-off-by: Marcelo Salazar R <[email protected]>

* Upgrade to alpine:3.2 and go 1.4

go 1.4 is now required (miekg/dns#197)

* Refactor bridge for better testability

bridge.New no longer attempts to ping an adapter, caller must now use the bridge Ping method.

A few simple tests have been added to the bridge pkg

* Removed unused attributes

* prebump

* Adding documentation link

* update wording for Weave product family

* updating documentation & CHANGELOG

* Fix specific port names not overriding port suffix

* Actually check metadata from port. Fix ENV variable order dependency

* Use exit status to determine if container was killed

Instead of using the "kill" and "stop" events, this uses the exit status to
check whether the container was terminated via a signal. This will be more
reliable since the "kill" event can also be sent for non-fatal signals such as
SIGHUP.

Fixes #248

* Fix releases link in README

* Align SPONSORS text

* Add more detailed usage regarding options placement

Go's "flag" module only parses options up until the first non-option argument,
so additional arguments are left unparsed in "flag.Args()". We only expect one
argument, but additional arguments were ignored, leading to some confusion
about options that were ignored.

This updates the Usage() message with the syntax showing the options before the
registry URI, as well as more detail if the registry argument is missing, or
additional arguments are found.

* Cleanup dangling services

When a service was previously registered into the service registry
and registrator exits without unregistering, registrator now queries
the backend to see which services were registered, and checks against
it's internal list to determine which should be unregistered.

* Support for Docker multi host networking

When using the Docker multi-host networking, IPAddress under NetworkSettings is set to an empty string and the container IP can be retrieved from NetworkSettings.Networks.

At this point it is assumed a single Docker network is associated with the container

* Update util.go

* Using NewVersionedClientFromEnv to create docker client

* Initial basic zookeeper backend for registrator

* Small docs refurbishment

* Note for ignoring individual service on container

* Add support for Consul unix sockets

* Change default port for etc2 backend to default 2379

* Update docs for etcd backend for default port

* *servicePort method
  add support hostip for overlay network

* gofmt bridge

* added Consul TCP Health Check

* removed sentence that was copied from HTTP health check

* Update CHANGELOG

* Add image size to readme.

Closes #290

* Release prep

* Add image size to docs

* bump

* Have the zookeeper backend use the host port for the service paths, allow publishing services if the base service path already exists, and allow publishing into the root of zookeeper. (#367)

CHANGELOG.md

@@ -10,13 +10,41 @@ All notable changes to this project will be documented in this file.
 
 ### Changed
 
+## [v7] - 2016-03-05
+### Fixed
+- Providing a SERVICE_NAME for a container with multiple ports exposed would cause services to overwrite each other
+- dd3ab2e Fix specific port names not overriding port suffix
+
+### Added
+- bridge.Ping - calls adapter.Ping
+- Consul TCP Health Check
+- Support for Consul unix sockets
+- Basic Zookeper backend
+- Support for Docker multi host networking
+- Default to tcp for PortType if not provided
+- Sync etcd cluster on service registration
+- Support hostip for overlay network
+- Cleanup dangling services
+- Startup backend service connection retry
+
+### Removed
+
+### Changed
+- Upgraded base image to alpine:3.2 and go 1.4
+- bridge.New returns an error instead of calling log.Fatal
+- bridge.New will not attempt to ping an adapter.
+- Specifying a SERVICE_NAME for containers exposing multiple ports will now result in a named service per port. #194
+- Etcd uses port 2379 instead of 4001 #340
+- Setup Docker client from environment
+- Use exit status to determine if container was killed
 
 ## [v6] - 2015-08-07
 ### Fixed
 - Support for etcd v0 and v2
 - Panic from invalid skydns2 URI.
 
 ### Added
+- Basic zookeeper adapter
 - Optional periodic resyncing of services from containers
 - More error logging for registries
 - Support for services on containers with `--net=host`
@@ -54,6 +82,7 @@ All notable changes to this project will be documented in this file.
 - Dropped Godeps for now
 
 
-[unreleased]: https://github.com/gliderlabs/registrator/compare/v6...HEAD
+[unreleased]: https://github.com/gliderlabs/registrator/compare/v7...HEAD
+[v7]: https://github.com/gliderlabs/registrator/compare/v6...v7
 [v6]: https://github.com/gliderlabs/registrator/compare/v5...v6
 [v5]: https://github.com/gliderlabs/registrator/compare/v0.4.0...v5

Dockerfile

@@ -1,4 +1,4 @@
-FROM gliderlabs/alpine:3.1
+FROM gliderlabs/alpine:3.2
 ENTRYPOINT ["/bin/registrator"]
 
 COPY . /go/src/github.com/gliderlabs/registrator

Dockerfile.dev

@@ -1,4 +1,4 @@
-FROM gliderlabs/alpine:3.1
+FROM gliderlabs/alpine:3.2
 CMD ["/bin/registrator"]
 
 ENV GOPATH /go

Makefile

@@ -1,11 +1,12 @@
 NAME=registrator
 VERSION=$(shell cat VERSION)
+DEV_RUN_OPTS ?= consul:
 
 dev:
 	docker build -f Dockerfile.dev -t $(NAME):dev .
 	docker run --rm \
 		-v /var/run/docker.sock:/tmp/docker.sock \
-		$(NAME):dev /bin/registrator consul:
+		$(NAME):dev /bin/registrator $(DEV_RUN_OPTS)
 
 build:
 	mkdir -p build
@@ -21,6 +22,7 @@ release:
 	glu hubtag gliderlabs/$(NAME) $(VERSION)
 
 docs:
+	boot2docker ssh "sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'" || true
 	docker run --rm -it -p 8000:8000 -v $(PWD):/work gliderlabs/pagebuilder mkdocs serve
 
 circleci:

README.md

@@ -4,6 +4,7 @@ Service registry bridge for Docker, sponsored by [Weave](http://weave.works).
 
 [![Circle CI](https://circleci.com/gh/gliderlabs/registrator.png?style=shield)](https://circleci.com/gh/gliderlabs/registrator)
 [![Docker Hub](https://img.shields.io/badge/docker-ready-blue.svg)](https://registry.hub.docker.com/u/gliderlabs/registrator/)
+[![ImageLayers Size](https://img.shields.io/imagelayers/image-size/gliderlabs/registrator/latest.svg)](https://imagelayers.io/?images=gliderlabs%2Fregistrator:latest)
 [![IRC Channel](https://img.shields.io/badge/irc-%23gliderlabs-blue.svg)](https://kiwiirc.com/client/irc.freenode.net/#gliderlabs)
 <br /><br />
 
@@ -13,12 +14,17 @@ supports pluggable service registries, which currently includes
 [Consul](http://www.consul.io/), [etcd](https://github.com/coreos/etcd) and
 [SkyDNS 2](https://github.com/skynetservices/skydns/).
 
+Full documentation available at http://gliderlabs.com/registrator
+
 ## Getting Registrator
 
 Get the latest release, master, or any version of Registrator via [Docker Hub](https://registry.hub.docker.com/u/gliderlabs/registrator/):
 
 	$ docker pull gliderlabs/registrator:latest
 
+Latest tag always points to the latest release. There is also a `:master` tag
+and version tags to pin to specific releases.
+
 ## Using Registrator
 
 The quickest way to see Registrator in action is our
@@ -42,12 +48,11 @@ discussing in [Slack](http://glider-slackin.herokuapp.com/).
 
 Also check out our Developer Guide on [Contributing
 Backends](https://gliderlabs.com/registrator/latest/dev/backends) and [Staging
-Releases](https://gliderlabs.com/registrator/latest/dev/releases.).
+Releases](https://gliderlabs.com/registrator/latest/dev/releases).
 
 ## Sponsors and Thanks
 
-Ongoing support of this project is made possible by [Weave](http://weave.works),
-the Docker SDN. Big thanks to Michael Crosby for
+Ongoing support of this project is made possible by [Weave](http://weave.works), the easiest way to connect, observe and control your containers. Big thanks to Michael Crosby for
 [skydock](https://github.com/crosbymichael/skydock) and the Consul mailing list
 for inspiration.
 

SPONSORS

@@ -1,2 +1,2 @@
-DigitalOcean 	http://digitalocean.com
+DigitalOcean  http://digitalocean.com
 Weaveworks    http://weave.works

VERSION

@@ -1 +1 @@
-v6
+v7

bridge/bridge.go

@@ -1,18 +1,22 @@
 package bridge
 
 import (
+	"errors"
 	"log"
 	"net"
 	"net/url"
 	"os"
 	"path"
+	"regexp"
 	"strconv"
 	"strings"
 	"sync"
 
 	dockerapi "github.com/fsouza/go-dockerclient"
 )
 
+var serviceIDPattern = regexp.MustCompile(`^(.+?):([a-zA-Z0-9][a-zA-Z0-9_.-]+):[0-9]+(?::udp)?$`)
+
 type Bridge struct {
 	sync.Mutex
 	registry       RegistryAdapter
@@ -22,28 +26,28 @@ type Bridge struct {
 	config         Config
 }
 
-func New(docker *dockerapi.Client, adapterUri string, config Config) *Bridge {
+func New(docker *dockerapi.Client, adapterUri string, config Config) (*Bridge, error) {
 	uri, err := url.Parse(adapterUri)
 	if err != nil {
-		log.Fatal("Bad adapter URI:", adapterUri)
+		return nil, errors.New("bad adapter uri: " + adapterUri)
 	}
 	factory, found := AdapterFactories.Lookup(uri.Scheme)
 	if !found {
-		log.Fatal("Unrecognized adapter:", adapterUri)
-	}
-	adapter := factory.New(uri)
-	err = adapter.Ping()
-	if err != nil {
-		log.Fatalf("%s: %s", uri.Scheme, err)
+		return nil, errors.New("unrecognized adapter: " + adapterUri)
 	}
+
 	log.Println("Using", uri.Scheme, "adapter:", uri)
 	return &Bridge{
 		docker:         docker,
 		config:         config,
-		registry:       adapter,
+		registry:       factory.New(uri),
 		services:       make(map[string][]*Service),
 		deadContainers: make(map[string]*DeadContainer),
-	}
+	}, nil
+}
+
+func (b *Bridge) Ping() error {
+	return b.registry.Ping()
 }
 
 func (b *Bridge) Add(containerId string) {
@@ -57,7 +61,7 @@ func (b *Bridge) Remove(containerId string) {
 }
 
 func (b *Bridge) RemoveOnExit(containerId string) {
-	b.remove(containerId, b.config.DeregisterCheck == "always" || b.didExitCleanly(containerId))
+	b.remove(containerId, b.shouldRemove(containerId))
 }
 
 func (b *Bridge) Refresh() {
@@ -97,8 +101,7 @@ func (b *Bridge) Sync(quiet bool) {
 
 	log.Printf("Syncing services on %d containers", len(containers))
 
-	// NOTE: This assumes reregistering will do the right thing, i.e. nothing.
-	// NOTE: This will NOT remove services.
+	// NOTE: This assumes reregistering will do the right thing, i.e. nothing..
 	for _, listing := range containers {
 		services := b.services[listing.ID]
 		if services == nil {
@@ -112,6 +115,47 @@ func (b *Bridge) Sync(quiet bool) {
 			}
 		}
 	}
+
+	// Clean up services that were registered previously, but aren't
+	// acknowledged within registrator
+	if b.config.Cleanup {
+		log.Println("Cleaning up dangling services")
+
+		extServices, err := b.registry.Services()
+		if err != nil {
+			log.Println("cleanup failed:", err)
+			return
+		}
+
+	Outer:
+		for _, extService := range extServices {
+			matches := serviceIDPattern.FindStringSubmatch(extService.ID)
+			if len(matches) != 3 {
+				// There's no way this was registered by us, so leave it
+				continue
+			}
+			serviceHostname := matches[1]
+			if serviceHostname != Hostname {
+				// ignore because registered on a different host
+				continue
+			}
+			serviceContainerName := matches[2]
+			for _, listing := range b.services {
+				for _, service := range listing {
+					if service.Name == extService.Name && serviceContainerName == service.Origin.container.Name[1:] {
+						continue Outer
+					}
+				}
+			}
+			log.Println("dangling:", extService.ID)
+			err := b.registry.Deregister(extService)
+			if err != nil {
+				log.Println("deregister failed:", extService.ID, err)
+				continue
+			}
+			log.Println(extService.ID, "removed")
+		}
+	}
 }
 
 func (b *Bridge) add(containerId string, quiet bool) {
@@ -176,28 +220,24 @@ func (b *Bridge) add(containerId string, quiet bool) {
 func (b *Bridge) newService(port ServicePort, isgroup bool) *Service {
 	container := port.container
 	defaultName := strings.Split(path.Base(container.Config.Image), ":")[0]
-	if isgroup {
-		defaultName = defaultName + "-" + port.ExposedPort
-	}
 
 	// not sure about this logic. kind of want to remove it.
-	hostname, err := os.Hostname()
-	if err != nil {
+	hostname := Hostname
+	if hostname == "" {
 		hostname = port.HostIP
-	} else {
-		if port.HostIP == "0.0.0.0" {
-			ip, err := net.ResolveIPAddr("ip", hostname)
-			if err == nil {
-				port.HostIP = ip.String()
-			}
+	}
+	if port.HostIP == "0.0.0.0" {
+		ip, err := net.ResolveIPAddr("ip", hostname)
+		if err == nil {
+			port.HostIP = ip.String()
 		}
 	}
 
 	if b.config.HostIp != "" {
 		port.HostIP = b.config.HostIp
 	}
 
-	metadata := serviceMetaData(container.Config, port.ExposedPort)
+	metadata, metadataFromPort := serviceMetaData(container.Config, port.ExposedPort)
 
 	ignore := mapDefault(metadata, "ignore", "")
 	if ignore != "" {
@@ -208,6 +248,9 @@ func (b *Bridge) newService(port ServicePort, isgroup bool) *Service {
 	service.Origin = port
 	service.ID = hostname + ":" + container.Name[1:] + ":" + port.ExposedPort
 	service.Name = mapDefault(metadata, "name", defaultName)
+	if isgroup && !metadataFromPort["name"] {
+		service.Name += "-" + port.ExposedPort
+	}
 	var p int
 	if b.config.Internal == true {
 		service.IP = port.ExposedIP
@@ -268,17 +311,41 @@ func (b *Bridge) remove(containerId string, deregister bool) {
 	delete(b.services, containerId)
 }
 
-func (b *Bridge) didExitCleanly(containerId string) bool {
+// bit set on ExitCode if it represents an exit via a signal
+var dockerSignaledBit = 128
+
+func (b *Bridge) shouldRemove(containerId string) bool {
+	if b.config.DeregisterCheck == "always" {
+		return true
+	}
 	container, err := b.docker.InspectContainer(containerId)
 	if _, ok := err.(*dockerapi.NoSuchContainer); ok {
 		// the container has already been removed from Docker
 		// e.g. probabably run with "--rm" to remove immediately
 		// so its exit code is not accessible
 		log.Printf("registrator: container %v was removed, could not fetch exit code", containerId[:12])
 		return true
-	} else if err != nil {
+	}
+
+	switch {
+	case err != nil:
 		log.Printf("registrator: error fetching status for container %v on \"die\" event: %v\n", containerId[:12], err)
 		return false
+	case container.State.Running:
+		log.Printf("registrator: not removing container %v, still running", containerId[:12])
+		return false
+	case container.State.ExitCode == 0:
+		return true
+	case container.State.ExitCode&dockerSignaledBit == dockerSignaledBit:
+		return true
 	}
-	return !container.State.Running && container.State.ExitCode == 0
+	return false
+}
+
+var Hostname string
+
+func init() {
+	// It's ok for Hostname to ultimately be an empty string
+	// An empty string will fall back to trying to make a best guess
+	Hostname, _ = os.Hostname()
 }

bridge/bridge_test.go

@@ -0,0 +1,23 @@
+package bridge
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestNewError(t *testing.T) {
+	bridge, err := New(nil, "", Config{})
+	assert.Nil(t, bridge)
+	assert.Error(t, err)
+}
+
+func TestNewValid(t *testing.T) {
+	Register(new(fakeFactory), "fake")
+	// Note: the following is valid for New() since it does not
+	// actually connect to docker.
+	bridge, err := New(nil, "fake://", Config{})
+
+	assert.NotNil(t, bridge)
+	assert.NoError(t, err)
+}

bridge/extpoints.go

@@ -139,4 +139,3 @@ func (ep *adapterFactoryExt) All() map[string]AdapterFactory {
 	}
 	return all
 }
-