connection timed out to Gotify from Watchtower in Docker

[oliver-cfc]

I don't really know what I'm doing, so bear that in mind

I'm running traefik, watchtower and gotify in docker containers. My gotify container is accessible at notify.mydomain.com

My watchtower logs show:

Failed to send Gotify notification:  
Post "https://notify.mydomain.com/message?token=12345678": 
dial tcp 192.168.55.198:443: connect: connection timed out

It's picking up the domain name and token from my watchtower docker-compose.yml (I wasn't sure whether it was needed so I've tried with the TLS_SKIP_VERIFY setting on and off and DOMAINNAME is picked up from a .env file)

    environment:
      WATCHTOWER_NOTIFICATIONS: gotify
      WATCHTOWER_NOTIFICATION_GOTIFY_URL: "https://notify.${DOMAINNAME}"
      WATCHTOWER_NOTIFICATION_GOTIFY_TOKEN: "12345678"
      WATCHTOWER_NOTIFICATION_GOTIFY_TLS_SKIP_VERIFY: "true"

What I assume is the problem is that traefik listens on 192.168.55.198:443 and reverse proxies the request to gotify so I'm wondering if watchtower is sending to the IP directly

If I run a test from the command line (same box, just not inside a docker container) all is good

$ http -f POST "https://notify.mydomain.com/message?token=12345678" 
title="my title" message="my message" priority="5"HTTP/1.1 200 OK
Content-Length: 118
Content-Type: application/json
Date: Sat, 24 Oct 2020 01:40:43 GMT

{
    "appid": 1,
    "date": "2020-10-23T21:40:43.773915744-04:00",
    "id": 6,
    "message": "my message",
    "priority": 5,
    "title": "my title"
}

And the gotify log

[GIN] 2020/10/23 - 21:40:43 | 200 |   17.602054ms |  192.168.55.198 | POST     "/message?token=12345678"

Am I on the right track? How can I get around it?

Thanks

[simskij]

Just to narrow it down further, would you mind launching a busybox or alpine container in your docker host and try to perform the same test there? It would allow us to know whether the error is inside the docker network or inside watchtower.

If they are expected to run on the same host indefinitely, I would skip the dns record altogether and just access it directly by container name after placing them on the same docker network.

Thanks! 🙇

[oliver-cfc]

Thanks for the advice!
I got the same error inside a different alpine container and switching to container name from DNS worked. I am expecting to run these on the same host/network indefinitely so this is a good fix for me.

[oliver-cfc]

So... I found out the root cause of the initial problem and I refer you to the opening line of the original post :)
My firewall on the docker host only allows tcp:443/80 from cloudflare IPs and the watchtower/my-test-container traffic to 443 was being rejected. As soon as I allowed the docker network through it all started working as expected.
Good to know I can always use containers names too though