Questions about healthcheck

[TChilderhose]

Hi there. Just started using your docker image and it's fantastic! It's saved me a few headaches setting up some new containers. Have a few questions about the healthcheck that I was hoping you could shed some light on

1. Was hoping for some clarification on HEALTH_OPENVPN_DURATION_INITIAL and HEALTH_OPENVPN_DURATION_ADDITION env variables as I see them in the Dockerfile but not in the docs. From my understanding :

• HEALTH_OPENVPN_DURATION_ADDITION is the additional delay for next error check. So with a value of 5, first failure will be retried in 11 seconds, then 16 seconds, then 21 seconds etc.
• HEALTH_OPENVPN_DURATION_INITIAL is the base amount of time between health checks? I see it get set to s.config.OpenVPN.Initial and then to s.vpn.healthyWait but then not really used. I'm wondering if this value is what is meant to be used for the hard-coded to 5 seconds timer?

gluetun/internal/healthcheck/health.go

Lines 47 to 52 in 87f4b9e

	// Success, check again in 5 seconds
	const period = 5 * time.Second
	timer := time.NewTimer(period)
	select {
	case <-ctx.Done():

I run Adguard Home for my dns resolver (similar to PiHole but with built-in DoH, DoT, DoQ, DNSCrypt so not afraid of DNS leakage) and have been noticing github.com getting resolved every 5 seconds which I would like to get down. But also brings up my next question.

2. I am unsure if the health check is actually really doing anything. From my reading I think resolver.LookupIP only does a dns lookup with not actually resolving anything

gluetun/internal/healthcheck/health.go

Lines 66 to 71 in d4ca5cf

	func healthCheck(ctx context.Context, resolver *net.Resolver) (err error) {
	// TODO use mullvad API if current provider is Mullvad
	const domainToResolve = "github.com"
	ips, err := resolver.LookupIP(ctx, "ip", domainToResolve)
	switch {
	case err != nil:

But with the default unbound caching (or Adguard Home in my case or PiHole), wont it just return the cached value and not actually test outbound connections past the first hit?

Maybe pinging a well-known dns server (1.1.1.1, 9.9.9.9, 8.8.8.8, 4.2.2.2) might be better? Ping packets are so small that it shouldn't really congest at all and should be fairly decent in terms of privacy.

3. I'm not really too familiar with go and it's entrypoints, but is this what docker is hitting for it's healthcheck? Wasn't sure if the docker endpoint drives the healthcheck or just gets a status of the background healthcheck
   

   gluetun/internal/healthcheck/handler.go

   Lines 26 to 36 in d4ca5cf

   	func (h *handler) ServeHTTP(responseWriter http.ResponseWriter, request *http.Request) {
   	if request.Method != http.MethodGet {
   	http.Error(responseWriter, "method not supported for healthcheck", http.StatusBadRequest)
   	return
   	}
   	if err := h.getErr(); err != nil {
   	http.Error(responseWriter, err.Error(), http.StatusInternalServerError)
   	return
   	}
   	responseWriter.WriteHeader(http.StatusOK)
   	}

Thanks again and stay safe my 🇨🇦 brother

[qdm12]

Hi there! Thanks for the detailed discussion and digging through the code!

The healthcheck code as you've seen does a dns resolution to github.com. You are right about the DNS cache, I was exactly thinking about it this morning. See the TODOs 1. below.

The healthcheck code runs every 5s when it's stable/healthy, and every second if it encountered a failure, until it succeeds. Now these times are independent from the VPN durations. I have to admit, I got confused myself looking at the code for a solid 2 minutes.

It's set here initially:

gluetun/internal/healthcheck/server.go

Line 35 in 87f4b9e

healthyWait: config.OpenVPN.Initial,

If it succeeds, it's reset here:

gluetun/internal/healthcheck/health.go

Line 26 in 87f4b9e

s.vpn.healthyWait = s.config.OpenVPN.Initial

If it fails for the first time:

gluetun/internal/healthcheck/health.go

Line 30 in 87f4b9e

s.vpn.healthyTimer = time.NewTimer(s.vpn.healthyWait)

If it fails after the initial time (6s), it's added 5s (configurable by env) here:

gluetun/internal/healthcheck/openvpn.go

Lines 17 to 23 in 87f4b9e

	func (s *Server) onUnhealthyOpenvpn(ctx context.Context) {
	s.logger.Info("program has been unhealthy for " +
	s.vpn.healthyWait.String() + ": restarting OpenVPN")
	_, _ = s.vpn.looper.ApplyStatus(ctx, constants.Stopped)
	_, _ = s.vpn.looper.ApplyStatus(ctx, constants.Running)
	s.vpn.healthyWait += s.config.OpenVPN.Addition
	s.vpn.healthyTimer = time.NewTimer(s.vpn.healthyWait)

Finally, the healthcheck HTTP server/client system is a bit convoluted but it works nicely! With the idea to one day have gluetun a single binary, see #588 and base that image on scratch instead of alpine 😉

The Docker healthcheck run a second ephemeral gluetun instance as a healthcheck client:

gluetun/Dockerfile

Line 173 in 87f4b9e

HEALTHCHECK --interval=5s --timeout=5s --start-period=10s --retries=1 CMD /entrypoint healthcheck

The client does a request to long running gluetun instance at http://127.0.0.1:9999

gluetun/internal/healthcheck/client.go

Lines 32 to 48 in 87f4b9e

	request, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
	if err != nil {
	return err
	}
	response, err := c.httpClient.Do(request)
	if err != nil {
	return err
	}
	defer response.Body.Close()
	if response.StatusCode == http.StatusOK {
	return nil
	}
	b, err := io.ReadAll(response.Body)
	if err != nil {
	return err
	}
	return fmt.Errorf("%w: %s: %s", ErrHTTPStatusNotOK, response.Status, string(b))

Which hits the HTTP server handler you saw, returning a status code 500 if there is an error and 200 otherwise.

All this is a bit convoluted I agree, but is designed to detect a bad connection quickly (report to the Docker healthcheck and HTTP server) but still wait long enough (and longer with each failure) for the VPN loop which usually takes about 4-5 seconds to setup (👀 Looking at you Openvpn, Wireguard takes 1 second or less).

TODOS

Now things I need to change:

• Change the healthcheck mechanism to avoid caching (leaning towards pinging for now)
  1. Use ping packets (but that requires root access, which I would ideally like to drop one day)
  2. Once my DoT+DoH Go API is stable & merged, use it with a Go DoT resolver without cache to do it.
• Configure the domain to resolve with a variable since github.com might be blocked depending on the location 🤔
• Rename Openvpn stuff in health code to vpn since it's now a 'vpn loop', not just openvpn.

Also thanks for:

• Adguard Home, I didn't know about it! It definitely is an awesome replacement to pihole. I might well inspire from it, since I'm developing a DoT+DoH server in Go to replace Unbound in gluetun and elsewhere. And DoQ, DNSCrypt look definitely interesting!

If you feel like contributing anything you feel like, you can use the dev container to setup your Go environment and get started. Let me know though, right now I push to master like a 🐷, I'm happy to do PRs if you want to contribute 😄

Cheers from Montreal! (also not Canadian, but aspiring to!)

[qdm12]

So... regarding the TODOS!...

1. It now uses ICMP packets to ping an address. I initially set it to 1.1.1.1 to avoid spamming DNS with resolutions every 5s, BUT then restored it to github.com to actually test the DNS is working.
2. You can set that address to something else with HEALTH_ADDRESS_TO_PING
3. The problem with ping is that you must run as root to do ICMP packets. Sending unprivileged UDP packets doesn't work without additional setup from the user which I want to avoid. Ideally one day I would like to run all this without root (if that's even possible considering all the root-ish traffic), but it might be possible with Linux capabilities such as setcap cap_net_raw=+ep /path/to/your/compiled/binary in this case.

EDIT: also a question for you, how do you run your Adguard Home resolver with gluetun? Are you running it with --network_mode="container:gluetun"?

[TChilderhose]

Wow, thanks for a detailed answer!

The healthcheck code runs every 5s when it's stable/healthy, and every second if it encountered a failure, until it succeeds. Now these times are independent from the VPN durations. I have to admit, I got confused myself looking at the code for a solid 2 minutes.

Ah I think I get it now. So those are times for when the VPN resets itself vs the actual checks healthcheck happens either 1 or 5 seconds.

Think my confusion came from how switch statements and timers work in Go. I'm so use to the C syntax where you need a break; otherwise each pattern is basically considered an or. e.g. I thought that both the one second timer and the healthTimer would call s.onUnhealthyOpenvpn(ctx) in this switch statement.

gluetun/internal/healthcheck/health.go

Lines 41 to 43 in 87f4b9e

	case <-timer.C:
	case <-s.vpn.healthyTimer.C:
	s.onUnhealthyOpenvpn(ctx)

But I see that the one second timer doesn't really do anything and just calls the for loop continue.

SideNote: Pretty neat you can basically wait for one of the blocking calls to finish in a switch statement like that, very cool.

The healthcheck code as you've seen does a dns resolution to github.com. You are right about the DNS cache, I was exactly thinking about it this morning. See the TODOs 1. below.

Haha that's pretty funny. Looked through the TODO and sounds good! You make a good point of leaving it at github.com to actually test the DNS as well, but appreciate the addition of HEALTH_ADDRESS_TO_PING!

I was actually looking at that go-ping repo you used myself and reading about that blob post about the unprivileged sockets, kind of sucks but I get it.

Re: Adguard Home

No problem! I actually like it a quite bit better that pihole. Has some pretty great features in it like choosing which dns server it uses base on regex.

Im not a huge fan of DoQ and DNSCrypt to be honest. I see why people may want to use them, but DoT is a pretty robust standard that everyone seems to be using and seems pretty good from my understanding. And then DoH is more flexible I guess? Good for browsers or people who have the TLS port blocked I guess.

I'll keep an eye out on your resolver repo for sure! If you have caching with an api to bust the cache for a query that would be awesome (and yeah fix this issue with testing connectivity). Its actually one of the features I have written down as something I would like in a resolver!

EDIT: also a question for you, how do you run your Adguard Home resolver with gluetun? Are you running it with --network_mode="container:gluetun"?

No. I was contemplating doing that, but am fine with my dns doing DoT on my main network. Everything in my house uses that as DNS so would like to reduce the amount things that it relys on for the time being.

Here is an example of my docker-compose. Basically just share the same network between the two containers, set the gluetun container DNS to the adguardhome and use the internal docker dns resolver inside gluetun

adguardhome:
   ...
  networks:
      some_network_name:
        ipv4_address: 10.1.0.100

gluetun:
   ...
  environment:
      ...
      - DOT=off
      - DNS_PLAINTEXT_ADDRESS=127.0.0.11
      ...
  dns:
      - 10.1.0.100
  networks:
      some_network_name:

networks:
  default:
    some_network_name:
      config:
        - subnet: 10.1.0.0/24

Pretty busy with work and a young one, but if I get some time I would more then happy to contribute. Would be an interesting entry into Go haha.

[qdm12]

Think my confusion came from how switch statements and timers work in Go.

Definitely. It's Go that adopted this strange switch/select way. All other languages I know of need break statements or they'll fallthrough, whereas Go is the opposite, since you need to specify it to fallthrough. As in I don't dislike it, it's just unusual.

I see why people may want to use them, but DoT is a pretty robust standard

Definitely. On top of that you sort of need DoT to resolve the DoH url from time to time. My only issue personally is that everyone can see I'm using DoT on port 853, whereas DoH goes unnoticed (although no traffic on port 53 would point to it too).

have caching with an api to bust the cache for a query that would be awesome

Please create a quick copy pasta issue on that github.com/qdm12/dns repo, I'll add that! Maybe post v2.0.0 though. Right now I'm working on DNSSEC to finally merge that beta branch, and it's a PITA 😅

I might get started on adding an http server api to control the DNS (like clear the cache elements). A UI would be nice too, of course.