Internal DNS servers return records with a TTL of 0

[jgallagher]

Our internal DNS servers currently don't set an explicit TTL, so we get the hickory-proto default of 0. This instructs clients to never cache results, which seems to be in conflict with connection pooling as we've started to integrate qorb. When we construct a qorb DnsResolver, we currently instruct it to ignore those 0-valued TTLs:

omicron/nexus/db-queries/src/db/pool.rs

Lines 63 to 66 in f3ab338

	DnsResolverConfig {
	hardcoded_ttl: Some(tokio::time::Duration::MAX),
	..Default::default()
	},

From this conversation on the Oximeter qorb integration PR:

smklein Oct 2, 2024

Yup! Right now, I think our internal DNS system is using a TTL of "zero", which means we'd see a backend, start to spin up connections for them, and then discard them immediately. Next time we query DNS, we'd start to create connections, then discard them immediately...

This behavior is tested here: https://github.com/oxidecomputer/qorb/blob/7c886a24369f458bdab4b9530e01f498da79e61f/src/resolvers/dns.rs#L712-L746 , but it sucks.

By ignoring TTLs, we still discard backends that are explicitly removed from DNS, but we only remove backends when DNS servers stop telling us about them successfully.

This might all be fine? We could decide qorb ignoring the TTL and doings its own periodic querying is exactly what we want, but I wanted to write this down in case we want to control TTLs from the DNS side (at which point we'd presumably want to stop overriding it in the qorb config(s)).

[rmustacc]

I think we should be proactively communicating the TTLs that we should be using otherwise as we spin up and spin down services you're not going to know how long you need to wait for it actively drain per se so at least it can't be discovered in internal DNS by subsequent connection pool mechanisms. If we have a fleet of qorb's that are all picking their own policy choices, then we'll have to go back and encode that knowledge somehow (and then add a bit more here). So while not a priority, I think the DNS server should be explicitly advertising this as a way to indicate to other clients what its intents are and to at least start to create some kind of explicit promise here.

Put differently, we should know that the moment say I remove an internal endpoint from DNS if I wait 1-2x the TTL it is safe for me to close my listen socket to minimize errors. There may still be ongoing connections, but there shouldn't be new connections going in because it should have expired from all DNS caches.