chore(flags): add retries to is_postgres_connected_check for flag matching paths

[havenbarnes]

Problem

We're still trying to root cause the recent spike in healthcheck_failed flag evaluation errors (conversation here https://posthog.slack.com/archives/C0185UNBSJZ/p1733369640212279), but for the short term this change will require 3 straight SELECT 1 failures before determining PG to be down when during flag matching 🩹

We see in Loki logs that this healthcheck is often failing with django.db.utils.OperationalError: consuming input failed: query_wait_timeout server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request., especially when the decide request is interacting with the PG writer.

We've also identified some over-reporting of healthcheck_failed when write paths fail, but short-circuiting other requests would only use read replicas. This will be addressed in a follow up PR

Changes

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

Yes

How did you test this code?

Hitting decide locally works as expected and with "errorsWhileComputingFlags": false,

[neilkakkar]

flyby that this will make some requests unacceptably slow. It's much better imo to have error computing flags fire (which doesn't mean none of the flags were evaluated, just that the ones relying on the db weren't), than to slow down the response by upto a second occasionally

[neilkakkar]

A better way to achieve the same as retries here would be to increase the query wait timeout setting, but ideally we'd tune the read replica connection settings and instances so these timeouts don't happen often. I bet when this issue happens it shows up on the pgbouncer graph as no. of waiting connections spiking (to pgbouncer, not to the db - that's been the case in the past)

[havenbarnes]

Yep that makes sense. Was thinking too much about customers who are dealing with our worst case