feat: Don't Label Users as Quota Limited for N Days After They Exceed Their Quota

[xrdt]

Problem

N = 7. We had discussed making N = 3 because 7 might just be too many days.

We would like to keep users' data for N days after they start to exceed their usage quotas. This will allow us to recover in a case where billing limits inadvertently become lower than current usage (INC-144). This PR introduces a new redis cache and additional metadata on the cached usage dict in posthog to record this temporary state.

Working on tests 🔨

Changes

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

[xrdt]

Background and requirements:

• We'd like to be able to toggle a data retention mechanism before we execute billing migrations. This will prevent us from inadvertently lowering billing limits and quota limiting users.
  • We should have telemetry into the following:
    • Who would have had data dropped had we not toggled this mechanism?
    • How much data would have been dropped? Aggregate and per org.
• Pipeline team will be looking into a plugin-server based implementation of billing / quota limits in Q2. This implementation will be a temporary mechanism that builds on existing quota limiting functionality in the billing api.

Approaches:

1. Temporarily move users into a separate redis set and mark when they should no longer be put into this set but into the default quota limiting set. This mechanism resets at the shorter of 7 days or the start of the new billing cycle.
   a. It's not clear how to support a toggling mechanism.
2. Use feature flags to toggle the suspension of quota limiting.
   a. This should be relatively simple and will allow us to have fine-grained toggling controls (all users and arbitrary subsets).

For both approaches, we can send events to posthog marking when a user should have been quota limited and when the temporary suspension of quota limiting ended. We can then use usage reports to estimate the usage that would have been dropped had quota limiting been in effect.

This PR partially implements approach 1. Here is a PR for approach 2.

@raquelmsmith More or less sums it up? It's looking like the feature flags approach makes the most sense, unless @neilkakkar identifies an issue.

[xrdt]

We're going to keep this strategy in place as a failsafe to complement the manual, feature-flag based retention trigger: #19996

[posthog-bot]

📸 UI snapshots have been updated

6 snapshot changes in total. 0 added, 6 modified, 0 deleted:

• chromium: 0 added, 6 modified, 0 deleted (wasn't pushed!)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[raquelmsmith]

Logic looks good here, just a couple questions about code hygiene.

I wonder what is the best way to use this and keep an eye on it. Like should we have an alert of some sort when the number of teams who would have been limited goes up?

[raquelmsmith]

When would we not have a billing period end?

[xrdt]

I can't imagine why we wouldn't have one. This is here simply to be consistent with the previous check if is_quota_limited and billing_period_end:

[xrdt]

It seems like we'd be most interested in keeping tabs on whether certain orgs are incorrectly having data retained. That would imply we're not correctly inserting and removing them from the data-retention redis set. I think we should create a dashboard similar to the existing quota limiting one, or simply add graphs for data retention to that dashboard.