chore(data-warehouse): Updated billing limits to work per sync

[Gilbert09]

Problem

In my opinion, we had a subpar experience for when users hit data warehouse billing limits. We're running a celery task every 20 mins to check whether a team has hit its limits and then "Pause" and "Cancel" existing jobs/schemas. Why was this bad?

• Users would have to wait 20 mins before they could resync once they upgraded their billing limits (with no info around this)
• A job being in a "Cancelled" state was confusing to users - already had a support ticket asking what this meant
• Users could go over their billing limit with subsequent jobs for up to 20 mins

Changes

• Remove the celery task
• Added billing limit check at the start of every data warehouse sync - this will cancel the job if the user has hit their billing limits and log this out
• Re-utilise the "cancelled" status and instead return "billing limits" as the job/schema/source status instead
• Removed the ad-hoc 500,000,000 monthly sync limit

Does this work well for both Cloud and self-hosted?

Yep

How did you test this code?

Added a unit test to ensure no rows get synced when limits are hit

[github-actions]

Size Change: 0 B

Total Size: 1.11 MB

ℹ️ View Unchanged

Filename	Size
frontend/dist/toolbar.js	1.11 MB

_{compressed-size-action}

[tomasfarias]

Just had some open question regarding whether we should instead have a separate status for billing limits reached instead of re-using cancelled.

But overall I don't think it would be a blocker.

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ ActivityError: Activity task timed out posthog.temporal.data_imports.external_data_job... View Issue
• ‼️ RuntimeError: Ignoring add command while deleting posthog.temporal.data_imports.external_data_job... View Issue
• ‼️ IntegrityError: insert or update on table "posthog_externaldatajob" violates foreign key constraint "posthog_exte... posthog.warehouse.external_data_source.jobs in ... View Issue
• ‼️ RuntimeError: Ignoring add command while deleting posthog.temporal.data_imports.external_data_job... View Issue
• ‼️ ActivityError: Activity task timed out posthog.temporal.data_imports.external_data_job... View Issue

_{Did you find this useful? React with a 👍 or 👎}