Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add separate celery queue for usage reports #20048

Merged
merged 12 commits into from
Jan 31, 2024

Conversation

raquelmsmith
Copy link
Member

@raquelmsmith raquelmsmith commented Jan 30, 2024

Problem

Usage reports cause a big spike in celery tasks, which then trigger an alert for the redis queue depth. We can move these to a separate queue with no alert so that they don't trigger it.

Changes

  • Create a usage_report queue
  • Add metrics for this queue so we can see when they are running (if there are issues with them running the first thing I look at is celery)

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

@raquelmsmith raquelmsmith marked this pull request as ready for review January 30, 2024 20:17
Copy link
Contributor

github-actions bot commented Jan 30, 2024

Size Change: 0 B

Total Size: 2.22 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 2.22 MB

compressed-size-action

Copy link
Contributor

@webjunkie webjunkie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

posthog/settings/sentry.py Outdated Show resolved Hide resolved
@@ -63,6 +64,14 @@
QUERY_RETRY_DELAY = 1
QUERY_RETRY_BACKOFF = 2

USAGE_REPORT_TASK_KWARGS = dict(
queue=CeleryQueue.USAGE_REPORTS.value,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@raquelmsmith just FYI - modified this to have each task use the queue. It doesn't work to only have the parent task on its own. The task that get created also need to specify their intended queue

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh that makes sense. Thanks!

@benjackwhite benjackwhite merged commit 51db8af into master Jan 31, 2024
73 checks passed
@benjackwhite benjackwhite deleted the feat/usage-report-celery-queue branch January 31, 2024 10:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants