Skip to content

Commit

Permalink
chore: Make Celery Queue Depth page less
Browse files Browse the repository at this point in the history
companion to https://github.com/PostHog/charts/pull/773

We've been paging for this daily for more than 10 days. In all of those cases there hasn't been any action and it resolves itself. This creates alert fatigue, so let's make the alert less sensitive.
We can additionally add a non-paging alert, if this is deemed useful.
  • Loading branch information
tiina303 authored Feb 12, 2024
1 parent 4ead4cb commit b4bae45
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions charts/posthog/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2528,12 +2528,12 @@ prometheus:
- alert: CeleryQueueDepth
expr: (max (posthog_celery_queue_depth)) > 1000
for: 10m
for: 60m
labels:
rotation: common
severity: critical
annotations:
summary: Celery job execution delayed for more than 10 minutes.
summary: Celery job execution delayed for more than 60 minutes.
description: |
The Celery jobs queue (stored in Redis) is filling up faster than it is consumed. This impacts our
monitoring, as some paging monitors depend on metrics exported by Celery jobs.
Expand Down

0 comments on commit b4bae45

Please sign in to comment.