Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

alerts: add CeleryNotReporting paging alert #712

Closed
wants to merge 4 commits into from
Closed

alerts: add CeleryNotReporting paging alert #712

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0f30657
alerts: add paging CeleryQueueDepth monitor
xvello Mar 17, 2023
a1e33e6
fix summary
xvello Mar 17, 2023
0a920a0
alerts: add CeleryNotReporting paging alert
xvello Mar 17, 2023
8f42009
Merge branch 'main' into xvello/celery-no-data
xvello Mar 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions charts/posthog/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2256,6 +2256,20 @@ prometheus:
- check the health of Postgres and Clickhouse, many jobs query them and could hang if these
stores get slow

- alert: CeleryNotReporting
expr: absent_over_time(posthog_celery_queue_depth[10m]) > 0
labels:
rotation: common
severity: critical
annotations:
summary: Celery has not pushed metrics for more than 10 minutes.
description: |
The posthog-worker pods have not reported points for the posthog_celery_queue_depth metric
in the last 10 minutes. Because many of our alerts rely on metrics reported by Celery jobs,
this is a critical alert (other issues could be hidden).
- make sure posthog-worker pods are healthy and check their logs
- look for recent code changes to posthog/celery.py and rollback if needed

- alert: KafkaDiskCritical
expr: min by (instance) (aws_msk_node_filesystem_free_bytes{mountpoint="/kafka/datalogs"}) < 536870912000
for: 2m
Expand Down
Loading