alerts: add paging CeleryQueueDepth monitor

[xvello]

Description

Add a monitor paging when Celery is getting slow and not processing jobs fast enough. No runbook yet, but I wrote basic investigation steps in the monitor description. It's not perfect as Celery is currently reporting its own health, and the metric will not be exported if Celery is completely down. We should also add a last_job_run_timestamp metric and alert on it. Or is there an easy way to alert on no-data in prometheus?

I tried to calibrate it so that:

• it would have triggered on the Feb 23rd ClickHouse incident
• it would not have triggered on the Feb 21st and 22nd pgbouncer spikes

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

How has this been tested?

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[xvello]

#712 is a "watching the watcher" follow-up blocked for now on infra work