Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(tasks): Add autoretry for too many simultaneous queries error in some tasks #21895

Merged
merged 3 commits into from
Apr 29, 2024

Conversation

webjunkie
Copy link
Contributor

Problem

We throw a lot of simultaneous queries errors. We can retry.

Fixes POSTHOG-DNR
Fixes POSTHOG-DNR
Fixes POSTHOG-TMV

Changes

  • use Celery autoretry_for
  • but for that to work, isolate the exception as a real one

Does this work well for both Cloud and self-hosted?

n/a

How did you test this code?

  • tested exception handling, added test
  • not sure how to test locally for real cause needs reproducing too many queries somehow

@webjunkie webjunkie changed the title Add autoretry for too many simultaneous queries error in some tasks fix(tasks): Add autoretry for too many simultaneous queries error in some tasks Apr 26, 2024
@webjunkie webjunkie requested a review from a team April 26, 2024 13:11
Copy link
Member

@Twixes Twixes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. More load if there's a period of lots of queries, but fewer user-facing errors for intermittent troubles, which seems like an alright treade-off

pass


CLICKHOUSE_SPECIFIC_ERROR_LOOKUP = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would also be good to move EstimatedQueryExecutionTimeTooLong and into this QuerySizeExceeded, in a separate PR

@webjunkie webjunkie merged commit 304540f into master Apr 29, 2024
99 of 100 checks passed
@webjunkie webjunkie deleted the fix/query-async-autoretry branch April 29, 2024 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants