Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Research adjusting the prefork setting for Celery and write an ADR proposing adjustments #1432

Open
3 tasks
ccostino opened this issue Nov 25, 2024 · 1 comment
Assignees
Labels
documentation Improvements or additions to documentation engineering

Comments

@ccostino
Copy link
Contributor

After our large batch send test, there might be some additional configuration we can make with Celery and the prefork worker pool to help improve stability, but we want to make sure it doesn't come at the expense of performance; or, if it does, what we can do to mitigate that.

In addition to the Celery documentation itself, there are some other guides (e.g, Celery School - The Prefork Worker Pool) that could be of help in figuring these pieces out.

Implementation Sketch and Acceptance Criteria

  • Do some more research into how best to configure Celery's concurrency options, especially with the prefork worker pool
  • Identify possible performance trade-offs and how they might be mitigated
  • Write up an ADR (Architectural Decision Record) with the proposed changes so that we can discuss them as a team

Security Considerations

  • We want to make sure that the site remains stable and resilient during increased usage, especially as we start to scale more.
@ccostino ccostino added documentation Improvements or additions to documentation engineering labels Nov 25, 2024
@ccostino ccostino moved this from 🌱 New to ⬇ Up-Next in Notify.gov product board Nov 25, 2024
@terrazoon terrazoon self-assigned this Nov 25, 2024
@terrazoon terrazoon moved this from ⬇ Up-Next to 🏗 In progress (WIP: ≤ 3 per person) in Notify.gov product board Nov 25, 2024
@terrazoon terrazoon moved this from 🏗 In progress (WIP: ≤ 3 per person) to 👀 In review in Notify.gov product board Dec 2, 2024
@ccostino ccostino moved this from 👀 In review to ⏱ Blocked/Waiting in Notify.gov product board Dec 2, 2024
@ccostino
Copy link
Contributor Author

ccostino commented Dec 2, 2024

We know this is ready to try out, but until we have more memory available to us in cloud.gov to properly test this in staging, we're not able to adequately test this just yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation engineering
Projects
Status: Blocked/Waiting
Development

No branches or pull requests

2 participants