LnD: investigation into restart submission report generation #7551
+11
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
colinbruce
force-pushed
the
lnd/restart-submissions-test
branch
from
6210f15 to
2f9a578
Compare
This allows us to check flow using open_search if an error occurs during state transitions. Rather than taking up DB space we can check logs and record output if needed
colinbruce
force-pushed
the
lnd/restart-submissions-test
branch
from
d5a1c98 to
ba220af
Compare
This sets up a new capusle with a concurrency of 2, this allows two applications to have their reports created at the same time. The current handling allows 5 and we saw crashes after a pause in CCMS submissions
colinbruce
force-pushed
the
lnd/restart-submissions-test
branch
from
ba220af to
26e98e2
Compare
|
naseberry
approved these changes
Jan 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
When CCMS submissions were restarted on the morning of 20 Dec the submissions restarted but the reports did not generate as expected.
This PR identified an issue where CCMS submissions have been paused during an evening downtime. Previously we had only seen 2 or 3 submissions maximum.
On this occasion there were 10 applications to restart. This turned out to be too many!
I recreated the situation, 10 paused applications, and turned the submissions back on. The first 5 (each sidekiq worker has 5 threads) requested a CCMS case reference and then tried to generate reports at the same time... the worker container ran out of memory and crashed. When a new pod was spawned, the next 5 started. This left all 10 applications trying to build reports and failing.
Working as a team we identified that Sidekiq Capsules should work for us, I restored a backup of the test branch with 10 paused applications and created a new report_creator queue with two threads. This allowed two applications to build reports simultaneously and successfully cleared the paused submissions
Checklist
Before you ask people to review this PR:
bundle exec rakegit rebase main.