Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: stop processing in create_usage_records job if it took more than 15min #2260

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

tanmoysrt
Copy link
Member

@tanmoysrt tanmoysrt commented Nov 6, 2024

In create_usage_records we process the usage of sites synchronously in a batch size of 200.

we hold lock on the record at the time of submission of usage record.

# Get a read lock on this invoice
# We're going to update the invoice and we don't want any other process to update it
invoice = team.get_upcoming_invoice(for_update=True)
if not invoice:
invoice = team.create_upcoming_invoice()

When there is some wait for lock on a record and it delay the overall job, it can exceed 15 minute. Meanwhile, next job started and start processing same record, hence more errors.

Multiple instance of same job running
Screenshot 2024-11-06 at 9 43 36 AM

If we can stop process further records after 15 minute, it can reduce the errors.

Copy link

codecov bot commented Nov 6, 2024

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 39.90%. Comparing base (2f435f8) to head (58fff1c).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
press/press/doctype/subscription/subscription.py 25.00% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2260      +/-   ##
==========================================
+ Coverage   29.94%   39.90%   +9.95%     
==========================================
  Files         371      371              
  Lines       28078    28082       +4     
==========================================
+ Hits         8408    11206    +2798     
+ Misses      19670    16876    -2794     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant