Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

677 refactor expunge #790

Open
wants to merge 14 commits into
base: develop
Choose a base branch
from
Open

677 refactor expunge #790

wants to merge 14 commits into from

Conversation

jsjiang
Copy link
Contributor

@jsjiang jsjiang commented Nov 13, 2024

@sfisher Hi Scott,

The like query is very heavy. The refactoring is focused on limiting the rows that will be scanned for each query.

The proc-expunge.py is the updated script. The original script is renamed and saved as proc-expunge_v1.py.

Here are the changes:

  • add data range and pagesize as optional parameters to the command
  • define default scan window to 3 days to reduce to be scanned records
  • use the created time range to find the mapped ID range (since the records are created in time sequence)
  • use the ID range to filter query set - filter on primary key is more efficient
  • process each query set in pagesize batches
  • increase default page size from 100 to 1000

Please review and let me know if you have quesitons.

Thank you

Jing

@jsjiang jsjiang requested a review from sfisher November 13, 2024 00:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant