Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(imports) possible fix for imports running out of memory #29169

Merged
merged 10 commits into from
Aug 29, 2024
Merged

fix(imports) possible fix for imports running out of memory #29169

merged 10 commits into from
Aug 29, 2024

Conversation

wezell
Copy link
Contributor

@wezell wezell commented Jul 9, 2024 •
edited by github-actions bot
Loading

This does a few things:

  • Removes reading the file in as a byte[] and instead passes around a File object
  • Changes how the import is run inline (helpful for glowroot traces) or to use the DotSubmitter thread pool. Config var is IMPORT_CONTENTLETS_ASYNC
  • Commit Granularity changed to 100, meaning every 100 contentlets will be committed and a new transaction will be started.
  • Copied CSV file to tmp dir.

ref: #29162

This PR fixes: #29162

@wezell wezell linked an issue Jul 9, 2024 that may be closed by this pull request
Slow performance with imports in some cases #29162
Closed
@github-actions GitHub Actions
Copy link

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the stale label Aug 23, 2024
@jdotcms jdotcms enabled auto-merge August 23, 2024 19:06
@erickgonzalez erickgonzalez disabled auto-merge August 27, 2024 16:39
@dotcms-sonarqube dotCMS-sonarqube
Copy link

Quality Gate passed Quality Gate passed

Issues
6 New issues
11 Fixed issues
2 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

wezell
wezell commented Aug 29, 2024
View reviewed changes
Copy link
Contributor Author

@wezell wezell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work Erick

@erickgonzalez erickgonzalez added this pull request to the merge queue Aug 29, 2024
Merged via the queue into master with commit 0e6f1cf Aug 29, 2024
30 of 31 checks passed
@erickgonzalez erickgonzalez deleted the issue-29162-slow-imports branch August 29, 2024 17:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jdotcms jdotcms jdotcms approved these changes

@jcastro-dotcms jcastro-dotcms jcastro-dotcms approved these changes

@erickgonzalez erickgonzalez erickgonzalez approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Slow performance with imports in some cases
4 participants
@wezell @erickgonzalez @jcastro-dotcms @jdotcms