Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting around startup memory spike #711

Merged

Conversation

mutantcornholio
Copy link
Contributor

Esbuild doesn't want to fix evanw/esbuild#3894, and we're hitting memory limits again, after upgrading papi to latest version.
This allows us to bypass binary loader and get back to normal memory usage during application startup.

I've created a small repro which fails with OOM on current papi version, but works fine with my changes.
papi-repro.tar.gz

Fixes #654

Copy link
Member

@josepot josepot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for this! I will let @voliva to double check this, but at a first glance this looks awesome. Thanks!

Copy link
Contributor

@voliva voliva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to use a text encoding more compact than hex?

Metadata types are quite big in general, and having a small change in encoding can have a significant impact on bundle sizes.

Also, the difference in memory usage with the reproduction case is in the same order of magnitude: The smallest V8's memory size I could do with the current version is 165mb, whereas this fix brings it down to 120mb.

Esbuild doesn't want to fix evanw/esbuild#3894,
and we're hitting memory limits again, after upgrading papi to latest
version.
This allows us to bypass binary loader and get back to normal memory
usage during application startup.
@mutantcornholio
Copy link
Contributor Author

Would it be possible to use a text encoding more compact than hex?

Okay, I've copied base64 decoder from esbuild itself, should be just the same now

Also, the difference in memory usage with the reproduction case is in the same order of magnitude: The smallest V8's memory size I could do with the current version is 165mb, whereas this fix brings it down to 120mb.

crank it up to five chains, it'll take 380mb for current version and 130mb for my fix

Copy link
Contributor

@voliva voliva left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great, thank you very much!

@voliva voliva added this pull request to the merge queue Sep 13, 2024
Merged via the queue into polkadot-api:main with commit f47e8b2 Sep 13, 2024
2 checks passed
@mutantcornholio mutantcornholio deleted the yuri/startup-memory-workaround branch September 13, 2024 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Outrageous peak memory usage with binary modules Insane startup memory usage due to esbuild issue
3 participants