Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Update Consumer Simulator to be more memory efficient #404

Closed
Tracked by #390
mattp-swirldslabs opened this issue Dec 12, 2024 · 2 comments · Fixed by #406
Closed
Tracked by #390

feat: Update Consumer Simulator to be more memory efficient #404

mattp-swirldslabs opened this issue Dec 12, 2024 · 2 comments · Fixed by #406
Assignees
Labels
Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. Simulator Issue related to Block Stream Simulator

Comments

@mattp-swirldslabs
Copy link
Contributor

mattp-swirldslabs commented Dec 12, 2024

As a Consumer Simulator User
I want to be able to consume the 10k TPS recording of real data with the setting blockStream.millisecondsPerBlock=500 without running out of memory in the simulator.
So that I can confirm end-to-end BN streaming.

  • While testing PBJ Issue #329, I found that the Consumer Simulator would run out of memory (16GB) before consuming all 4058 blocks of the 10k TPS recording
  • On a local branch, I was able to keep the Consumer Simulator at around 1 GB of memory while consuming the entire 4058 blocks of the recording to test #329. I was able to fix the memory issue by using the "Unparsed" BlockItem strategy we're using on the BN server. However, after a stand up discussion, we determined that we should NOT use the "Unparsed" strategy for the Consumer Simulator.
  • Instead of "Unparsed" types, we need to find ways to conserve memory without changing the existing protoc types if possible.

Tech Notes

  • Getting to the point where you can see the OutOfMemoryError on the Consumer Simulator may require the Helidon fixes coming in 4.2.0-SNAPSHOT. Here's how you can get these HTTP/2 fixes to run the 10k TPS recording successfully from the producer -> BN -> consumer:
  1. Prerequisite: install Maven. I used version 3.8.7
  2. On a terminal cd to a directory to clone the Helidon repo
  3. git clone [email protected]:helidon-io/helidon.git && cd helidon && mvn install -DskipTests
  4. In settings.gradle.kts change val helidonVersion = "4.1.1" to `val helidonVersion = "4.2.0-SNAPSHOT"
  5. Build and deploy the BN container as you usually would
@mattp-swirldslabs mattp-swirldslabs added Simulator Issue related to Block Stream Simulator Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. labels Dec 12, 2024
@jsync-swirlds
Copy link
Member

Some brief thoughts on possible approaches:

  1. Write blocks to disk in the simulator, only keep a small number (2-3) of blocks in memory at a time.
  2. Drop blocks once they're read and (possibly) verified
    • This is the easiest, just don't keep things around, but we'll eventually need to be able to do more than just receive the block; we need to verify and possibly do other processing. We should be able to reuse the persistence and verification services from Block Node itself to do these things.
  3. Have some sort of configurable option to not parse block items, just write them to disk, or hash them, as received (possibly not manageable with protoc generated gRPC methods).

All of these involve reducing the number of blocks in memory, which I suspect is central to making the memory impact reasonable.

@georgi-l95
Copy link
Contributor

Did some testing, yes, heap grows uncontrollably.
image
image

@georgi-l95 georgi-l95 self-assigned this Dec 13, 2024
@georgi-l95 georgi-l95 linked a pull request Dec 13, 2024 that will close this issue
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature Enhancement Enhancing an existing feature driven by business requirements. Typically backwards compatible. Simulator Issue related to Block Stream Simulator
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants