Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Fix Simulator unbounded memory growth #406

Merged
merged 5 commits into from
Dec 19, 2024
Merged

chore: Fix Simulator unbounded memory growth #406

merged 5 commits into from
Dec 19, 2024

Conversation

georgi-l95
Copy link
Contributor

@georgi-l95 georgi-l95 commented Dec 13, 2024 •
edited
Loading

Description:
This PR aims to resolve unbounded memory growth within the simulator, when running in consumer mode. It's was caused by the uncontrollable grow of the lastKnownStatuses list, when running in "high-speed" mode.

  • Changed to ArrayDeque with predefined capacity.
  • Added new blockStreamConfig called lastKnownStatusesCapacity, set to be 10 by default

Related issue(s):

Fixes #404

Notes for reviewer:
Here is image of the heap size before the fix:
image
Here is image of the heap size after the fix:
image

Checklist

  • Documented (Code comments, README, etc.)
  • Tested (unit, integration, etc.)

@georgi-l95 georgi-l95 added the Simulator Issue related to Block Stream Simulator label Dec 13, 2024
@georgi-l95 georgi-l95 added this to the 0.3.0 milestone Dec 13, 2024
@georgi-l95 georgi-l95 self-assigned this Dec 13, 2024
@georgi-l95 georgi-l95 requested a review from a team as a code owner December 13, 2024 14:05
@georgi-l95 georgi-l95 linked an issue Dec 13, 2024 that may be closed by this pull request
feat: Update Consumer Simulator to be more memory efficient #404
Closed
@georgi-l95 georgi-l95 changed the title bug: Fix Simulator memory leak chore: Fix Simulator memory leak Dec 13, 2024
@georgi-l95 georgi-l95 changed the title chore: Fix Simulator memory leak chore: Fix Simulator unbounded memory growth Dec 13, 2024
mattp-swirldslabs
mattp-swirldslabs previously approved these changes Dec 13, 2024
View reviewed changes
Copy link
Contributor

@mattp-swirldslabs mattp-swirldslabs left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I added a few comments and questions. Also, I'm seeing a few javadoc gaps when I build it:
image

simulator/src/main/java/com/hedera/block/simulator/grpc/impl/ConsumerStreamObserver.java Outdated Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/grpc/impl/PublishStreamObserver.java Outdated Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/grpc/impl/PublishStreamObserver.java Outdated Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/grpc/impl/ConsumerStreamObserver.java Outdated Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/config/data/BlockStreamConfig.java Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/config/data/BlockStreamConfig.java Outdated Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/grpc/impl/ConsumerStreamGrpcClientImpl.java Outdated Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/grpc/impl/ConsumerStreamObserver.java Outdated Show resolved Hide resolved
@georgi-l95
Fix Simulator memory leak
10acc2d 
Signed-off-by: georgi-l95 <[email protected]>
@georgi-l95
add additional unit test
3eaa3ba 
Signed-off-by: georgi-l95 <[email protected]>

revert app.properties change

Signed-off-by: georgi-l95 <[email protected]>

fix e2e test

Signed-off-by: georgi-l95 <[email protected]>
@georgi-l95
address feedback
d232852 
Signed-off-by: georgi-l95 <[email protected]>
@georgi-l95 georgi-l95 force-pushed the 404-feat-update-consumer-simulator-to-be-more-memory-efficient branch from 713b515 to d232852 Compare December 16, 2024 12:47
@georgi-l95
map env variables
8e8bc26 
Signed-off-by: georgi-l95 <[email protected]>
ata-nas
ata-nas previously approved these changes Dec 16, 2024
View reviewed changes
Copy link
Contributor

@ata-nas ata-nas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good @georgi-l95, just a couple of nits to consider.

simulator/src/main/java/com/hedera/block/simulator/grpc/impl/ConsumerStreamObserver.java Outdated Show resolved Hide resolved
simulator/src/main/java/com/hedera/block/simulator/grpc/impl/PublishStreamObserver.java Outdated Show resolved Hide resolved
@georgi-l95
address feedback
9305185 
Signed-off-by: georgi-l95 <[email protected]>
@georgi-l95 georgi-l95 requested a review from ata-nas December 16, 2024 13:50
ata-nas
ata-nas approved these changes Dec 16, 2024
View reviewed changes
Copy link
Contributor

@ata-nas ata-nas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

simulator/src/main/java/com/hedera/block/simulator/grpc/impl/ConsumerStreamObserver.java Show resolved Hide resolved
@georgi-l95 georgi-l95 merged commit 3ad4efc into main Dec 19, 2024
10 checks passed
@georgi-l95 georgi-l95 deleted the 404-feat-update-consumer-simulator-to-be-more-memory-efficient branch December 19, 2024 09:24
@codecov Codecov
Copy link

codecov bot commented Dec 19, 2024

Codecov Report

Attention: Patch coverage is 82.60870% with 4 lines in your changes missing coverage. Please review.

Project coverage is 96.73%. Comparing base (4201258) to head (9305185).
Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
...ck/simulator/grpc/impl/ConsumerStreamObserver.java 50.00% 1 Missing and 1 partial ⚠️
...ock/simulator/grpc/impl/PublishStreamObserver.java 50.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files 
@@             Coverage Diff              @@
##               main     #406      +/-   ##
============================================
- Coverage     96.99%   96.73%   -0.26%     
  Complexity      400      400              
============================================
  Files            80       80              
  Lines          1429     1441      +12     
  Branches         94       95       +1     
============================================
+ Hits           1386     1394       +8     
- Misses           32       34       +2     
- Partials         11       13       +2
Files with missing lines Coverage Δ
...config/SimulatorMappedConfigSourceInitializer.java 100.00% <ø> (ø)
...block/simulator/config/data/BlockStreamConfig.java 100.00% <100.00%> (ø)
...dera/block/simulator/config/data/StreamStatus.java 100.00% <100.00%> (ø)
...ulator/grpc/impl/ConsumerStreamGrpcClientImpl.java 100.00% <100.00%> (ø)
...mulator/grpc/impl/PublishStreamGrpcClientImpl.java 90.56% <100.00%> (+0.18%) ⬆️
...ck/simulator/grpc/impl/ConsumerStreamObserver.java 93.54% <50.00%> (-6.46%) ⬇️
...ock/simulator/grpc/impl/PublishStreamObserver.java 88.88% <50.00%> (-11.12%) ⬇️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ata-nas ata-nas ata-nas approved these changes

@AlfredoG87 AlfredoG87 Awaiting requested review from AlfredoG87

@mattp-swirldslabs mattp-swirldslabs Awaiting requested review from mattp-swirldslabs

Assignees

@georgi-l95 georgi-l95

Labels
Simulator Issue related to Block Stream Simulator
Projects
None yet
Milestone
0.3.0
Development

Successfully merging this pull request may close these issues.

feat: Update Consumer Simulator to be more memory efficient
3 participants
@georgi-l95 @ata-nas @mattp-swirldslabs