Skip to content
This repository has been archived by the owner on Jul 4, 2024. It is now read-only.

Bugfix: Massive block heights causing OOM with historical block flow #206

Merged

Conversation

hyperschwartz
Copy link
Contributor

Description

When an incredibly large block range is sent into the historical block flow, all of the height values are converted to a list and then immediately chunked, causing an OOM in some circumstances:

java.lang.OutOfMemoryError: Java heap space
	at kotlin.collections.CollectionsKt___CollectionsKt.windowed(_Collections.kt:3333)
	at kotlin.collections.CollectionsKt___CollectionsKt.chunked(_Collections.kt:3115)
	at tech.figure.eventstream.stream.flows.HistoricalBlockHeaderFlowKt$toBlockMeta$1.invokeSuspend(HistoricalBlockHeaderFlow.kt:91)
	at tech.figure.eventstream.stream.flows.HistoricalBlockHeaderFlowKt$toBlockMeta$1.invoke(HistoricalBlockHeaderFlow.kt)
	at tech.figure.eventstream.stream.flows.HistoricalBlockHeaderFlowKt$toBlockMeta$1.invoke(HistoricalBlockHeaderFlow.kt)

This fix creates a sequence of the incoming values for processing, instead of converting the entire input range to a list in the first place.

@hyperschwartz hyperschwartz requested a review from a team as a code owner December 28, 2023 20:04
@hyperschwartz hyperschwartz merged commit 6990f2c into main Dec 28, 2023
1 check passed
@hyperschwartz hyperschwartz deleted the jschwartz/fix-memory-overloads-with-large-historical-flows branch December 28, 2023 20:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants