You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When starting data extraction on a large block range or on a job handling large amounts of data, the program will consume all the available memory, possibly rendering the host PC irresponsive if left for too long.
This hasn't been tested extensively but is visible with any sizable workload sent to the tool.
Proposition
Several safeguards could be put in place to prevent this (ranging from most amount of work to least):
Monitor resources used, warn and possibly take actions (pause asyncio tasks, send processed blocks to post-process, save state, etc.) when certain threshold are attained.
Just warn in case of high memory usage detected (so still include monitoring from the first option).
Warn preemptively by judging the size of the workload at the beginning.
Only warn in the documentation.
The text was updated successfully, but these errors were encountered:
Problem
When starting data extraction on a large block range or on a job handling large amounts of data, the program will consume all the available memory, possibly rendering the host PC irresponsive if left for too long.
This hasn't been tested extensively but is visible with any sizable workload sent to the tool.
Proposition
Several safeguards could be put in place to prevent this (ranging from most amount of work to least):
asyncio
tasks, send processed blocks to post-process, save state, etc.) when certain threshold are attained.The text was updated successfully, but these errors were encountered: