Skip to content

StatAnalysis Memory and Thresholding Issues #1076

Discussion options

You must be logged in to vote

@lindsayrblank, good news. I found a bug with a simple one-line fix that'll solve this excessive memory use problem.

The void ClimoCDFInfo::set_cdf_ta(int n_bin, bool &center) function fails to initialize the cdf_ta array. Each time we call it, that arrays grows by 21 elements. Your job calls it 22,530 times, once for each combination of station id and variable name. So by the end, it has length 473,130 and we have 22,530 copies of it. That's what's hogging all the memory. Thanks for finding this issue!

Testing with a patch, Stat-Analysis consumes around 2 GB for this job. And technically, it could consume less than 1/4 of that. The issue is the NumArray class... it allocates memory in bl…

Replies: 2 comments 7 replies

Comment options

You must be logged in to vote
3 replies
@lindsayrblank
Comment options

@georgemccabe
Comment options

@JohnHalleyGotway
Comment options

Comment options

You must be logged in to vote
4 replies
@lindsayrblank
Comment options

@JohnHalleyGotway
Comment options

@lindsayrblank
Comment options

@JohnHalleyGotway
Comment options

Answer selected by JohnHalleyGotway
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment