RandomForest def predictRFStat(rf, X, statDictionary,n_jobs) optimization (memory and CPU)

[miranov25]

The memory consumption in the predictRFStat is too big because all trees are evaluated for all of the points

2 changes are needed:

• running in chunks -to keep memory in limit
  • parameters chunkSize or maxMemory
• optimization of the CPU - statistics should also be calculated in parallel
  • Currently only trees are calculated in parallel

There are 2 implementations of the predictRFStat, predictRFStatNew . This was a quick hack after scikit changed some implementation. The hack was not documented and remained in the code.

As the transition happened 6 month ago, maybe we can make New version default and remove the obsolete version...

[miranov25]

Start with predictRFStatPar so we can compare it with the previous version and later replace it ...

Benchmark:

• CPU time
• Memory
• results

[miranov25]

To save memory the predictRFStat should be optionally split to the chunk
If we consider that to be a function responsibility of the function - new parameter maxRows to be added

If maxRow is set then the function will be called in chunks with maxRows and all data will be merged at the end.

Memory consumption - nRowsxnTrees which can be significantly smaller than current nEntries x nTrees