add multithreading to speed up #9
Conversation
| for perm = 1:n_permutations | ||
|
|
||
|
|
||
| Threads.@threads for perm = 1:n_permutations |
There was a problem hiding this comment.
there might be a problem of race-condition, as we currently filter inplace. That is dat_filtered is now shared between threads and might not indicate a unique permutation. I'm not entirely sure how much performance gain the filtering inplace has over allocating a new array.
There was a problem hiding this comment.
(couldnt mark it, but I mean the filter_fast! function below (or whatever it is called exctly)
|
work on inplace changes of dat |
|
hey, nice work there. Can you check with e.g. https://github.com/LilithHafner/Chairmarks.jl the performance a bit more systematic? I would be interested especially in the filt! vs filt difference, both one-threaded. I'm specifically asking because we could also pre-allocate one array per thread, which is with e.g. 4 threads (is that how many you used?) much smaller than the number of permutations (100 x n_predictors) There are also some threds on discourse if you really want to digg deeper (but I think GPU is the way to go). search "julia threads preallocation" or similar. Also https://github.com/JuliaFolds/FLoops.jl might be a good fit - but it hasnt seen updates in quite some time. |
|
@behinger thank you! I am back from conferences and vacations and have more time now, so I will look into it this week |
|
@behinger 1 thread, inplace changes (fast_filter!) 198 s I will try to look into Floops too, if neccessary |
|
cool! interesting that the inpace filtering doesnt really help. I think right now best is to discuss with Vlad, we try to sidestep the filtering and just take some (right now even non-overlapping) means instead |
Use Threads to run fast_filter across permutations and channels in parallel: Issue #7