Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add multithreading to speed up #9

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

add multithreading to speed up #9

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e2cd509
add multithreading to speed up
rainsummer613 May 19, 2024
9b3d522
Merge branch 'main' into multiproc
vladdez May 19, 2024
eb9a3e5
remove inplace changes where multithreading
rainsummer613 May 19, 2024
d0b06a6
remove inplace change from one more function
rainsummer613 May 19, 2024
5af4bb5
remove unnecessary variable d_emp in mult_chan_pattern_detector_proba…
rainsummer613 May 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 12 additions & 11 deletions src/pattern_detection.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,11 @@ function slow_filter(img)
return filtered_data
end

function fast_filter!(dat_filtered, kernel, dat) #

function fast_filter(kernel, dat) #
#r = Images.ImageFiltering.ComputationalResources.CPU1(Images.ImageFiltering.FIR())
DSP.filt!(dat_filtered, kernel[1].data.parent, dat)
return dat_filtered
filter_result = DSP.filt(kernel[1].data.parent, dat)
return filter_result
end

function single_chan_pattern_detector(dat, func, evts)
Expand Down Expand Up @@ -42,26 +43,26 @@ function mult_chan_pattern_detector_probability(dat, stat_function, evts; n_perm
d_perm = similar(dat, size(dat, 1), n_permutations)
@debug "starting permutation loop"
# We permute data for all events in advance
for ch = 1:size(dat, 1)
for perm = 1:n_permutations


Threads.@threads for perm = 1:n_permutations
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there might be a problem of race-condition, as we currently filter inplace. That is dat_filtered is now shared between threads and might not indicate a unique permutation. I'm not entirely sure how much performance gain the filtering inplace has over allocating a new array.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(couldnt mark it, but I mean the filter_fast! function below (or whatever it is called exctly)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point!
can you please have a look at the new commits on multiproc branch? eb9a3e5 and d0b06a6

I measured time with and without threads and inplace operations.
inplace+ threading- 240 s
inplace- threading+ 50 s

for ch = 1:size(dat, 1)
sortix = shuffle(1:size(dat_filtered, 1))
d_perm[ch, perm] = stat_function(
fast_filter!(dat_filtered, kernel, @view(dat_padded[ch, sortix, :])),
fast_filter(kernel, @view(dat_padded[ch, sortix, :])),
)
@show ch, perm
end
end
mean_d_perm = mean(d_perm, dims = 2)[:, 1]

for n in names(evts)
Threads.@threads for n in names(evts)
sortix = sortperm(evts[!, n])
col = fill(NaN, size(dat, 1))
for ch = 1:size(dat, 1)
fast_filter!(dat_filtered, kernel, @view(dat_padded[ch, sortix, :]))
d_emp = stat_function(dat_filtered)
col[ch] = abs(stat_function(
fast_filter(kernel, @view(dat_padded[ch, sortix, :]))
) - mean_d_perm[ch])

col[ch] = abs(d_emp - mean_d_perm[ch])
print(ch, " ")
end
println(n)
Expand Down
17 changes: 17 additions & 0 deletions src/runner.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# FOR MULTITHREADING:
# run: >julia -t [n_threads]
# instead of [n_threads] write a desired number of threads (<= amount of CPU cores)

include("setup.jl")
include("pattern_detection.jl")

Expand Down Expand Up @@ -29,6 +33,13 @@ fid = h5open("data/mult.hdf5", "r")
dat2 = read(fid["data"]["mult.hdf5"])
close(fid)

# Data for multiple channels (only fixations)
# 128 channels x 769 time x 2508 events

fid = h5open("data/data_fixations.hdf5", "r")
dat_fix = read(fid["data"]["data_fixations.hdf5"])
close(fid)


# PATTERN DECTECTION 1
# for single channel data
Expand Down Expand Up @@ -65,6 +76,12 @@ evts_init = CSV.read("data/events_init.csv", DataFrame)
evts_d = mult_chan_pattern_detector_probability(dat2[:, :, ix], Images.entropy, evts)
end

# PATTERN DETECTION 4 (FOR FIXATIONS ONLY)
# 10 cores: 50 s
@time begin
evts_d = mult_chan_pattern_detector_probability(dat_fix, Images.entropy, evts)
end

begin
f = Figure()
ax = CairoMakie.Axis(f[1, 1], xlabel = "Channels", ylabel = "Sorting event variables")
Expand Down