Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable meta parallel Circuitscape - call Circuitscape on several INI files at once with 1 process each #248

Closed
wants to merge 5 commits into from
Closed

Enable meta parallel Circuitscape - call Circuitscape on several INI files at once with 1 process each #248

wants to merge 5 commits into from

Conversation

ranjanan
Copy link
Member

@ranjanan ranjanan commented Jun 16, 2020 •
edited
Loading

Should fix #236

@ranjanan
Copy link
Member Author

@VLucet and @vlandau could you check if this PR works for a list of INI files?

@VLucet
Copy link
Contributor

VLucet commented Jun 17, 2020

Hi @ranjanan, thanks for enabling this. One question that comes to mind: could julia recognize that the input is an array when calling compute and dispatch to compute_metaparallel, and would that be a good design idea or not?

In all cases, I gave it a try, and first, I run into the same problem than I mentioned to @ViralBShah in #236 , when Circuitscape is only installed in the project environment (and not in the default environment), I get the following error, presumably because the workers inherit the default environment (which does not see Circuitscape).

julia> compute_metaparallel(ini_list, 5)
[ Info: 2020-06-17 21:37:32 : Starting up 5 processes
ERROR: On worker 2:
ArgumentError: Package Circuitscape not found in current path:
- Run `import Pkg; Pkg.add("Circuitscape")` to install the Circuitscape package.

require at ./loading.jl:892
eval at ./boot.jl:331
top-level scope at none:0
eval at ./boot.jl:331
#101 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:290
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:88
#94 at ./task.jl:358

...and 4 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:316
 [2] macro expansion at ./task.jl:335 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/macros.jl:217
 [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/macros.jl:201 [inlined]
 [5] myaddprocs(::Int64) at /opt/julia/packages/Circuitscape/z0iGj/src/utils.jl:146
 [6] compute_metaparallel(::Array{String,1}, ::Int64) at /opt/julia/packages/Circuitscape/z0iGj/src/utils.jl:110
 [7] top-level scope at REPL[8]:1

Once the same version of circuitscape is installed (I made sure to install fro the same sha on the last commit on RA/metapar) in the default environment, it works fine :

julia> compute_metaparallel(ini_list, 5)
[ Info: 2020-06-17 21:46:20 : Starting up 5 processes
[ Info: 2020-06-17 21:46:32 : Launching 29 Circuitscape jobs among 5 processes
[ Info: 2020-06-17 21:48:08 : Precision used: Double
[ Info: 2020-06-17 21:48:08 : Precision used: Double
[ Info: 2020-06-17 21:48:10 : Precision used: Double
[ Info: 2020-06-17 21:48:10 : Precision used: Double
[ Info: 2020-06-17 21:48:10 : Precision used: Double
[ Info: 2020-06-17 21:48:16 : Reading maps
[ Info: 2020-06-17 21:48:17 : Reading maps
[ Info: 2020-06-17 21:48:18 : Reading maps
[ Info: 2020-06-17 21:48:19 : Reading maps
[ Info: 2020-06-17 21:48:19 : Reading maps
[ Info: 2020-06-17 21:48:21 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:22 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:23 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:24 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:24 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:34 : Total number of pair solves = 1
[ Info: 2020-06-17 21:48:34 : Solving pair 1 of 1
[ Info: 2020-06-17 21:49:01 : Solver used: AMG accelerated by CG
[ Info: 2020-06-17 21:49:01 : Graph has 6239984 nodes, 2 focal points and 1 connected components
[ Info: 2020-06-17 21:49:16 : Time taken to construct preconditioner = 13.576540371 seconds
[...]

However I then downgraded Circuitscape to the release version in the default environment and it appears that when the versions in the project and in the default environment do not coincide, we have errors. I know that sounds like I might have done something wrong.

julia> compute_metaparallel(ini_list, 5)
[ Info: 2020-06-17 22:12:06 : Starting up 5 processes
[ Info: 2020-06-17 22:12:17 : Launching 29 Circuitscape jobs among 5 processes
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Reading maps
ERROR: On worker 2:
Base.InvalidCharError{Char}('\xe6')
invalid_char at ./char.jl:85
UInt32 at ./char.jl:130
convert at ./char.jl:180 [inlined]
cconvert at ./essentials.jl:390 [inlined]
lowercase at ./strings/unicode.jl:245
map at ./strings/basic.jl:574
lowercase at ./strings/unicode.jl:531 [inlined]
_guess_file_type at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:153
_ascii_grid_read_header at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:113
_ascii_grid_reader at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:90
read_cellmap at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:68
load_raster_data at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:410
raster_pairwise at /opt/julia/packages/Circuitscape/3Rn8u/src/raster/pairwise.jl:18
_compute at /opt/julia/packages/Circuitscape/3Rn8u/src/run.jl:42
macro expansion at ./util.jl:234 [inlined]
compute at /opt/julia/packages/Circuitscape/3Rn8u/src/run.jl:73
#104 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:294
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:79
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:294 [inlined]
#103 at ./task.jl:358
Stacktrace:
 [1] (::Base.var"#726#728")(::Task) at ./asyncmap.jl:178
 [2] foreach(::Base.var"#726#728", ::Array{Any,1}) at ./abstractarray.jl:1919
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::Array{Dict{String,String},1}) at ./asyncmap.jl:178
 [4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#204#207"{Distributed.WorkerPool}, ::Function, ::Array{Dict{String,String},1}) at ./asyncmap.jl:154
 [5] async_usemap(::Distributed.var"#188#190"{Distributed.var"#188#189#191"{Distributed.WorkerPool,typeof(compute)}}, ::Array{Dict{String,String},1}; ntasks::Function, batch_size::Nothing) at ./asyncmap.jl:103
 [6] #asyncmap#710 at ./asyncmap.jl:81 [inlined]
 [7] pmap(::Function, ::Distributed.WorkerPool, ::Array{Dict{String,String},1}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Array{Any,1}, retry_check::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:126
 [8] pmap(::Function, ::Distributed.WorkerPool, ::Array{Dict{String,String},1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:101
 [9] pmap(::Function, ::Array{Dict{String,String},1}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:156
 [10] pmap at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:156 [inlined]
 [11] compute_metaparallel(::Array{String,1}, ::Int64) at /opt/julia/packages/Circuitscape/z0iGj/src/utils.jl:120
 [12] top-level scope at REPL[12]:1

@vlandau
Copy link
Member

vlandau commented Jun 17, 2020

could julia recognize that the input is an array when calling compute

I had to create a customized function that accepts arrays for Omniscape. See Omniscape.jl/src/functions.jl#L306. Note that is specifically for advanced mode. You may have to create a similar function for your use case.

@ViralBShah ViralBShah marked this pull request as draft August 11, 2023 16:19
@ViralBShah ViralBShah closed this Jul 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Best way to paralellize Circuitscape?
4 participants
@ranjanan @VLucet @vlandau @ViralBShah