Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functions used for Code Ocean pipeline #10

Open
wants to merge 85 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
2fd57b7
Use spikeinterface functions for DeepInteprolation training and infer…
alejoe91 Jul 11, 2023
a9072c4
Add minimal requirements
alejoe91 Jul 11, 2023
38e4128
Add separate scripts
alejoe91 Jul 14, 2023
976a73c
Create pipeline folder
alejoe91 Jul 18, 2023
ead2343
update run_spike_sorting
alejoe91 Jul 18, 2023
8594f49
Add generate_job_config function
alejoe91 Jul 18, 2023
694eaa0
Propagate JSON files to output
alejoe91 Jul 18, 2023
92e91f0
Reduce validation interval to 100ms
alejoe91 Jul 18, 2023
ece9cf3
Added collect_results script
alejoe91 Jul 18, 2023
44162b3
Change spike sorting output flders
alejoe91 Jul 18, 2023
555d9a9
Remove unused imports
alejoe91 Jul 18, 2023
67d207e
Oups
alejoe91 Jul 18, 2023
3b15043
fix sorting paths
alejoe91 Jul 18, 2023
4cd35c1
fix sorting paths 1
alejoe91 Jul 18, 2023
0a8390a
fix sorting paths 2
alejoe91 Jul 18, 2023
4eb9e45
Debug
alejoe91 Jul 18, 2023
28ed0e7
Debug1
alejoe91 Jul 18, 2023
4445737
Remove results dict
alejoe91 Jul 18, 2023
26636f4
Try to resolve base_path
alejoe91 Jul 19, 2023
60b7f21
Add session to print
alejoe91 Jul 19, 2023
cdde356
Change output folders and add super training
alejoe91 Jul 19, 2023
576ee38
Improve prints
alejoe91 Jul 19, 2023
e1e62e1
finalize super training
alejoe91 Jul 19, 2023
2001656
Fixes
alejoe91 Jul 19, 2023
ded56d1
Change relative paths
alejoe91 Jul 20, 2023
4abf984
Fix model_folder names
alejoe91 Jul 20, 2023
ff149d5
Fix inference sub folders
alejoe91 Jul 20, 2023
4c28a8f
Propagate models to results
alejoe91 Jul 20, 2023
64f4271
Set training verbose to false
alejoe91 Jul 20, 2023
e3e2232
Scale number of GPUs
alejoe91 Jul 20, 2023
4ee7afa
Adjust paths
alejoe91 Jul 20, 2023
203c3f7
Fix collect capsule
alejoe91 Jul 20, 2023
649a237
Fix collect capsule 1
alejoe91 Jul 20, 2023
f71c1dc
Fix collect capsule 2
alejoe91 Jul 20, 2023
2ca3796
Fix nb_gpus
alejoe91 Jul 20, 2023
c88957a
Specify steps per epoch
alejoe91 Jul 20, 2023
606ab18
Add remove_excess_spikes curation
alejoe91 Jul 21, 2023
c78168b
Extend to simulated data
alejoe91 Jul 21, 2023
a1e8f17
Steps per epoch 10 in debug mode
alejoe91 Jul 21, 2023
f3827c4
Move depth order later
alejoe91 Jul 21, 2023
54731b0
Move depth order later 2
alejoe91 Jul 21, 2023
a533616
Oups!
alejoe91 Jul 21, 2023
e065b00
Reintroduce debug in inference
alejoe91 Jul 21, 2023
3eadac0
Fix sorting import
alejoe91 Jul 21, 2023
01b2b0b
Optimize inference
alejoe91 Jul 22, 2023
1e496c1
Save sim to binary
alejoe91 Jul 22, 2023
1de9909
Fix zscore binary
alejoe91 Jul 22, 2023
67f43d9
Fix zscore binary 1
alejoe91 Jul 22, 2023
5d44817
Fix zscore binary 2
alejoe91 Jul 22, 2023
6b4edb7
Fix sorting eval sim
alejoe91 Jul 22, 2023
3a41f68
Fix df concatenation
alejoe91 Jul 24, 2023
60a08f1
Fix debug mode for sorting GT and add inference parallel params
alejoe91 Jul 24, 2023
c89e7ab
Add debug print
alejoe91 Jul 24, 2023
c649d27
Final cmp fix
alejoe91 Jul 24, 2023
218a57a
Add unit id column and sort columns GT
alejoe91 Jul 24, 2023
12d80a8
Add unit id column and sort columns GT 1
alejoe91 Jul 24, 2023
fab66b7
Add unit id column and sort columns GT 2
alejoe91 Jul 24, 2023
e049bb3
Debug paths
alejoe91 Jul 24, 2023
f4c8a0a
Remove resolve
alejoe91 Jul 24, 2023
b3e2257
Remove resolved and update super training
alejoe91 Jul 24, 2023
15a7489
Super-training: add probe option
alejoe91 Jul 24, 2023
3acf580
Don't max out CPU
alejoe91 Jul 24, 2023
9ce301b
Steps per epoch in super-training
alejoe91 Jul 24, 2023
6693240
Add default probeset
alejoe91 Jul 24, 2023
1d57918
Limit n_njobs
alejoe91 Jul 24, 2023
f38240d
Set n_jobs with params
alejoe91 Jul 25, 2023
bf26914
Handle sorting errors in run_sorting function
alejoe91 Jul 31, 2023
c2672ae
Oups
alejoe91 Jul 31, 2023
122707b
Protect against small mismatches in sampling frequency
alejoe91 Aug 2, 2023
96abe6a
Correcly save session level results
alejoe91 Aug 29, 2023
fec2c53
Merge branch 'use-si-functions' of github.com:Jad-Selman/ephys-deepin…
alejoe91 Aug 29, 2023
8067f75
A round of black
alejoe91 Aug 29, 2023
e4d473d
Extend training
alejoe91 Sep 5, 2023
83008fe
Extend unit level and matched unit level results
alejoe91 Sep 15, 2023
05b74f7
wrong argument
alejoe91 Sep 15, 2023
339cc4a
Fixes
alejoe91 Sep 15, 2023
db18cd9
Use scratch folder for waveforms and n_jobs=1 for QM
alejoe91 Sep 15, 2023
8950ad7
Fix data loading in spike sorting pipeline
alejoe91 Sep 15, 2023
9fbb1d9
pre-compute sparsity
alejoe91 Sep 18, 2023
26c50b8
Add filter_option in JSOn and skip NN metrics
alejoe91 Sep 18, 2023
313211a
Cleanup
alejoe91 Sep 18, 2023
84dc31e
Save sparse waveforms in results
alejoe91 Sep 18, 2023
8196427
filter_options -> filter_option
alejoe91 Sep 18, 2023
53754d2
Remove correct folder
alejoe91 Sep 19, 2023
2a4f63b
Add metrics to matched-units csv
alejoe91 Sep 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Limit n_njobs
alejoe91 committed Jul 24, 2023
commit 1d57918dbde2ec7ee2532722dcbb8db79e608339
4 changes: 2 additions & 2 deletions pipeline/run_inference.py
Original file line number Diff line number Diff line change
@@ -37,7 +37,7 @@
##### DEFINE DATASETS AND FOLDERS #######
from sessions import all_sessions_exp, all_sessions_sim

n_jobs = os.cpu_count() - 4
n_jobs = int(0.7 * (os.cpu_count()))

job_kwargs = dict(n_jobs=n_jobs, progress_bar=True, chunk_duration="1s")

@@ -78,7 +78,7 @@
pre_post_omission = 1
desired_shape = (192, 2)
# play around with these
inference_n_jobs = os.cpu_count() - 4
inference_n_jobs = int(0.7 * (os.cpu_count()))
inference_chunk_duration = "1s"
inference_predict_workers = 1
inference_memory_gpu = 2000 # MB