Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NWB revisions - PR 2 of 3 #71

Merged
merged 157 commits into from
Oct 19, 2022
Merged
Changes from 1 commit
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
f740aef
added NWB export
Aug 18, 2021
042cc46
copied `ephys` to `ephys-no-curation`, added `recording_duration`, make
Sep 19, 2021
a39a9b1
remove `Curation`
Sep 19, 2021
590310e
make `clustering_output_dir` user-input optional, auto infer
Sep 19, 2021
adfad95
minor bug fix
Sep 19, 2021
b7c0845
improve kilosort loading routine - add `validate()` method
Sep 20, 2021
de84ce0
helper for `ProbeInsertion` - `auto_generate_entries(session_key)`
Sep 20, 2021
4397dd7
no-curation, store processed data in user-specified `processed_data_d…
Sep 21, 2021
5420ae0
using `find_full_path` for session dir as well - improve robustness
Sep 21, 2021
70e93b3
Update ephys_no_curation.py
Sep 21, 2021
a7e7554
bugfix
Sep 22, 2021
0d56456
automate ClusteringTask insertion
Sep 23, 2021
047bfa1
triggering Kilosort with ecephys_spike_sorting package
Sep 30, 2021
60105da
remove `_required_packages_paths`
Sep 30, 2021
a9326ea
minor stylistic improvements
Sep 30, 2021
149ef3f
Apply suggestions from code review
Sep 30, 2021
9f98195
typo fix
Oct 1, 2021
a24bd1a
Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…
Oct 1, 2021
ba28637
handles a weird windows/unix path incompatibility (even with pathlib)
Oct 1, 2021
5cdc994
fix missing `clustering_method`
Oct 1, 2021
6c646bb
flag to create spike sorting output dir
Oct 1, 2021
dd01fd2
Update kilosort_trigger.py
Oct 1, 2021
f699ce7
minor tweak/improvements in kilosort triggering
Oct 1, 2021
6bf0eb1
bugfix in triggering ecephys_spike_sorting
Oct 1, 2021
6788180
handle cases where `fileTimeSecs` is not available
Oct 1, 2021
d63561f
bugfix
Oct 2, 2021
375e437
missing generate module json
Oct 2, 2021
c593baf
change default `noise_template_use_rf` to False
Oct 2, 2021
69c5e51
minor bugfix
Oct 4, 2021
a6cae12
add kilosort 2.5 as default content
Oct 4, 2021
a0ea9f7
make variable naming consistent
Oct 4, 2021
acdab12
include `clustering_method` into the calculation of `param_set_hash`
Oct 4, 2021
38c5be6
updating `kilosort_repository` depending on which KSVer to be used
Oct 4, 2021
28c4452
bugfix
Oct 4, 2021
94aade7
bugfix, convert path to string
Oct 4, 2021
ae966aa
log the folder creation
Oct 5, 2021
b9f4e92
add debugging lines
Oct 5, 2021
044c389
bugfix - no dir created
Oct 5, 2021
0f0c212
Update kilosort.py
Oct 6, 2021
67341d8
new version 0.1.0b1
Oct 6, 2021
c2ee64f
improve clusteringtask and waveform ingestion routine
Oct 7, 2021
dc7ddd9
minor cleanup
Oct 7, 2021
408532c
stage tracking and resumable kilosort run
Oct 7, 2021
55bec01
minor bugfix
Oct 7, 2021
f34e95d
bugfix logging for kilosort triggering
Oct 7, 2021
19b704b
duration for each module run
Oct 7, 2021
beaf765
check `ap.bin` file validity before triggering kilosort (based on fil…
Oct 8, 2021
74e3ead
Apply suggestions from code review
Oct 11, 2021
e8ffe17
stylistic improvements, addressing code review comments
Oct 11, 2021
d07f830
handle NWB export with multiple curated clusterings from one session
Nov 2, 2021
19616ef
Update nwb.py
Nov 2, 2021
09c1e60
minor bugfix
Nov 2, 2021
3ebdf23
Merge pull request #2 from ttngu207/nwb-export
Nov 2, 2021
b25f065
add nwb export to `no-curation` ephys
Nov 2, 2021
49ca0be
first prototype for running the ecephys_pipeline with OpenEphys
Nov 9, 2021
199a2ba
bugfix for running kilosort for Open Ephys data
Nov 9, 2021
58b5984
specify additional recording-info as part of the `params`
Nov 11, 2021
ddc3b94
add `neuropixels UHD` probe type
Nov 18, 2021
df599fb
triggering kilosort analysis for open-ephys
Nov 18, 2021
819ff19
first prototype for pykilosort
Nov 19, 2021
81d99c8
use_C_waves=False for OpenEphys
Nov 19, 2021
96931a4
using fork of pyopenephys
Nov 22, 2021
b6f8f99
minor bugfix in running pykilosort
Nov 22, 2021
4250220
update openephys loader
Nov 22, 2021
af2b18b
fix package requirement formatting error
Nov 22, 2021
e325a30
handles format differences between npx1 vs 3A
Nov 22, 2021
db3027b
supporting `neuropixels UHD` in `ephys.EphysRecording`
Nov 23, 2021
f55a6a7
bugfix in electrode sites design for Neuropixels UHD probe
Nov 23, 2021
d105419
search recording channels for Open Ephys based on channel names
Nov 23, 2021
0d16e7e
bugfix openephys loader
Nov 23, 2021
c213325
bugfix - extract recording datetime (instead of using experiment date…
Nov 23, 2021
adffe34
bugfix, timedelta as seconds
Nov 23, 2021
43d6614
handles extraction of `connected` channels for NP_PROBE format in Ope…
Nov 23, 2021
da10c66
clusters extraction - check `cluster_group.tsv` and `cluster_KSLabel.…
Nov 25, 2021
02069c9
bugfix in triggering pykilosort
Nov 25, 2021
142459d
handle missing `sample_rate` from pykilosort params.py
Nov 25, 2021
1c39568
for pykilosort's probe, provide both Nchan and NchanTOT
Nov 25, 2021
2022e91
openephys loader - handles signalchain and processor as single elemen…
Nov 26, 2021
02c4b67
handles multi-probe for older OE version
Nov 26, 2021
8dff08b
NWB export fix, specifying which ephys module
Nov 29, 2021
6107e8e
version bump
Nov 29, 2021
dae36d1
fix NWB export - null `waveforms` - back to version 0.1.0b1
Nov 29, 2021
8657d58
version bump
Nov 29, 2021
aa47c8a
include probe as part of the electrode_group name for uniqueness
Nov 29, 2021
31bba8c
bugfix in assigning unit electrode indices
ttngu207 Dec 7, 2021
f9a4754
Merge pull request #3 from ttngu207/nwb-export
Dec 7, 2021
f2881ca
all three ephys files up to date
A-Baji Jan 24, 2022
c56cd18
applied requested changes
A-Baji Jan 24, 2022
9dd6b42
Merge pull request #4 from A-Baji/no-curation
Jan 24, 2022
4951b39
DEPRECATING NWB EXPORT
Jan 26, 2022
b4832ea
Merge branch 'main' of https://github.com/datajoint/element-array-eph…
Jan 26, 2022
d66368c
minor bugfix
Jan 27, 2022
779b2fb
name tweak
Jan 27, 2022
f4052cc
Apply suggestions from code review
Jan 27, 2022
03cab02
rename `sess_dir` -> `session_dir`
Jan 31, 2022
cb1a041
Update requirements.txt
Jan 31, 2022
7320f9f
Update requirements.txt
Jan 31, 2022
f0b3d4a
Update requirements.txt
Jan 31, 2022
bdcfa46
handle older open ephys format for single probe
Feb 18, 2022
83649f5
remove unintended prototyping work
Feb 18, 2022
c2f8aea
delete nwb export - rename `ephys` module -> `ephys_acute`
Feb 25, 2022
e0a9a4f
Update __init__.py
Mar 14, 2022
769de13
bugfix
Mar 16, 2022
39c8579
configurable `paramset_idx` for auto ClusteringTask generation
Mar 25, 2022
85c7c8b
Update openephys.py
May 12, 2022
11a12ba
update openephys loader - handle new open ephys format
May 20, 2022
cd5fe70
handles new probe naming in latest Open Ephys format
May 20, 2022
af864d7
Apply suggestions from code review
May 23, 2022
47babf3
added assertion - safeguard against failed loading of continuous.dat
May 23, 2022
7fecff1
Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…
May 23, 2022
ddd4095
Merge remote-tracking branch 'upstream/run_kilosort' into no-curation
May 23, 2022
364f80e
Merge pull request #69 from ttngu207/no-curation
kabilar May 23, 2022
a578d85
Update requirements.txt
May 23, 2022
f4b917d
`kilosort2` also as part of the `contents` for ClusteringMethod
May 23, 2022
f11e016
bugfix for LFP electrode mapping
May 24, 2022
747c15f
Merge pull request #70 from ttngu207/no-curation
sidhulyalkar May 24, 2022
49bba8a
WIP: nwb bugfix
CBroz1 May 25, 2022
b62fd12
WIP: NWB fix - prevent SQL err by restricting key
CBroz1 May 26, 2022
07604e2
update open ephys loader to handle "STREAM" in latest format
May 26, 2022
e1c9b35
WIP: nwb line length, Readme mention of
CBroz1 May 31, 2022
4e367d7
added loading of electrode location for new openephys format
Jun 1, 2022
9272ee6
Merge pull request #73 from ttngu207/no-curation
sidhulyalkar Jun 1, 2022
67039ac
more robust loading of openephys format
Jun 3, 2022
d298b07
Merge pull request #77 from ttngu207/no-curation
Jun 3, 2022
720e355
Update README.md
CBroz1 Jun 3, 2022
9a59e57
improve logic for running kilosort modules in a resumable fashion
Jun 7, 2022
b97566e
code cleanup, minor bugfix
Jun 7, 2022
3f1ee37
bugfix, match new implementation for openephys
Jun 9, 2022
4a3e0bf
Issue #11
CBroz1 Jun 15, 2022
bd6d7e4
Merge branch 'rk' of https://github.com/CBroz1/element-array-ephys in…
CBroz1 Jun 15, 2022
d102f6f
Issue #63
CBroz1 Jun 15, 2022
e43e5d5
Add Code of Conduct
CBroz1 Jun 16, 2022
70e0b1c
Add contact info to Code of Conduct
CBroz1 Jun 17, 2022
0c77826
improve kilosort calls, handle spaces in paths
Jun 30, 2022
b71b459
remove space escaping character
Jun 30, 2022
a3c5c2f
improve error message
Jul 19, 2022
fd331bd
Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…
Jul 19, 2022
13d74ad
Merge branch 'run_kilosort' of https://github.com/datajoint/element-a…
CBroz1 Jul 21, 2022
0221848
WIP: version bump pynwb to 2.0
CBroz1 Jul 21, 2022
d9c3887
Merge pull request #86 from CBroz1/rk
kabilar Jul 22, 2022
ad8436e
Merge pull request #88 from ttngu207/no-curation
kabilar Jul 25, 2022
b2bd0ee
implement data compression using `mtscomp` for openephys and spikeglx…
Jul 25, 2022
97f3d21
garbage collect openephys objects
Aug 12, 2022
d8aea04
garbage collect openephys objects
Aug 15, 2022
c153e7f
Merge pull request #89 from ttngu207/no-curation
sidhulyalkar Aug 15, 2022
f9f18d0
Update kilosort_triggering.py
Aug 30, 2022
5905392
bugfix in running catgt
Sep 2, 2022
987231b
enable catgt run
Sep 2, 2022
aed42ca
bugfix
Sep 2, 2022
fbdbe24
bugfix
Sep 2, 2022
35da39b
`missing_ok` arg only available in python 3.8+
Sep 2, 2022
c33d1b0
bugfix - path search for catgt output
Sep 2, 2022
f60ba3d
improve error handling
Sep 2, 2022
23ca7ca
Merge pull request #90 from ttngu207/no-curation
Sep 2, 2022
71d87ae
Update kilosort_triggering.py
Sep 14, 2022
6757ef7
Merge pull request #91 from ttngu207/no-curation
Sep 14, 2022
0e94252
Ensure Path type for get_spikeglx_meta_filepath
CBroz1 Sep 15, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
bugfix openephys loader
Thinh Nguyen committed Nov 23, 2021
commit 0d16e7ed61ec7911d97db7077b6f11498709cf73
5 changes: 3 additions & 2 deletions element_array_ephys/ephys_no_curation.py
Original file line number Diff line number Diff line change
@@ -385,7 +385,7 @@ def make(self, key):
oe_probe = get_openephys_probe_data(key)

lfp_channel_ind = np.r_[
len(oe_probe.lfp_meta['channels_ids'])-1:0:-self._skip_channel_counts]
len(oe_probe.lfp_meta['channels_indices'])-1:0:-self._skip_channel_counts]

lfp = oe_probe.lfp_timeseries[:, lfp_channel_ind] # (sample x channel)
lfp = (lfp * np.array(oe_probe.lfp_meta['channels_gains'])[lfp_channel_ind]).T # (channel x sample)
@@ -402,7 +402,8 @@ def make(self, key):
probe_electrodes = {key['electrode']: key
for key in electrode_query.fetch('KEY')}

electrode_keys.extend(probe_electrodes[channel_idx] for channel_idx in oe_probe.lfp_meta['channels_ids'])
electrode_keys.extend(probe_electrodes[channel_idx]
for channel_idx in oe_probe.lfp_meta['channels_indices'])
else:
raise NotImplementedError(f'LFP extraction from acquisition software'
f' of type {acq_software} is not yet implemented')
2 changes: 1 addition & 1 deletion element_array_ephys/readers/openephys.py
Original file line number Diff line number Diff line change
@@ -102,7 +102,7 @@ def load_probe_data(self):

meta.update(**continuous_info,
channels_indices=channels_indices,
channels_ids=analog_signal.channels_ids,
channels_ids=analog_signal.channel_ids,
channels_names=analog_signal.channel_names,
channels_gains=analog_signal.gains)