No curation ephys AND kilosort triggering #36

ttngu207 · 2021-09-30T17:22:21Z

Fix #34
Fix #40
Fix #37

…ir` if provided

dimitri-yatsenko · 2021-09-30T17:58:56Z

element_array_ephys/ephys_no_curation.py

+        # search session dir and determine acquisition software
+        for ephys_pattern, ephys_acq_type in zip(['*.ap.meta', '*.oebin'],
+                                                 ['SpikeGLX', 'Open Ephys']):
+            ephys_meta_filepaths = [fp for fp in sess_dir.rglob(ephys_pattern)]


Suggested change

ephys_meta_filepaths = [fp for fp in sess_dir.rglob(ephys_pattern)]

ephys_meta_filepaths = list(sess_dir.rglob(ephys_pattern))

dimitri-yatsenko · 2021-09-30T18:00:19Z

element_array_ephys/ephys_no_curation.py

+        for ephys_pattern, ephys_acq_type in zip(['*.ap.meta', '*.oebin'],
+                                                 ['SpikeGLX', 'Open Ephys']):
+            ephys_meta_filepaths = [fp for fp in sess_dir.rglob(ephys_pattern)]
+            if len(ephys_meta_filepaths):


Suggested change

if len(ephys_meta_filepaths):

if ephys_meta_filepaths:

element_array_ephys/ephys_no_curation.py

Co-authored-by: Dimitri Yatsenko <[email protected]>

…ay-ephys into no-curation

…time)

…nEphys

…tsv`

…t or list

bendichter · 2021-11-30T01:20:29Z

element_array_ephys/export/nwb.py

+
+    # add additional columns to the electrodes table
+    electrodes_query = probe.ProbeType.Electrode * probe.ElectrodeConfig.Electrode
+    for additional_attribute in ['shank', 'shank_col', 'shank_row']:


shank would probably map to electrode_group

bendichter · 2021-11-30T01:21:15Z

element_array_ephys/export/nwb.py

+    # add additional columns to the units table
+    units_query = ephys.EphysRecording * ephys.ClusteringTask @ ephys.CuratedClustering.Unit
+    for additional_attribute in ['cluster_quality_label', 'spike_count', 'sampling_rate',
+                                 'spike_times', 'spike_sites', 'spike_depths']:


spike_sites would probably map to electrodes

bendichter · 2021-11-30T01:22:01Z

element_array_ephys/export/nwb.py

+                        if ephys_device_name in nwbfile.devices
+                        else nwbfile.create_device(name=ephys_device_name))
+
+        electrode_group = nwbfile.create_electrode_group(


I would create one electrode group per shank

bendichter · 2021-11-30T01:25:23Z

element_array_ephys/export/nwb.py

+
+        # ---- Local Field Potential ----
+        if ephys.LFP.Electrode & curated_clustering_key:
+            nwb_lfp = pynwb.ecephys.LFP(name=f'probe_{electrode_config["probe"]} - LFP')


I don't know if you need to name this according to a probe. The underlying ElectricalSeries will have an electrodes attribute that points to a subregion of the electrodes table

bendichter · 2021-11-30T01:26:20Z

element_array_ephys/export/nwb.py

+
+            nwb_lfp.create_electrical_series(
+                name='processed_electrical_series',
+                data=lfp_data * 1e-6,  # convert to Volts


don't convert to volts, especially if the data is ints. Instead, include 1e-6 as a conversion factor.

bendichter · 2021-11-30T01:27:10Z

element_array_ephys/probe.py

+        neuropixels_probes_config = {
+            'neuropixels 1.0 - 3A': dict(site_count=960, col_spacing=32, row_spacing=20,
+                                         white_spacing=16, col_count=2,
+                                         shank_count=1, shank_spacing=0),
+            'neuropixels 1.0 - 3B': dict(site_count=960, col_spacing=32, row_spacing=20,
+                                         white_spacing=16, col_count=2,
+                                         shank_count=1, shank_spacing=0),
+            'neuropixels UHD': dict(site_count=384, col_spacing=6, row_spacing=6,
+                                    white_spacing=0, col_count=8,
+                                    shank_count=1, shank_spacing=0),
+            'neuropixels 2.0 - SS': dict(site_count=1280, col_spacing=32, row_spacing=15,
+                                         white_spacing=0, col_count=2,
+                                         shank_count=1, shank_spacing=250),
+            'neuropixels 2.0 - MS': dict(site_count=1280, col_spacing=32, row_spacing=15,
+                                         white_spacing=0, col_count=2,
+                                         shank_count=4, shank_spacing=250)
+        }


It might be better to use ProbeInterface for this

bendichter · 2021-11-30T20:55:02Z

element_array_ephys/export/nwb.py

+    electrodes_query = probe.ProbeType.Electrode * probe.ElectrodeConfig.Electrode
+    for additional_attribute in ['shank', 'shank_col', 'shank_row']:
+        nwbfile.add_electrode_column(
+            name=electrodes_query.heading.attributes[additional_attribute].name,


A column named "name" might be illegal. Could we change this to "label"?

bendichter · 2021-11-30T20:55:42Z

element_array_ephys/export/nwb.py

+    for additional_attribute in ['cluster_quality_label', 'spike_count', 'sampling_rate',
+                                 'spike_times', 'spike_sites', 'spike_depths']:
+        nwbfile.add_unit_column(
+            name=units_query.heading.attributes[additional_attribute].name,


here again, "name" might be an illegal name for a column

CBroz1 · 2021-12-03T01:21:01Z

element_array_ephys/export/nwb.py

+    # add additional columns to the units table
+    units_query = ephys.EphysRecording * ephys.ClusteringTask @ ephys.CuratedClustering.Unit
+    for additional_attribute in ['cluster_quality_label', 'spike_count', 'sampling_rate',
+                                 'spike_times', 'spike_sites', 'spike_depths']:


Getting the following error regarding spike times:

UserWarning: Column 'spike_times' is predefined in Units with index=True which does not match the entered index argument. The predefined index spec will be ignored. Please ensure the new column complies with the spec. This will raise an error in a future version of HDMF.
warn(msg)

You don't need to add 'spike_times', it is by default part of the Units table. Just omit "spike_times" this from the add_unit_column calls and keep it in the add_unit calls

bugfix in assigning unit electrode indices

ttngu207 · 2022-01-20T22:28:03Z

I suggest closing this PR, removing NWB export and issue a new PR.
As NWB export is implemented and PR'ed on #44 .

A fresh PR would also make it easier to review.
Thoughts @kabilar @CBroz1 ?

kabilar · 2022-01-21T23:20:47Z

Hi @ttngu207,

Happy to review in this pull request or a new one. You're right. We should remove the NWB export from the pull request. I will review this work next week.

all three ephys files up to date

ttngu207 · 2022-01-26T20:19:37Z

Per discussion with @kabilar, I'm closing this PR, removing the NWB export, and reissue as new PR

Thinh Nguyen added 14 commits August 18, 2021 15:16

added NWB export

f740aef

copied ephys to ephys-no-curation, added recording_duration, make

042cc46

remove Curation

a39a9b1

make clustering_output_dir user-input optional, auto infer

590310e

minor bug fix

adfad95

improve kilosort loading routine - add validate() method

b7c0845

helper for ProbeInsertion - auto_generate_entries(session_key)

de84ce0

no-curation, store processed data in user-specified `processed_data_d…

4397dd7

…ir` if provided

using find_full_path for session dir as well - improve robustness

5420ae0

Update ephys_no_curation.py

70e93b3

bugfix

a7e7554

automate ClusteringTask insertion

0d56456

triggering Kilosort with ecephys_spike_sorting package

047bfa1

remove _required_packages_paths

60105da

ttngu207 requested review from kabilar, dimitri-yatsenko, iamamutt and yambottle September 30, 2021 17:22

dimitri-yatsenko reviewed Sep 30, 2021

View reviewed changes

minor stylistic improvements

a9326ea

dimitri-yatsenko reviewed Sep 30, 2021

View reviewed changes

element_array_ephys/ephys_no_curation.py Outdated Show resolved Hide resolved

dimitri-yatsenko reviewed Sep 30, 2021

View reviewed changes

element_array_ephys/ephys_no_curation.py Outdated Show resolved Hide resolved

dimitri-yatsenko reviewed Sep 30, 2021

View reviewed changes

element_array_ephys/ephys_no_curation.py Outdated Show resolved Hide resolved

dimitri-yatsenko reviewed Sep 30, 2021

View reviewed changes

element_array_ephys/ephys_no_curation.py Outdated Show resolved Hide resolved

Thinh Nguyen and others added 5 commits September 30, 2021 13:56

Apply suggestions from code review

149ef3f

Co-authored-by: Dimitri Yatsenko <[email protected]>

typo fix

9f98195

Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…

a24bd1a

…ay-ephys into no-curation

handles a weird windows/unix path incompatibility (even with pathlib)

ba28637

fix missing clustering_method

5cdc994

Thinh Nguyen added 14 commits November 23, 2021 14:44

bugfix - extract recording datetime (instead of using experiment date…

c213325

…time)

bugfix, timedelta as seconds

adffe34

handles extraction of connected channels for NP_PROBE format in Ope…

43d6614

…nEphys

clusters extraction - check cluster_group.tsv and `cluster_KSLabel.…

da10c66

…tsv`

bugfix in triggering pykilosort

02069c9

handle missing sample_rate from pykilosort params.py

142459d

for pykilosort's probe, provide both Nchan and NchanTOT

1c39568

openephys loader - handles signalchain and processor as single elemen…

2022e91

…t or list

handles multi-probe for older OE version

02c4b67

NWB export fix, specifying which ephys module

8dff08b

version bump

6107e8e

fix NWB export - null waveforms - back to version 0.1.0b1

dae36d1

version bump

8657d58

include probe as part of the electrode_group name for uniqueness

aa47c8a

bendichter reviewed Nov 30, 2021

View reviewed changes

CBroz1 reviewed Dec 3, 2021

View reviewed changes

CBroz1 mentioned this pull request Dec 3, 2021

Create notebook for example usage of NWB export datajoint/workflow-array-ephys#28

Closed

ttngu207 and others added 2 commits December 7, 2021 15:22

bugfix in assigning unit electrode indices

31bba8c

Merge pull request #3 from ttngu207/nwb-export

f9a4754

bugfix in assigning unit electrode indices

kabilar assigned kabilar and iamamutt Dec 21, 2021

A-Baji and others added 3 commits January 24, 2022 10:42

all three ephys files up to date

f2881ca

applied requested changes

c56cd18

Merge pull request #4 from A-Baji/no-curation

9dd6b42

all three ephys files up to date

ttngu207 closed this Jan 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No curation ephys AND kilosort triggering #36

No curation ephys AND kilosort triggering #36

ttngu207 commented Sep 30, 2021 •

edited

Loading

dimitri-yatsenko Sep 30, 2021

dimitri-yatsenko Sep 30, 2021

bendichter Nov 30, 2021

bendichter Nov 30, 2021

bendichter Nov 30, 2021

bendichter Nov 30, 2021

bendichter Nov 30, 2021

bendichter Nov 30, 2021

bendichter Nov 30, 2021

bendichter Nov 30, 2021

CBroz1 Dec 3, 2021

bendichter Dec 3, 2021

ttngu207 commented Jan 20, 2022

kabilar commented Jan 21, 2022

ttngu207 commented Jan 26, 2022

	ephys_meta_filepaths = [fp for fp in sess_dir.rglob(ephys_pattern)]
	ephys_meta_filepaths = list(sess_dir.rglob(ephys_pattern))

No curation ephys AND kilosort triggering #36

No curation ephys AND kilosort triggering #36

Conversation

ttngu207 commented Sep 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ttngu207 commented Jan 20, 2022

kabilar commented Jan 21, 2022

ttngu207 commented Jan 26, 2022

ttngu207 commented Sep 30, 2021 •

edited

Loading