Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NWB revisions - PR 2 of 3 #71

Merged
merged 157 commits into from
Oct 19, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
157 commits
Select commit Hold shift + click to select a range
f740aef
added NWB export
Aug 18, 2021
042cc46
copied `ephys` to `ephys-no-curation`, added `recording_duration`, make
Sep 19, 2021
a39a9b1
remove `Curation`
Sep 19, 2021
590310e
make `clustering_output_dir` user-input optional, auto infer
Sep 19, 2021
adfad95
minor bug fix
Sep 19, 2021
b7c0845
improve kilosort loading routine - add `validate()` method
Sep 20, 2021
de84ce0
helper for `ProbeInsertion` - `auto_generate_entries(session_key)`
Sep 20, 2021
4397dd7
no-curation, store processed data in user-specified `processed_data_d…
Sep 21, 2021
5420ae0
using `find_full_path` for session dir as well - improve robustness
Sep 21, 2021
70e93b3
Update ephys_no_curation.py
Sep 21, 2021
a7e7554
bugfix
Sep 22, 2021
0d56456
automate ClusteringTask insertion
Sep 23, 2021
047bfa1
triggering Kilosort with ecephys_spike_sorting package
Sep 30, 2021
60105da
remove `_required_packages_paths`
Sep 30, 2021
a9326ea
minor stylistic improvements
Sep 30, 2021
149ef3f
Apply suggestions from code review
Sep 30, 2021
9f98195
typo fix
Oct 1, 2021
a24bd1a
Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…
Oct 1, 2021
ba28637
handles a weird windows/unix path incompatibility (even with pathlib)
Oct 1, 2021
5cdc994
fix missing `clustering_method`
Oct 1, 2021
6c646bb
flag to create spike sorting output dir
Oct 1, 2021
dd01fd2
Update kilosort_trigger.py
Oct 1, 2021
f699ce7
minor tweak/improvements in kilosort triggering
Oct 1, 2021
6bf0eb1
bugfix in triggering ecephys_spike_sorting
Oct 1, 2021
6788180
handle cases where `fileTimeSecs` is not available
Oct 1, 2021
d63561f
bugfix
Oct 2, 2021
375e437
missing generate module json
Oct 2, 2021
c593baf
change default `noise_template_use_rf` to False
Oct 2, 2021
69c5e51
minor bugfix
Oct 4, 2021
a6cae12
add kilosort 2.5 as default content
Oct 4, 2021
a0ea9f7
make variable naming consistent
Oct 4, 2021
acdab12
include `clustering_method` into the calculation of `param_set_hash`
Oct 4, 2021
38c5be6
updating `kilosort_repository` depending on which KSVer to be used
Oct 4, 2021
28c4452
bugfix
Oct 4, 2021
94aade7
bugfix, convert path to string
Oct 4, 2021
ae966aa
log the folder creation
Oct 5, 2021
b9f4e92
add debugging lines
Oct 5, 2021
044c389
bugfix - no dir created
Oct 5, 2021
0f0c212
Update kilosort.py
Oct 6, 2021
67341d8
new version 0.1.0b1
Oct 6, 2021
c2ee64f
improve clusteringtask and waveform ingestion routine
Oct 7, 2021
dc7ddd9
minor cleanup
Oct 7, 2021
408532c
stage tracking and resumable kilosort run
Oct 7, 2021
55bec01
minor bugfix
Oct 7, 2021
f34e95d
bugfix logging for kilosort triggering
Oct 7, 2021
19b704b
duration for each module run
Oct 7, 2021
beaf765
check `ap.bin` file validity before triggering kilosort (based on fil…
Oct 8, 2021
74e3ead
Apply suggestions from code review
Oct 11, 2021
e8ffe17
stylistic improvements, addressing code review comments
Oct 11, 2021
d07f830
handle NWB export with multiple curated clusterings from one session
Nov 2, 2021
19616ef
Update nwb.py
Nov 2, 2021
09c1e60
minor bugfix
Nov 2, 2021
3ebdf23
Merge pull request #2 from ttngu207/nwb-export
Nov 2, 2021
b25f065
add nwb export to `no-curation` ephys
Nov 2, 2021
49ca0be
first prototype for running the ecephys_pipeline with OpenEphys
Nov 9, 2021
199a2ba
bugfix for running kilosort for Open Ephys data
Nov 9, 2021
58b5984
specify additional recording-info as part of the `params`
Nov 11, 2021
ddc3b94
add `neuropixels UHD` probe type
Nov 18, 2021
df599fb
triggering kilosort analysis for open-ephys
Nov 18, 2021
819ff19
first prototype for pykilosort
Nov 19, 2021
81d99c8
use_C_waves=False for OpenEphys
Nov 19, 2021
96931a4
using fork of pyopenephys
Nov 22, 2021
b6f8f99
minor bugfix in running pykilosort
Nov 22, 2021
4250220
update openephys loader
Nov 22, 2021
af2b18b
fix package requirement formatting error
Nov 22, 2021
e325a30
handles format differences between npx1 vs 3A
Nov 22, 2021
db3027b
supporting `neuropixels UHD` in `ephys.EphysRecording`
Nov 23, 2021
f55a6a7
bugfix in electrode sites design for Neuropixels UHD probe
Nov 23, 2021
d105419
search recording channels for Open Ephys based on channel names
Nov 23, 2021
0d16e7e
bugfix openephys loader
Nov 23, 2021
c213325
bugfix - extract recording datetime (instead of using experiment date…
Nov 23, 2021
adffe34
bugfix, timedelta as seconds
Nov 23, 2021
43d6614
handles extraction of `connected` channels for NP_PROBE format in Ope…
Nov 23, 2021
da10c66
clusters extraction - check `cluster_group.tsv` and `cluster_KSLabel.…
Nov 25, 2021
02069c9
bugfix in triggering pykilosort
Nov 25, 2021
142459d
handle missing `sample_rate` from pykilosort params.py
Nov 25, 2021
1c39568
for pykilosort's probe, provide both Nchan and NchanTOT
Nov 25, 2021
2022e91
openephys loader - handles signalchain and processor as single elemen…
Nov 26, 2021
02c4b67
handles multi-probe for older OE version
Nov 26, 2021
8dff08b
NWB export fix, specifying which ephys module
Nov 29, 2021
6107e8e
version bump
Nov 29, 2021
dae36d1
fix NWB export - null `waveforms` - back to version 0.1.0b1
Nov 29, 2021
8657d58
version bump
Nov 29, 2021
aa47c8a
include probe as part of the electrode_group name for uniqueness
Nov 29, 2021
31bba8c
bugfix in assigning unit electrode indices
ttngu207 Dec 7, 2021
f9a4754
Merge pull request #3 from ttngu207/nwb-export
Dec 7, 2021
f2881ca
all three ephys files up to date
A-Baji Jan 24, 2022
c56cd18
applied requested changes
A-Baji Jan 24, 2022
9dd6b42
Merge pull request #4 from A-Baji/no-curation
Jan 24, 2022
4951b39
DEPRECATING NWB EXPORT
Jan 26, 2022
b4832ea
Merge branch 'main' of https://github.com/datajoint/element-array-eph…
Jan 26, 2022
d66368c
minor bugfix
Jan 27, 2022
779b2fb
name tweak
Jan 27, 2022
f4052cc
Apply suggestions from code review
Jan 27, 2022
03cab02
rename `sess_dir` -> `session_dir`
Jan 31, 2022
cb1a041
Update requirements.txt
Jan 31, 2022
7320f9f
Update requirements.txt
Jan 31, 2022
f0b3d4a
Update requirements.txt
Jan 31, 2022
bdcfa46
handle older open ephys format for single probe
Feb 18, 2022
83649f5
remove unintended prototyping work
Feb 18, 2022
c2f8aea
delete nwb export - rename `ephys` module -> `ephys_acute`
Feb 25, 2022
e0a9a4f
Update __init__.py
Mar 14, 2022
769de13
bugfix
Mar 16, 2022
39c8579
configurable `paramset_idx` for auto ClusteringTask generation
Mar 25, 2022
85c7c8b
Update openephys.py
May 12, 2022
11a12ba
update openephys loader - handle new open ephys format
May 20, 2022
cd5fe70
handles new probe naming in latest Open Ephys format
May 20, 2022
af864d7
Apply suggestions from code review
May 23, 2022
47babf3
added assertion - safeguard against failed loading of continuous.dat
May 23, 2022
7fecff1
Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…
May 23, 2022
ddd4095
Merge remote-tracking branch 'upstream/run_kilosort' into no-curation
May 23, 2022
364f80e
Merge pull request #69 from ttngu207/no-curation
kabilar May 23, 2022
a578d85
Update requirements.txt
May 23, 2022
f4b917d
`kilosort2` also as part of the `contents` for ClusteringMethod
May 23, 2022
f11e016
bugfix for LFP electrode mapping
May 24, 2022
747c15f
Merge pull request #70 from ttngu207/no-curation
sidhulyalkar May 24, 2022
49bba8a
WIP: nwb bugfix
CBroz1 May 25, 2022
b62fd12
WIP: NWB fix - prevent SQL err by restricting key
CBroz1 May 26, 2022
07604e2
update open ephys loader to handle "STREAM" in latest format
May 26, 2022
e1c9b35
WIP: nwb line length, Readme mention of
CBroz1 May 31, 2022
4e367d7
added loading of electrode location for new openephys format
Jun 1, 2022
9272ee6
Merge pull request #73 from ttngu207/no-curation
sidhulyalkar Jun 1, 2022
67039ac
more robust loading of openephys format
Jun 3, 2022
d298b07
Merge pull request #77 from ttngu207/no-curation
Jun 3, 2022
720e355
Update README.md
CBroz1 Jun 3, 2022
9a59e57
improve logic for running kilosort modules in a resumable fashion
Jun 7, 2022
b97566e
code cleanup, minor bugfix
Jun 7, 2022
3f1ee37
bugfix, match new implementation for openephys
Jun 9, 2022
4a3e0bf
Issue #11
CBroz1 Jun 15, 2022
bd6d7e4
Merge branch 'rk' of https://github.com/CBroz1/element-array-ephys in…
CBroz1 Jun 15, 2022
d102f6f
Issue #63
CBroz1 Jun 15, 2022
e43e5d5
Add Code of Conduct
CBroz1 Jun 16, 2022
70e0b1c
Add contact info to Code of Conduct
CBroz1 Jun 17, 2022
0c77826
improve kilosort calls, handle spaces in paths
Jun 30, 2022
b71b459
remove space escaping character
Jun 30, 2022
a3c5c2f
improve error message
Jul 19, 2022
fd331bd
Merge branch 'no-curation' of https://github.com/ttngu207/element-arr…
Jul 19, 2022
13d74ad
Merge branch 'run_kilosort' of https://github.com/datajoint/element-a…
CBroz1 Jul 21, 2022
0221848
WIP: version bump pynwb to 2.0
CBroz1 Jul 21, 2022
d9c3887
Merge pull request #86 from CBroz1/rk
kabilar Jul 22, 2022
ad8436e
Merge pull request #88 from ttngu207/no-curation
kabilar Jul 25, 2022
b2bd0ee
implement data compression using `mtscomp` for openephys and spikeglx…
Jul 25, 2022
97f3d21
garbage collect openephys objects
Aug 12, 2022
d8aea04
garbage collect openephys objects
Aug 15, 2022
c153e7f
Merge pull request #89 from ttngu207/no-curation
sidhulyalkar Aug 15, 2022
f9f18d0
Update kilosort_triggering.py
Aug 30, 2022
5905392
bugfix in running catgt
Sep 2, 2022
987231b
enable catgt run
Sep 2, 2022
aed42ca
bugfix
Sep 2, 2022
fbdbe24
bugfix
Sep 2, 2022
35da39b
`missing_ok` arg only available in python 3.8+
Sep 2, 2022
c33d1b0
bugfix - path search for catgt output
Sep 2, 2022
f60ba3d
improve error handling
Sep 2, 2022
23ca7ca
Merge pull request #90 from ttngu207/no-curation
Sep 2, 2022
71d87ae
Update kilosort_triggering.py
Sep 14, 2022
6757ef7
Merge pull request #91 from ttngu207/no-curation
Sep 14, 2022
0e94252
Ensure Path type for get_spikeglx_meta_filepath
CBroz1 Sep 15, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: 'bug'
assignees: ''

---

## Bug Report

### Description

A clear and concise description of what is the overall operation that is intended to be
performed that resulted in an error.

### Reproducibility
Include:
- OS (WIN | MACOS | Linux)
- DataJoint Element Version
- MySQL Version
- MySQL Deployment Strategy (local-native | local-docker | remote)
- Minimum number of steps to reliably reproduce the issue
- Complete error stack as a result of evaluating the above steps

### Expected Behavior
A clear and concise description of what you expected to happen.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Additional Research and Context
Add any additional research or context that was conducted in creating this report.

For example:
- Related GitHub issues and PR's either within this repository or in other relevant
repositories.
- Specific links to specific lines or a focus within source code.
- Relevant summary of Maintainers development meetings, milestones, projects, etc.
5 changes: 5 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
blank_issues_enabled: false
contact_links:
- name: DataJoint Contribution Guideline
url: https://docs.datajoint.org/python/community/02-Contribute.html
about: Please make sure to review the DataJoint Contribution Guidelines
57 changes: 57 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
name: Feature request
about: Suggest an idea for a new feature
title: ''
labels: 'enhancement'
assignees: ''

---

## Feature Request

### Problem

A clear and concise description how this idea has manifested and the context. Elaborate
on the need for this feature and/or what could be improved. Ex. I'm always frustrated
when [...]

### Requirements

A clear and concise description of the requirements to satisfy the new feature. Detail
what you expect from a successful implementation of the feature. Ex. When using this
feature, it should [...]

### Justification

Provide the key benefits in making this a supported feature. Ex. Adding support for this
feature would ensure [...]

### Alternative Considerations

Do you currently have a work-around for this? Provide any alternative solutions or
features you've considered.

### Related Errors
Add any errors as a direct result of not exposing this feature.

Please include steps to reproduce provided errors as follows:
- OS (WIN | MACOS | Linux)
- DataJoint Element Version
- MySQL Version
- MySQL Deployment Strategy (local-native | local-docker | remote)
- Minimum number of steps to reliably reproduce the issue
- Complete error stack as a result of evaluating the above steps

### Screenshots
If applicable, add screenshots to help explain your feature.

### Additional Research and Context
Add any additional research or context that was conducted in creating this feature request.

For example:
- Related GitHub issues and PR's either within this repository or in other relevant
repositories.
- Specific links to specific lines or a focus within source code.
- Relevant summary of Maintainers development meetings, milestones, projects, etc.
- Any additional supplemental web references or links that would further justify this
feature request.
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.


## [0.1.0b4] - 2021-11-29
### Added
+ Processing with Kilosort and pyKilosort for Open Ephys and SpikeGLX


## [0.1.0b0] - 2021-05-07
### Added
+ First beta release
Expand Down
133 changes: 133 additions & 0 deletions CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@

# Contributor Covenant Code of Conduct

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

## Our Standards

Examples of behavior that contributes to a positive environment for our
community include:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
community

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or advances of
any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting

## Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.

Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.

## Scope

This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
[[email protected]](mailto:[email protected]).
All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the
reporter of any incident.

## Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:

### 1. Correction

**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.

**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.

### 2. Warning

**Community Impact**: A violation through a single incident or series of
actions.

**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.

### 3. Temporary Ban

**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.

**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.

### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within the
community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].

Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].

For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].

[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,19 @@
+ This repository features DataJoint pipeline design for extracellular array electrophysiology,
with Neuropixels probe and Kilosort spike sorting method.

+ The pipeline presented here is not a complete pipeline by itself, but rather a modular
design of tables and dependencies specific to the extracellular electrophysiology workflow.
+ This module is designed to be integrated into a complete data pipeline and provides
data tables and computational dependencies specific to extracellular electrophysiology
processing and analysis.

+ This modular pipeline element can be flexibly attached downstream
to any particular design of experiment session, thus assembling a fully functional
ephys pipeline.

+ This Element provides three schema options, selected via the DataJoint config file, with `dj.config['custom']['ephys_mode']`
+ `acute` probe insertion, with curated clustering
+ `chronic` probe insertion, with curated clustering
+ `no-curation`, with kilosort triggered clustering

+ See the [Element Array Electrophysiology documentation](https://elements.datajoint.org/description/array_ephys/) for the background information and development timeline.

+ For more information on the DataJoint Elements project, please visit https://elements.datajoint.org. This work is supported by the National Institutes of Health.
Expand Down
16 changes: 14 additions & 2 deletions element_array_ephys/__init__.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,14 @@
# ephys_acute as default
import element_array_ephys.ephys_acute as ephys
import datajoint as dj
import logging
import os


dj.config['enable_python_native_blobs'] = True


def get_logger(name):
log = logging.getLogger(name)
log.setLevel(os.getenv('LOGLEVEL', 'INFO'))
return log

from . import ephys_acute as ephys
Loading