Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add table definitions #2

Merged
merged 20 commits into from
Mar 3, 2022
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -123,3 +123,4 @@ docker-compose.y*ml

# notes
temp*
*/temp*
8 changes: 4 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@

Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.

## [0.1.0c0] - 2021-11-03
## 0.0.0b0 - Unreleased
CBroz1 marked this conversation as resolved.
Show resolved Hide resolved
### Added
+ First draft begins
+ First beta release

## [0.1.0b0] - 2021-00-00
## 0.0.0a0 - 2021-11-15
CBroz1 marked this conversation as resolved.
Show resolved Hide resolved
### Added
CBroz1 marked this conversation as resolved.
Show resolved Hide resolved
+ First beta release
+ First draft based on [Cajal](https://github.com/cajal/pipeline) and [Kavli Institute](https://github.com/kavli-ntnu/dj-docs) precursor projects
kabilar marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 3 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Contribution Guidelines

This project follows the [DataJoint Contribution Guidelines](https://docs.datajoint.org/python/community/02-Contribute.html). Please reference the link for more full details.
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2021 DataJoint
Copyright (c) 2022 DataJoint

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
Expand Down
98 changes: 65 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,65 @@
# element-trial-behavior
This repository is a work in progress. It serves as a draft of a DataJoints element for trial-based behavior for our U24 itiative.

## Notes:
I looked at the structure for `element-array-ephys` for general principle on how to call and load files. I mirrored the main DataJoint implementation as split from 'readers'. I incorporated feedback from project-specific `behavior.py` elsewhere in table development.

## To do:
- [ ] Support functions
- [ ] Other elements/workflows pull `find_full_path` and `find_root_directory` either from their own `__init__.py` files or from `element-data-loader.utils`. Which is best practice?
- [ ] `workflow-array-ephys` relies on the linking module for functions to get root and session directories, but the MAP project defines these internally. Which is best practice?
- [ ] Table definitions: Discuss table structure
- [ ] Decide supported filetypes
- [ ] BPOD
- [ ] Kepec standard, TBD
- [ ] Generalizable CSV with user-determined column name to DJ variable name correspondence?
- [ ] Contact the [BPod team](https://github.com/sanworks/)
- [ ] Already an implementation of loading to Python?
- [ ] Create joint sustainability roadmap
- [ ] Contact Kepec team - joint sustainability roadmap
- [ ] Analysis package
- [ ] Load processed data to table structure
- [ ] Trigger analysis on raw data import
- [ ] Quality control metrics
- [ ] GitHub Actions for PyPI release
- [ ] example workflow
- [ ] Integration tests with pytest
- [ ] Tutorials in text format (i.e. Jupyter notebook)
- [ ] Tutorial in video format
- [ ] Docker for tests
- [ ] Example dataset(s) for public release, in DJ Archive
- [ ] NWB export
- [ ] README
- [ ] RRID
# DataJoint Element - Experimental trials
This repository is a work in progress not yet ready for public release.
It serves as a draft of a DataJoint element for trialized experiments behavior
for our U24 itiative.

## Element architecture

In both of the following diagrams, the trial element starts immediately downstream from
CBroz1 marked this conversation as resolved.
Show resolved Hide resolved
***Session***. In one case, Sessions are first segmented into trials, and then
segmented into events. This might be appropriate, for example, in a paradigm with
repeated conditions and response behaviors associated with different conditions. In the
next, Sessions are directly upstream from both Trials and Events. This might be appropropriate for a paradigm that recorded events within naturalistic free behavior. We provide an
[example workflow](https://github.com/datajoint/workflow-trial/) with a
[pipeline script](https://github.com/datajoint/workflow-trial/blob/main/workflow_trial/pipeline.py)
that models combining this Element with the corresponding
[Element-Session](https://github.com/datajoint/element-session).

### Trial Schema

![trial schema](./images/diagram_trial.svg)

### Event Schema
![event schema](./images/diagram_event.svg)

## Installation

+ Install `element-trial`
```
pip install element-trial
```

+ Upgrade `element-trial` previously installed with `pip`
```
pip install --upgrade element-trial
```

<!---
+ Install `element-interface`

+ `element-interface` is a dependency of `element-trial`, however it is not
contained within `requirements.txt`.

```
pip install "element-interface @ git+https://github.com/datajoint/element-interface"
```
-->

## Usage

### Element activation

To activate the `element-trial`, one need to provide:

1. Schema names for the event or trial module
2. Upstream Session table: A set of keys identifying a recording session (see [
Element-Session](https://github.com/datajoint/element-session)).
3. Utility functions. See
[example definitions here](https://github.com/datajoint/workflow-trial/blob/main/workflow_trial/paths.py)

For more detail, check the docstring of the `element-trial`:
```python
from element_trial import event, trial
help(event.activate)
help(trial.activate)
```
26 changes: 0 additions & 26 deletions dj_local_conf_example.json

This file was deleted.

Empty file added element_trial/__init__.py
Empty file.
156 changes: 156 additions & 0 deletions element_trial/event.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
"""Events are linked directly to session"""

import datajoint as dj
import inspect
import importlib

schema = dj.schema()

_linking_module = None


def activate(schema_name, *, create_schema=True, create_tables=True,
linking_module=None):
"""
activate(schema_name, *, create_schema=True, create_tables=True,
linking_module=None)
:param schema_name: schema name on the database server to activate
the `behavior` element
:param create_schema: when True (default), create schema in the
database if it does not yet exist.
:param create_tables: when True (default), create tables in the
database if they do not yet exist.
:param linking_module: a module (or name) containing the required
dependencies to activate the `session` element:
Upstream tables:
+ Session: parent table to ProbeInsertion, typically
identifying a recording session.
Functions:
+ get_trial_root_data_dir() -> list
Retrieve the root data director(y/ies) with behavioral
recordings (e.g., bpod files) for all subject/sessions.
:return: a string for full path to the root data directory
+ get_trial_sess_dir(session_key: dict) -> str
Retrieve the session directory containing the recording(s)
for a given Session
:param session_key: a dictionary of one Session `key`
:return: a string for full path to the session directory
"""
if isinstance(linking_module, str):
linking_module = importlib.import_module(linking_module)
assert inspect.ismodule(linking_module), "The argument 'dependency' must"\
+ " be a module or module name"

schema.activate(schema_name, create_schema=create_schema,
create_tables=create_tables,
add_objects=linking_module.__dict__)

# -------------- Functions required by the element-trial ---------------


def get_trial_root_data_dir() -> list:
"""
All data paths, directories in DataJoint Elements are recommended to be
stored as relative paths, with respect to some user-configured "root"
directory, which varies from machine to machine

get_trial_root_data_dir() -> list
This user-provided function retrieves the list of possible root data
directories containing the behavioral data for all subjects/sessions
:return: a string for full path to the behavioral root data directory,
or list of strings for possible root data directories
"""
return _linking_module.get_trial_root_data_dir()


def get_trial_sess_dir(session_key: dict) -> str:
"""
get_trial_sess_dir(session_key: dict) -> str
Retrieve the session directory, with all recordings for a given Session
:param session_key: a dictionary of one Session `key`
:return: a string for full path to the session directory
"""
return _linking_module.get_trial_sess_dir(session_key)

# ----------------------------- Table declarations ----------------------


@schema
class BehaviorRecording(dj.Manual):
definition = """
-> Session
recording_id : varchar(16)
---
recording_notes : varchar(256)
"""

class BehaviorFile(dj.Part):
definition = """
-> master
filepath : varchar(16)
"""


@schema
class TrialType(dj.Lookup):
definition = """
trial_type : varchar(16)
---
trial_type_description : varchar(256)
"""


@schema
class Trial(dj.Imported):
definition = """
-> Session
trial : smallint # trial number (1-based indexing)
---
-> TrialType
start_time : float # (second) relative to recording start
stop_time : float # (second) relative to recording start
"""

class TrialVariable(dj.Part):
definition = """
-> master
variable_name: varchar(16)
---
variable_value: varchar(2000)
"""


@schema
class EventType(dj.Lookup):
definition = """
event_type : varchar(16)
---
event_type_description='': varchar(256)
"""


@schema
class Event(dj.Imported):
definition = """ #
-> Session
event_start_time : float # (second) relative to recording start
---
-> EventType
event_end_time=null: float # (second) relative to recording start
"""

class EventVariable(dj.Part):
definition = """
-> master
variable_name: varchar(16)
---
variable_value: varchar(2000)
"""


@schema
class TrialEvent(dj.Imported):
definition = """
-> Trial
-> Event
"""
Empty file.
Loading