Add table definitions

.gitignore

@@ -123,3 +123,4 @@ docker-compose.y*ml
 
 # notes
 temp*
+*/temp*

CHANGELOG.md

@@ -2,10 +2,10 @@
 
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
-## [0.1.0c0] - 2021-11-03
+## [0.1.0b0] - [unreleased]
 ### Added
-+ First draft begins
++ First beta release
 
-## [0.1.0b0] - 2021-00-00
+## [0.1.0a0] - 2021-11-15
 ### Added
-+ First beta release
++ First draft begins

CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Contribution Guidelines
+
+This project follows the [DataJoint Contribution Guidelines](https://docs.datajoint.io/python/community/02-Contribute.html). Please reference the link for more full details.

README.md

@@ -1,33 +1,62 @@
-# element-trial-behavior
-This repository is a work in progress. It serves as a draft of a DataJoints element for trial-based behavior for our U24 itiative.
-
-## Notes:
-I looked at the structure for `element-array-ephys` for general principle on how to call and load files. I mirrored the main DataJoint implementation as split from 'readers'. I incorporated feedback from project-specific `behavior.py` elsewhere in table development.
-
-## To do:
-- [ ] Support functions
-   - [ ] Other elements/workflows pull `find_full_path` and `find_root_directory` either from their own `__init__.py` files or from `element-data-loader.utils`. Which is best practice?
-   - [ ] `workflow-array-ephys` relies on the linking module for functions to get root and session directories, but the MAP project defines these internally.  Which is best practice?
-- [ ] Table definitions: Discuss table structure
-- [ ] Decide supported filetypes
-   -  [ ] BPOD
-   -  [ ] Kepec standard, TBD
-   -  [ ] Generalizable CSV with user-determined column name to DJ variable name correspondence?
-- [ ] Contact the [BPod team](https://github.com/sanworks/)
-   - [ ] Already an implementation of loading to Python?
-   - [ ] Create joint sustainability roadmap
-- [ ] Contact Kepec team - joint sustainability roadmap
-- [ ] Analysis package
-   - [ ] Load processed data to table structure
-   - [ ] Trigger analysis on raw data import
-- [ ] Quality control metrics
-- [ ] GitHub Actions for PyPI release
-- [ ] example workflow
-   - [ ] Integration tests with pytest
-   - [ ] Tutorials in text format (i.e. Jupyter notebook)
-   - [ ] Tutorial in video format
-   - [ ] Docker for tests
-   - [ ] Example dataset(s) for public release, in DJ Archive
-   - [ ] NWB export
-   - [ ] README
-- [ ] RRID
+# DataJoint Element - Experimental trials
+This repository is a work in progress not yet ready for public release.
+It serves as a draft of a DataJoint element for trialized experiments behavior
+for our U24 itiative.
+
+Work in progress.
+
+## Element architecture
+
+In both of the following diagrams, the trial element starts immediately downstream from ***Session***. In one case, Sessions are first segmented into trials, and then segmented into events. This might be appropriate, for example, in a paradigm with repeated conditions and response behaviors associated with different conditions. In the next, Sessions are directly upstream from Events. This might be appropropriate for a paradigm that recorded events within naturalistic free behavior.
+We  provide an [example workflow](https://github.com/datajoint/workflow-trial/) with a
+[pipeline script](https://github.com/datajoint/workflow-trial/blob/main/workflow_trial/pipeline.py) that models combining this Element with the corresponding [Element-Session](https://github.com/datajoint/element-session).
+
+
+<!---
+![element-trial diagram](images/attached_trial_element_trialized.svg)
+![element-trial diagram](images/attached_trial_element_events.svg)
+-->
+
+## Installation
+
++ Install `element-trial`
+    ```
+    pip install element-trial
+    ```
+
++ Upgrade `element-trial` previously installed with `pip`
+    ```
+    pip install --upgrade element-trial
+    ```
+
+<!---
++ Install `element-interface`
+
+    + `element-interface` is a dependency of `element-trial`, however it is not contained within `requirements.txt`.
+
+    ```
+    pip install "element-interface @ git+https://github.com/datajoint/element-interface"
+    ```
+-->
+
+## Usage
+
+### Element activation
+
+To activate the `element-trial`, one need to provide:
+
+1. Schema names for the event or trial module
+
+2. Upstream Session table: A set of keys identifying a recording session (see [Element-Session](https://github.com/datajoint/element-session)).
+3. Utility functions. See example definitions here](https://github.com/datajoint/workflow-trial/blob/main/workflow_trial/paths...
+
+For more detail, check the docstring of the `element-trial`:
+```python
+from element_trial import event, trial
+help(event.activate)
+help(trial.activate)
+```
+
+### Example usage
+
+See [this project](https://github.com/datajoint/workflow-trial) for an example usage of this Array Electrophysiology Element.

element_trial/__init__.py

@@ -0,0 +1,4 @@
+"""
+Two schemas, trial and event. Trial for fully trialized, segmented.
+Event for events independent of trials, like an act of naturaistic behavior.
+"""

element_trial/event.py

@@ -0,0 +1 @@
+# Copy from trial, remove 'EventTrialized'

element_trial/export/nwb.py

@@ -0,0 +1,78 @@
+from pynwb import NWBFile
+
+from element_session import session
+from element_trial import trial
+
+
+def trial_to_nwb(trial_key):
+    scan_query = session.Session & trial.TrialEvent & trial_key
+    # https://github.com/nwb-extensions/ndx-events-record
+    return NWBFile(scan_query)
+
+
+""" From CHEN 2017 https://github.com/vathes/DJ-NWB-Chen-2017
+
+# =============== TrialSet ====================
+# NWB 'trial' (of type dynamic table) by default comes with three mandatory
+# attributes: 'start_time' and 'stop_time'. Other trial-related information
+# needs to be added in to the trial-table as additional columns (with column
+# name and column description)
+
+dj_trial = experiment.SessionTrial * experiment.BehaviorTrial
+skip_adding_columns = experiment.Session.primary_key + ['trial_uid', 'trial']
+
+if experiment.SessionTrial & session_key:
+    # Get trial descriptors from TrialSet.Trial and TrialStimInfo
+    trial_columns = [{'name': tag,
+              'description': re.sub('\s+:|\s+', ' ', re.search(
+                f'(?<={tag})(.*)', str(dj_trial.heading)).group()).strip()}
+             for tag in dj_trial.heading.names
+             if tag not in skip_adding_columns + ['start_time', 'stop_time']]
+
+    # Add new table columns to nwb trial-table for trial-label
+    for c in trial_columns:
+        nwbfile.add_trial_column(**c)
+
+    # Add entry to the trial-table
+    for trial in (dj_trial & session_key).fetch(as_dict=True):
+        trial['start_time'] = float(trial['start_time'])
+        trial['stop_time'] = (float(trial['stop_time']) if
+                              trial['stop_time'] else 5.0)
+        [trial.pop(k) for k in skip_adding_columns]
+        trial['early_lick'] = True if trial['early_lick'] == 'early' else False
+        nwbfile.add_trial(**trial)
+
+# ===========================================================================
+# ============================= BEHAVIOR TRIAL EVENTS =======================
+# ===========================================================================
+
+behavior = nwbfile.create_processing_module(
+    'behavior', 'Time of behavioral events in this session')
+behav_event = pynwb.behavior.BehavioralEvents(name='BehavioralEvents')
+behavior.add_data_interface(behav_event)
+
+for trial_event_type in \
+        (experiment.TrialEventType & \
+            experiment.TrialEvent & session_key).fetch('trial_event_type'):
+    event_times, trial_starts = \
+        (experiment.TrialEvent * experiment.SessionTrial
+            & session_key & {'trial_event_type': trial_event_type}).fetch(
+        'trial_event_time', 'start_time')
+
+    if trial_event_type == 'sample':
+        description = 'Timestamps: beginning of the sampling on each trial.'
+    elif trial_event_type == 'delay':
+        description = 'Timestamps: beginning of the delay on each trial.'
+    elif trial_event_type == 'go':
+        description = 'Time stamps of the go cue signal on each trial.'
+
+    if len(event_times) > 0:
+        event_times = np.hstack(event_times.astype(float)
+                                + trial_starts.astype(float))
+        behav_event.create_timeseries(
+            name=trial_event_type, unit='a.u.', conversion=1.0,
+            data=np.full_like(event_times, 1),
+            timestamps=event_times,
+            description=description)
+
+"""

element_trial/readers/bpod.py

@@ -0,0 +1,76 @@
+''' bpod_loader.py
+For eventual inclusion in element-data-loader
+'''
+
+import pathlib
+import scipy.io as spio
+
+
+class BPod:
+    """ Parse a bpod file into the following objects
+    fields: top-level bpod fields
+    number_trials: total number of trials
+    trial_start_times: list of floats, seconds relative to session start
+    trial_types: array of tinyint designating condition number
+    """
+    def __init__(self, full_dir):
+        try:  # check for file in path
+            self.filepath = next(pathlib.Path(full_dir).glob('*.mat'))
+        except StopIteration:
+            raise FileNotFoundError(f'No .mat file found at: {full_dir}')
+        self.data = load_bpod_matfile(self.filepath)  # see helper below
+
+    @property
+    def fields(self):
+        if self._fields is None:
+            self._fields = list(self.data.keys())
+        return self._fields
+
+    @property
+    def number_trials(self):
+        if self._number_trials is None:
+            self._number_trials = self.data['nTrials']
+        return self._number_trials
+
+    @property
+    def trial_start_times(self):
+        if self._trial_start_times is None:
+            self._trial_start_times = list(self.data['TrialStartTimestamp'])
+        return self._trial_start_times
+
+    @property
+    def trial_types(self):
+        if self._trial_types is None and 'TrialTypes' in self.fields:
+            self._trial_types = self.data['TrialTypes']
+        return self._trial_types
+
+    '''
+    @property
+    def [each relevant bpod property](self)]:
+        if self.[relevant property] is None:
+            self.[relevant property] = self.file.[specific structure]
+    '''
+
+# --------------------- HELPER LOADER FUNCTIONS -----------------
+
+
+def load_bpod_matfile(mat_filepath):
+    """
+    Loading routine for behavioral file, bpod .mat
+    """
+    # loadmat optionally takes mdict, existing dictionary which it loads into
+    # simplify_cells returns a simplified dict structure, instead of reversible
+    #                mat-like files
+    # squeeze_me compresses matrix dimensions
+    mat_file = spio.loadmat(mat_filepath.as_posix(),squeeze_me=True,
+                            struct_as_record=False,simplify_cells=True)
+    # bpod files load as dict with the following keys
+    # __header__    : mat version, flatform, creation date
+    # __version__   : file version
+    # __globals__
+    # SessionData   : mat_struct
+    if 'SessionData' in mat_file.keys():
+        return mat_file['SessionData']
+    else:
+        raise FileNotFoundError('.mat file missing SessionData'
+                                + f'field at: {mat_filepath}')