Add table definitions

.gitignore

@@ -123,3 +123,4 @@ docker-compose.y*ml
 
 # notes
 temp*
+*/temp*

CHANGELOG.md

@@ -2,10 +2,10 @@
 
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
-## [0.1.0c0] - 2021-11-03
+## [0.1.0b0] - [unreleased]
 ### Added
-+ First draft begins
++ First beta release
 
-## [0.1.0b0] - 2021-00-00
+## [0.1.0c0] - 2021-11-15
 ### Added
-+ First beta release
++ First draft begins

CONTRIBUTING.md

@@ -0,0 +1,3 @@
+# Contribution Guidelines
+
+This project follows the [DataJoint Contribution Guidelines](https://docs.datajoint.io/python/community/02-Contribute.html). Please reference the link for more full details.

README.md

@@ -1,22 +1,72 @@
-# element-trial-behavior
-This repository is a work in progress. It serves as a draft of a DataJoints element for trial-based behavior for our U24 itiative.
+# element-trial
+This repository is a work in progress not yet ready for public release.
+It serves as a draft of a DataJoint element for trialized experiments behavior
+for our U24 itiative.
 
-## Notes:
-I looked at the structure for `element-array-ephys` for general principle on how to call and load files. I mirrored the main DataJoint implementation as split from 'readers'. I incorporated feedback from project-specific `behavior.py` elsewhere in table development.
+Work in progress.
+
+## Schemas
+`trial` should be activated if all events are tied to specific trials.
+`event` should be activated if events occur independently of trials.
+
+The current draft of `trial` contains all proposed tables. When we're sure of
+the contents, including make functions, one would copy the contents of the
+`trial` schema over to events, and then remove the trialized version of the
+event table.
+
+## Filetypes
+
+### Bpod
+MATLAB is the only officially supported environment. SciPy has a lot of support
+for loading matlab files. In the current draft, SciPy is used to import Bpod
+.mat files as embedded dictionaries. Micheal Wulf shared the Bpod files that
+are currently available via djarchive as `workflow-trial`, revision 0.0.0b1.
+They were shared as a diverse set of examples.
+
+Under `element_trial/readers/`, there are two files: `Bpod.py` and
+`Bpod_fields_notes.py`. The former is a draft of an eventual reader that could
+be folded into `element_data_loader`. It handles general ingestion for fields
+that are consisitent across examples. The latter highlights just how different
+Bpod files can be. A fully realized loader may need to pull directly from
+Bpod 'raw' fields to reconstruct the experiement, as many other fields are
+organized differently across trials.
+
+Development for this reader was partially conducted within
+`workflow_trial/notebooks/1_.ipynb`. This contains code snipits for exploring
+the loaded Bpod data.
+
+Ecosystem: The SanWorks team
+([git repositories here](https://github.com/sanworks?tab=repositories)) offers
+no offical python support, suggesting instead that users
+[export to JSON](https://sanworks.io/forum/showthread.php?tid=626&pid=1169).
+The PyBpod project ([github](https://github.com/pyBpod/pyBpod),
+[docs](https://pyBpod.readthedocs.io/en/v1.8.1/)) offers a python-based GUI
+alternative for running Bpod hardware. So far as I could tell, they do not
+offer any usefull ingestion functions we would want to incorporate.
+
+
+### Alternate generalized format
+
+We previously discussed also supporing a univariate csv format that would
+contain 4 values: timestamp on, timestamp off, data type and value. This has
+not yet been implemented.
 
 ## To do:
-- [ ] Support functions
-   - [ ] Other elements/workflows pull `find_full_path` and `find_root_directory` either from their own `__init__.py` files or from `element-data-loader.utils`. Which is best practice?
-   - [ ] `workflow-array-ephys` relies on the linking module for functions to get root and session directories, but the MAP project defines these internally.  Which is best practice?
-- [ ] Table definitions: Discuss table structure
-- [ ] Decide supported filetypes
-   -  [ ] BPOD
-   -  [ ] Kepec standard, TBD
-   -  [ ] Generalizable CSV with user-determined column name to DJ variable name correspondence?
-- [ ] Contact the [BPod team](https://github.com/sanworks/)
-   - [ ] Already an implementation of loading to Python?
-   - [ ] Create joint sustainability roadmap
-- [ ] Contact Kepec team - joint sustainability roadmap
+- [X] Support functions
+   - [X] Pull `find_full_path` and `find_root_directory` from `element-data-loader.utils`.
+   - [X] To find root/session dirs, refer to config file if exists. If not, linking module
+- [ ] Table definitions
+   - [X] Discuss table structure
+	 - [X] Decide supported filetypes:
+      - Bpod
+      - Generalizable 'Univariate' CSV: timestamp on, off, data type, value
+    - [ ] Develop ingestion functions
+- [ ] Test tables with example data
+   - Aeon, bonsai system API
+   - Kepecs group provided Bpod files
+- [X] Contact the [Bpod team](https://github.com/sanworks/)
+   - [X] Already an implementation of loading to Python - No
+   - [X] Create joint sustainability roadmap - Unlikely
 - [ ] Analysis package
    - [ ] Load processed data to table structure
    - [ ] Trigger analysis on raw data import
@@ -27,7 +77,26 @@ I looked at the structure for `element-array-ephys` for general principle on how
    - [ ] Tutorials in text format (i.e. Jupyter notebook)
    - [ ] Tutorial in video format
    - [ ] Docker for tests
-   - [ ] Example dataset(s) for public release, in DJ Archive
+   - [X] Example dataset(s) for public release
+   - [ ] Example dataset loaded to DJ Archive
    - [ ] NWB export
    - [ ] README
 - [ ] RRID
+
+## Installation
+
+```
+pip install element-trial
+```
+
+if you have an older version of ***element-trial*** installed, using `pip`, upgrade with
+
+```
+pip install --upgrade element-trial
+```
+
+Install `element-data-loader`
+
+```
+element-data-loader @ git+https://github.com/datajoint/element-data-loader
+```

element_trial/__init__.py

@@ -0,0 +1,13 @@
+__author__ = "DataJoint"
+__date__ = "December, 2021"
+__version__ = "0.1.0c0"
+
+__all__ = ['__author__', '__version__', '__date__']
+
+import datajoint as dj
+dj.config['enable_python_native_blobs'] = True
+
+"""
+Two schemas, trial and event. Trial for fully trialized, segmented.
+Event for events independent of trials, like an act of naturaistic behavior.
+"""

element_trial/event.py

@@ -0,0 +1 @@
+# Copy from trial, remove 'EventTrialized'

element_trial/export/__init__.py

@@ -0,0 +1,7 @@
+
+
+__author__ = "DataJoint"
+__date__ = "November, 2021"
+__version__ = "0.1.0c0"
+
+__all__ = ['__author__', '__version__', '__date__']

element_trial/export/nwb.py

@@ -0,0 +1,83 @@
+from pynwb import NWBFile
+
+from element_session import session
+from element_trial import trial
+
+
+def trial_to_nwb(trial_key):
+    scan_query = session.Session & trial.TrialEvent & trial_key
+    # https://github.com/nwb-extensions/ndx-events-record
+    return NWBFile(scan_query)
+
+
+""" From CHEN 2017 https://github.com/vathes/DJ-NWB-Chen-2017
+
+
+# ===========================================================================
+# ============================= BEHAVIOR TRIALS =============================
+# ===========================================================================
+
+# =============== TrialSet ====================
+# NWB 'trial' (of type dynamic table) by default comes with three mandatory
+# attributes: 'start_time' and 'stop_time'. Other trial-related information
+# needs to be added in to the trial-table as additional columns (with column
+# name and column description)
+
+dj_trial = experiment.SessionTrial * experiment.BehaviorTrial
+skip_adding_columns = experiment.Session.primary_key + ['trial_uid', 'trial']
+
+if experiment.SessionTrial & session_key:
+    # Get trial descriptors from TrialSet.Trial and TrialStimInfo
+    trial_columns = [{'name': tag,
+              'description': re.sub('\s+:|\s+', ' ', re.search(
+                f'(?<={tag})(.*)', str(dj_trial.heading)).group()).strip()}
+             for tag in dj_trial.heading.names
+             if tag not in skip_adding_columns + ['start_time', 'stop_time']]
+
+    # Add new table columns to nwb trial-table for trial-label
+    for c in trial_columns:
+        nwbfile.add_trial_column(**c)
+
+    # Add entry to the trial-table
+    for trial in (dj_trial & session_key).fetch(as_dict=True):
+        trial['start_time'] = float(trial['start_time'])
+        trial['stop_time'] = (float(trial['stop_time']) if
+                              trial['stop_time'] else 5.0)
+        [trial.pop(k) for k in skip_adding_columns]
+        trial['early_lick'] = True if trial['early_lick'] == 'early' else False
+        nwbfile.add_trial(**trial)
+
+# ===========================================================================
+# ============================= BEHAVIOR TRIAL EVENTS =======================
+# ===========================================================================
+
+behavior = nwbfile.create_processing_module(
+    'behavior', 'Time of behavioral events in this session')
+behav_event = pynwb.behavior.BehavioralEvents(name='BehavioralEvents')
+behavior.add_data_interface(behav_event)
+
+for trial_event_type in \
+        (experiment.TrialEventType & \
+            experiment.TrialEvent & session_key).fetch('trial_event_type'):
+    event_times, trial_starts = \
+        (experiment.TrialEvent * experiment.SessionTrial
+            & session_key & {'trial_event_type': trial_event_type}).fetch(
+        'trial_event_time', 'start_time')
+
+    if trial_event_type == 'sample':
+        description = 'Timestamps: beginning of the sampling on each trial.'
+    elif trial_event_type == 'delay':
+        description = 'Timestamps: beginning of the delay on each trial.'
+    elif trial_event_type == 'go':
+        description = 'Time stamps of the go cue signal on each trial.'
+
+    if len(event_times) > 0:
+        event_times = np.hstack(event_times.astype(float)
+                                + trial_starts.astype(float))
+        behav_event.create_timeseries(
+            name=trial_event_type, unit='a.u.', conversion=1.0,
+            data=np.full_like(event_times, 1),
+            timestamps=event_times,
+            description=description)
+
+"""

element_trial/readers/bpod.py

@@ -0,0 +1,76 @@
+''' bpod_loader.py
+For eventual inclusion in element-data-loader
+'''
+
+import pathlib
+import scipy.io as spio
+
+
+class BPod:
+    """ Parse a bpod file into the following objects
+    fields: top-level bpod fields
+    number_trials: total number of trials
+    trial_start_times: list of floats, seconds relative to session start
+    trial_types: array of tinyint designating condition number
+    """
+    def __init__(self, full_dir):
+        try:  # check for file in path
+            self.filepath = next(pathlib.Path(full_dir).glob('*.mat'))
+        except StopIteration:
+            raise FileNotFoundError(f'No .mat file found at: {full_dir}')
+        self.data = load_bpod_matfile(self.filepath)  # see helper below
+
+    @property
+    def fields(self):
+        if self._fields is None:
+            self._fields = list(self.data.keys())
+        return self._fields
+
+    @property
+    def number_trials(self):
+        if self._number_trials is None:
+            self._number_trials = self.data['nTrials']
+        return self._number_trials
+
+    @property
+    def trial_start_times(self):
+        if self._trial_start_times is None:
+            self._trial_start_times = list(self.data['TrialStartTimestamp'])
+        return self._trial_start_times
+
+    @property
+    def trial_types(self):
+        if self._trial_types is None and 'TrialTypes' in self.fields:
+            self._trial_types = self.data['TrialTypes']
+        return self._trial_types
+
+    '''
+    @property
+    def [each relevant bpod property](self)]:
+        if self.[relevant property] is None:
+            self.[relevant property] = self.file.[specific structure]
+    '''
+
+# --------------------- HELPER LOADER FUNCTIONS -----------------
+
+
+def load_bpod_matfile(mat_filepath):
+    """
+    Loading routine for behavioral file, bpod .mat
+    """
+    # loadmat optionally takes mdict, existing dictionary which it loads into
+    # simplify_cells returns a simplified dict structure, instead of reversible
+    #                mat-like files
+    # squeeze_me compresses matrix dimensions
+    mat_file = spio.loadmat(mat_filepath.as_posix(),squeeze_me=True,
+                            struct_as_record=False,simplify_cells=True)
+    # bpod files load as dict with the following keys
+    # __header__    : mat version, flatform, creation date
+    # __version__   : file version
+    # __globals__
+    # SessionData   : mat_struct
+    if 'SessionData' in mat_file.keys():
+        return mat_file['SessionData']
+    else:
+        raise FileNotFoundError('.mat file missing SessionData'
+                                + f'field at: {mat_filepath}')