Merge pull request #28 from catalystneuro/Neuropixels/add-tutorials

Add tutorials for electrophysiology and fiber photometry

src/constantinople_lab_to_nwb/fiber_photometry/tutorials/fiber_photometry_run_session_to_nwb.ipynb

@@ -0,0 +1,133 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "90ae0edb-2fab-47dc-ad31-e79cf2905733",
+   "metadata": {},
+   "source": [
+    "# Fiber photometry data conversion to NWB\n",
+    "\n",
+    "This notebook demonstrates how to convert an example session from the Fiber photometry dataset to NWB.\n",
+    "\n",
+    "## Fiber photometry dataset\n",
+    "\n",
+    "This dataset contains fiber photometry recordings during decision-making behavioral task in rats. Deeplabcut software (v.2.2.3) was used for tracking the behavior ports (right port, central port, and left port) and 6 body parts (right ear, nose, left ear, mid-point along the right torso, mid-point along the left torso, and base of the tail). Video data were acquired using cameras attached to the ceiling of behavior rigs to capture the top-down view of the arena (Doric USB3 behavior camera, Sony IMX290, recorded with Doric Neuroscience Studio v6 software). The fluorescence from activity-dependent (GRAB-DA and GRAB-ACh) and activity-independent (isosbestic or mCherry) signals was acquired simultaneously via demodulation and downsampled on-the-fly by a factor of 25 to ~481.9 Hz. The recorded demodulated fluorescence was corrected for photobleaching and motion using Two-channel motion artifact correction (3) with mCherry or isosbestic signal as the activity-independent channel. The behavioral tasks were conducted in a high-throughput facility where rats were trained in increasingly complex protocols. Trials were initiated by a nose-poke in a lit center port and required maintaining a center fixation for 0.8 to 1.2 seconds, during which a tone indicated the possible reward size. A subsequent side LED indicated the potential reward location, followed by a delay period drawn from an exponential distribution (mean = 2.5 s). Rats could opt out at any time by poking the unlit port, restarting the trial. Catch trials, where the delay period only ended if the rat opted out, constituted 15-25% of the trials. Rats received penalties for premature fixation breaks. Additionally, the tasks introduced semi-observable hidden states by varying reward statistics across uncued blocks (high, low, and mixed), structured hierarchically, with blocks transitioning after 40 successfully completed trials.\n",
+    "\n",
+    "\n",
+    "This notebook demonstrates how to convert an example session to NWB.\n",
+    "\n",
+    "This dataset have the following data streams:\n",
+    "- Raw behavior: Bpod output (.mat), Video (.mp4)\n",
+    "- Processed behavior: DeepLabCut output (.h5)\n",
+    "- Fiber photometry: Doric (.doric or .csv)\n",
+    "\n",
+    "## Notes on the conversion\n",
+    "\n",
+    "The conversion notes is located in `src/constantinople_lab_to_nwb/fiber_photometry/fiber_photometry_notes.md`. This file contains information about the expected file structure and the conversion process.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "9309ffc0-aedb-4c21-9d20-8442813c29e2",
+   "metadata": {},
+   "source": [
+    "## Convert a single session to NWB\n",
+    "\n",
+    "The `fiber_photometry_convert_session.py` script defines the `session_to_nwb` function that converts a session of fiber photometry data to NWB."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ca3883d4-c9df-4024-83fa-fb2c1082cc1f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pathlib import Path\n",
+    "from constantinople_lab_to_nwb.fiber_photometry.fiber_photometry_convert_session import session_to_nwb\n",
+    "from neuroconv.utils import load_dict_from_file, dict_deep_update\n",
+    "\n",
+    "from constantinople_lab_to_nwb.fiber_photometry import FiberPhotometryNWBConverter\n",
+    "from ndx_pose import PoseEstimation\n",
+    "\n",
+    "from constantinople_lab_to_nwb.utils import get_subject_metadata_from_rat_info_folder\n",
+    "\n",
+    "# Parameters for conversion\n",
+    "\n",
+    "# The path to the raw fiber photometry file (.doric or .csv).\n",
+    "doric_file_path=\"/Volumes/T9/Constantinople/Preprocessed_data/J069/Raw/J069_ACh_20230809_HJJ_0002.doric\"\n",
+    "# The path to the NWB file to write.\n",
+    "nwbfile_path=\"/Volumes/T9/Constantinople/nwbfiles/J069_ACh_20230809_HJJ_0002.nwb\"\n",
+    "# The path to the DLC output (.h5), optional\n",
+    "dlc_file_path=\"/Volumes/T9/Constantinople/DeepLabCut/J069/J069-2023-08-09_rig104cam01_0002compDLC_resnet50_GRAB_DA_DMS_RIG104DoricCamera_J029May12shuffle1_500000.h5\"\n",
+    "# The path to the behavior camera recording (.mp4), optional\n",
+    "video_file_path=\"/Volumes/T9/Constantinople/Compressed Videos/J069/J069-2023-08-09_rig104cam01_0002comp.mp4\"\n",
+    "# The raw behavior data from Bpod (contains data for a single session)\n",
+    "bpod_file_path=\"/Volumes/T9/Constantinople/raw_Bpod/J069/DataFiles/J069_RWTautowait2_20230809_131216.mat\"\n",
+    "\n",
+    "# Load the default fiber photometry metadata from the yaml file\n",
+    "# For .doric files use \"doric_fiber_photometry_metadata.yaml\"\n",
+    "# For .csv files use \"doric_csv_fiber_photometry_metadata.yaml\"\n",
+    "current_dir = Path(__file__).parent if '__file__' in globals() else Path.cwd().parent\n",
+    "fiber_photometry_metadata_file_path = current_dir / \"metadata\" / \"doric_fiber_photometry_metadata.yaml\"\n",
+    "fiber_photometry_metadata = load_dict_from_file(fiber_photometry_metadata_file_path)\n",
+    "\n",
+    "# Get subject metadata from rat registry\n",
+    "rat_registry_folder_path = \"/Volumes/T9/Constantinople/Rat_info\"\n",
+    "subject_metadata = get_subject_metadata_from_rat_info_folder(\n",
+    "    folder_path=rat_registry_folder_path,\n",
+    "    subject_id=\"J069\",\n",
+    "    date=\"2023-08-09\",\n",
+    ")\n",
+    "\n",
+    "# Optional parameters\n",
+    "\n",
+    "# Whether to overwrite an existing NWB file.\n",
+    "overwrite = True\n",
+    "# Whether to print verbose output.\n",
+    "verbose = True\n",
+    "\n",
+    "session_to_nwb(\n",
+    "    nwbfile_path=nwbfile_path,\n",
+    "    raw_fiber_photometry_file_path=doric_file_path,\n",
+    "    fiber_photometry_metadata=fiber_photometry_metadata,\n",
+    "    raw_behavior_file_path=bpod_file_path,\n",
+    "    subject_metadata=subject_metadata,\n",
+    "    dlc_file_path=dlc_file_path,\n",
+    "    video_file_path=video_file_path,\n",
+    "    overwrite=overwrite,\n",
+    "    verbose=verbose,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "879cd669-a5a5-4111-9aef-ca13fb9ed2b8",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

src/constantinople_lab_to_nwb/schierek_embargo_2024/tutorials/ephys_run_session_to_nwb.ipynb

@@ -0,0 +1,207 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "352a2fb2-54fe-4418-ae34-eda86223c180",
+   "metadata": {},
+   "source": [
+    "# Neuropixels data conversion to NWB\n",
+    "\n",
+    "This notebook demonstrates how to convert an example session from the Neuropixels dataset to NWB.\n",
+    "\n",
+    "## Neuropixels dataset\n",
+    "\n",
+    "This dataset contains in vivo extracellular electrophysiology recordings from rats performing a value-based decision-making task. Neural data were acquired using Neuropixels probes (384 channels, 30 kHz sampling rate) with Neuropix-PXI hardware and OpenEphys, and preprocessed using Kilosort 2.5 with manual curation in Phy. Trials were initiated by a nose-poke in a lit center port and required maintaining a center fixation for 0.8 to 1.2 seconds, during which a tone indicated the possible reward size. A subsequent side LED indicated the potential reward location, followed by a delay period drawn from an exponential distribution (mean = 2.5 s). Rats could opt out at any time by poking the unlit port, restarting the trial. Catch trials, where the delay period only ended if the rat opted out, constituted 15-25% of the trials. Rats received penalties for premature fixation breaks. Additionally, the tasks introduced semi-observable hidden states by varying reward statistics across uncued blocks (high, low, and mixed), structured hierarchically, with blocks transitioning after 40 successfully completed trials.\n",
+    "This notebook demonstrates how to convert an example session to NWB.\n",
+    "\n",
+    "This dataset have the following data streams:\n",
+    "- Behavior: Bpod output (.mat)\n",
+    "- Recording AP, LFP: OpenEphys (binary format)\n",
+    "- Units: Phy output\n",
+    "\n",
+    "## Notes on the conversion\n",
+    "\n",
+    "The conversion notes is located in `src/constantinople_lab_to_nwb/schierek_embargo_2024/schierek_embargo_2024_notes.md`. This file contains information about the expected file structure and the conversion process.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b221cfff-c7e5-44e7-bd3a-024bf2f26f67",
+   "metadata": {},
+   "source": [
+    "## Convert a single session to NWB\n",
+    "\n",
+    "The `schierek_embargo_2024_convert_session.py` script defines the `session_to_nwb` function that converts a session of Neuropixels data to NWB."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "695d7f2c-5b40-4e0d-b4b7-38546a4dce52",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2024-12-04T12:39:08.133836Z",
+     "start_time": "2024-12-04T12:39:08.128903Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from constantinople_lab_to_nwb.schierek_embargo_2024.schierek_embargo_2024_convert_session import session_to_nwb\n",
+    "from constantinople_lab_to_nwb.utils import get_subject_metadata_from_rat_info_folder\n",
+    "\n",
+    "# Parameters for conversion\n",
+    "\n",
+    "# The path to the NWB file to write.\n",
+    "nwbfile_path = \"/Volumes/T9/Constantinople/nwbfiles/sub-J076_ephys.nwb\"\n",
+    "# The OpenEphys recording folder path (make sure to include the 'Record Node #' in the folder path)\n",
+    "folder_path = \"/Volumes/T9/Constantinople/Ephys Data/J076_2023-12-12_14-52-04/Record Node 117\"\n",
+    "# The name of the *raw* recording stream (e.g. )\n",
+    "raw_stream_name = \"Neuropix-PXI-119.ProbeA-AP\"\n",
+    "# The name of the *LFP* recording stream (e.g. )\n",
+    "lfp_stream_name = \"Neuropix-PXI-119.ProbeA-LFP\"\n",
+    "# The path to the processed spike sorting file (.mat). This file contains the \"SU\" named struct that contains the processed spike data.\n",
+    "# This file also contains the \"S\" named struct containing the processed behavior data.\n",
+    "spike_sorting_mat_file_path = \"/Volumes/T9/Constantinople/Ephys Data/J076_2023-12-12.mat\"\n",
+    "# The path to the raw Bpod data\n",
+    "bpod_file_path = \"/Volumes/T9/Constantinople/raw_Bpod/J076/DataFiles/J076_RWTautowait2_20231212_145250.mat\"\n",
+    "\n",
+    "# Task specific parameters\n",
+    "\n",
+    "# The column name mapping is used to rename the columns in the processed data to more descriptive column names.\n",
+    "# New (unseen) task parameters should be added here.\n",
+    "column_name_mapping = dict(\n",
+    "    NoseInCenter=\"nose_in_center\",\n",
+    "    TrainingStage=\"training_stage\",\n",
+    "    Block=\"block_type\",\n",
+    "    BlockLengthAd=\"num_trials_in_adaptation_blocks\",\n",
+    "    BlockLengthTest=\"num_trials_in_test_blocks\",\n",
+    "    ProbCatch=\"catch_percentage\",\n",
+    "    RewardDelay=\"reward_delay\",\n",
+    "    RewardAmount=\"reward_volume_ul\",\n",
+    "    WaitForPoke=\"wait_for_center_poke\",\n",
+    "    hits=\"is_rewarded\",\n",
+    "    vios=\"is_violation\",\n",
+    "    optout=\"is_opt_out\",\n",
+    "    wait_time=\"wait_time\",\n",
+    "    wait_thresh=\"wait_time_threshold\",\n",
+    "    wait_for_cpoke=\"wait_for_center_poke\",\n",
+    "    zwait_for_cpoke=\"z_scored_wait_for_center_poke\",\n",
+    "    RewardedSide=\"rewarded_port\",\n",
+    "    Cled=\"center_poke_times\",\n",
+    "    Lled=\"left_poke_times\",\n",
+    "    Rled=\"right_poke_times\",\n",
+    "    l_opt=\"left_opt_out_times\",\n",
+    "    r_opt=\"right_opt_out_times\",\n",
+    "    ReactionTime=\"reaction_time\",\n",
+    "    slrt=\"short_latency_reaction_time\",\n",
+    "    iti=\"inter_trial_interval\",\n",
+    ")\n",
+    "# The column descriptions are used to add descriptions to the columns in the processed data.\n",
+    "# New (unseen) task parameter descriptions should be added here.\n",
+    "column_descriptions = dict(\n",
+    "    NoseInCenter=\"The time in seconds when the animal is required to maintain center port to initiate the trial (uniformly drawn from 0.8 - 1.2 seconds).\",\n",
+    "    TrainingStage=\"The stage of the training.\",\n",
+    "    Block=\"The block type (High, Low or Test). High and Low blocks are high reward (20, 40, or 80μL) or low reward (5, 10, or 20μL) blocks. Test blocks are mixed blocks.\",\n",
+    "    BlockLengthAd=\"The number of trials in each high reward (20, 40, or 80μL) or low reward (5, 10, or 20μL) blocks.\",\n",
+    "    BlockLengthTest=\"The number of trials in each mixed blocks.\",\n",
+    "    ProbCatch=\"The percentage of catch trials.\",\n",
+    "    RewardDelay=\"The delay in seconds to receive reward, drawn from exponential distribution with mean = 2.5 seconds.\",\n",
+    "    RewardAmount=\"The volume of reward in microliters.\",\n",
+    "    hits=\"Whether the subject received reward for each trial.\",\n",
+    "    vios=\"Whether the subject violated the trial by not maintaining center poke for the time required by 'nose_in_center'.\",\n",
+    "    optout=\"Whether the subject opted out for each trial.\",\n",
+    "    WaitForPoke=\"The time (s) between side port poke and center poke.\",\n",
+    "    wait_time=\"The wait time for the subject for for each trial in seconds, after removing outliers.\"\n",
+    "        \" For hit trials (when reward was delivered) the wait time is equal to the reward delay.\"\n",
+    "        \" For opt-out trials, the wait time is equal to the time waited from trial start to opting out.\",\n",
+    "    wait_for_cpoke=\"The time between side port poke and center poke in seconds, includes the time when the subject is consuming the reward.\",\n",
+    "    zwait_for_cpoke=\"The z-scored wait_for_cpoke using all trials.\",\n",
+    "    RewardedSide=\"The rewarded port (Left or Right) for each trial.\",\n",
+    "    Cled=\"The time of center port LED on/off for each trial (2 x ntrials).\",\n",
+    "    Lled=\"The time of left port LED on/off for each trial (2 x ntrials).\",\n",
+    "    Rled=\"The time of right port LED on/off for each trial (2 x ntrials).\",\n",
+    "    l_opt=\"The time of left port entered/exited for each trial (2 x ntrials).\",\n",
+    "    r_opt=\"The time of right port entered/exited for each trial (2 x ntrials).\",\n",
+    "    ReactionTime=\"The reaction time in seconds.\",\n",
+    "    slrt=\"The short-latency reaction time in seconds.\",\n",
+    "    iti=\"The time to initiate trial in seconds (the time between the end of the consummatory period and the time to initiate the next trial).\",\n",
+    "    wait_thresh=\"The threshold in seconds to remove wait-times (mean + 1*std of all cumulative wait-times).\",\n",
+    ")\n",
+    "\n",
+    "\n",
+    "# Get subject metadata from rat registry\n",
+    "rat_registry_folder_path = \"/Volumes/T9/Constantinople/Rat_info\"\n",
+    "subject_metadata = get_subject_metadata_from_rat_info_folder(\n",
+    "    folder_path=rat_registry_folder_path,\n",
+    "    subject_id=\"J076\",\n",
+    "    date=\"2023-12-12\",\n",
+    ")\n",
+    "\n",
+    "# Ephys registry file path (constains metadata for the neuropixels probe)\n",
+    "ephys_registry_file_path = \"/Volumes/T9/Constantinople/Ephys Data/Ephys_registry.mat\"\n",
+    "\n",
+    "\n",
+    "# Optional parameters\n",
+    "\n",
+    "# Whether to run a stub test conversion. The stubbed file will only contain a small portion of data.\n",
+    "# When running the full conversion stub_test should be disabled.\n",
+    "stub_test = True\n",
+    "# Whether to overwrite an existing NWB file.\n",
+    "overwrite = True\n",
+    "\n",
+    "# Run the `session_to_nwb` function with the parameters to convert to NWB\n",
+    "session_to_nwb(\n",
+    "    nwbfile_path=nwbfile_path,\n",
+    "    openephys_recording_folder_path=folder_path,\n",
+    "    ap_stream_name=raw_stream_name,\n",
+    "    lfp_stream_name=lfp_stream_name,\n",
+    "    processed_spike_sorting_file_path=spike_sorting_mat_file_path,\n",
+    "    raw_behavior_file_path=bpod_file_path,\n",
+    "    column_name_mapping=column_name_mapping,\n",
+    "    column_descriptions=column_descriptions,\n",
+    "    ephys_registry_file_path=ephys_registry_file_path,\n",
+    "    subject_metadata=subject_metadata,\n",
+    "    stub_test=stub_test,\n",
+    "    overwrite=overwrite,\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "9e70b3e5-de74-4f15-bec0-688151d80ce4",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "37140fad-10d0-436d-a4a8-452083d33d10",
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}