From 4f58c68562820c583ac04fccd8a394e729adafdb Mon Sep 17 00:00:00 2001 From: Kushal Bakshi <52367253+kushalbakshi@users.noreply.github.com> Date: Tue, 28 Nov 2023 12:35:46 -0600 Subject: [PATCH 01/14] Update diagram_flowchart.drawio --- images/diagram_flowchart.drawio | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/images/diagram_flowchart.drawio b/images/diagram_flowchart.drawio index b259ce3e..998e0220 100644 --- a/images/diagram_flowchart.drawio +++ b/images/diagram_flowchart.drawio @@ -1,6 +1,6 @@ - + - + From 4a4104c92a370162a4cb51642d337b3567ef28fe Mon Sep 17 00:00:00 2001 From: Kushal Bakshi <52367253+kushalbakshi@users.noreply.github.com> Date: Tue, 28 Nov 2023 12:38:23 -0600 Subject: [PATCH 02/14] Added diagram_flowchart.svg --- images/diagram_flowchart.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/images/diagram_flowchart.svg b/images/diagram_flowchart.svg index e8b1df4b..b4f1c1c5 100644 --- a/images/diagram_flowchart.svg +++ b/images/diagram_flowchart.svg @@ -1,4 +1,4 @@ -
Generate quality
control metrics
Generate quality...
Optionally
curate units
Optionally...
Acquire Neuropixels data with OpenEphys
 or SpikeGLX
Acquire Neuropixels...
Enter metadata
into pipeline
Enter metadata...
Process with
Kilosort
Process with...
Visualize



 
Visualize...

 Export & publish

 
 
Export & publish...
Synchronize data modalities
& exploratory
analysis
Synchronize data mo...Text is not SVG - cannot display \ No newline at end of file +
Generate quality
control metrics
Generate quality...
Optionally
curate units
Optionally...
Acquire Neuropixels data with OpenEphys
 or SpikeGLX
Acquire Neuropixels...
Enter metadata
into pipeline
Enter metadata...
Process with
Kilosort
Process with...
Visualize



 
Visualize...

 Export & publish

 
 
Export & publish...
Synchronize data modalities
& exploratory
analysis
Synchronize data mo...Text is not SVG - cannot display \ No newline at end of file From bf63fe1b6887075a123de99a5b00fd20f9ee9561 Mon Sep 17 00:00:00 2001 From: Kushal Bakshi <52367253+kushalbakshi@users.noreply.github.com> Date: Tue, 28 Nov 2023 12:39:23 -0600 Subject: [PATCH 03/14] Added diagram_flowchart.svg --- images/diagram_flowchart.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/images/diagram_flowchart.svg b/images/diagram_flowchart.svg index b4f1c1c5..eed92b1c 100644 --- a/images/diagram_flowchart.svg +++ b/images/diagram_flowchart.svg @@ -1,4 +1,4 @@ -
Generate quality
control metrics
Generate quality...
Optionally
curate units
Optionally...
Acquire Neuropixels data with OpenEphys
 or SpikeGLX
Acquire Neuropixels...
Enter metadata
into pipeline
Enter metadata...
Process with
Kilosort
Process with...
Visualize



 
Visualize...

 Export & publish

 
 
Export & publish...
Synchronize data modalities
& exploratory
analysis
Synchronize data mo...Text is not SVG - cannot display \ No newline at end of file +
Generate quality
control metrics
Generate quality...
Optionally
curate units
Optionally...
Acquire Neuropixels data with OpenEphys
 or SpikeGLX
Acquire Neuropixels...
Enter metadata
into pipeline
Enter metadata...
Process with
Kilosort
Process with...
Visualize



 
Visualize...

 Export & publish

 
 
Export & publish...
Synchronize data modalities
& exploratory
analysis
Synchronize data mo...Text is not SVG - cannot display \ No newline at end of file From 87175289eb3a0aa1d401bb41f5b9f7e73c62659a Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 28 Nov 2023 13:06:52 -0600 Subject: [PATCH 04/14] Minor updates for formatting and structure --- notebooks/tutorial.ipynb | 745 ++++++--------------------------------- 1 file changed, 108 insertions(+), 637 deletions(-) diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb index 9cbec92c..38cd8396 100644 --- a/notebooks/tutorial.ipynb +++ b/notebooks/tutorial.ipynb @@ -5,100 +5,56 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# DataJoint Elements for Array Electrophysiology with NeuroPixels" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### **Open-source Data Pipeline for Processing and Analyzing Extracellular Electrophysiology Datasets**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This tutorial aims to provide a comprehensive understanding of the open-source data pipeline by `element-array-ephys`. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![flowchart](../images/diagram_flowchart.svg)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The package is designed to **process NeuroPixels ephys data** with **OpenEphys** and spike sorted with **Kilosort**. The following Diagram corresponds to the `ephys_acute` module:" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![pipeline](../images/attached_array_ephys_element_acute.svg)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "By the end of this tutorial, you will have a clear grasp of how to set up and integrate the `element-array-ephys` into your specific research projects and your lab." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### **Key Components and Objectives**\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "**- Setup**\n", + "# DataJoint Elements for Array Electrophysiology with NeuroPixels\n", "\n", - "**- Designing the DataJoint Pipeline**\n", + "#### Open-source Data Pipeline for Processing and Analyzing Extracellular Electrophysiology Datasets\n", "\n", - "**- Step 1: Insert Example Data into Subject and Session tables**\n", + "Welcome to the tutorial for the DataJoint Element for extracellular array electrophysiology. This\n", + "tutorial aims to provide a comprehensive understanding of the open-source data pipeline\n", + "created using `element-array-ephys`.\n", "\n", - "**- Step 2: Register the Electrophysiology Recording information for each Probe**\n", + "This package is designed to seamlessly process and ingest extracellular electrophysiology\n", + "data, along with its associated probe and recording metadata. By the end of this\n", + "tutorial you will have a clear grasp on setting up and integrating `element-array-ephys`\n", + "into your specific research projects and lab. \n", "\n", - "**- Step 3: Run the Clustering Task**\n", + "![flowchart](https://github.com/datajoint/element-array-ephys/blob/main/images/diagram_flowchart.svg)\n", "\n", - "**- Step 4: Curate the Clustering Results (Optional)**\n", + "### Prerequisites\n", "\n", - "**- Step 5: Visualize the Results**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### **Setup**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This tutorial examines the `ephys_acute` module applied to physiological recordings and automatic ingestion of spike sorting results.\n", + "Please see the [datajoint tutorials GitHub\n", + "repository](https://github.com/datajoint/datajoint-tutorials/tree/main) before\n", + "proceeding.\n", "\n", - "The goal is to store, track and manage different curations of the spike sorting results and unit-level visualization results.\n", + "A basic understanding of the following DataJoint concepts will be beneficial to your\n", + "understanding of this tutorial: \n", + "1. The `Imported` and `Computed` tables types in `datajoint-python`.\n", + "2. The functionality of the `.populate()` method. \n", "\n", - "The results of this Element example can be combined with other modalities to create a complete customizable data pipeline for your specific lab or study. For instance, you can combine `element-array-ephys` with `element-calcium-imaging` and `element-deeplabcut` to characterize the neural activity." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's start this tutorial by importing the packages necessary to run the data pipeline." + "#### **Tutorial Overview**\n", + "\n", + "+ Setup\n", + "+ *Activate* the DataJoint pipeline.\n", + "+ *Insert* subject, session, and probe metadata.\n", + "+ *Populate* electrophysiology recording metadata.\n", + "+ Run the clustering task.\n", + "+ Curate the results (optional).\n", + "+ Visualize the results.\n", + "\n", + "### **Setup**\n", + "\n", + "This tutorial examines extracellular electrophysiology data acquired with `OpenEphys`\n", + "and spike-sorted using Kilosort 2.5. The goal is to store, track\n", + "and manage sessions of array electrophysiology data, including spike sorting results and\n", + "unit-level visualizations. \n", + "\n", + "The results of this Element example can be combined with **other modalities** to create\n", + "a complete customizable data pipeline for your specific lab or study. For instance, you\n", + "can combine `element-array-ephys` with `element-calcium-imaging` and\n", + "`element-deeplabcut` to characterize the neural activity along with markless\n", + "pose-estimation during behavior.\n", + "\n", + "Let's start this tutorial by importing the packages necessary to run the notebook." ] }, { @@ -117,7 +73,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "This codespace provides a local database private to you for experimentation. Let's connect to the database server:" + "If the tutorial is run in Codespaces, there is a local database private to you for\n", + "experimentation. Let's connect to the database server." ] }, { @@ -152,16 +109,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### **Design the DataJoint Pipeline**" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This tutorial presumes that the `element-array-ephys` has been pre-configured and instantiated, with the database linked downstream to pre-existing `subject` and `session` tables. \n", + "### **Activate the DataJoint Pipeline**\n", "\n", - "Now, we will proceed to import the essential schemas required to construct this data pipeline, with particular attention to the primary components: `probe` and `ephys`." + "This tutorial activates the `ephys-acute.py` module from `element-array-ephys`, along\n", + "with upstream dependencies from `element-animal` and `element-session`. Please refer to the\n", + "[`tutorial_pipeline.py`](./tutorial_pipeline.py) for the source code." ] }, { @@ -185,7 +137,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can represent a diagram of some of the upstream and downstream dependencies connected to these `probe` and `ephys` schemas:" + "We can represent the tables in the `probe` and `ephys` schemas as well as some of the\n", + "upstream dependencies to `session` and `subject` schemas as a diagram." ] }, { @@ -657,465 +610,18 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "As evident, this data pipeline is fairly comprehensive, encompassing several tables associated with different Array Electrophysiology components like ephys recording, probe, and clustering. A few tables, such as `Subject` or `Session`, while integral to the pipeline, fall outside the scope of `element-array-ephys` tutorial as they are upstream. \n", + "As evident from the diagram, this data pipeline encompasses tables associated with\n", + "recording and probe metadata, results of clustering, and optional curation of clustering\n", + "results. A few tables, such as `subject.Subject` or `session.Session`,\n", + "while important for a complete pipeline, fall outside the scope of `element-array-ephys`\n", + "tutorial, and will therefore, not be explored extensively here. The primary focus of\n", + "this tutorial will be on the `probe` and `ephys` schemas.\n", + "\n", + "### **Insert subject, session, and probe metadata**\n", "\n", - "Our focus in this tutorial will be primarily on the two core schemas:" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "image/svg+xml": [ - "\n", - "\n", - "%3\n", - "\n", - "\n", - "\n", - "ephys.LFP.Electrode\n", - "\n", - "\n", - "ephys.LFP.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.WaveformSet.PeakWaveform\n", - "\n", - "\n", - "ephys.WaveformSet.PeakWaveform\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusteringTask\n", - "\n", - "\n", - "ephys.ClusteringTask\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.Clustering\n", - "\n", - "\n", - "ephys.Clustering\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusteringTask->ephys.Clustering\n", - "\n", - "\n", - "\n", - "\n", - "ephys.QualityMetrics\n", - "\n", - "\n", - "ephys.QualityMetrics\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.QualityMetrics.Waveform\n", - "\n", - "\n", - "ephys.QualityMetrics.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.QualityMetrics->ephys.QualityMetrics.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.QualityMetrics.Cluster\n", - "\n", - "\n", - "ephys.QualityMetrics.Cluster\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.QualityMetrics->ephys.QualityMetrics.Cluster\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit->ephys.WaveformSet.PeakWaveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit->ephys.QualityMetrics.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.WaveformSet.Waveform\n", - "\n", - "\n", - "ephys.WaveformSet.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit->ephys.WaveformSet.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit->ephys.QualityMetrics.Cluster\n", - "\n", - "\n", - "\n", - "\n", - "ephys.LFP\n", - "\n", - "\n", - "ephys.LFP\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.LFP->ephys.LFP.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "ephys.Curation\n", - "\n", - "\n", - "ephys.Curation\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering\n", - "\n", - "\n", - "ephys.CuratedClustering\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.Curation->ephys.CuratedClustering\n", - "\n", - "\n", - "\n", - "\n", - "ephys.Clustering->ephys.Curation\n", - "\n", - "\n", - "\n", - "\n", - "ephys.AcquisitionSoftware\n", - "\n", - "\n", - "ephys.AcquisitionSoftware\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.EphysRecording\n", - "\n", - "\n", - "ephys.EphysRecording\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.AcquisitionSoftware->ephys.EphysRecording\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ProbeInsertion\n", - "\n", - "\n", - "ephys.ProbeInsertion\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ProbeInsertion->ephys.EphysRecording\n", - "\n", - "\n", - "\n", - "\n", - "ephys.InsertionLocation\n", - "\n", - "\n", - "ephys.InsertionLocation\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ProbeInsertion->ephys.InsertionLocation\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode->ephys.LFP.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode->ephys.CuratedClustering.Unit\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode->ephys.WaveformSet.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.EphysRecording->ephys.ClusteringTask\n", - "\n", - "\n", - "\n", - "\n", - "ephys.EphysRecording->ephys.LFP\n", - "\n", - "\n", - "\n", - "\n", - "ephys.EphysRecording.EphysFile\n", - "\n", - "\n", - "ephys.EphysRecording.EphysFile\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.EphysRecording->ephys.EphysRecording.EphysFile\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusteringMethod\n", - "\n", - "\n", - "ephys.ClusteringMethod\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusteringParamSet\n", - "\n", - "\n", - "ephys.ClusteringParamSet\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusteringMethod->ephys.ClusteringParamSet\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering->ephys.QualityMetrics\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering->ephys.CuratedClustering.Unit\n", - "\n", - "\n", - "\n", - "\n", - "ephys.WaveformSet\n", - "\n", - "\n", - "ephys.WaveformSet\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.CuratedClustering->ephys.WaveformSet\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusteringParamSet->ephys.ClusteringTask\n", - "\n", - "\n", - "\n", - "\n", - "probe.ProbeType.Electrode\n", - "\n", - "\n", - "probe.ProbeType.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "probe.ProbeType.Electrode->probe.ElectrodeConfig.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "ephys.WaveformSet->ephys.WaveformSet.PeakWaveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.WaveformSet->ephys.WaveformSet.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "probe.Probe\n", - "\n", - "\n", - "probe.Probe\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "probe.Probe->ephys.ProbeInsertion\n", - "\n", - "\n", - "\n", - "\n", - "probe.ProbeType\n", - "\n", - "\n", - "probe.ProbeType\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "probe.ProbeType->probe.ProbeType.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "probe.ProbeType->probe.Probe\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig\n", - "\n", - "\n", - "probe.ElectrodeConfig\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "probe.ProbeType->probe.ElectrodeConfig\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusterQualityLabel\n", - "\n", - "\n", - "ephys.ClusterQualityLabel\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "ephys.ClusterQualityLabel->ephys.CuratedClustering.Unit\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig->probe.ElectrodeConfig.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig->ephys.EphysRecording\n", - "\n", - "\n", - "\n", - "" - ], - "text/plain": [ - "" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "dj.Diagram(probe) + dj.Diagram(ephys)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This diagram represents an example of the `element-array-ephys` pipeline." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### **Step 1 - Insert Example Data into Subject and Session tables**" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ "Let's start with the first table in the schema diagram (i.e. `subject.Subject` table).\n", "\n", - "To know what data to insert into the table, we can view its dependencies and attributes using the `.describe()` and `.heading` functions." + "To know what data to insert into the table, we can view its dependencies and attributes using the `.describe()` and `.heading` methods." ] }, { @@ -1558,14 +1064,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### **Step 2: Register the Electrophysiology Recording information for each Probe**\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Every experimental session produces a set of data files. The purpose of the `SessionDirectory` table is to locate these files. It references a directory path relative to a root directory, defined in `dj.config[\\\"custom\\\"]`. More information about `dj.config` is provided in the [Documentation](https://datajoint.com/docs/elements/user-guide/)." + "Every experimental session produces a set of data files. The purpose of the `SessionDirectory` table is to locate these files. It references a directory path relative to a root directory, defined in `dj.config[\\\"custom\\\"]`. More information about `dj.config` is provided in the [documentation](https://datajoint.com/docs/elements/user-guide/)." ] }, { @@ -2055,7 +1554,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "In the upcoming cells, populate the `ephys.EphysRecording` table and its part table `ephys.EphysRecording.EphysFile` will extract and store the recording information from a given experimental session.\n" + "### **Populate electrophysiology recording metadata**\n", + "\n", + "In the upcoming cells, populate the `ephys.EphysRecording` table and its part table `ephys.EphysRecording.EphysFile` will extract and store the recording information from a given experimental session." ] }, { @@ -2509,18 +2010,13 @@ "ephys.EphysRecording.EphysFile()" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### **Step 3: Run the Clustering Task**" - ] - }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ + "### **Step 3: Run the Clustering Task**\n", + "\n", "We're almost ready to spike sort the data with `kilosort`. An important step before\n", "processing is managing the parameters which will be used in that step. To do so, we will\n", "define the kilosort parameters in a dictionary and insert them into a DataJoint table\n", @@ -2790,14 +2286,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### **Step 4: Curate the Clustering Results (Optional)**" - ] - }, - { - "attachments": {}, - "cell_type": "markdown", - "metadata": {}, - "source": [ + "### **Curate the results (Optional)**\n", + "\n", "While spike sorting is completed in the above step, you can optionally curate\n", "the output of image processing using the `Curation` table. For this demo, we\n", "will simply use the results of the spike sorting output from the `Clustering` task." @@ -2892,7 +2382,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### **Step 5: Visualization of Results**" + "### **Visualize the results**" ] }, { @@ -2942,7 +2432,7 @@ "plt.plot(lfp_average)\n", "plt.title(\"Average LFP Waveform for Insertion 1\")\n", "plt.xlabel(\"Samples\")\n", - "plt.ylabel(\"microvolts (uV)\");" + "plt.ylabel(\"microvolts (uV)\")" ] }, { @@ -2997,7 +2487,15 @@ "y = np.hstack([np.full_like(s, u) for u, s in zip(units, unit_spiketimes)])\n", "plt.plot(x, y, \"|\")\n", "plt.xlabel(\"Time (s)\")\n", - "plt.ylabel(\"Unit\");" + "plt.ylabel(\"Unit\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Below we will use two queries to fetch *all* of the information about a single unit and\n", + "plot the unit waveform." ] }, { @@ -3252,7 +2750,7 @@ " unit_data[\"peak_electrode_waveform\"],\n", ")\n", "plt.xlabel(\"Time (ms)\")\n", - "plt.ylabel(r\"Voltage ($\\mu$V)\");" + "plt.ylabel(r\"Voltage ($\\mu$V)\")" ] }, { @@ -3261,66 +2759,32 @@ "source": [ "## Summary\n", "\n", - "Following this tutorial, we've efficiently: \n", - "- covered the essential functionality of `element-array-ephys`\n", - "- acquired the skills to register the electrophysiology recordings for each probe in each experimental session\n", - "- insert data into tables\n", - "- execute the spike sort with Kilosort\n", - "- visualize the results " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Documentation and DataJoint tutorials" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For detailed documentation on `element-array-ephys`:\n", - "\n", - "[`DataJoint Element for Extracellular Electrophysiology - Documentation`](https://datajoint.com/docs/elements/element-array-ephys/)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For detailed documentation and tutorials on general DataJoint principles that support collaboration, automation, reproducibility, and visualizations:\n", + "Following this tutorial, we have: \n", + "+ Covered the essential functionality of `element-array-ephys`.\n", + "+ Learned how to manually insert data into tables.\n", + "+ Executed and ingested results of spike sorting with Kilosort.\n", + "+ Visualized the results. \n", "\n", - "[`DataJoint for Python - Interactive Tutorials`](https://github.com/datajoint/datajoint-tutorials) covers fundamentals, including table tiers, query operations, fetch operations, automated computations with the make function, and more.\n", - "\n", - "[`DataJoint for Python - Documentation`](https://datajoint.com/docs/core/datajoint-python/0.14/)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Run this tutorial on your own data" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Throughout this notebook, we've used DataJoint to work with database tables and keep\n", - "data organized and automate analyses to increase efficiency of data processing. We've\n", - "inserted data into tables, used queries to retrieve, manipulate, and visualize ephys data.\n", + "#### Documentation and DataJoint Tutorials\n", "\n", - "Remember, this is just the beginning. As you grow familiar with DataJoint, you'll\n", - "uncover even more ways to harness its capabilities for your specific research needs. \n", + "+ [Detailed documentation on\n", + " `element-array-ephys`.](https://datajoint.com/docs/elements/element-array-ephys/)\n", + "+ [General `datajoint-python`\n", + " tutorials.](https://github.com/datajoint/datajoint-tutorials) covering fundamentals,\n", + " such as table tiers, query operations, fetch operations, automated computations with the\n", + " make function, and more.\n", + "+ [Documentation for\n", + " `datajoint-python`.](https://datajoint.com/docs/core/datajoint-python/)\n", "\n", + "##### Run this tutorial on your own data\n", "\n", "To run this tutorial notebook on your own data, please use the following steps:\n", - "- Download the mysql-docker image for DataJoint and run the container according to the\n", - " instructions provide in the repository.\n", - "- Create a fork of this repository to your GitHub account.\n", - "- Clone the repository and open the files using your IDE.\n", - "- Add a code cell immediately after the first code cell in the notebook - we will setup\n", + "+ Download the [mysql-docker image for\n", + " DataJoint](https://github.com/datajoint/mysql-docker) and run the container according\n", + " to the instructions provide in the repository.\n", + "+ Create a fork of this repository to your GitHub account.\n", + "+ Clone the repository and open the files using your IDE.\n", + "+ Add a code cell immediately after the first code cell in the notebook - we will setup\n", " the local connection using this cell. In this cell, type in the following code. \n", "\n", "```python\n", @@ -3334,8 +2798,15 @@ "dj.conn()\n", "```\n", "\n", - "- Run this code block above and proceed with the rest of the notebook." + "+ Run the code block above and proceed with the rest of the notebook." ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] } ], "metadata": { From f424a0f7726e8dcfafe99b2f053af194d856a536 Mon Sep 17 00:00:00 2001 From: Kushal Bakshi <52367253+kushalbakshi@users.noreply.github.com> Date: Tue, 28 Nov 2023 19:16:33 +0000 Subject: [PATCH 05/14] Update path to flowchart diagram --- notebooks/tutorial.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb index 38cd8396..54e9ba8f 100644 --- a/notebooks/tutorial.ipynb +++ b/notebooks/tutorial.ipynb @@ -18,7 +18,7 @@ "tutorial you will have a clear grasp on setting up and integrating `element-array-ephys`\n", "into your specific research projects and lab. \n", "\n", - "![flowchart](https://github.com/datajoint/element-array-ephys/blob/main/images/diagram_flowchart.svg)\n", + "![flowchart](../images/diagram_flowchart.svg)\n", "\n", "### Prerequisites\n", "\n", From 6323d4ee7ac4cd07b494ce93cca8ac159c0bf843 Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 28 Nov 2023 13:17:53 -0600 Subject: [PATCH 06/14] Fix heading styling --- notebooks/tutorial.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb index 38cd8396..47388cc2 100644 --- a/notebooks/tutorial.ipynb +++ b/notebooks/tutorial.ipynb @@ -2015,7 +2015,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### **Step 3: Run the Clustering Task**\n", + "### **Run the Clustering Task**\n", "\n", "We're almost ready to spike sort the data with `kilosort`. An important step before\n", "processing is managing the parameters which will be used in that step. To do so, we will\n", From da2239dbc06dc018f78823c6c36a7f83ff48a5d4 Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 28 Nov 2023 13:24:48 -0600 Subject: [PATCH 07/14] Minor fixes to README --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index b59fda94..85277136 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,9 @@ environment and notebooks to learn the pipeline. ## Getting Started + Please fork this repository. + + Clone the repository to your computer. + ```bash git clone https://github.com//element-array-ephys.git ``` @@ -38,6 +40,7 @@ environment and notebooks to learn the pipeline. ```bash pip install -e . ``` + + [Interactive tutorial on GitHub Codespaces](https://github.com/datajoint/element-array-ephys#interactive-tutorial) From 43ff0d42a1abe89f6bf3f3f334b32773ca0d2c37 Mon Sep 17 00:00:00 2001 From: Kushal Bakshi <52367253+kushalbakshi@users.noreply.github.com> Date: Tue, 28 Nov 2023 19:37:41 +0000 Subject: [PATCH 08/14] Run tutorial notebook --- notebooks/tutorial.ipynb | 716 +++++++++++++++++++-------------------- 1 file changed, 358 insertions(+), 358 deletions(-) diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb index 7f79de22..cfee4903 100644 --- a/notebooks/tutorial.ipynb +++ b/notebooks/tutorial.ipynb @@ -86,8 +86,8 @@ "name": "stderr", "output_type": "stream", "text": [ - "[2023-11-27 15:06:43,279][INFO]: Connecting root@fakeservices.datajoint.io:3306\n", - "[2023-11-27 15:06:43,287][INFO]: Connected root@fakeservices.datajoint.io:3306\n" + "[2023-11-28 19:33:47,134][INFO]: Connecting root@fakeservices.datajoint.io:3306\n", + "[2023-11-28 19:33:47,142][INFO]: Connected root@fakeservices.datajoint.io:3306\n" ] }, { @@ -125,7 +125,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "[2023-11-27 15:06:47,930][WARNING]: lab.Project and related tables will be removed in a future version of Element Lab. Please use the project schema.\n" + "[2023-11-28 19:33:49,428][WARNING]: lab.Project and related tables will be removed in a future version of Element Lab. Please use the project schema.\n" ] } ], @@ -149,447 +149,447 @@ { "data": { "image/svg+xml": [ - "\n", + "\n", "\n", "%3\n", - "\n", - "\n", + "\n", + "\n", "\n", - "ephys.LFP.Electrode\n", - "\n", - "\n", - "ephys.LFP.Electrode\n", + "ephys.LFP\n", + "\n", + "\n", + "ephys.LFP\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.WaveformSet.PeakWaveform\n", - "\n", - "\n", - "ephys.WaveformSet.PeakWaveform\n", + "\n", + "\n", + "ephys.LFP.Electrode\n", + "\n", + "\n", + "ephys.LFP.Electrode\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.ClusteringTask\n", - "\n", - "\n", - "ephys.ClusteringTask\n", + "\n", + "\n", + "ephys.LFP->ephys.LFP.Electrode\n", + "\n", + "\n", + "\n", + "\n", + "ephys.EphysRecording.EphysFile\n", + "\n", + "\n", + "ephys.EphysRecording.EphysFile\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.Clustering\n", - "\n", - "\n", - "ephys.Clustering\n", + "\n", + "\n", + "ephys.QualityMetrics.Cluster\n", + "\n", + "\n", + "ephys.QualityMetrics.Cluster\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.ClusteringTask->ephys.Clustering\n", - "\n", - "\n", - "\n", + "\n", "\n", - "ephys.QualityMetrics\n", - "\n", - "\n", - "ephys.QualityMetrics\n", + "probe.ProbeType.Electrode\n", + "\n", + "\n", + "probe.ProbeType.Electrode\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.QualityMetrics.Waveform\n", - "\n", - "\n", - "ephys.QualityMetrics.Waveform\n", + "\n", + "\n", + "probe.ElectrodeConfig.Electrode\n", + "\n", + "\n", + "probe.ElectrodeConfig.Electrode\n", "\n", "\n", "\n", - "\n", + "\n", "\n", - "ephys.QualityMetrics->ephys.QualityMetrics.Waveform\n", - "\n", + "probe.ProbeType.Electrode->probe.ElectrodeConfig.Electrode\n", + "\n", "\n", - "\n", - "\n", - "ephys.QualityMetrics.Cluster\n", - "\n", - "\n", - "ephys.QualityMetrics.Cluster\n", + "\n", + "\n", + "ephys.Curation\n", + "\n", + "\n", + "ephys.Curation\n", "\n", "\n", "\n", - "\n", + "\n", + "\n", + "ephys.CuratedClustering\n", + "\n", + "\n", + "ephys.CuratedClustering\n", + "\n", + "\n", + "\n", + "\n", "\n", - "ephys.QualityMetrics->ephys.QualityMetrics.Cluster\n", - "\n", + "ephys.Curation->ephys.CuratedClustering\n", + "\n", "\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit\n", - "\n", - "\n", - "ephys.CuratedClustering.Unit\n", + "\n", + "\n", + "ephys.ProbeInsertion\n", + "\n", + "\n", + "ephys.ProbeInsertion\n", "\n", "\n", "\n", - "\n", + "\n", + "\n", + "ephys.InsertionLocation\n", + "\n", + "\n", + "ephys.InsertionLocation\n", + "\n", + "\n", + "\n", + "\n", "\n", - "ephys.CuratedClustering.Unit->ephys.WaveformSet.PeakWaveform\n", - "\n", + "ephys.ProbeInsertion->ephys.InsertionLocation\n", + "\n", "\n", - "\n", + "\n", + "\n", + "ephys.EphysRecording\n", + "\n", + "\n", + "ephys.EphysRecording\n", + "\n", + "\n", + "\n", + "\n", "\n", - "ephys.CuratedClustering.Unit->ephys.QualityMetrics.Waveform\n", - "\n", + "ephys.ProbeInsertion->ephys.EphysRecording\n", + "\n", "\n", - "\n", - "\n", - "ephys.WaveformSet.Waveform\n", - "\n", - "\n", - "ephys.WaveformSet.Waveform\n", + "\n", + "\n", + "probe.ElectrodeConfig\n", + "\n", + "\n", + "probe.ElectrodeConfig\n", "\n", "\n", "\n", - "\n", + "\n", "\n", - "ephys.CuratedClustering.Unit->ephys.WaveformSet.Waveform\n", - "\n", + "probe.ElectrodeConfig->probe.ElectrodeConfig.Electrode\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.CuratedClustering.Unit->ephys.QualityMetrics.Cluster\n", - "\n", + "probe.ElectrodeConfig->ephys.EphysRecording\n", + "\n", "\n", - "\n", - "\n", - "session.Session\n", - "\n", - "\n", - "session.Session\n", + "\n", + "\n", + "ephys.WaveformSet.PeakWaveform\n", + "\n", + "\n", + "ephys.WaveformSet.PeakWaveform\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.ProbeInsertion\n", - "\n", - "\n", - "ephys.ProbeInsertion\n", + "\n", + "\n", + "ephys.CuratedClustering.Unit\n", + "\n", + "\n", + "ephys.CuratedClustering.Unit\n", "\n", "\n", "\n", - "\n", + "\n", "\n", - "session.Session->ephys.ProbeInsertion\n", - "\n", - "\n", - "\n", - "\n", - "ephys.LFP\n", - "\n", - "\n", - "ephys.LFP\n", - "\n", - "\n", + "ephys.CuratedClustering.Unit->ephys.QualityMetrics.Cluster\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.LFP->ephys.LFP.Electrode\n", - "\n", - "\n", - "\n", - "\n", - "ephys.Curation\n", - "\n", - "\n", - "ephys.Curation\n", - "\n", - "\n", + "ephys.CuratedClustering.Unit->ephys.WaveformSet.PeakWaveform\n", + "\n", "\n", - "\n", - "\n", - "ephys.CuratedClustering\n", - "\n", - "\n", - "ephys.CuratedClustering\n", + "\n", + "\n", + "ephys.WaveformSet.Waveform\n", + "\n", + "\n", + "ephys.WaveformSet.Waveform\n", "\n", "\n", "\n", - "\n", + "\n", "\n", - "ephys.Curation->ephys.CuratedClustering\n", - "\n", - "\n", - "\n", - "\n", - "ephys.Clustering->ephys.Curation\n", - "\n", + "ephys.CuratedClustering.Unit->ephys.WaveformSet.Waveform\n", + "\n", "\n", - "\n", - "\n", - "ephys.AcquisitionSoftware\n", - "\n", - "\n", - "ephys.AcquisitionSoftware\n", + "\n", + "\n", + "ephys.QualityMetrics.Waveform\n", + "\n", + "\n", + "ephys.QualityMetrics.Waveform\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.EphysRecording\n", - "\n", - "\n", - "ephys.EphysRecording\n", + "\n", + "\n", + "ephys.CuratedClustering.Unit->ephys.QualityMetrics.Waveform\n", + "\n", + "\n", + "\n", + "\n", + "ephys.WaveformSet\n", + "\n", + "\n", + "ephys.WaveformSet\n", "\n", "\n", "\n", - "\n", + "\n", "\n", - "ephys.AcquisitionSoftware->ephys.EphysRecording\n", - "\n", + "ephys.WaveformSet->ephys.WaveformSet.PeakWaveform\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.ProbeInsertion->ephys.EphysRecording\n", - "\n", + "ephys.WaveformSet->ephys.WaveformSet.Waveform\n", + "\n", "\n", - "\n", - "\n", - "ephys.InsertionLocation\n", - "\n", - "\n", - "ephys.InsertionLocation\n", + "\n", + "\n", + "probe.ProbeType\n", + "\n", + "\n", + "probe.ProbeType\n", "\n", "\n", "\n", - "\n", + "\n", "\n", - "ephys.ProbeInsertion->ephys.InsertionLocation\n", - "\n", + "probe.ProbeType->probe.ProbeType.Electrode\n", + "\n", "\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode\n", - "\n", + "\n", + "\n", + "probe.ProbeType->probe.ElectrodeConfig\n", + "\n", "\n", + "\n", + "\n", + "probe.Probe\n", + "\n", + "\n", + "probe.Probe\n", + "\n", "\n", - "\n", - "\n", - "probe.ElectrodeConfig.Electrode->ephys.LFP.Electrode\n", - "\n", "\n", - "\n", + "\n", "\n", - "probe.ElectrodeConfig.Electrode->ephys.CuratedClustering.Unit\n", - "\n", + "probe.ProbeType->probe.Probe\n", + "\n", "\n", - "\n", + "\n", "\n", - "probe.ElectrodeConfig.Electrode->ephys.WaveformSet.Waveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.EphysRecording->ephys.ClusteringTask\n", - "\n", - "\n", - "\n", - "\n", - "ephys.EphysRecording->ephys.LFP\n", - "\n", + "probe.Probe->ephys.ProbeInsertion\n", + "\n", "\n", - "\n", - "\n", - "ephys.EphysRecording.EphysFile\n", - "\n", - "\n", - "ephys.EphysRecording.EphysFile\n", + "\n", + "\n", + "ephys.AcquisitionSoftware\n", + "\n", + "\n", + "ephys.AcquisitionSoftware\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.EphysRecording->ephys.EphysRecording.EphysFile\n", - "\n", + "\n", + "\n", + "ephys.AcquisitionSoftware->ephys.EphysRecording\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.ClusteringMethod\n", - "\n", - "\n", - "ephys.ClusteringMethod\n", + "ephys.QualityMetrics\n", + "\n", + "\n", + "ephys.QualityMetrics\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.ClusteringParamSet\n", - "\n", - "\n", - "ephys.ClusteringParamSet\n", - "\n", + "\n", + "\n", + "ephys.QualityMetrics->ephys.QualityMetrics.Cluster\n", + "\n", "\n", + "\n", + "\n", + "ephys.QualityMetrics->ephys.QualityMetrics.Waveform\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.ClusteringMethod->ephys.ClusteringParamSet\n", - "\n", + "ephys.CuratedClustering->ephys.CuratedClustering.Unit\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.CuratedClustering->ephys.QualityMetrics\n", - "\n", + "ephys.CuratedClustering->ephys.WaveformSet\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.CuratedClustering->ephys.CuratedClustering.Unit\n", - "\n", + "ephys.CuratedClustering->ephys.QualityMetrics\n", + "\n", "\n", - "\n", - "\n", - "ephys.WaveformSet\n", - "\n", - "\n", - "ephys.WaveformSet\n", + "\n", + "\n", + "subject.Subject\n", + "\n", + "\n", + "subject.Subject\n", "\n", "\n", "\n", - "\n", - "\n", - "ephys.CuratedClustering->ephys.WaveformSet\n", - "\n", + "\n", + "\n", + "session.Session\n", + "\n", + "\n", + "session.Session\n", + "\n", "\n", - "\n", - "\n", - "ephys.ClusteringParamSet->ephys.ClusteringTask\n", - "\n", "\n", - "\n", - "\n", - "probe.ProbeType.Electrode\n", - "\n", - "\n", - "probe.ProbeType.Electrode\n", - "\n", + "\n", + "\n", + "subject.Subject->session.Session\n", + "\n", "\n", + "\n", + "\n", + "probe.ElectrodeConfig.Electrode->ephys.CuratedClustering.Unit\n", + "\n", "\n", - "\n", + "\n", "\n", - "probe.ProbeType.Electrode->probe.ElectrodeConfig.Electrode\n", - "\n", + "probe.ElectrodeConfig.Electrode->ephys.WaveformSet.Waveform\n", + "\n", "\n", - "\n", + "\n", "\n", - "ephys.WaveformSet->ephys.WaveformSet.PeakWaveform\n", - "\n", - "\n", - "\n", - "\n", - "ephys.WaveformSet->ephys.WaveformSet.Waveform\n", - "\n", + "probe.ElectrodeConfig.Electrode->ephys.LFP.Electrode\n", + "\n", "\n", - "\n", - "\n", - "subject.Subject\n", - "\n", - "\n", - "subject.Subject\n", + "\n", + "\n", + "ephys.ClusteringMethod\n", + "\n", + "\n", + "ephys.ClusteringMethod\n", "\n", "\n", "\n", - "\n", - "\n", - "subject.Subject->session.Session\n", - "\n", - "\n", - "\n", - "\n", - "probe.Probe\n", - "\n", - "\n", - "probe.Probe\n", + "\n", + "\n", + "ephys.ClusteringParamSet\n", + "\n", + "\n", + "ephys.ClusteringParamSet\n", "\n", "\n", "\n", - "\n", + "\n", + "\n", + "ephys.ClusteringMethod->ephys.ClusteringParamSet\n", + "\n", + "\n", + "\n", + "\n", + "ephys.EphysRecording->ephys.LFP\n", + "\n", + "\n", + "\n", "\n", - "probe.Probe->ephys.ProbeInsertion\n", - "\n", + "ephys.EphysRecording->ephys.EphysRecording.EphysFile\n", + "\n", "\n", - "\n", - "\n", - "probe.ProbeType\n", - "\n", - "\n", - "probe.ProbeType\n", + "\n", + "\n", + "ephys.ClusteringTask\n", + "\n", + "\n", + "ephys.ClusteringTask\n", "\n", "\n", "\n", - "\n", + "\n", "\n", - "probe.ProbeType->probe.ProbeType.Electrode\n", - "\n", + "ephys.EphysRecording->ephys.ClusteringTask\n", + "\n", "\n", - "\n", + "\n", "\n", - "probe.ProbeType->probe.Probe\n", - "\n", - "\n", - "\n", - "\n", - "probe.ElectrodeConfig\n", - "\n", - "\n", - "probe.ElectrodeConfig\n", - "\n", - "\n", + "session.Session->ephys.ProbeInsertion\n", + "\n", "\n", - "\n", + "\n", "\n", - "probe.ProbeType->probe.ElectrodeConfig\n", - "\n", + "ephys.ClusteringParamSet->ephys.ClusteringTask\n", + "\n", "\n", "\n", - "\n", + "\n", "ephys.ClusterQualityLabel\n", - "\n", - "\n", - "ephys.ClusterQualityLabel\n", + "\n", + "\n", + "ephys.ClusterQualityLabel\n", "\n", "\n", "\n", "\n", "\n", "ephys.ClusterQualityLabel->ephys.CuratedClustering.Unit\n", - "\n", + "\n", "\n", - "\n", + "\n", + "\n", + "ephys.Clustering\n", + "\n", + "\n", + "ephys.Clustering\n", + "\n", + "\n", + "\n", + "\n", "\n", - "probe.ElectrodeConfig->probe.ElectrodeConfig.Electrode\n", - "\n", + "ephys.Clustering->ephys.Curation\n", + "\n", "\n", - "\n", + "\n", "\n", - "probe.ElectrodeConfig->ephys.EphysRecording\n", - "\n", + "ephys.ClusteringTask->ephys.Clustering\n", + "\n", "\n", "\n", "" ], "text/plain": [ - "" + "" ] }, "execution_count": 4, @@ -626,7 +626,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 5, "metadata": {}, "outputs": [ { @@ -716,7 +716,7 @@ " (Total: 0)" ] }, - "execution_count": 6, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } @@ -727,7 +727,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "metadata": {}, "outputs": [ { @@ -750,7 +750,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "metadata": {}, "outputs": [ { @@ -765,7 +765,7 @@ "subject_description=\"\" : varchar(1024) # " ] }, - "execution_count": 8, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -786,7 +786,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 8, "metadata": {}, "outputs": [ { @@ -880,7 +880,7 @@ " (Total: 1)" ] }, - "execution_count": 9, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -902,7 +902,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 9, "metadata": {}, "outputs": [ { @@ -921,7 +921,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 10, "metadata": {}, "outputs": [ { @@ -932,7 +932,7 @@ "session_datetime : datetime # " ] }, - "execution_count": 11, + "execution_count": 10, "metadata": {}, "output_type": "execute_result" } @@ -959,7 +959,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 11, "metadata": {}, "outputs": [], "source": [ @@ -968,7 +968,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 12, "metadata": {}, "outputs": [ { @@ -1050,7 +1050,7 @@ " (Total: 1)" ] }, - "execution_count": 13, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -1069,7 +1069,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 13, "metadata": {}, "outputs": [ { @@ -1155,7 +1155,7 @@ " (Total: 1)" ] }, - "execution_count": 14, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } @@ -1180,7 +1180,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 14, "metadata": {}, "outputs": [ { @@ -1266,7 +1266,7 @@ " (Total: 1)" ] }, - "execution_count": 15, + "execution_count": 14, "metadata": {}, "output_type": "execute_result" } @@ -1289,7 +1289,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 15, "metadata": {}, "outputs": [ { @@ -1311,7 +1311,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 16, "metadata": {}, "outputs": [ { @@ -1325,7 +1325,7 @@ "probe : varchar(32) # unique identifier for this model of probe (e.g. serial number)" ] }, - "execution_count": 17, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } @@ -1336,7 +1336,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 17, "metadata": {}, "outputs": [ { @@ -1426,7 +1426,7 @@ " (Total: 1)" ] }, - "execution_count": 18, + "execution_count": 17, "metadata": {}, "output_type": "execute_result" } @@ -1451,7 +1451,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 18, "metadata": {}, "outputs": [ { @@ -1541,7 +1541,7 @@ " (Total: 1)" ] }, - "execution_count": 19, + "execution_count": 18, "metadata": {}, "output_type": "execute_result" } @@ -1561,7 +1561,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 19, "metadata": {}, "outputs": [ { @@ -1660,7 +1660,7 @@ " (Total: 0)" ] }, - "execution_count": 20, + "execution_count": 19, "metadata": {}, "output_type": "execute_result" } @@ -1671,7 +1671,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 20, "metadata": {}, "outputs": [ { @@ -1758,7 +1758,7 @@ " (Total: 0)" ] }, - "execution_count": 21, + "execution_count": 20, "metadata": {}, "output_type": "execute_result" } @@ -1769,14 +1769,14 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "EphysRecording: 100%|██████████| 1/1 [00:01<00:00, 1.51s/it]\n" + "EphysRecording: 100%|██████████| 1/1 [00:01<00:00, 1.25s/it]\n" ] } ], @@ -1794,7 +1794,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 22, "metadata": {}, "outputs": [ { @@ -1900,7 +1900,7 @@ " (Total: 1)" ] }, - "execution_count": 23, + "execution_count": 22, "metadata": {}, "output_type": "execute_result" } @@ -1911,7 +1911,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 23, "metadata": {}, "outputs": [ { @@ -2001,7 +2001,7 @@ " (Total: 1)" ] }, - "execution_count": 24, + "execution_count": 23, "metadata": {}, "output_type": "execute_result" } @@ -2028,7 +2028,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 24, "metadata": {}, "outputs": [ { @@ -2043,7 +2043,7 @@ "params : longblob # dictionary of all applicable parameters" ] }, - "execution_count": 25, + "execution_count": 24, "metadata": {}, "output_type": "execute_result" } @@ -2054,7 +2054,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 25, "metadata": {}, "outputs": [ { @@ -2148,7 +2148,7 @@ " (Total: 1)" ] }, - "execution_count": 26, + "execution_count": 25, "metadata": {}, "output_type": "execute_result" } @@ -2203,7 +2203,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 26, "metadata": {}, "outputs": [ { @@ -2219,7 +2219,7 @@ "task_mode=\"load\" : enum('load','trigger') # 'load': load computed analysis results, 'trigger': trigger computation" ] }, - "execution_count": 27, + "execution_count": 26, "metadata": {}, "output_type": "execute_result" } @@ -2250,7 +2250,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 27, "metadata": {}, "outputs": [], "source": [ @@ -2267,14 +2267,14 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 28, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "Clustering: 100%|██████████| 1/1 [00:00<00:00, 2.71it/s]\n" + "Clustering: 100%|██████████| 1/1 [00:00<00:00, 3.27it/s]\n" ] } ], @@ -2295,7 +2295,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 29, "metadata": {}, "outputs": [ { @@ -2315,7 +2315,7 @@ "curation_note=\"\" : varchar(2000) # " ] }, - "execution_count": 30, + "execution_count": 29, "metadata": {}, "output_type": "execute_result" } @@ -2326,7 +2326,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 30, "metadata": {}, "outputs": [], "source": [ @@ -2345,7 +2345,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 31, "metadata": {}, "outputs": [ { @@ -2354,9 +2354,9 @@ "text": [ "CuratedClustering: 0%| | 0/1 [00:00 Date: Tue, 28 Nov 2023 14:11:06 -0600 Subject: [PATCH 09/14] Markdown structural edits --- notebooks/tutorial.ipynb | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb index cfee4903..54a2123e 100644 --- a/notebooks/tutorial.ipynb +++ b/notebooks/tutorial.ipynb @@ -7,13 +7,13 @@ "source": [ "# DataJoint Elements for Array Electrophysiology with NeuroPixels\n", "\n", - "#### Open-source Data Pipeline for Processing and Analyzing Extracellular Electrophysiology Datasets\n", + "#### Open-source data pipeline for processing and analyzing extracellular electrophysiology datasets.\n", "\n", "Welcome to the tutorial for the DataJoint Element for extracellular array electrophysiology. This\n", "tutorial aims to provide a comprehensive understanding of the open-source data pipeline\n", "created using `element-array-ephys`.\n", "\n", - "This package is designed to seamlessly process and ingest extracellular electrophysiology\n", + "This package is designed to seamlessly process, ingest, and track extracellular electrophysiology\n", "data, along with its associated probe and recording metadata. By the end of this\n", "tutorial you will have a clear grasp on setting up and integrating `element-array-ephys`\n", "into your specific research projects and lab. \n", @@ -48,8 +48,8 @@ "and manage sessions of array electrophysiology data, including spike sorting results and\n", "unit-level visualizations. \n", "\n", - "The results of this Element example can be combined with **other modalities** to create\n", - "a complete customizable data pipeline for your specific lab or study. For instance, you\n", + "The results of this Element can be combined with **other modalities** to create\n", + "a complete, customizable data pipeline for your specific lab or study. For instance, you\n", "can combine `element-array-ephys` with `element-calcium-imaging` and\n", "`element-deeplabcut` to characterize the neural activity along with markless\n", "pose-estimation during behavior.\n", @@ -73,8 +73,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "If the tutorial is run in Codespaces, there is a local database private to you for\n", - "experimentation. Let's connect to the database server." + "If the tutorial is run in Codespaces, a private, local database server is created and\n", + "made available for you. This is where we will insert and store our processed results.\n", + "Let's connect to the database server." ] }, { @@ -613,7 +614,7 @@ "As evident from the diagram, this data pipeline encompasses tables associated with\n", "recording and probe metadata, results of clustering, and optional curation of clustering\n", "results. A few tables, such as `subject.Subject` or `session.Session`,\n", - "while important for a complete pipeline, fall outside the scope of `element-array-ephys`\n", + "while important for a complete pipeline, fall outside the scope of the `element-array-ephys`\n", "tutorial, and will therefore, not be explored extensively here. The primary focus of\n", "this tutorial will be on the `probe` and `ephys` schemas.\n", "\n", From b979feca44468a33601af36c1db3f917993844df Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 28 Nov 2023 15:00:56 -0600 Subject: [PATCH 10/14] Remove PyPI versioning in setup --- setup.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/setup.py b/setup.py index abd3a92d..64311078 100644 --- a/setup.py +++ b/setup.py @@ -41,11 +41,11 @@ ], extras_require={ "elements": [ - "element-animal>=0.1.8", - "element-event>=0.2.3", - "element-interface>=0.4.0", - "element-lab>=0.3.0", - "element-session>=0.1.5", + "elemment-animal @ git+https://github.com/datajoint/element-animal.git", + "element-event @ git+https://github.com/datajoint/element-event.git", + "element-interface @ git+https://github.com/datajoint/element-interface.git", + "element-lab @ git+https://github.com/datajoint/element-lab.git", + "element-session @ git+https://github.com/datajoint/element-session.git", ], "nwb": ["dandi", "neuroconv[ecephys]", "pynwb"], "tests": ["pre-commit", "pytest", "pytest-cov"], From 0dbdde70a27054f5489399daca4743f40c34ce29 Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 28 Nov 2023 15:02:43 -0600 Subject: [PATCH 11/14] Move dj_config setup to `tutorial_pipeline.py` --- element_array_ephys/__init__.py | 18 ------------------ notebooks/tutorial_pipeline.py | 20 ++++++++++++++++++-- 2 files changed, 18 insertions(+), 20 deletions(-) diff --git a/element_array_ephys/__init__.py b/element_array_ephys/__init__.py index 99acf32a..1c0c7285 100644 --- a/element_array_ephys/__init__.py +++ b/element_array_ephys/__init__.py @@ -1,19 +1 @@ -import os -import datajoint as dj - -if "custom" not in dj.config: - dj.config["custom"] = {} - -# overwrite dj.config['custom'] values with environment variables if available - -dj.config["custom"]["database.prefix"] = os.getenv( - "DATABASE_PREFIX", dj.config["custom"].get("database.prefix", "") -) - -dj.config["custom"]["ephys_root_data_dir"] = os.getenv( - "EPHYS_ROOT_DATA_DIR", dj.config["custom"].get("ephys_root_data_dir", "") -) - -db_prefix = dj.config["custom"].get("database.prefix", "") - from . import ephys_acute as ephys diff --git a/notebooks/tutorial_pipeline.py b/notebooks/tutorial_pipeline.py index 2163d315..a61df2da 100644 --- a/notebooks/tutorial_pipeline.py +++ b/notebooks/tutorial_pipeline.py @@ -1,7 +1,9 @@ +import os +import pathlib import datajoint as dj from element_animal import subject from element_animal.subject import Subject -from element_array_ephys import db_prefix, probe, ephys_acute as ephys +from element_array_ephys import probe, ephys_acute as ephys from element_lab import lab from element_lab.lab import Lab, Location, Project, Protocol, Source, User from element_lab.lab import Device as Equipment @@ -9,9 +11,23 @@ from element_session import session_with_datetime as session from element_session.session_with_datetime import Session import element_interface -import pathlib +if "custom" not in dj.config: + dj.config["custom"] = {} + +# overwrite dj.config['custom'] values with environment variables if available + +dj.config["custom"]["database.prefix"] = os.getenv( + "DATABASE_PREFIX", dj.config["custom"].get("database.prefix", "") +) + +dj.config["custom"]["ephys_root_data_dir"] = os.getenv( + "EPHYS_ROOT_DATA_DIR", dj.config["custom"].get("ephys_root_data_dir", "") +) + +db_prefix = dj.config["custom"].get("database.prefix", "") + # Declare functions for retrieving data def get_ephys_root_data_dir(): """Retrieve ephys root data directory.""" From 16d36d56fad9e0e5b97af2fa57df065a664917cf Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 28 Nov 2023 15:03:35 -0600 Subject: [PATCH 12/14] Black formatting --- notebooks/tutorial_pipeline.py | 1 + 1 file changed, 1 insertion(+) diff --git a/notebooks/tutorial_pipeline.py b/notebooks/tutorial_pipeline.py index a61df2da..23a544b4 100644 --- a/notebooks/tutorial_pipeline.py +++ b/notebooks/tutorial_pipeline.py @@ -28,6 +28,7 @@ db_prefix = dj.config["custom"].get("database.prefix", "") + # Declare functions for retrieving data def get_ephys_root_data_dir(): """Retrieve ephys root data directory.""" From b919ca34432c3189d934a3a75bdb071fe8bcb6b9 Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 28 Nov 2023 15:05:42 -0600 Subject: [PATCH 13/14] Fix typo in setup.py --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 64311078..0ff4bf08 100644 --- a/setup.py +++ b/setup.py @@ -41,7 +41,7 @@ ], extras_require={ "elements": [ - "elemment-animal @ git+https://github.com/datajoint/element-animal.git", + "element-animal @ git+https://github.com/datajoint/element-animal.git", "element-event @ git+https://github.com/datajoint/element-event.git", "element-interface @ git+https://github.com/datajoint/element-interface.git", "element-lab @ git+https://github.com/datajoint/element-lab.git", From 1ea7c89993465eaa34e02864f654c654b9e7285c Mon Sep 17 00:00:00 2001 From: kushalbakshi Date: Tue, 19 Dec 2023 13:29:35 -0600 Subject: [PATCH 14/14] Minor fixes and updates to notebook --- notebooks/tutorial.ipynb | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb index 54a2123e..954766ca 100644 --- a/notebooks/tutorial.ipynb +++ b/notebooks/tutorial.ipynb @@ -112,7 +112,7 @@ "source": [ "### **Activate the DataJoint Pipeline**\n", "\n", - "This tutorial activates the `ephys-acute.py` module from `element-array-ephys`, along\n", + "This tutorial activates the `ephys_acute.py` module from `element-array-ephys`, along\n", "with upstream dependencies from `element-animal` and `element-session`. Please refer to the\n", "[`tutorial_pipeline.py`](./tutorial_pipeline.py) for the source code." ] @@ -1065,7 +1065,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Every experimental session produces a set of data files. The purpose of the `SessionDirectory` table is to locate these files. It references a directory path relative to a root directory, defined in `dj.config[\\\"custom\\\"]`. More information about `dj.config` is provided in the [documentation](https://datajoint.com/docs/elements/user-guide/)." + "Every experimental session produces a set of data files. The purpose of the `SessionDirectory` table is to locate these files. It references a directory path relative to a root directory, defined in `dj.config[\"custom\"]`. More information about `dj.config` is provided in the [documentation](https://datajoint.com/docs/elements/user-guide/)." ] }, { @@ -1557,7 +1557,8 @@ "source": [ "### **Populate electrophysiology recording metadata**\n", "\n", - "In the upcoming cells, populate the `ephys.EphysRecording` table and its part table `ephys.EphysRecording.EphysFile` will extract and store the recording information from a given experimental session." + "In the upcoming cells, the `.populate()` method will automatically extract and store the\n", + "recording metadata for each experimental session in the `ephys.EphysRecording` table and its part table `ephys.EphysRecording.EphysFile`." ] }, { @@ -2194,8 +2195,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Now that we've inserted kilosort parameters into the `ClusteringParamSet` table,\n", - "we're almost ready to sort our data. DataJoint uses a `ClusteringTask` table to\n", + "DataJoint uses a `ClusteringTask` table to\n", "manage which `EphysRecording` and `ClusteringParamSet` should be used during processing. \n", "\n", "This table is important for defining several important aspects of\n", @@ -2235,18 +2235,12 @@ "metadata": {}, "source": [ "The `ClusteringTask` table contains two important attributes: \n", - "+ `paramset_idx` \n", - "+ `task_mode` \n", - "\n", - "The `paramset_idx` attribute tracks\n", - "your kilosort parameter sets. You can choose the parameter set using which \n", - "you want spike sort ephys data. For example, `paramset_idx=0` may contain\n", - "default parameters for kilosort processing whereas `paramset_idx=1` contains your custom parameters for sorting. This\n", - "attribute tells the `Processing` table which set of parameters you are processing in a given `populate()`.\n", - "\n", - "The `task_mode` attribute can be set to either `load` or `trigger`. When set to `load`,\n", - "running the processing step initiates a search for exisiting kilosort output files. When set to `trigger`, the\n", - "processing step will run kilosort on the raw data. " + "+ `paramset_idx` - Allows the user to choose the parameter set with which you want to\n", + " run spike sorting.\n", + "+ `task_mode` - Can be set to `load` or `trigger`. When set to `load`, running the\n", + " Clustering step initiates a search for existing output files of the spike sorting\n", + " algorithm defined in `ClusteringParamSet`. When set to `trigger`, the processing step\n", + " will run spike sorting on the raw data." ] }, { @@ -2266,6 +2260,13 @@ ")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's call populate on the `Clustering` table which checks for kilosort results since `task_mode=load`." + ] + }, { "cell_type": "code", "execution_count": 28,