From 34f12adfc14db14248237d387bf9fa0ea961786c Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Wed, 31 Jan 2024 16:59:50 -0500 Subject: [PATCH 01/10] Update README.md --- README.md | 88 +++---------------------------------------------------- 1 file changed, 4 insertions(+), 84 deletions(-) diff --git a/README.md b/README.md index 31e54759..45f7b0d8 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,9 @@ Developed by [CatalystNeuro](http://catalystneuro.com/). `pip install roiextractors` ## Usage: + +See [documentation](https://roiextractors.readthedocs.io/en/latest/) for details. + ### Supported file types: #### Imaging 1. HDF5 @@ -31,74 +34,8 @@ Developed by [CatalystNeuro](http://catalystneuro.com/). 45. Numpy (a data format for manual input of optical physiology data as various numpy datasets) #### Functionality: -Interconversion amongst the various data formats as well as conversion to the NWB format and back. - -#### Features: -1. SegmentationExtractor object: - * `seg_obj.get_channel_names()` : - List of optical channel names - * `seg_obj.get_num_channels()` : - Number of channels - * `seg_obj.get_movie_framesize()`: - (height, width) of raw movie - * `seg_obj.get_movie_location()`: - Location of storage of movie/tiff images - * `seg_obj.get_image_masks(self, roi_ids=None)`: - Image masks as (ht, wd, num_rois) with each value as the weight given during segmentation operation. - * `seg_obj.get_pixel_masks(roi_ids=None)`: - Get pixel masks as (total_pixels(ht*wid), no_rois) - * `seg_obj.get_traces(self, roi_ids=None, start_frame=None, end_frame=None)`: - df/F trace as (num_rois, num_frames) - * `seg_obj.get_sampling_frequency()`: - Sampling frequency of movie/df/F trace. - * `seg_obj.get_roi_locations()`: - Centroid pixel location of the ROI (Regions Of Interest) as (x,y). - * `seg_obj.get_num_rois()`: - Total number of ROIs after segmentation operation. - * `seg_obj.get_roi_ids()`: - Any integer tags associated with an ROI, defaults to `0:num_of_rois` - -#### SegmentationExtractor object creation: -```python -import roiextractors -import numpy as np +This package provides a common API for various optical imaging and segmentation formats streamline conversion and data analysis. -seg_obj_cnmfe = roiextractors.CnmfeSegmentationExtractor('cnmfe_filename.mat') # cnmfe -seg_obj_extract = roiextractors.ExtractSegmentationExtractor('extract_filename.mat') # extract -seg_obj_sima = roiextractors.SimaSegmentationExtractor('sima_filename.sima') # SIMA -seg_obj_numpy = roiextractors.NumpySegmentationExtractor( - filepath = 'path-to-file', - masks=np.random.rand(movie_size[0],movie_size[1],no_rois), - signal=np.random.randn(num_rois,num_frames), - roi_idx=np.random.randint(no_rois,size=[1,no_rois]), - no_of_channels=None, - summary_image=None, - channel_names=['Blue'], -) # Numpy object -seg_obj_nwb = roiextractors.NwbSegmentationExtractor( - filepath_of_nwb, optical_channel_name=None, # optical channel to extract and store info from - imaging_plane_name=None, image_series_name=None, # imaging plane to extract and store data from - processing_module_name=None, - neuron_roi_response_series_name=None, # roi_response_series name to extract and store data from - background_roi_response_series_name=None) # nwb object -``` -#### Data format conversion: SegmentationExtractor to NWB: -```python -roiextractors.NwbSegmentationExtractor.write_segmentation(seg_obj, saveloc, - propertydict=[{'name': 'ROI feature 1, - 'description': 'additional attribute of each ROI', - 'data': np.random.rand(1,no_rois), - 'id': seg_obj.get_roi_ids()}, - {'name': 'ROI feature 2, - 'description': 'additional attribute of each ROI', - 'data': np.random.rand(1,no_rois), - 'id': seg_obj.get_roi_ids()}], - nwbfile_kwargs={'session_description': 'nwbfiledesc', - 'experimenter': 'experimenter name', - 'lab': 'test lab', - 'session_id': 'test sess id'}, - emission_lambda=400.0, excitation_lambda=500.0) -``` ## Example Datasets: Example datasets are maintained at https://gin.g-node.org/CatalystNeuro/ophys_testing_data. @@ -116,23 +53,6 @@ gin get-content To update data later, `cd` into the test directory and run `gin get-content` - -## Class descriptions: - -* **SegmentationExtractor:** An abstract class that contains all the meta-data and output data from the ROI segmentation operation when applied to the pre-processed data. It also contains methods to read from and write to various data formats ouput from the processing pipelines like SIMA, CaImAn, Suite2p, CNNM-E. - -* **NumpySegmentationExtractor:** Contains all data coming from a file format for which there is currently no support. To construct this, all data must be entered manually as arguments. - -* **CnmfeSegmentationExtractor:** This class inherits from the SegmentationExtractor class, having all its funtionality specifically applied to the dataset output from the 'CNMF-E' ROI segmentation method. - -* **ExtractSegmentationExtractor:** This class inherits from the SegmentationExtractor class, having all its funtionality specifically applied to the dataset output from the 'EXTRACT' ROI segmentation method. - -* **SimaSegmentationExtractor:** This class inherits from the SegmentationExtractor class, having all its funtionality specifically applied to the dataset output from the 'SIMA' ROI segmentation method. - -* **NwbSegmentationExtractor:** Extracts data from the NWB data format. Also implements a static method to write any format specific object to NWB. - -* **Suite2PSegmentationExtractor:** Extracts data from suite2p format. - ## Troubleshooting ##### Installing SIMA with python>=3.7: Will need a manual installation for package dependency **SIMA** since it does not currently support python 3.7: From 5a276dd5dd056a3e4efec8958ae3aec172f82d79 Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Wed, 31 Jan 2024 17:06:20 -0500 Subject: [PATCH 02/10] Update support list --- README.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 45f7b0d8..a488ef4e 100644 --- a/README.md +++ b/README.md @@ -21,20 +21,23 @@ See [documentation](https://roiextractors.readthedocs.io/en/latest/) for details ### Supported file types: #### Imaging 1. HDF5 -2. TIFF -3. STK -4. FLI -5. SBX +2. Miniscope +3. NWB +4. SBX +5. Micromanager +6. Bruker +7. ScanImage #### Segmentation 1. [calciumImagingAnalysis](https://github.com/bahanonu/calciumImagingAnalysis) (CNMF-E, EXTRACT) 2. [SIMA](http://www.losonczylab.org/sima/1.3.2/) 3. [NWB](https://pynwb.readthedocs.io/en/stable/) 4. [suite2p](https://github.com/MouseLand/suite2p) -45. Numpy (a data format for manual input of optical physiology data as various numpy datasets) +5. Numpy (a data format for manual input of optical physiology data as various numpy datasets) +6. [CaImAn](https://github.com/flatironinstitute/CaImAn) #### Functionality: -This package provides a common API for various optical imaging and segmentation formats streamline conversion and data analysis. +This package provides a common API for various optical imaging and segmentation formats to streamline conversion and data analysis. ## Example Datasets: Example datasets are maintained at https://gin.g-node.org/CatalystNeuro/ophys_testing_data. From bdca04c78ff771ed9e179d12060c14efb74409da Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Thu, 1 Feb 2024 16:29:20 -0500 Subject: [PATCH 03/10] Update README.md to mirror NeuroConv's format --- README.md | 83 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 54 insertions(+), 29 deletions(-) diff --git a/README.md b/README.md index a488ef4e..01943ebc 100644 --- a/README.md +++ b/README.md @@ -5,41 +5,62 @@ [![documentation](https://readthedocs.org/projects/roiextractors/badge/?version=latest)](https://roiextractors.readthedocs.io/en/latest/) [![License](https://img.shields.io/pypi/l/pynwb.svg)](https://github.com/catalystneuro/roiextractors/license.txt) -# ROI Extractors -Python-based module for extracting from, converting between, and handling recorded and optical imaging data from several file formats. Inspired by [SpikeExtractors](https://github.com/SpikeInterface/spikeextractors). +# ROIExtractors +

+

Automatically read optical imaging/segmentation data into a common API

+

+

+ Explore our documentation ยป +

-Developed by [CatalystNeuro](http://catalystneuro.com/). + -## Getting Started: -#### Installation: -`pip install roiextractors` +## Table of Contents -## Usage: +- [About](#about) +- [Installation](#installation) +- [Documentation](#documentation) +- [Funding](#funding) -See [documentation](https://roiextractors.readthedocs.io/en/latest/) for details. +## About -### Supported file types: -#### Imaging -1. HDF5 -2. Miniscope -3. NWB -4. SBX -5. Micromanager -6. Bruker -7. ScanImage +ROIExtractors provides a common API for various optical imaging and segmentation formats to streamline conversion and data analysis. ROI stands for Region Of Interest, which is the region in a set of acquired fluorescence images which the segmentation software has determined as a neuron. -#### Segmentation -1. [calciumImagingAnalysis](https://github.com/bahanonu/calciumImagingAnalysis) (CNMF-E, EXTRACT) -2. [SIMA](http://www.losonczylab.org/sima/1.3.2/) -3. [NWB](https://pynwb.readthedocs.io/en/stable/) -4. [suite2p](https://github.com/MouseLand/suite2p) -5. Numpy (a data format for manual input of optical physiology data as various numpy datasets) -6. [CaImAn](https://github.com/flatironinstitute/CaImAn) +Features: -#### Functionality: -This package provides a common API for various optical imaging and segmentation formats to streamline conversion and data analysis. +* Reads data from 10+ popular optical imaging and segmentation data formats into a common API. +* Extracts relevant metadata from each format. +* Provides a handy set of methods to analyze data such as `get_roi_locations()` -## Example Datasets: +## Installation + +To install the latest stable release of **roiextractors** though PyPI, type: +```shell +pip install roiextractors +``` + +For more flexibility we recommend installing the latest version directly from GitHub. The following commands create an environment with all the required dependencies and the latest updates: + +```shell +git clone https://github.com/catalystneuro/roiextractors +cd roiextractors +conda env create roiextractors_env +conda activate roiextractors_env +pip install -e . +``` +Note that this will install the package in [editable mode](https://pip.pypa.io/en/stable/cli/pip_install/#editable-installs). + +Finally, if you prefer to avoid `conda` altogether, the following commands provide a clean installation within the current environment: +```shell +pip install git+https://github.com/catalystneuro/roiextractors.git@main +``` + +## Documentation +See our [ReadTheDocs page](https://roiextractors.readthedocs.io/en/latest/) for full documentation, including a gallery of all supported formats. + +## Developer Guide + +### Example Datasets Example datasets are maintained at https://gin.g-node.org/CatalystNeuro/ophys_testing_data. To download test data on your machine, @@ -56,13 +77,17 @@ gin get-content To update data later, `cd` into the test directory and run `gin get-content` -## Troubleshooting +### Troubleshooting ##### Installing SIMA with python>=3.7: Will need a manual installation for package dependency **SIMA** since it does not currently support python 3.7: 1. Download SIMA wheels distribution [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/#sima). 2. `pip install ` 3. `pip install roiextractors` -### Funded by +## Funding +ROIExtractors is funded by * Stanford University as part of the Ripple U19 project (U19NS104590). * LBNL as part of the NWB U24 (U24NS120057). + +## License +ROIExtractors is distributed under the BSD3 License. See [LICENSE](https://github.com/catalystneuro/roiextractors/blob/main/LICENSE.txt) for more information. From bf5885ebde351475060340646b40cc8781da57d9 Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Thu, 1 Feb 2024 16:29:56 -0500 Subject: [PATCH 04/10] add license to toc --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 01943ebc..ffdc872a 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,7 @@ - [Installation](#installation) - [Documentation](#documentation) - [Funding](#funding) +- [License](#license) ## About From 18206b7a8cb073b558f38ca091e063c4ed67f671 Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Thu, 1 Feb 2024 18:40:03 -0500 Subject: [PATCH 05/10] removed developer guide --- README.md | 26 -------------------------- 1 file changed, 26 deletions(-) diff --git a/README.md b/README.md index ffdc872a..d1f7b201 100644 --- a/README.md +++ b/README.md @@ -59,32 +59,6 @@ pip install git+https://github.com/catalystneuro/roiextractors.git@main ## Documentation See our [ReadTheDocs page](https://roiextractors.readthedocs.io/en/latest/) for full documentation, including a gallery of all supported formats. -## Developer Guide - -### Example Datasets -Example datasets are maintained at https://gin.g-node.org/CatalystNeuro/ophys_testing_data. - -To download test data on your machine, - -1. Install the gin client (instructions [here](https://gin.g-node.org/G-Node/Info/wiki/GIN+CLI+Setup#linux)) -2. Use gin to download data. -```shell -gin get CatalystNeuro/ophys_testing_data -cd ophys_testing_data -gin get-content -``` - -3. Change the file at `roiextractors/tests/gin_test_config.json` to point to the path of this test data - -To update data later, `cd` into the test directory and run `gin get-content` - -### Troubleshooting -##### Installing SIMA with python>=3.7: -Will need a manual installation for package dependency **SIMA** since it does not currently support python 3.7: -1. Download SIMA wheels distribution [here](https://www.lfd.uci.edu/~gohlke/pythonlibs/#sima). -2. `pip install ` -3. `pip install roiextractors` - ## Funding ROIExtractors is funded by * Stanford University as part of the Ripple U19 project (U19NS104590). From dd2ccc5e3ae1d48a531e96cdde5fe9d80db3ddbe Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Thu, 1 Feb 2024 19:03:07 -0500 Subject: [PATCH 06/10] Update contribute.rst with developer guide info from the readme --- docs/source/contribute.rst | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/source/contribute.rst b/docs/source/contribute.rst index edcd1122..22adaa6d 100644 --- a/docs/source/contribute.rst +++ b/docs/source/contribute.rst @@ -1,7 +1,38 @@ Contribute: =========== -To contribute to Roiextractors to help us develop an api to handle a new data format for optophysiology: +Example Datasets +---------------- + +Example datasets are maintained at https://gin.g-node.org/CatalystNeuro/ophys_testing_data. + +To download test data on your machine, + +1. Install the gin client (instructions `here `_) +2. Use gin to download data: + + .. code-block:: bash + + gin get CatalystNeuro/ophys_testing_data + cd ophys_testing_data + gin get-content + +3. Change the file at ``roiextractors/tests/gin_test_config.json`` to point to the path of this test data + +To update data later, ``cd`` into the test directory and run ``gin get-content`` + +Troubleshooting +--------------- + +Installing SIMA with python>=3.7: +________________________________ +Will need a manual installation for package dependency **SIMA** since it does not currently support python 3.7: + +1. Download SIMA wheels distribution `here `_. +2. `pip install ` +3. `pip install roiextractors` + +More details on how to construct individual extractors can be found here: .. toctree:: :maxdepth: 1 From 2aff36de922427045475f8cd4319309fba34aca9 Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Thu, 1 Feb 2024 19:14:13 -0500 Subject: [PATCH 07/10] Update supported formats to live table --- docs/source/compatible.rst | 23 +++-------------------- 1 file changed, 3 insertions(+), 20 deletions(-) diff --git a/docs/source/compatible.rst b/docs/source/compatible.rst index 80663ea3..ef436d3e 100644 --- a/docs/source/compatible.rst +++ b/docs/source/compatible.rst @@ -1,25 +1,8 @@ Supported formats ================= -Roiextractors supports various image formats obtained from different 2photon acquisition systems. -It also supports well known post processing cell extraction/segmentation pipelines. If you use any other format and would like to see it being supported we would love to know! (:doc:`Contact `) +The following is a live record of all the formats supported by ROIExtractors. -Imaging -------- -1. HDF5 -2. TIFF -3. STK -4. FLI +.. raw:: html -Segmentation ------------- -#. `calciumImagingAnalysis `_ (CNMF-E, EXTRACT) -#. `Caiman `_ -#. `SIMA `_ -#. `NWB `_ -#. `suite2p `_ -#. Numpy (for any currently un-supported format) - -Example Datasets ----------------- -Example datasets for each fo the file formats can be downloaded `here `_ + From faa5cb39d9b6c2409ef898dcce21d00c96973f92 Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Thu, 1 Feb 2024 19:16:50 -0500 Subject: [PATCH 08/10] specify ophys tab in support table link --- docs/source/compatible.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/compatible.rst b/docs/source/compatible.rst index ef436d3e..225664fd 100644 --- a/docs/source/compatible.rst +++ b/docs/source/compatible.rst @@ -1,8 +1,8 @@ Supported formats ================= -The following is a live record of all the formats supported by ROIExtractors. +The following is a live record of all the formats supported by ROIExtractors as well as other relevant ecosystem support. .. raw:: html - + From 7788cb7dc185c33a6b134f7f5262a436921694fb Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Mon, 5 Feb 2024 13:17:03 -0500 Subject: [PATCH 09/10] rollback compatible.rst --- docs/source/compatible.rst | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/docs/source/compatible.rst b/docs/source/compatible.rst index 225664fd..80663ea3 100644 --- a/docs/source/compatible.rst +++ b/docs/source/compatible.rst @@ -1,8 +1,25 @@ Supported formats ================= -The following is a live record of all the formats supported by ROIExtractors as well as other relevant ecosystem support. +Roiextractors supports various image formats obtained from different 2photon acquisition systems. +It also supports well known post processing cell extraction/segmentation pipelines. If you use any other format and would like to see it being supported we would love to know! (:doc:`Contact `) -.. raw:: html +Imaging +------- +1. HDF5 +2. TIFF +3. STK +4. FLI - +Segmentation +------------ +#. `calciumImagingAnalysis `_ (CNMF-E, EXTRACT) +#. `Caiman `_ +#. `SIMA `_ +#. `NWB `_ +#. `suite2p `_ +#. Numpy (for any currently un-supported format) + +Example Datasets +---------------- +Example datasets for each fo the file formats can be downloaded `here `_ From 0a4440a41ca19b28b09ffb3fbf8cde4115efa817 Mon Sep 17 00:00:00 2001 From: Paul Adkisson Date: Mon, 5 Feb 2024 13:18:06 -0500 Subject: [PATCH 10/10] rollback contribute.rst --- docs/source/contribute.rst | 33 +-------------------------------- 1 file changed, 1 insertion(+), 32 deletions(-) diff --git a/docs/source/contribute.rst b/docs/source/contribute.rst index 22adaa6d..edcd1122 100644 --- a/docs/source/contribute.rst +++ b/docs/source/contribute.rst @@ -1,38 +1,7 @@ Contribute: =========== -Example Datasets ----------------- - -Example datasets are maintained at https://gin.g-node.org/CatalystNeuro/ophys_testing_data. - -To download test data on your machine, - -1. Install the gin client (instructions `here `_) -2. Use gin to download data: - - .. code-block:: bash - - gin get CatalystNeuro/ophys_testing_data - cd ophys_testing_data - gin get-content - -3. Change the file at ``roiextractors/tests/gin_test_config.json`` to point to the path of this test data - -To update data later, ``cd`` into the test directory and run ``gin get-content`` - -Troubleshooting ---------------- - -Installing SIMA with python>=3.7: -________________________________ -Will need a manual installation for package dependency **SIMA** since it does not currently support python 3.7: - -1. Download SIMA wheels distribution `here `_. -2. `pip install ` -3. `pip install roiextractors` - -More details on how to construct individual extractors can be found here: +To contribute to Roiextractors to help us develop an api to handle a new data format for optophysiology: .. toctree:: :maxdepth: 1