[Feature]: Allow for incomplete data ingestion

[RichieHakim]

What would you like to see added to nwb-conversion-tools?

I would like to use roiextractors as a bridge for ingesting data from multiple data sources in a package I am working on (https://github.com/RichieHakim/ROICaT).

Ttime series fluorescence fluorescence is not necessary for my pipeline to work. For example, for Suite2p output data, I only ingest the stat.npy and ops.npy files and nothing else. Given the large size of the F.npy and similar files compared to the small size of stat.npy and ops.npy, it would be advantageous to allow the roiextractors classes to allow for incomplete data ingestion, i.e. importing directories containing only stat.npy and ops.npy files but not F.npy etc.

Is your feature request related to a problem?

No response

What solution would you like?

Allow for incomplete data ingestion. If traces are missing, only throw a warning, not an error. Perhaps provide an argument like allow_incomplete_data.

Do you have any interest in helping implement the feature?

Yes, but I would need guidance.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you ensured this bug was not already reported?

[bendichter]

@RichieHakim nice to meet you! It's very cool that you want to use ROI Extractors for this. So far we have only used it as an ingestion tool for NeuroConv, but it is intended to be general-purpose and you tool looks like a great alternative use-case. And of course it would be great to get a small community together so we can benefit from each other's work in API standardization. In principle, I think your request is reasonable, but if course the devil is in the details. @h-mayorquin and @CodyCBakerPhD what do you think? How difficult would it be to implement something like this?

[h-mayorquin]

I took a look at this. I think it would be straighforward. We already have in our dummy segmentation extractor (for testing purposes) functionality for not loading the diverse traces:

roiextractors/src/roiextractors/testing.py

Lines 54 to 67 in 062f69e

	def generate_dummy_segmentation_extractor(
	num_rois: int = 10,
	num_frames: int = 30,
	num_rows: int = 25,
	num_columns: int = 25,
	sampling_frequency: float = 30.0,
	has_summary_images: bool = True,
	has_raw_signal: bool = True,
	has_dff_signal: bool = True,
	has_deconvolved_signal: bool = True,
	has_neuropil_signal: bool = True,
	rejected_list: Optional[list] = None,
	) -> SegmentationExtractor:
	"""

Even the IO which we don't really uses has logic for this implemented:

roiextractors/src/roiextractors/extractors/suite2p/suite2psegmentationextractor.py

Lines 160 to 166 in 062f69e

	if segmentation_object.get_traces(name="raw") is not None:
	np.save(save_path / "F.npy", segmentation_object.get_traces(name="raw").T)
	if segmentation_object.get_traces(name="neuropil") is not None:
	np.save(save_path / "Fneu.npy", segmentation_object.get_traces(name="neuropil").T)
	if segmentation_object.get_traces(name="deconvolved") is not None:
	np.save(
	save_path / "spks.npy",

What remains is to decide what kind of API would be good for this. The simplest is a single flag that omits the traces but maybe we can think on a more general configuration. For example, we could use a set of flags as in the generate_dummy_segmentation_extractor or a dictionary of parts of the extractor to omit which I think might be more succint.

[RichieHakim]

Thank you both for your responses. We greatly appreciate any help you are willing to provide and we are happy to chat if that would be useful.

I wrote a PR related to this: #227
However, I noticed that the other ingestion classes don't all properly ingest all of the data I would need. These include only two things: roi_masks and FOV_images (like a mean FOV image or 'sample image').

For example with CaimanSegmentationExtractor it seems like it only attempts to ingest a 'correlation image' if a field (['estimators']['Cn']) is present. Otherwise it gives up. In the example data you have, it does not find this field. See below:

roiextractors/src/roiextractors/extractors/caiman/caimansegmentationextractor.py

Line 51 in 062f69e

self._image_correlation = self._summary_image_read()

roiextractors/src/roiextractors/extractors/caiman/caimansegmentationextractor.py

Lines 78 to 80 in 062f69e

	def _summary_image_read(self):
	if self._dataset_file["estimates"].get("Cn"):
	return np.array(self._dataset_file["estimates"]["Cn"])

Is it possible to make this more robust? It seems like this may be the case with the other extractors as well. I do not have experience with other extraction methods, nor do I have example output files from different pipelines beyond the ones you provide in the gin repo. As mentioned above the goal would be to ingest an roiextractors segmentation object as such:

roicat_data.FOV_image = roiextractors_segmentation_obj.get_image(name='mean')
roicat_data.masks = ingestion_method(roiextractors_segmentation_obj.get_roi_pixel_masks())

or similar. Do you think this is possible? If so, we are willing to help where possible.

[CodyCBakerPhD]

@RichieHakim Sorry for the delayed response on this

We spoke about it at our group meeting and decided the simplest course of action for now would be to guarantee completely lazy reading for all extractors, which ought to be trivial and we were quite surprised it wasn't already implemented across the board on ROIExtractors. We'll try to get that in place soon

The long-term desired API would be to introduce the notion of a 'stream' to ROIExtractors, analogous to the NEO and SpikeInterface projects in the extracellular electrophysiology sector; for SegmentationExtractors, each stream would be a different ROI response series and a single extractor would correspond one-to-one with each series for simplicity, which grants the level of granularity requested in this original issue