Correct Neuroconv Schema Inconsistency

[garrettmflynn]

This PR extends #406 to see if we could get a successful conversion to run by fixing the GUIDE's interpretation of the returned schema. Transforming the schema so the anomalous properties field in a array type is transformed into the items fields works to allow conversions.

Note: The NWB Inspector doesn't like the resulting file, though, as the following error is thrown:

[main-process]: [2023-09-29 11:59:28,094] ERROR in app: Exception on /neuroconv/inspect_file [POST]
Traceback (most recent call last):
  File "/Users/garrettflynn/Documents/Github/nwb-guide/pyflask/apis/neuroconv.py", line 150, in post
    list(
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/nwbinspector/nwbinspector.py", line 555, in inspect_nwbfile
    validation_errors = pynwb.validate(io=io)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/utils.py", line 648, in func_call
    return func(**pargs)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/pynwb/validate.py", line 132, in validate
    validation_errors = _validate_helper(io=io, namespace=namespace or CORE_NAMESPACE)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/pynwb/validate.py", line 29, in _validate_helper
    return validator.validate(builder)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/utils.py", line 644, in func_call
    return func(args[0], **pargs)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 250, in validate
    return validator.validate(builder)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/utils.py", line 644, in func_call
    return func(args[0], **pargs)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 425, in validate
    errors.extend(self.__validate_children(builder))
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 451, in __validate_children
    yield from self.__validate_child_builder(child_spec, child_builder, parent_builder)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 500, in __validate_child_builder
    yield from child_validator.validate(child_builder)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/utils.py", line 644, in func_call
    return func(args[0], **pargs)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 425, in validate
    errors.extend(self.__validate_children(builder))
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 451, in __validate_children
    yield from self.__validate_child_builder(child_spec, child_builder, parent_builder)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 500, in __validate_child_builder
    yield from child_validator.validate(child_builder)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/utils.py", line 644, in func_call
    return func(args[0], **pargs)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 388, in validate
    dtype = get_type(data)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/hdmf/validate/validator.py", line 129, in get_type
    return get_type(data[0])
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/h5py/_hl/dataset.py", line 831, in __getitem__
    selection = sel.select(self.shape, args, dataset=self)
  File "/opt/anaconda3/envs/nwb-guide/lib/python3.9/site-packages/h5py/_hl/selections.py", line 82, in select
    return selector.make_selection(args)
  File "h5py/_selector.pyx", line 282, in h5py._selector.Selector.make_selection
  File "h5py/_selector.pyx", line 151, in h5py._selector.Selector.apply_args
IndexError: Index (0) out of range for empty dimension

[CodyCBakerPhD]

I too see that error in the dev console, but it ought to be caught by the inspector and rendered back as a ERROR type of inspector message

[CodyCBakerPhD]

I can confirm this is still a problem with something regarding the NWB GUIDE modifying the default metadata of that interface

Running the conversion gallery: https://neuroconv.readthedocs.io/en/main/conversion_examples_gallery/segmentation/caiman.html

produces a file without validation issues (or errors)

My idea for the quick fix was just to pop Ophys from the metadata schema that gets returned to the front-end, and deep update what the front-end does successfully handle right now to the defaults during the run conversion step

[garrettmflynn]

Fixed. Just for our future reference, this was throwing an error on pynwb.validate because TwoPhotonSeries was auto-populated to have an empty array—of which the 0th index was attempting to be accessed (despite the schema not having a minItems entry).

[CodyCBakerPhD]

Works great now!