[Bug]: mismatch between timestamps and frames in miniscope video #376

laurelrr · 2024-10-17T21:05:39Z

What happened?

I am working with this repo: https://github.com/catalystneuro/tye-lab-to-nwb/tree/main/src/tye_lab_to_nwb/ast_ophys
and attempting to convert Miniscope calcium recordings but got the same error on 2 subjects.

Steps to Reproduce

python convert_session.py

Traceback

Traceback (most recent call last):
  File "/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_carl.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

Operating System

Linux

Python Executable

Conda

Python Version

3.11

Package Versions

environment_for_issue.txt

Code of Conduct

Yes

Duplicated Issue Check

Yes

h-mayorquin · 2024-10-18T15:53:45Z

@weiglszonja. @alessandratrapani (tagging both of you as we have worked on this)

I discussed this with @laurelrr it seems that one of the sessions for the Tye lab has a session where the timestamps do not match the number of frames. So there is an error here:

roiextractors/src/roiextractors/imagingextractor.py

Lines 189 to 198 in 2cf9235

    
               def set_times(self, times: ArrayType) -> None: 
        
                   """Set the recording times (in seconds) for each frame. 
        
                   Parameters 
        
                   ---------- 
        
                   times: array-like 
        
                       The times in seconds for each frame 
        
                   """ 
        
                   assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!" 
        
                   self._times = np.array(times).astype("float64")

The timestamps are obtained from a csv that is specific to miniscope and aggregated here:

https://github.com/catalystneuro/neuroconv/blob/52cd6aa0ea46c0d143739767da4951ade17c3b74/src/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py#L80-L85

And the frames are the sum of the frames of each individual extractor here:

roiextractors/src/roiextractors/multiimagingextractor.py

Lines 40 to 47 in 2cf9235

    
           self._start_frames, self._end_frames = [], [] 
        
           num_frames = 0 
        
           for imaging_extractor in self._imaging_extractors: 
        
               self._start_frames.append(num_frames) 
        
               num_frames = num_frames + imaging_extractor.get_num_frames() 
        
               self._end_frames.append(num_frames) 
        
           self._num_frames = num_frames

Which in turn are obtained from the video directly using OpenCV.

roiextractors/src/roiextractors/extractors/miniscopeimagingextractor/miniscopeimagingextractor.py

Lines 105 to 107 in 2cf9235

    
           def get_image_size(self) -> Tuple[int, int]: 
        
               return self._image_size[:-1]

So I think that either the timestamps in the csv are wrong or there are some drop frames in some videos. @laurelrr will see how he can share the data with us so we can verify that.

Meanwhile, what should we do in the case there are dropped frames? I think dropping frames is somehow common in acquisition systems that involve video so I think we should handle that case gracefully but I am not sure how.

alessandratrapani · 2024-10-21T10:37:11Z

The miniscope TimeStamps.csv has a dedicated column "Buffer_Index" to signal problems with frame acquisition, e.g., dropped frames. We can start by looking at this piece of data.

h-mayorquin · 2024-10-21T17:45:36Z

OK, this happens because there are some files that match the *.avi wildcard but not in the intended way in one of the segments:

├── 13_59_04
│   ├── BehavCam_2
│   │   ├── 0.avi
│   │   ├── 1.avi
│   │   ├── 2.avi
│   │   ├── 3.avi
│   │   ├── 4.avi
│   │   ├── 5.avi
│   │   ├── 6.avi
│   │   ├── 7.avi
│   │   ├── 8.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   ├── metaData.json
│   ├── Miniscope
│   │   ├── 0.avi
│   │   ├── 0_corrected.avi
│   │   ├── 1.avi
│   │   ├── 2.avi
│   │   ├── 3.avi
│   │   ├── 4.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   └── notes.csv
└── G2B4_C5-6852_L_yeti_DISC6UNP+TONES_msCamAll_ffd.mat

See the 0_corrected.avi

Can/should we improve the extractor to not match things with letters?

laurelrr · 2024-11-21T21:00:40Z

OK, my fix for this was to make a copy of the data folder and mv 0_corrected.avi 0.avi for each instance of a mislabeled filename.
I assume I did not need to reprocess any of the other downstream *.mat files that are used in convert_session.py, but please correct me if that seems wrong.
I'm trying it out now and will let you know the results.

laurelrr · 2024-11-21T21:40:07Z

OK, my idea did not seem to fix the issue. I will confer with my lab mates and see what else might have been affected.

laurelrr · 2024-11-22T18:56:13Z

After conferring with the team, it looks like we need only the 0.avi behavCam and the 0_corrected.avi miniscope file (which I have renamed to 0.avi). I set this up in my copy of the original folder. it now looks like this:

├── 13_59_04
│   ├── BehavCam_2
│   │   ├── 0.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   ├── metaData.json
│   ├── Miniscope
│   │   ├── 0.avi   (previously, 0_corrected.avi)
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   └── notes.csv
└── G2B4_C5-6852_L_yeti_DISC6UNP+TONES_msCamAll_ffd.mat

I am trying to run pulling from the updated miniscope folder, but still see the same error message:

Traceback (most recent call last):
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_yeti.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

Is there something else that should be updated, beyond just deleting extraneous files?

laurelrr · 2024-12-02T23:56:03Z

I also tried cropping the timeStamps.csv file to the correct length and tried running again but this also gives the same error:

Traceback (most recent call last):
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_yeti.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

laurelrr · 2024-12-02T23:56:46Z

Is your code making use of the metaData.json information when loading in the files?

weiglszonja · 2024-12-03T10:15:35Z

Is your code making use of the metaData.json information when loading in the files?

Yes, we are extracting the information from "recordingStartTime". But I don't see how that would affect the mismatch in frames. Can you share this session in its current form?

laurelrr · 2024-12-04T00:18:27Z

OK, I think I have figured out the issue. For some unknown reason, the Miniscope reset its frame rate from 15 fps to 60 fps partway through the recording. To correct this aberrant recording, my colleague used ffmpeg to change frame rate of the video but had not updated the timestamps file. I corrected the timestamps to match the video (and frame rate) and this now processes as expected.

If you wanted to make a more interpretable error message, listing the folder where the first instance of mismatch between the timestamps file and video frame rates would be helpful for your users. As always, thanks for the help!

weiglszonja · 2024-12-04T09:04:58Z

Very interesting, I'm glad to hear it works now!

Thank you for the feedback!

laurelrr · 2024-12-04T22:02:52Z

I'm getting a different error with our second dataset that seems related to the timestamps issue (subject had same framerate of 15 fps switching to 60 fps and was corrected post-processing).
I corrected the timestamps file in the same way as before and after that I could create the nwb file. But I got 2 critical errors in the nwbinspector report coming from check_timestamps_match_first_dimension:
drd1_6860_unpaired_inspector_result.txt

The miniscope dataset associated with this subject is about 6GB. Is there a way for me to share with you?

laurelrr added the bug Something isn't working label Oct 17, 2024

h-mayorquin mentioned this issue Oct 18, 2024

Improve error when settings timestamps #377

Merged

h-mayorquin assigned h-mayorquin, weiglszonja and alessandratrapani Nov 23, 2024

laurelrr closed this as completed Dec 4, 2024

laurelrr reopened this Dec 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: mismatch between timestamps and frames in miniscope video #376

[Bug]: mismatch between timestamps and frames in miniscope video #376

laurelrr commented Oct 17, 2024

h-mayorquin commented Oct 18, 2024

alessandratrapani commented Oct 21, 2024

h-mayorquin commented Oct 21, 2024

laurelrr commented Nov 21, 2024 •

edited

Loading

laurelrr commented Nov 21, 2024

laurelrr commented Nov 22, 2024

laurelrr commented Dec 2, 2024

laurelrr commented Dec 2, 2024

weiglszonja commented Dec 3, 2024

laurelrr commented Dec 4, 2024

weiglszonja commented Dec 4, 2024

laurelrr commented Dec 4, 2024

[Bug]: mismatch between timestamps and frames in miniscope video #376

[Bug]: mismatch between timestamps and frames in miniscope video #376

Comments

laurelrr commented Oct 17, 2024

What happened?

Steps to Reproduce

Traceback

Operating System

Python Executable

Python Version

Package Versions

Code of Conduct

Duplicated Issue Check

h-mayorquin commented Oct 18, 2024

alessandratrapani commented Oct 21, 2024

h-mayorquin commented Oct 21, 2024

laurelrr commented Nov 21, 2024 • edited Loading

laurelrr commented Nov 21, 2024

laurelrr commented Nov 22, 2024

laurelrr commented Dec 2, 2024

laurelrr commented Dec 2, 2024

weiglszonja commented Dec 3, 2024

laurelrr commented Dec 4, 2024

weiglszonja commented Dec 4, 2024

laurelrr commented Dec 4, 2024

laurelrr commented Nov 21, 2024 •

edited

Loading