MesoscopePreprocess tasks erroring because last frame corrupt

[samupicard]

A number of MesoscopePreprocess tasks have errored with the log below, likely because the acquisition was interrupted due to disk storage being full. This leads to an issue with the final frame of the final tiff stack not being written to disk properly.

Would be great to have some kind of workaround for this as it has happened in a handful of important sessions. Might be difficult to solve this in the suite2p code itself, but perhaps we can have a separate task that checks the final tif stack, and removes any corrupt final frames, before it is passed to the bin_per_plane task.

In these cases the MesoscopeSync task may also need a workaround (i.e. ignore the final frame counter(s) in timeline).

2024-11-16 01:25:33,993,993 INFO [mesoscope_tasks.py:774] Extracting tif data per plane
2024-11-16 01:32:32,890,890 ERROR [tasks.py:267] Traceback (most recent call last):
File "/home/ibladmin/Documents/PYTHON/envs/suite2p/lib/python3.10/site-packages/ibllib/pipes/tasks.py", line 257, in run
outputs = self._run(**kwargs)
File "/home/ibladmin/Documents/PYTHON/envs/suite2p/lib/python3.10/site-packages/ibllib/pipes/mesoscope_tasks.py", line 776, in _run
plane_folders, _ = self.bin_per_plane(metadata, save_folder=save_folder, save_path0=self.session_path)
File "/home/ibladmin/Documents/PYTHON/envs/suite2p/lib/python3.10/site-packages/ibllib/pipes/mesoscope_tasks.py", line 677, in bin_per_plane
ret = suite2p.io.mesoscan_to_binary(ops.copy())
File "/home/ibladmin/Documents/PYTHON/envs/suite2p/lib/python3.10/site-packages/suite2p/io/tiff.py", line 330, in mesoscan_to_binary
tif, Ltif = open_tiff(file, use_sktiff)
File "/home/ibladmin/Documents/PYTHON/envs/suite2p/lib/python3.10/site-packages/suite2p/io/tiff.py", line 92, in open_tiff
tif = ScanImageTiffReader(file)
File "/home/ibladmin/Documents/PYTHON/envs/suite2p/lib/python3.10/site-packages/ScanImageTiffReader/__init__.py", line 173, in __init__
self.open(filename)
File "/home/ibladmin/Documents/PYTHON/envs/suite2p/lib/python3.10/site-packages/ScanImageTiffReader/__init__.py", line 222, in open
raise Exception(self.__ctx.log.decode("utf-8","strict"))
Exception: Got unrecognized element type in a tiff tag.

Examples of sessions that error like this are:

• MesoscopePreprocess 65464454-caf0-4dbf-83a3-8ec3fc854b82 SP061/2024-11-15/001 Errored
• MesoscopePreprocess e52aa4c4-c179-4ae3-b52e-38ebc8b40332 SP061/2024-11-18/001 Errored