Protect against corrupted channels in legacy openephys.

[samuelgarcia]

@weiglszonja : Here this raise an error when a file is corrupted (bad timestamp)

[zm711]

So I don't know openephys very well at all, but I'm wondering if there would ever be a case where the timestamps could be corrupted but the other channels are okay (ie warn the user of the corruption but then try to salvage the data and let the end-user decide whether to continue using that dataset). But maybe timestamp corruption always indicates that the whole file is messed up and shouldn't be used. I just don't know personally.

[samuelgarcia]

@zm711 : I get a data example from @weiglszonja where only on channelf ile is corrupted and prevent the entire opening. Other channels are OK (but with gaps).

Skipping the channel would be tedious in the code because it is a mix of index and key.
So the raise erreur indicate a clear solution for the user I think no ?

[zm711]

In this issue #1438, someone is asking to ignore corrupted channel and just use samples. If it's too annoying to work around I think that's fine, but it seems like at least some users would like to still use datasets even if some channels are corrupt. I'm fine with just raising the error though. Just wanted to keep it on your radar.

[samuelgarcia]

I recently add zeros padding when gaps are detected. So we cannot ignore this otherwise events could be not aligned with traces. But if one channel have bad timstamps that are not monotically increasing then we cannot do anything. Ignoring the gaps like it used to be the case for a while is not good I think.

Here the PR is for the legacy openephys format.
The issue you are mentionning for the "new" binary openephys format. No ?

[zm711]

Yes it is for the binary/new format. My point is just that some people regardless of the corruption status of their data may want to try to salvage something. That doesn't mean we have to support it, but I think it does come up.

[weiglszonja]

My point is just that some people regardless of the corruption status of their data may want to try to salvage something. That doesn't mean we have to support it, but I think it does come up.

I think what @samuelgarcia did here still provides a way for the user to salvage the dataset, they would have to move out the corrupted file from the folder and rerun. But maybe I misunderstood your comment here.

[zm711]

Hi @weiglszonja!

If that's all it takes then shouldn't that be in the error message? Again, I don't use openephys so I might be saying nonsensical things, but if we can still parse the files by removing the corrupted files shouldn't we just warn the user and do that? Or let them know that that is a possibility? Or we say that as a library that we don't do that because if part of the file/folder of files is corrupted we don't want to potentially return other data being corrupted.