Add support for Biocam brw v4.x files, closes #1324

[mahlzahn]

• missing: proper test file

[alejoe91]

Thanks Robert! Can you generate and share a very small test file for the new format?

[mahlzahn]

Thanks Robert! Can you generate and share a very small test file for the new format?

Hi Alessio, I will upload it and modify the concerning tests as soon as my colleague does new recordings. That should be until mid-October. (Same for SpikeInterface/probeinterface#216)

[matbonfanti]

Hi @mahlzahn, thanks for the work you have done! My collaborators and I are looking forward to using neo with our data.
Could we help you in any way? E.g. we could record and make available a short recording for the unittest...

[mahlzahn]

@matbonfanti You can already try it if you checkout the branch (but this means you don't pick up the latest developments from master):

pip install git+https://github.com/mahlzahn/python-neo@add_brw_v4.x_support

@alejoe91 I created the test file and I am waiting for the next steps to do the upload, see https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/issues/111.

Actually, I found there already another test file with a brw4.x file but not with the full raw data and this PR does not include support for such kind of recordings.

[matbonfanti]

thanks @mahlzahn, I have already tried your version of the code, reading a test recording from my collaborators. I can confirm it works fine (but this recording is too long to be used as test data).

[alejoe91]

@matbonfanti You can already try it if you checkout the branch (but this means you don't pick up the latest developments from master):

pip install git+https://github.com/mahlzahn/python-neo@add_brw_v4.x_support

@alejoe91 I created the test file and I am waiting for the next steps to do the upload, see https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/issues/111.

Actually, I found there already another test file with a brw4.x file but not with the full raw data and this PR does not include support for such kind of recordings.

That's great! If you send me the test files I can make a PR to gin. Let me know if instead you prefer to do it yourself!

[mahlzahn]

That's great! If you send me the test files I can make a PR to gin. Let me know if instead you prefer to do it yourself!

I already prepared all and now I wait for the access rights and then I create the PR myself.

[alejoe91]

@mahlzahn you should be able to open a PR now on GIN ;)

[pep8speaks]

Hello @mahlzahn! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2023-11-02 09:59:25 UTC

[b-grimaud]

Hi, any ETA on this ? I was trying to make my own version locally before finding this, great to see it being supported !

[alejoe91]

This should be ready for final review since @mahlzahn affed the test files too!

[b-grimaud]

Just noticed that, I also had a sample uploaded to the GIN a few months back.

Any reason not to calculate the gain with the bit depth as it is for brw v3.x ? I just get it with bit_depth = int(np.log2(rec_vars['ValueConverter']['MaxDigitalValue'])). They're very close numerically but on that kind of data I'm always a bit cautious.

I'll test with HerdingSpikes to see if there's the same scaling issues as with the original unfiltered neo readout.

[mahlzahn]

Any reason not to calculate the gain with the bit depth as it is for brw v3.x ? I just get it with bit_depth = int(np.log2(rec_vars['ValueConverter']['MaxDigitalValue'])). They're very close numerically but on that kind of data I'm always a bit cautious.

Yes, the brw v4.x file format documentation explicitly states how to calculate the analog value on page 7:

AnalogValue = MinAnalogValue + DigitalValue * (MaxAnalogValue – MinAnalogValue) /
(MaxDigitalValue – MinDigitalValue)

I just followed this.

[b-grimaud]

Opening, filtering and reading .brw v4.x worked just fine for me !

Still getting scaling issues as before :

• Through neo/SI, no filtering :
  

• Through neo/SI, with bandpass filter :
  
  

Some OK, some not, might just be saturation, hard to tell without the ability to filter in HerdingSpikes.

• Compared to HerdingSpikes with my own sort of functional reader for brw v4.x :
  

Will try again with another sorter to see if this is HS-specific.

[b-grimaud]

The issue seems to persist, even before sorting.

I loaded one of my recording, and through SpikeInterface :

traces = recording.get_traces(channel_ids=['45','100','1000','2486','78'], end_frame=4096)
plt.plot(traces)

Without filtering :

WIth bandpass filtering :

Looks like the same 2048 scaling issue.
Would be curious to see if this is reproducible on someone else's recording to see if this could be due to some hardware/recording parameters issue.

EDIT :

Tested on the test file uploaded by @mahlzahn to GIN, similar results :

Raw traces :

Bandpass filtered :

[samuelgarcia]

Hi Robert.
thank you for adding this new format version.
Sorry for the delay of merging!