Is it possible to reuse sorter forlder?

[borrepp]

Hello,

We are using spikeInterface to sort our monkey data. We record with 32-channel probes, and typical recordings last ~2 hrs. It takes a lot of time (~2.5 hrs; 5 minutes per channel approx) to create the sorting folder when using, for example, "waveclus". Even if the spikeInterface binary folder already exists.

This waiting time might not be a big problem if it is only one time per recording. But, we would like to run the same sorter with different parameters, i.e., "detect_threshold". This means it will create the recording files again. Is there a way to reuse the folder and run the same (or different) sorter with other parameters? Or, could we reduce the writing process every time we test a different parameter or use a different sorter?

To give some information, I'm running the package on a Windows computer with 64GB of RAM, i9-13th generation, and NVIDIA card RTX4090 with 16GB.

Thanks in advance for your help.
Pepe

[zm711]

Hey @borrepp, we are super bad about responding to discussion! Sorry. Opening an issue is way better. In fact this could be a discussion point. If your recording is binary compatible our wrappers for Kilosort for example should detect that and not rewrite the recording again. It seems that waveclus requires a specific format for it to read files which necessitates the writing. I guess we could add a check for waveclus such that if you have the file it could reuse, but this would be tricky since each sorter has its own rules. Would you like to open an issue and make this feature request?

[borrepp]

Hello Zach @zm711

Thanks for your response, you are right, I have been trying other sorters and the setup of the recording is super fast, for example:

kilosort4 & spykingcircus2 less than a minute (as you said, detected the binary and didn't write)
mountainsort5 ~2 minutes (it rewrites the binary into hdf5)

I think we are going to skip "Waveclus" for now. So, I'm not sure if it is worth asking for a parallel processing step for writing channels in the case of Waveclus? What do you think?

[zm711]

Waveclus isn't being super actively developed, so not sure if it is worth pursuing unless you really want to use it. It would definitely be lower on the priority list. The other tricky part I would need to read up on is that the .hdf5 format might not tell us whether it is a random hdf5 vs an appropriately prepared file, so I'm not sure if making the check would be super easy.

Mountainsort5 does what kilosort does, but it does it inside the spikeinterface wrapper (ie, kilosort uses the binary file then writes the temp_wh.dat as its intermediate file). MS5 takes the recording you give it preprocesses then writes that preprocessing with spikeinterface multiprocessing. So needing to write that preprocessed intermediate file is just part of the MS5 workflow (just like Kilosort writing its temp_wh.data). So both of those are pretty optimized for their workflows.

[borrepp]

I will ask my PI whether he really wants to use Waveclus. If so, I will open an issue. But I think we will get very good sorting performance with either Kilosort, MS5, or SpikingCircus.

By the way, while trying to run Kilosort4, I found a couple of errors in the kilosort4 wrapper:
a) One related to a missing parameter called "save_preprocessed_copy", I solved it by setting the value to False (similar to the original kilosort4 pipeline) and adding it as an argument when "initialize_ops" is called. (line194)
b) An error with the number of outputs when "get_run_parameters" is called (line 197). I solved this by adding two more outputs ("_")

My question is, should I "commit" these changes through GitHub? I'm relatively new to Git Hub, so I don't know if that is the proper way to proceed or if I should open an issue and explain what I did.

Thanks for all your feedback Zach (@zm711)
Best
Pepe

[zm711]

Kilosort4 has had a lot of updates (multiple per week). So our current codebase (0.100.8 and 0.100.0rec0) should support up to 4.12. What version of spikeinterface are you using. If it is below 0.100.8 then I would suggest updating and seeing if there error is still there. If it is still there you could do one of two things.

1. just open the PR explaining what you did. (This could be risky because maybe the developers are already working on it and so yours gets closed)
2. open an issue with the problem and your solution and see if they want you to open a PR.

The group here is pretty chill so just opening a PR isn't the end of the world, but it is generally more polite and less risky to open an issue first. Even better would be to search if someone else reported the issue and then just comment on it that you have the problem too and that you think you have a solution.

But first up for KS4 is what version of spikeinterface are you using and which version of kilosort4.

[borrepp]

Got it,

The spikeinterface version that I'm using is 0.101.0 (calling spikeinterface.version) and kilosort4 version is 4.0.13 (calling si.sorters.print_sorter_versions())

I will search if the same issue has been reported, if not I will open an issue.

Thanks again for all your help ¡¡¡
Best
Pepe