Migrating caches / base folders between drives

[kevsch88]

Hi,

tldr: Trying to migrate curated waveforms folder to new drive, but no luck making it refer to location of recording on the new drive via save() method.

I am working with Neuropixels recordings, have a lot of experience with ephys data but am newer to larger scale datasets and its related tribulations. Overall, It's been a pleasure working with SpikeInterface so far to set up our sorting and analysis pipeline, I've mostly run into one obstacle regarding saving/loading cached data, which led me to wonder if there was a recommended way to migrate a folder with processed and sorted outputs onto a different drive.

My reason for it is that currently I have a short-term storage solution that is faster for the processing stages, but need to move the large components (ie, caches) to slow long-term storage (and/or network) drives once I've extracted what I need for analysis after post-processing.

I very well might be adding unnecessary steps, but what I am trying to do is relocate the cleaned waveforms folder, which would seem to require doing so for the recording and sorting caches as well, to maintain access. I have played around with different options and relative path kwargs but I can't seem to be able to get the newly saved metadata of the clean waveforms folder to point to the new drive, even if the recording and sorting caches themselves are loaded from there.

I don't think anything is not working as intended, so am I missing something, is there an ideal order for saving that might aid with this or is this just a manual json editing job?

Thanks in advance for your time in any case, and great work with SI, I am looking forward to its growth!
Kevin

[kevsch88]

PS: To avoid potentially making anyone repeat themselves, I wanted to add that I had previously found the answers to this issue, which seemed similar but not quite what I've been doing.

If the answer is the same though then it sounds like I can dump the recording and sorting to new drive but just directly copy the waveforms_folder (which is already relative) to the new "base_folder", rather than use the waveform extractor's save method, which raises the drive path error if you try to keep paths relative but pointing to the new sorting directory.

[alejoe91]

Hi @kevsch88

Sorry for our late reply. The issue you mentioned #1618 is indeed what yuou should do. I also added a section in the documentation to better describe this very common use case: https://spikeinterface.readthedocs.io/en/latest/modules/core.html#waveformextractor (see the IMPORTANT note!)

Basically, you can move the waveform folder directly as long as you also move the sorting file/folder (and optionally the recording file/folder) to have the same relative path with respect to the waveform folder.

Hope this is clear :)

Cheers
Alessio