MergeUnitsSorting returns an error

[rat-h]

I have directories with preprocessed data, saved sorting results, extracted waveforms, and precomputed quality metrics for multiple recordings. I'm going over each directory and auto-merge units. Note These recordings have (naturally) low spike rates, so I set minimum_spikes to the minimum spikes in units.

import sys, os, json
import psutil
import spikeinterface.full as si
import pandas as pd
from numpy import ceil
from spikeinterface.curation import MergeUnitsSorting, get_potential_auto_merge

for x in sys.argv[1:]:
    try:
        we = si.load_waveforms(x+'/waveforms')
    except BaseException as e:
        print(f"Cannot load waveform from {x}/waveforms: {e}")
        continue
    try:
         m = pd.read_pickle(x+'/qualitymetrics.pkl')
    except BaseException as e:
        print(f"Cannot load waveform from {x}/qualitymetrics.pkl: {e}")
        continue
    minnumspike = int( ceil(m['num_spikes'].min()) )
    merges = get_potential_auto_merge(we,
              minimum_spikes=minnumspike,
              maximum_distance_um=50.,
              peak_sign="neg",
              bin_ms=0.5,
              window_ms=100.,
              corr_diff_thresh=0.16,
              template_diff_thresh=0.25,
              censored_period_ms=0., 
              refractory_period_ms=2.0,
              contamination_threshold=0.01, 
              num_channels=5, num_shift=5,
              firing_contamination_balance=1.5)
    print(x)
    print(merges)
    print(len(merges))
    print()
    # exit(0)
    if len(merges) == 0: continue
    try:
        sorting = si.load_extractor(x+'/sorting-saved')
    except BaseException as e:
        print(f"Cannot load sorting from {x}/sorting-saved: {e}")
        continue
    try:
        preproc_saved = si.load_extractor(x+'/preprocessed')
    except BaseException as e:
        print(f"Cannot load prerpocessed data from {x}/preprocessed: {e}")
        continue
    clean_sorting = MergeUnitsSorting(sorting, merges)
    sorting_saved = clean_sorting.save(folder=x+"/sorting-clean")

If there are any candidates for merging, the MergeUnitsSorting crashes with an error:

python test-selfmerge.py tds2-continuous-T1-20230902-155148-crx24x70-2335067-final-archive-147 
tds2-continuous-T1-20230902-155148-crx24x70-2335067-final-archive-147
[(135, 392), (211, 1247), (237, 751), (237, 1020), (239, 658), (243, 261), (243, 938), (261, 938), (311, 638), (392, 938), (459, 617), (514, 1491), (519, 879), (519, 938), (598, 751), (598, 1020), (619, 1052), (648, 857), (658, 664), (683, 910), (683, 1278), (683, 1354), (751, 1020), (751, 1052), (789, 920), (789, 1066), (789, 1290), (843, 910), (879, 938), (880, 938), (907, 1548), (1020, 1052), (1033, 1114), (1054, 1144), (1144, 1343), (1203, 1387)]
36

Traceback (most recent call last):
  File "/home/rth/spikeinterface/test-selfmerge.py", line 52, in <module>
    clean_sorting = MergeUnitsSorting(sorting, merges)
  File "/home/rth/.local/apps/spikes/lib/python3.10/site-packages/spikeinterface/curation/mergeunitssorting.py", line 48, in __init__
    keep_unit_ids = [u for u in parents_unit_ids if u not in all_removed_ids]
  File "/home/rth/.local/apps/spikes/lib/python3.10/site-packages/spikeinterface/curation/mergeunitssorting.py", line 48, in <listcomp>
    keep_unit_ids = [u for u in parents_unit_ids if u not in all_removed_ids]
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

It feels like I did something silly, but I couldn't find what. Any ideas?

[maxjuv]

Hello,
I've got the same issue. get_potential_auto_merge returns a list of tuples, while MergeUnitsSorting requires a lit of lists.
I added this to my code to solve the issue :

unit_to_merge_list = []
for units_tuple in merges :
    unit_to_merge_list.append(list(units_tuple))
clean_sorting = MergeUnitsSorting(sorting, unit_to_merge_list)

[rat-h]

@maxjuv, you are right! A conversion of tuples to lists fixes this problem

merges = [ list(units_tuple) for units_tuple in merges ]

Thanks a lot!

P.S. why get_potential_auto_merge returns tuples?

[alejoe91]

Can you guys check this #1995 and see if it solves the issue (avoiding the extra tuple to list conversion)?

[rat-h]

@alejoe91 Yes, it does.