KS4: CUDNN_STATUS_NOT_SUPPORTED error

[a-zmz]

Hi,

I upgraded my spikeinterface to "0.101.0rc0" tried to run kilosort 4.0.12 with singularity image. However, I got the following error:

---------------------------------------------------------------------------                                                                                                                                                                                                                                                                                      
SpikeSortingError                         Traceback (most recent call last)                                                                                                                                                                   
File ~/Documents/git/analysis/analysis/preprocessing/test_ks4.py:119                                                                                                                                                                          
    116 os.chdir(preprocess_folder)                                                                                                                                                                                                           
    118 # sort spikes                                                                                                                                                                                                                         
--> 119 sorting = ss.run_sorter(                                                                                                                                                                                                              
    120     sorter_name='kilosort4',                                                                                                                                                                                                          
    121     recording=ap_mcd,                                                                                                                                                                                                                 
    122     folder=str(base_folder/"ks4_full"),                                                                                                                                                                                               
    123     singularity_image=True,                                                                                                                                                                                                           
    124     remove_existing_folder=True,                                                                                                                                                                                                      
    125     #**ks4_params,                                                                                                                                                                                                                    
    126 )                                                                                                                                                                                                                                     
    128 # curate                                                                                                                                                                                                                              
    129 sorting = scur.remove_duplicated_spikes(                                                                                                                                                                                              
    130     sorting,                                                                                                                                                                                                                          
    131     censored_period_ms=0.3,                                                                                                                                                                                                           
    132     method="keep_first_iterative",                                                                                                                                                                                                    
    133 )                                                                                                                                                                                                                                     
       
File ~/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py:169, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    167         else:
    168             container_image = singularity_image
--> 169     return run_sorter_container(
    170         container_image=container_image,
    171         mode=mode,
    172         **common_kwargs,
    173     )
    175 return run_sorter_local(**common_kwargs)

File ~/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py:595, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, delete_container_files, extra_requirements, installation_mode, spikeinterface_version, spikeinterface_folder_source, **sorter_params)
    593 if run_error:
    594     if raise_error:
--> 595         raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    596 else:
    597     if with_output:

SpikeSortingError: Spike sorting in singularity failed with the following error:
  0%|          | 0/844 [00:00<?, ?it/s]/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py:134: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)
  B = conv1d(X.unsqueeze(1), W, padding=nt//2)
100%|██████████| 844/844 [03:44<00:00,  3.76it/s]
100%|██████████| 844/844 [03:44<00:00,  3.76it/s]
100%|██████████| 96/96 [02:00<00:00,  1.25s/it]
/home/miniconda3/lib/python3.11/site-packages/kilosort/template_matching.py:78: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)
  X = conv1d(X.unsqueeze(1), ops['wTEMP'].unsqueeze(1), padding=ops['nt']//2)
100%|██████████| 844/844 [01:07<00:00, 12.50it/s]
100%|██████████| 96/96 [01:47<00:00,  1.12s/it]
/home/amz/running_data/npx/raw/20240711_az_PANT06/si_preprocess/in_container_python_base/lib/python3.11/site-packages/spikeinterface/core/basesorting.py:264: UserWarning: The registered recording will not be persistent on disk, but only available in memory
  warnings.warn("The registered recording will not be persistent on disk, but only available in memory")

> /home/amz/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py(595)run_sorter_container()
    593     if run_error:
    594         if raise_error:
--> 595             raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    596     else:
    597         if with_output:

Could anyone please help? Thanks!

Best,
Arthur

[zm711]

Hey @a-zmz,

this is an error in the kilosort runtime. You can see the error here is within the kilosort package.

%|          | 0/844 [00:00<?, ?it/s]/home/miniconda3/lib/python3.11/site-packages/kilosort/spikedetect.py:134: UserWarning: Plan failed with a cudnnException: CUDNN_BACKEND_EXECUTION_PLAN_DESCRIPTOR: cudnnFinalize Descriptor Failed cudnn_status: CUDNN_STATUS_NOT_SUPPORTED (Triggered internally at ../aten/src/ATen/native/cudnn/Conv_v8.cpp:919.)

It would probably be best to open the issue directly on their repo so they can let you know what to do for the cudnn errors. Then if they say the problem is something with our wrapper you can bring the issues here. But since this error is inside kilosort they would know better than us.

[h-mayorquin]

Is that the complete error traces/

Also, isn't that the wrong version of spikeinterface for trying kilosort4?

See
#3192

[zm711]

I think we had the ks4 fixes by rc0, but maybe not. You could also try from main in that case.

[a-zmz]

thanks, i will try 0.100.8

[a-zmz]

Is that the complete error traces/

Also, isn't that the wrong version of spikeinterface for trying kilosort4?

See #3192

yes that is the complete error trace

[a-zmz]

hello, just tried with spikeinterface 0.100.8, got this error:

SpikeSortingError                         Traceback (most recent call last)                                          
File ~/Documents/git/analysis/analysis/preprocessing/test_ks4.py:119                                                 
    116 os.chdir(preprocess_folder)                                                                                  
    118 # sort spikes                                                                                                
--> 119 sorting = ss.run_sorter(                                                                                     
    120     sorter_name='kilosort4',
  120     sorter_name='kilosort4',                                                                                                                                                                                        16:43:46 [60/6372]
    121     recording=ap_mcd,                                                                                                                                                                                                                 
    122     folder=str(base_folder/"ks4_full"),                                                                                                                                                                                               
    123     singularity_image=True,                                                                                                                                                                                                           
    124     remove_existing_folder=True,                                                                                                                                                                                                      
    125     #**ks4_params,                                                                                                                                                                                                                    
    126 )                                                                                                                                                                                                                                     
    128 # curate                                                                                                                                                                                                                              
    129 sorting = scur.remove_duplicated_spikes(                                                                                                                                                                                              
    130     sorting,                                                                                                                                                                                                                          
    131     censored_period_ms=0.3,                                                                                                                                                                                                           
    132     method="keep_first_iterative",                                                                                                                                                                                                    
    133 )                                                                                                                                                                                                                                     
                                                                                                                                                                                                                                              
File ~/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py:169, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularit
y_image, delete_container_files, with_output, **sorter_params)                                                                                                                                                                                
    167         else:                                                                                                                                                                                                                         
    168             container_image = singularity_image                                                                                                                                                                                       
--> 169     return run_sorter_container(                                                                                                                                                                                                      
    170         container_image=container_image,                                                                                                                                                                                              
    171         mode=mode,                                                                                                                                                                                                                    
    172         **common_kwargs,                                                                                                                                                                                                              
    173     )                                                                                                                                                                                                                                 
    175 return run_sorter_local(**common_kwargs)                                                                                                                                                                                              
                                                                                                                                                                                                                                              
File ~/.conda/envs/ks4/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py:595, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, rais
e_error, with_output, delete_container_files, extra_requirements, installation_mode, spikeinterface_version, spikeinterface_folder_source, **sorter_params)                                                                                   
    593 if run_error:                                                                                                                                                                                                                         
    594     if raise_error:                                                                                                                                                                                                                   
--> 595         raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")                                                                                                                     
    596 else:                                                                                                                                                                                                                                 
    597     if with_output:                                                                                                                                                                                                                   
                                                                                                                                                                                                                                              
SpikeSortingError: Spike sorting in singularity failed with the following error:                                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                                            
  File "/home/amz/running_data/npx/raw/20240711_az_PANT06/si_preprocess/in_container_sorter_script.py", line 23, in <module>                                                                                                                  
    sorting = run_sorter_local(                                                                                                                                                                                                               
              ^^^^^^^^^^^^^^^^^                                                                                                                                                                                                               
  File "/home/amz/running_data/npx/raw/20240711_az_PANT06/si_preprocess/in_container_python_base/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 223, in run_sorter_local                                             
    SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)                                                                                                                                                        
  File "/home/amz/running_data/npx/raw/20240711_az_PANT06/si_preprocess/in_container_python_base/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 179, in set_params_to_folder                                        
    raise AttributeError("Bad parameters: " + str(bad_params))                                                                                                                                                                                
AttributeError: Bad parameters: ['folder'] 

[h-mayorquin]

It seems that your have a keyword argument folder that is not properly absorbed by the run_sorter caller and instead is proagated to the sorter where is not a valid parameter.

Can you show me how you are calling the sorter? (the code that you run)

[a-zmz]

  118     # sort spikes
  119     sorting = ss.run_sorter(
  120         sorter_name='kilosort4',
  121         recording=ap_mcd,
  122         folder=str(base_folder/"ks4_full"),
  123         singularity_image=True,
  124         remove_existing_folder=True,
  125         #**ks4_params,
  126     )

the folder parameter was initially a Path, but then i got this error:
TypeError: Object of type PosixPath is not JSON serializable

so i converted it into string, and got the previous error

[zm711]

That's an error we got when we switched run_sorter and run_sorter_by_property to both use the folder argument rather than output_folder. could you try

output_folder

instead of folder and see what happens.

If someone has a stale docker/singularity then they won't quite have the right spikeinterface. Alessio said we need to find a way to force rebuild for this as parameters change. But we've been too busy to work on that.

[a-zmz]

i see, thanks i'll try

[h-mayorquin]

Related:

#3156

And:

#3103

If it not too much to ask maybe you coul run from main with the suggestions that @zm711 gave on #3192

Otherwise, a release is coming that should contain all of those fixes in the next week.

[a-zmz]

yea i could indeed try to run from main, and good to know a release is coming soon, thanks very much!