-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bypass bessel filter for Chirp sweeps sampled at 12.5K #546
Comments
I think I have code for this in my chirp update, not positive though. I'll update here as I get that merged in. |
@MatthewAitken even after adding the new chirp stimuli to the EphysRawStimulusNames table we are still encountering the same issue and crash during feature extraction for any experiments that contain these new chirp sweeps. I checked the stimulus_ontology files since adding the new chirps to the EphysRawStimulusNames table and they are now present in the stimulus_ontology file so I am not sure why we still have the same issue. I haven't had a chance to look into this very much right now, other than to just confirm that the issue still exists. Let me know if you have any questions. Here is a recent example experiment log file: error message: |
@MatthewAitken I am now unable to install or update ipfx since implementing the fix for this issue and updating to 1.07. Here is the error I am getting: ` Preparing metadata (setup.py) ... error × python setup.py egg_info did not run successfully. note: This error originates from a subprocess, and is likely not a problem with pip. × Encountered error while generating package metadata. note: This is an issue with the package mentioned above, not pip. Is anyone else encountering this issue when attempting to install or update ipfx? Also, let me know if you need more information about the issue. |
Apologies coming in here as an external user of ipfx. I actually encountered this issue recently when trying to deploy ipfx to a fresh python install. |
@smestern thanks for the tip! For some reason this still didn't work with a conda environment for me but switching to a venv environment it did. |
Describe the use case that is addressed by this feature.
The inclusion of our new Chirp stimulus sets is causing feature extraction to fail because we are collecting them at a sampling rate of 12.5 K and IPFX has the Bessel filter set to 10 K for all current clamp sweeps.
Detailed description:
We have new chirp stimulus sets with slightly different names than the previous names; C2CHIRPA180503, C2CHIRPB180503, C2CHIRPC180503, C2CHIRPD180503 (we have added ‘A’, ‘B’, ‘C’, and ‘D’ between C2CHIRP and the date, 180503 as they are each ran at different holding potentials.
We are running these chirp sweeps at a sampling rate of 12.5 K with our bessel filter set to 6 K, but this is causing issues with IPFX and causing feature extraction to crash. The issue appears to stem from the call to data_set_features.extract_data_set_features(), which collects all current clamp sweeps and then runs them through data_set_features.extract_sweep_features() to get sweep_features (dictionary) and features_states[‘sweep_features_state’] (dictionary). The extract_sweep_features() function calls data_set_features.extractors_for_sweeps() to create a feature_extractor.SpikeFeatureExtractor object for each sweep then and calls the process() method from the class instance which calls time_series_utils.calculate_dvdt(), and this is where the issue is coming from.
The filter parameter is preset to 10.0 in the SpikeFeatureExtractor class, so the filter coefficient (filter_coeff in time_series_utils.calculate_dvdt()) is being determined to be outside of the acceptable range, which is 0 < filter_coeff <= 1, and throws a ValueError with the message “Bessel coeff 1.6 is outside of valid range [0,1); cannot filter sampling frequency 12.5 KHz with cutoff frequency 10 KHz”. This leaves sweep_features as an empty dictionary and the feature_states[‘sweep_features_state’] dictionary has ‘failed_fx’ set to True and the ‘fail_fx_message’ contains the ValueError message.
The filter_coeff is calculated as (filter * 1000) / (samp_rate / 2), which with a bessel fitler of 10,000 and a sampling rate of 12.5K comes to a filter_coeff of 1.6. The solution here is to just bypass the bessel filter by setting the filter parameter to None when creating the SpikeFeatureExtractor object for each of the chirp sweeps. This should not cause any issues as the sweeps are already filtered at 6kHz during data collection.
We will probably want to also update the chirp_stimulus_codes in the run_chirp_fv_extraction.CollectChirpFeatureVectorParameters class to include the new chirp stimulus codes.
Currently this issue is causing all of feature extraction to crash with a KeyError because the sweep_features dictionary is being returned as an empty dictionary from the extract_sweep_features() function after encountering the ValueError thrown by caluculate_dvdt(). After calling extract_sweep_features, extract_data_set_features tries to iterate over the sweep numbers in cell_features[‘long_squares’][‘subthreshold_sweeps’] to get the ‘peak_deflect’ values and assign it to the corresponding sweep number in the sweep_features dictionary, but because that dictionary has been returned back as an empty dictionary feature extraction crashes with a KeyError on the first attempt through the loop.
Describe the solution you'd like
A solution to this feature request will prevent feature extraction from crashing due to the sampling rate of the new Chirp sweeps by setting the filter None for the chirp sweeps.
We would like the Bessel filter to be bypassed for all for all new chirp sweeps that are filtered at 12.5 K
Describe alternatives you've considered
I don’t see any other solution to this issue other than sampling at a higher rate, but that will produce larger file sizes and we have already started collecting data at 12.5 K for the new chirp sweeps
Additional context
I have attached a jupyter notebook that I used to troubleshoot this issue which contains all of the outputs from the cells which will hopefully make it easy to follow along)
Do you want to work on this issue?
I can help out with this issue but I don’t have the time to develop a solution entirely on my own.
new_chirp_fx_error.zip
The text was updated successfully, but these errors were encountered: