Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unequal number of datapoints in NWB conversion #473

Open
ozsat opened this issue Sep 4, 2020 · 2 comments
Open

unequal number of datapoints in NWB conversion #473

ozsat opened this issue Sep 4, 2020 · 2 comments
Labels

Comments

@ozsat
Copy link

ozsat commented Sep 4, 2020

I noticed that when I convert DAT to NWB (ipfx.bin.run_x_to_nwb_conversion.convert), the voltage samples and current samples are not equal. In the current samples, 5 of the given points are always missing from the beginning. It seems to be a major problem because the ipfx cant process those NWB files.
I checked the raw data and it contains an equal number of points (I also attached a picture that shows the discrepancy in the NWB file).

voltage_samples_nwb
stimulus_samples_nwb

When I run run_pipeline on the generated NWB data the following error message appears:

INFO:root:
INFO:root:***** ***** ***** Extract QC features ***** ***** *****
INFO:root:=======================================================
INFO:root:
INFO:root:Ontology is not provided, using default E:\anaconda3\lib\site-packages\ipfx\defaults\stimulus_ontology.json
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:28: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
blowout_sweep_number = data_set.get_sweep_number(ontology.blowout_names)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpinbath
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpbreakn
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Long square
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Rheobase
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Short square
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Capacitance
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Chirp test
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: Chirp
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\stimulus.py:88: UserWarning: Could not find stimulus: extpexpend
warnings.warn("Could not find stimulus: %s" % tag)
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:57: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
bath_sweep_number = data_set.get_sweep_number(ontology.bath_names)
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:99: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
seal_sweep_number = data_set.get_sweep_number(ontology.seal_names,"VoltageClamp")
E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py:151: VisibleDeprecationWarning: Function get_sweep_number is deprecated. call .get_sweep_numbers()[-1] instead
breakin_sweep_number = data_set.get_sweep_number(ontology.breakin_names,"VoltageClamp")
WARNING:root:Could not compute input/access resistance ratio (sr: None, ir:: None)
WARNING:root:Blowout is not available
WARNING:root:Electrode 0 is not available
WARNING:root:Seal is not available
WARNING:root:Breakin sweep not found
Traceback (most recent call last):
File "e:/ipfx/File_reader.py", line 74, in
output_dir=Output_Full_Path,
File "e:/ipfx/File_reader.py", line 60, in run_pipelinefrom_nwb_file
write_spikes)
File "E:\anaconda3\lib\site-packages\ipfx\bin\run_pipeline.py", line 27, in run_pipeline
stimulus_ontology_file
File "E:\anaconda3\lib\site-packages\ipfx\bin\run_sweep_extraction.py", line 69, in run_sweep_extraction
sweep_features = sweep_qc_features(ds)
File "E:\anaconda3\lib\site-packages\ipfx\qc_feature_extractor.py", line 278, in sweep_qc_features
sweep = data_set.sweep(sweep_num)
File "E:\anaconda3\lib\site-packages\ipfx\dataset\ephys_data_set.py", line 240, in sweep
enforce_equal_length=True,
File "E:\anaconda3\lib\site-packages\ipfx\dataset\ephys_data_set.py", line 431, in _voltage_current
f"found {len(voltage)} voltage samples, "
ValueError: found 302000 voltage samples, but 301995 current samples

The DAT file as an example: https://drive.google.com/file/d/1GgNunvFPqRpjz9A6DAj9jF7uFA_owmtR/view?usp=sharing
The generated NWB: https://drive.google.com/file/d/1eHzVY_2F6fyWNtE0R1OyhkAtZ2a-JMWt/view?usp=sharing

Environment:

  • Windows 10
  • ipfx 1.0.0
  • python 3.7
@tmchartrand
Copy link
Collaborator

@ozsat I'm guessing you closed this based on using the fix implemented on my fork? we should keep this issue open to track the work of merging that fix into the primary repository.
If you closed it for some other reason, please let us know!

@ozsat ozsat reopened this Oct 15, 2020
@wbwakeman wbwakeman mentioned this issue Oct 16, 2020
5 tasks
t-b pushed a commit to byte-physics/x-to-nwb that referenced this issue Dec 6, 2020
@tmchartrand
Copy link
Collaborator

This is fixed at the new x-to-nwb conversion repo (see above), but should maybe be left for reference until the faulty code is removed here (#492).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants