Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sgi.insert_sessions failing for ImportedSpikes #1204

Open
pauladkisson opened this issue Jan 2, 2025 · 3 comments · May be fixed by #1205
Open

sgi.insert_sessions failing for ImportedSpikes #1204

pauladkisson opened this issue Jan 2, 2025 · 3 comments · May be fixed by #1205
Labels
bug Something isn't working merge To do with merge tables NWB ingestion Problems with loading nwb files into spyglass

Comments

@pauladkisson
Copy link

pauladkisson commented Jan 2, 2025 •
edited
Loading

Describe the bug
I'm getting an error with spyglass.data_import.insert_sessions for a simple nwb file with just a units table.

To Reproduce

  1. Start the database
docker run --name spyglass-db -v dj-vol:/var/lib/mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=tutorial datajoint/mysql
  1. Generate test nwb file
from pynwb.testing.mock.file import mock_NWBFile
import numpy as np
from pynwb import NWBHDF5IO

nwbfile = mock_NWBFile()
num_units = 3
for unit in range(num_units):
    spike_times = np.sort(np.random.rand(100))
    nwbfile.add_unit(id=unit, spike_times=spike_times)
nwbfile_path = "/Volumes/T7/CatalystNeuro/Spyglass/raw/test.nwb"
with NWBHDF5IO(nwbfile_path, mode='w') as io:
    io.write(nwbfile)
  1. Import spyglass
import os
import datajoint as dj
# change to the upper level folder to detect dj_local_conf.json
if os.path.basename(os.getcwd()) == "notebooks":
    os.chdir("..")
dj.config.load("dj_local_conf.json")  # load config for database connection info
import spyglass.common as sgc # this import connects to the database
import spyglass.data_import as sgi
from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename
  1. Insert sessions
nwb_file_name = "test.nwb"
sgi.insert_sessions(nwb_file_name, rollback_on_fail=True, raise_err=True)
Error Stack 
---------------------------------------------------------------------------
DataJointError                            Traceback (most recent call last)
Cell In[3], [line 2](vscode-notebook-cell:?execution_count=3&line=2)
      [1](vscode-notebook-cell:?execution_count=3&line=1) nwb_file_name = "test.nwb"
----> [2](vscode-notebook-cell:?execution_count=3&line=2) sgi.insert_sessions(nwb_file_name, rollback_on_fail=True, raise_err=True)

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:77, in insert_sessions(nwb_file_names, rollback_on_fail, raise_err)
     [75](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:75) copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)
     [76](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:76) Nwbfile().insert_from_relative_file_name(out_nwb_file_name)
---> [77](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:77) return populate_all_common(
     [78](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:78)     out_nwb_file_name,
     [79](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:79)     rollback_on_fail=rollback_on_fail,
     [80](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:80)     raise_err=raise_err,
     [81](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/data_import/insert_sessions.py:81) )

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:150, in populate_all_common(nwb_file_name, rollback_on_fail, raise_err)
    [126](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:126) table_lists = [
    [127](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:127)     [  # Tables that can be inserted in a single transaction
    [128](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:128)         Session,
   (...)
    [146](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:146)     ],
    [147](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:147) ]
    [149](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:149) for tables in table_lists:
--> [150](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:150)     single_transaction_make(
    [151](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:151)         tables=tables,
    [152](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:152)         nwb_file_name=nwb_file_name,
    [153](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:153)         raise_err=raise_err,
    [154](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:154)         error_constants=error_constants,
    [155](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:155)     )
    [157](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:157) err_query = InsertError & error_constants
    [158](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:158) nwbfile_query = Nwbfile & {"nwb_file_name": nwb_file_name}

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:91, in single_transaction_make(tables, nwb_file_name, raise_err, error_constants)
     [89](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:89) except Exception as err:
     [90](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:90)     if raise_err:
---> [91](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:91)         raise err
     [92](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:92)     log_insert_error(
     [93](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:93)         table=table, err=err, error_constants=error_constants
     [94](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:94)     )

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:88, in single_transaction_make(tables, nwb_file_name, raise_err, error_constants)
     [86](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:86) for pop_key in (key_source & file_restr).fetch("KEY"):
     [87](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:87)     try:
---> [88](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:88)         table().make(pop_key)
     [89](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:89)     except Exception as err:
     [90](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/common/populate_all_common.py:90)         if raise_err:

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:49, in ImportedSpikeSorting.make(self, key)
     [46](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:46)         logger.warn("No units found in NWB file")
     [47](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:47)         return
---> [49](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:49) from spyglass.spikesorting.spikesorting_merge import (
     [50](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:50)     SpikeSortingOutput,
     [51](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:51) )  # noqa: F401
     [53](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:53) key["object_id"] = nwbfile.units.object_id
     [55](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/imported.py:55) self.insert1(key, skip_duplicates=True, allow_direct_insert=True)

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:7
      [4](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:4) from ripple_detection import get_multiunit_population_firing_rate
      [6](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:6) from spyglass.spikesorting.imported import ImportedSpikeSorting  # noqa: F401
----> [7](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:7) from spyglass.spikesorting.v0.spikesorting_curation import (  # noqa: F401
      [8](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:8)     CuratedSpikeSorting,
      [9](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:9) )
     [10](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:10) from spyglass.spikesorting.v1 import ArtifactDetectionSelection  # noqa: F401
     [11](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:11) from spyglass.spikesorting.v1 import (
     [12](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:12)     CurationV1,
     [13](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:13)     MetricCurationSelection,
     [14](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:14)     SpikeSortingRecordingSelection,
     [15](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:15)     SpikeSortingSelection,
     [16](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/spikesorting_merge.py:16) )

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:1
----> [1](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:1) from spyglass.spikesorting.v0.curation_figurl import (  # noqa: F401
      [2](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:2)     CurationFigurl,
      [3](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:3)     CurationFigurlSelection,
      [4](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:4) )
      [5](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:5) from spyglass.spikesorting.v0.sortingview import (  # noqa: F401
      [6](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:6)     SortingviewWorkspace,
      [7](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:7)     SortingviewWorkspaceSelection,
      [8](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:8) )
      [9](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:9) from spyglass.spikesorting.v0.spikesorting_artifact import (  # noqa: F401
     [10](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:10)     ArtifactDetection,
     [11](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:11)     ArtifactDetectionParameters,
     [12](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:12)     ArtifactDetectionSelection,
     [13](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:13)     ArtifactRemovedIntervalList,
     [14](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/__init__.py:14) )

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:10
      [7](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:7) from sortingview.SpikeSortingView import SpikeSortingView
      [9](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:9) from spyglass.spikesorting.utils import _reformat_metrics
---> [10](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:10) from spyglass.spikesorting.v0.spikesorting_curation import Curation
     [11](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:11) from spyglass.spikesorting.v0.spikesorting_recording import (
     [12](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:12)     SpikeSortingRecording,
     [13](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:13) )
     [14](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/curation_figurl.py:14) from spyglass.spikesorting.v0.spikesorting_sorting import SpikeSorting

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:30
     [26](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:26) from spyglass.settings import waveforms_dir
     [27](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:27) from spyglass.spikesorting.v0.merged_sorting_extractor import (
     [28](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:28)     MergedSortingExtractor,
     [29](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:29) )
---> [30](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:30) from spyglass.spikesorting.v0.spikesorting_recording import (
     [31](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:31)     SortInterval,
     [32](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:32)     SpikeSortingRecording,
     [33](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:33) )
     [34](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:34) from spyglass.utils import SpyglassMixin, logger
     [36](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_curation.py:36) from .spikesorting_recording import SortGroup

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:35
     [29](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:29) from spyglass.utils.dj_helper_fn import dj_replace
     [31](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:31) schema = dj.schema("spikesorting_recording")
     [34](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:34) @schema
---> [35](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:35) class SortGroup(SpyglassMixin, dj.Manual):
     [36](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:36)     definition = """
     [37](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:37)     # Set of electrodes that will be sorted together
     [38](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:38)     -> Session
   (...)
     [41](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:41)     sort_reference_electrode_id = -1: int  # the electrode to use for reference. -1: no reference, -2: common median
     [42](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:42)     """
     [44](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/spyglass/spikesorting/v0/spikesorting_recording.py:44)     class SortGroupElectrode(SpyglassMixin, dj.Part):

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:176, in Schema.__call__(self, cls, context)
    [172](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:172)     raise DataJointError(
    [173](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:173)         "The schema decorator should not be applied to Part tables."
    [174](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:174)     )
    [175](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:175) if self.is_activated():
--> [176](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:176)     self._decorate_master(cls, context)
    [177](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:177) else:
    [178](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:178)     self.declare_list.append((cls, context))

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:187, in Schema._decorate_master(self, cls, context)
    [181](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:181) def _decorate_master(self, cls, context):
    [182](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:182)     """
    [183](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:183) 
    [184](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:184)     :param cls: the master class to process
    [185](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:185)     :param context: the class' declaration context
    [186](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:186)     """
--> [187](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:187)     self._decorate_table(
    [188](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:188)         cls, context=dict(context, self=cls, **{cls.__name__: cls})
    [189](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:189)     )
    [190](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:190)     # Process part tables
    [191](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:191)     for part in ordered_dir(cls):

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:225, in Schema._decorate_table(self, table_class, context, assert_declared)
    [223](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:223) is_declared = instance.is_declared
    [224](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:224) if not is_declared and not assert_declared and self.create_tables:
--> [225](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:225)     instance.declare(context)
    [226](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:226)     self.connection.dependencies.clear()
    [227](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/schemas.py:227) is_declared = is_declared or instance.is_declared

File /opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:96, in Table.declare(self, context)
     [89](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:89) """
     [90](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:90) Declare the table in the schema based on self.definition.
     [91](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:91) 
     [92](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:92) :param context: the context for foreign key resolution. If None, foreign keys are
     [93](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:93)     not allowed.
     [94](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:94) """
     [95](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:95) if self.connection.in_transaction:
---> [96](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:96)     raise DataJointError(
     [97](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:97)         "Cannot declare new tables inside a transaction, "
     [98](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:98)         "e.g. from inside a populate/make call"
     [99](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:99)     )
    [100](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:100) # Enforce strict CamelCase #1150
    [101](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:101) if not is_camel_case(self.class_name):

DataJointError: Cannot declare new tables inside a transaction, e.g. from inside a populate/make call

Expected behavior
Successful import (or at least an informative error if I'm doing something wrong here).

Additional context
relevant package versions:

datajoint==0.14.3
pynwb==2.8.2
h5py==3.12.1
hdmf==3.14.4
@samuelbray32
Copy link
Collaborator

samuelbray32 commented Jan 2, 2025 •
edited
Loading

Immediate Solution
The error is being raised because the table hasn't been declared in the fresh database before the poopulate call is being executed. As an immediate fix for you, I think running

from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput

before the insert_sessions call should declare the tables in the database and resolve this.

Spyglass Fix

  • This stems from the fact that SpikeSortingOutput is imported inside the ImportedSpikeSorting.make() function rather than the top of the file
    • This was done to prevent a circular import
    • But, it prevents SpikesortingOutput (and it's upstream tables in the spikesorting pipelines) from being imported and declared automatically in a fresh database like this
  • Solution: Import the merge table inside the insert_sessions function before starting the transaction
  • Note: Should think about other merge tables where this could potentially happen too

@samuelbray32 samuelbray32 added bug Something isn't working merge To do with merge tables NWB ingestion Problems with loading nwb files into spyglass labels Jan 2, 2025
@pauladkisson
Copy link
Author

Immediate Solution The error is being raised because the table hasn't been declared in the fresh database before the poopulate call is being executed. As an immediate fix for you, I think running

from spyglass.spikesorting.spikesorting_merge import SpikeSortingOutput

before the insert_sessions call should declare the tables in the database and resolve this.

Thanks! That did the trick.

@samuelbray32
Copy link
Collaborator

Related to #872

@samuelbray32 samuelbray32 linked a pull request Jan 3, 2025 that will close this issue
Open
7 tasks
@pauladkisson pauladkisson mentioned this issue Jan 9, 2025
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working merge To do with merge tables NWB ingestion Problems with loading nwb files into spyglass
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure merge tables declared in new database LorenFrankLab/spyglass
2 participants
@samuelbray32 @pauladkisson