Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split large rec files between multiple SpikegadgetsRawIO iterators #50

Merged
merged 18 commits into from
Nov 1, 2023

Conversation

samuelbray32
Copy link
Collaborator

Excessively large iterators can cause memory issues due to lading large chunks of data into memory during get_analogsignal_multiplexed or get_digitalsignal.

This merge splits the memmap of these large rec files into multiple SpikegadgetsRawIO iterators. The ordering of these is tracked by the RecDataChunkIterator so almost any use case doesn't need to worry about this.

If trying too get info for a specific epoch, note that this means there is not a 1:1 relation between iterators and epochs. Use the file_info pandas dataframe to determine which epoch each belongs to (see add_epochs for example)

@samuelbray32 samuelbray32 requested a review from rly October 2, 2023 22:26
@codecov
Copy link

codecov bot commented Oct 2, 2023

Codecov Report

Attention: 100 lines in your changes are missing coverage. Please review.

Comparison is base (cf7561e) 94.39% compared to head (d859f90) 90.10%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #50      +/-   ##
==========================================
- Coverage   94.39%   90.10%   -4.29%     
==========================================
  Files          25       25              
  Lines        2157     2275     +118     
==========================================
+ Hits         2036     2050      +14     
- Misses        121      225     +104     
Files Coverage Δ
src/spikegadgets_to_nwb/convert_dios.py 97.43% <100.00%> (+0.21%) ⬆️
...pikegadgets_to_nwb/tests/test_convert_intervals.py 100.00% <100.00%> (ø)
src/spikegadgets_to_nwb/convert_ephys.py 84.84% <40.00%> (-11.41%) ⬇️
src/spikegadgets_to_nwb/spike_gadgets_raw_io.py 72.32% <9.27%> (-19.57%) ⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@edeno edeno self-requested a review October 30, 2023 18:52
src/spikegadgets_to_nwb/convert_ephys.py Outdated Show resolved Hide resolved
src/spikegadgets_to_nwb/spike_gadgets_raw_io.py Outdated Show resolved Hide resolved
src/spikegadgets_to_nwb/convert_ephys.py Show resolved Hide resolved
src/spikegadgets_to_nwb/spike_gadgets_raw_io.py Outdated Show resolved Hide resolved
@edeno edeno merged commit 0082936 into main Nov 1, 2023
8 of 10 checks passed
@edeno edeno deleted the split_iterator branch November 1, 2023 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants