Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add data read #85

Draft
wants to merge 69 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
1e1f041
Define base classes for reading
oruebel Aug 31, 2024
e58f50f
First draft of reading datasets and attributes
oruebel Aug 31, 2024
250a66b
Split reading of attribute and dataset to separate functions
oruebel Aug 31, 2024
e1bfa2e
Remove debug print
oruebel Aug 31, 2024
4aedefd
Add functions to construct ReadDatasetWrapper and ReadAttributeWrappe…
oruebel Aug 31, 2024
4dae171
Fix formatting
oruebel Aug 31, 2024
140809e
Add test for using the ReadDatasetWrapper
oruebel Aug 31, 2024
4d7b1ea
Start refactor containers for read
oruebel Aug 31, 2024
2db4bfe
Fix format
oruebel Aug 31, 2024
b2959d2
Read ElectricalSeries.data example working
oruebel Sep 1, 2024
37d30d3
Revert change to tests used for debugging
oruebel Sep 1, 2024
6cfe416
Fix codespell issue
oruebel Sep 1, 2024
3d86aff
Move read example to an example file
oruebel Sep 1, 2024
3fc1d7a
Fix bug in Container::create
oruebel Sep 1, 2024
b09a4fa
Add example for data read
oruebel Sep 1, 2024
970f4c8
Fix spelling error
oruebel Sep 1, 2024
26fbae9
Add user docs for data read
oruebel Sep 1, 2024
4299f99
Update read user docs
oruebel Sep 1, 2024
9c9301c
Add toc
oruebel Sep 2, 2024
447337a
Add read software design figure and more details on the design
oruebel Sep 2, 2024
e3d4330
Some adjustment to the edges in the fig
oruebel Sep 2, 2024
77f0861
Some adjustment to the edges in the fig
oruebel Sep 2, 2024
5190731
Add intro for data read page
oruebel Sep 2, 2024
f08e42f
Make Container::create inline
oruebel Sep 3, 2024
1a754b9
Make DataBlock::fromGeneric inline
oruebel Sep 3, 2024
ebfbdf1
Implement function to allow us to get the storage object type for a g…
oruebel Sep 3, 2024
3022861
Rename getObjectType to getStroageObjectType
oruebel Sep 3, 2024
6ba32a5
Fix reading of attributes and add TimeSeries.resolution read
oruebel Sep 3, 2024
1fb5d95
Add example for reading an attribute
oruebel Sep 3, 2024
676b978
Add ReadDataWrapperBase as common base class for ReadDatasetWrapper a…
oruebel Sep 3, 2024
14c8362
Fix docstring
oruebel Sep 3, 2024
d2b1c23
Minor fixed to docs
oruebel Sep 3, 2024
78825f2
Mention ReadDataWrapperBase in the read dox page
oruebel Sep 3, 2024
85db320
Merge branch 'main' into add_read
oruebel Sep 3, 2024
5c2faae
Fix unit tests after merge
oruebel Sep 3, 2024
c481260
Remove NWBFile.identifierText property
oruebel Sep 3, 2024
b418149
Allow setting of value type template parameter for ReadDataWrapperBas…
oruebel Sep 4, 2024
1f3ad24
Save the std::type_index as part of the DataBlock to support runtime …
oruebel Sep 4, 2024
99b5d4e
Fix #88 move io classes to their own module
oruebel Sep 5, 2024
04117e9
Refactor to add IO namespace to match folder structure
oruebel Sep 5, 2024
82a4c08
Move HDF5RecordingData to its own hpp/cpp files
oruebel Sep 5, 2024
6554e29
Moved read I/O classes to new ReadIO.hpp
oruebel Sep 5, 2024
bff7121
Refactor the read data wrappers to use a single ReadDataWrapper class…
oruebel Sep 5, 2024
53365ff
Replace redundant ReadObjectType with StorageObjectType
oruebel Sep 6, 2024
c66b3bc
Add traits for ReadObjectWrapper to prevent instantiation for Group a…
oruebel Sep 6, 2024
0da4800
Fix formatting
oruebel Sep 6, 2024
d161c47
Fix spelling error
oruebel Sep 6, 2024
58e5afb
Inline functions in ReadIO.hpp
oruebel Sep 6, 2024
b7cc149
Some minor fixes
oruebel Sep 6, 2024
b9c6c9b
Merge branch 'main' into add_read
oruebel Sep 7, 2024
69c3c4e
Update ElectrodGroup to provide standard Container constructor and mo…
oruebel Sep 8, 2024
e75c7ad
Fix formatting
oruebel Sep 8, 2024
5ab8464
Merge branch 'main' into add_read
oruebel Sep 9, 2024
803dfcf
Fix ElectricalSeries
oruebel Sep 9, 2024
cdffc74
Run test workflows on all PRs
oruebel Sep 19, 2024
89f6d29
Fix static assert for ReadDataWrapper
oruebel Sep 19, 2024
5608c5a
Sync add_read with main branch (#101)
oruebel Sep 19, 2024
c7ced4e
Merge branch 'main' into add_read
oruebel Sep 19, 2024
dd01ad3
Merge branch 'main' into add_read
oruebel Sep 20, 2024
968fcf0
Fix formatting and docs error after merge with main
oruebel Sep 20, 2024
bfaabb8
Update docs/pages/userdocs/read.dox
oruebel Sep 24, 2024
a439212
Update src/io/ReadIO.hpp
oruebel Sep 24, 2024
5f28fb0
Update src/io/ReadIO.hpp
oruebel Sep 24, 2024
a9725b2
Rename ReadDataWrapper.is_dataset
oruebel Sep 24, 2024
e70949c
Update src/io/hdf5/HDF5IO.hpp
oruebel Sep 24, 2024
d1da5d1
Remove duplicate code in NWBFile
oruebel Sep 24, 2024
12a86a5
Remove outdated ToDo item
oruebel Sep 24, 2024
ae66f33
Merge branch 'main' into add_read
oruebel Oct 22, 2024
69df3e0
Fix build error after merge
oruebel Oct 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/codespell.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
- main

pull_request:
branches:
- main
#branches:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uncomment workflow triggers before merging

Copy link
Contributor Author

@oruebel oruebel Sep 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I had modified this is because it would'nt trigger workflows if they didn't target the main branch. I think it would be useful to run the tests on all PRs, even if they target branches other than main

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, that makes sense. We can remove the branch section entirely then or yes just leave it commented out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either way is fine. I just commented them out here, because I wanted to see what tests are working/failing on this PR. We can change workflows also in a separate PR and remove this change here before merging if you prefer.

#- main
workflow_dispatch:

jobs:
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/doxygen-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@ name: Doxygen build test

on:
pull_request:
branches:
- main
#branches:
#- main

jobs:
test:
Expand All @@ -17,7 +17,7 @@ jobs:
uses: actions/checkout@v4
with:
submodules: "true"

- name: Install dependencies - macos
run: brew install hdf5 boost catch2 doxygen graphviz

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
- main

pull_request:
branches:
- main
#branches:
#- main
workflow_dispatch:

jobs:
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ on:
- main

pull_request:
branches:
- main
#branches:
#- main
workflow_dispatch:

jobs:
Expand All @@ -22,7 +22,7 @@ jobs:
fail-fast: false
matrix:
os: [macos-latest, ubuntu-latest]

runs-on: ${{ matrix.os }}

steps:
Expand All @@ -39,7 +39,7 @@ jobs:
git checkout "v3.5.3"
cmake -Bbuild -H. -DBUILD_TESTING=OFF
sudo cmake --build build/ --target install

- name: Install dependencies - macos
if: matrix.os == 'macos-latest'
run: brew install hdf5 boost catch2
Expand Down Expand Up @@ -67,7 +67,7 @@ jobs:

sanitize:
needs: test

runs-on: ubuntu-latest

steps:
Expand Down
5 changes: 3 additions & 2 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,10 @@ include(cmake/variables.cmake)

add_library(
aqnwb_aqnwb
src/BaseIO.cpp
src/io/BaseIO.cpp
src/Channel.cpp
src/hdf5/HDF5IO.cpp
src/io/hdf5/HDF5IO.cpp
src/io/hdf5/HDF5RecordingData.cpp
src/nwb/NWBFile.cpp
src/nwb/RecordingContainers.cpp
src/nwb/base/TimeSeries.cpp
Expand Down
4 changes: 4 additions & 0 deletions docs/Doxyfile.in
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ EXTRACT_ALL = YES
RECURSIVE = YES
OUTPUT_DIRECTORY = "@DOXYGEN_OUTPUT_DIRECTORY@"

# Also show private members in the docs,
EXTRACT_PRIVATE = YES
# HIDE_UNDOC_MEMBERS = YES

# Enable Markdown support
MARKDOWN_SUPPORT = YES

Expand Down
1 change: 1 addition & 0 deletions docs/pages/1_userdocs.dox
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@
* - \subpage user_install_page
* - \subpage workflow
* - \subpage hdf5io
* - \subpage read_page
*/
2 changes: 1 addition & 1 deletion docs/pages/devdocs/documentation.dox
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@
*
* To display a code snippet from our example in the documentation we can use the
* ``\snippet <file> <label>`` Doxygen command in our ``*.dox`` documentation file.
* For example to just show the line where we create the \ref AQNWB::HDF5::HDF5IO "HDF5IO
* For example to just show the line where we create the \ref AQNWB::IO::HDF5::HDF5IO "HDF5IO"
* object we use:
*
* \code{.sh}
Expand Down
10 changes: 5 additions & 5 deletions docs/pages/userdocs/hdf5io.dox
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@
*
* \section hdf5io_swmr Single-Writer Multiple-Reader (SWMR) Mode
*
* The \ref AQNWB::HDF5::HDF5IO "HDF5IO" I/O backend uses by default SWMR mode while recording data.
* The \ref AQNWB::IO::HDF5::HDF5IO "HDF5IO" I/O backend uses by default SWMR mode while recording data.
* Using SWMR, one process can write to the HDF5 file and multiple other processes can read
* from the file concurrently while ensuring that the readers see a consistent view of the data.
*
Expand All @@ -106,7 +106,7 @@
* node [shape=box, style=filled];
*
* // Define the writer node with light blue fill color
* writer [label="AqNWB HDF5IO\n(SWMR Write)", fillcolor=lightblue, URL="\ref AQNWB::HDF5::HDF5IO"];
* writer [label="AqNWB HDF5IO\n(SWMR Write)", fillcolor=lightblue, URL="\ref AQNWB::IO::HDF5::HDF5IO"];
* // Define the HDF5 file node with a distinct shape
* hdf5_file [label="\n.nwb\n(HDF5 File)\n ", shape=folder, fillcolor=lightgray, URL="\ref AQNWB::NWB::NWBFile"];
* // Define reader nodes with light green fill color
Expand Down Expand Up @@ -148,11 +148,11 @@
* \note
* While SWMR mode ensures data integrity, some data loss may still occur if the application crashes.
* Only data that has been completely written and flushed to disk will be readable. To manually
* flush data to disk use \ref AQNWB::HDF5::HDF5IO::flush "HDF5IO::flush".
* flush data to disk use \ref AQNWB::IO::HDF5::HDF5IO::flush "HDF5IO::flush".
*
* \subsection hdf5io_swmr_workflow Writing an NWB file with SWMR mode
*
* SWMR mode is enabled when calling \ref AQNWB::HDF5::HDF5IO::startRecording "HDF5IO::startRecording".
* SWMR mode is enabled when calling \ref AQNWB::IO::HDF5::HDF5IO::startRecording "HDF5IO::startRecording".
* Once SWMR mode is enabled, no new data objects (Datasets, Groups, Attributes etc.)
* can be created, but we can only add and set values to existing data objects. Since other
* processes may read from the HDF5 file, it is not possible to intermittently disable
Expand All @@ -168,7 +168,7 @@
* This workflow is applicable to a wide range of data acquisition use-cases. However,
* for use cases that require creation of new Groups and Datasets during acquisition,
* you can disable the use of SWMR mode by setting `disableSWMRMode=true` when
* constructing the \ref AQNWB::HDF5::HDF5IO object.
* constructing the \ref AQNWB::IO::HDF5::HDF5IO object.
*
* \warning
* While disabling SWMR mode allows Groups and Datasets to be created during and after
Expand Down
Loading
Loading