Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add support for SQLiteStore for ZarrIO #66

Closed
wants to merge 88 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
9d26290
Add support for using select user-defined zarr stores
oruebel Jan 5, 2023
0995543
Update resolution of references to work also for file-based Zarr stores
oruebel Jan 5, 2023
1a94f17
Update test_io_zarr.py to allow file-based Zarr stores
oruebel Jan 5, 2023
88d5dfb
Add SQLLite test draft
oruebel Jan 5, 2023
7076b82
Updated changelog
oruebel Jan 5, 2023
2bb8d78
Merge branch 'dev' into add/alternate_stores
oruebel Jan 6, 2023
d9a3a75
Merge branch 'dev' into add/alternate_stores
oruebel Jan 6, 2023
1dca7df
Add ZarrIO.file property ease implementation of tests
oruebel Jan 6, 2023
022cf46
Refactored ZarrIO tests for consistency and to run all backends via d…
oruebel Jan 7, 2023
0a59d9c
Update NWBZarrIO to support the new path options from ZarrIO
oruebel Jan 7, 2023
9b77f40
Minor changes to tests and comments
oruebel Jan 7, 2023
6a6b8f0
Update test_io_convert.py to test with all supported zarr.storage bac…
oruebel Jan 7, 2023
4c11f8c
Added docs on how to integrate new backends stores with ZarrIO
oruebel Jan 7, 2023
9a28380
Clarify the docs to integrate stores
oruebel Jan 7, 2023
6fc99cb
Update storage docs to add missing reserved links and groups
oruebel Jan 7, 2023
c85052c
Add DEFAULT_SPEC_LOC_DIR and SUPPORTED_ZARR_STORES module variable of…
oruebel Jan 7, 2023
a03bc32
Minor fixes to TempStore tests
oruebel Jan 7, 2023
e3e06a3
Add Mixin and test cases to test convertion between Zarr and Zarr
oruebel Jan 7, 2023
40c23ca
Update ZarrIO tutorial to describe using custom data stores
oruebel Jan 7, 2023
03b16d4
Update Changelog
oruebel Jan 7, 2023
55daea8
Attempt to fix Windows tests
oruebel Jan 7, 2023
fd00185
Add note on why we set dir on TempStore
oruebel Jan 7, 2023
954cf4f
Remove commented code
oruebel Jan 8, 2023
18191d5
Fix bad test setup
oruebel Jan 8, 2023
bc4386f
Set store paths in child classes
oruebel Jan 8, 2023
b18c41e
Added some more details to integrating_data_stores.rst
oruebel Jan 8, 2023
731deeb
Added SQLiteStore support and tests (some link tests still failing)
oruebel Jan 8, 2023
9c231d4
Fix flake8
oruebel Jan 8, 2023
29dd7f6
Update path calculation for links to fix SQLite linking
oruebel Jan 8, 2023
b4af48c
Fix failing test case
oruebel Jan 8, 2023
c4e4c8b
Move resources readme to avoid including it in the docs
oruebel Jan 8, 2023
39bd120
Filter warnings in NWB conversion tutorial
oruebel Jan 8, 2023
be3d156
Updated changelog
oruebel Jan 8, 2023
87f7c54
Update changelog to add PR links
oruebel Jan 8, 2023
8e54adb
Attempt to fix file access conflict in test suite for Windows
oruebel Jan 8, 2023
998409c
Attempt to fix Permission issues on Windows
oruebel Jan 8, 2023
1ea405a
Attempt to fix Permission issues on Windows
oruebel Jan 8, 2023
e64cd06
Avoid explicit use of zarr.open and use file from ZarrIO.file instead
oruebel Jan 8, 2023
9da70dd
Attempt to catch permission issues on Windows during tests
oruebel Jan 8, 2023
07a9b8c
Attempt to catch permission issues on Windows during tests
oruebel Jan 8, 2023
82ba7ca
Add __del__ and __exit__ to ensure stores are closed on exit/delete
oruebel Jan 9, 2023
df0891b
Catch error on multiple close of SQLite store
oruebel Jan 9, 2023
be1fc14
Fix tests accessing closed SQLite store
oruebel Jan 9, 2023
ba8d74a
Make sure io is delete on TestZarrWriter tests
oruebel Jan 9, 2023
7f19f45
Do not catch permission error on tests to ease debugging on windows CI
oruebel Jan 9, 2023
79c9b4d
Close stores opened to resolve references on ZarrIO close
oruebel Jan 9, 2023
b131006
Add docs for tracking opened stores
oruebel Jan 9, 2023
c9d7261
Merge branch 'dev' into add/alternate_stores
oruebel Jan 11, 2023
7595087
Merge branch 'dev' into add/alternate_stores
oruebel Jan 11, 2023
8d61358
Minor text fixes
rly Jan 17, 2023
c96aad6
Minor text fixes
rly Jan 17, 2023
4a316c8
Minor text fixes
rly Jan 17, 2023
dcbae13
Minor text fixes
rly Jan 17, 2023
2f36fd7
Minor text fixes
rly Jan 17, 2023
5dbff7e
Minor text edits
rly Jan 17, 2023
220ad36
Increase HDMF version to 3.5
oruebel Jan 17, 2023
86cfef2
Removed filepath param from get_builder_exists_on_disk
oruebel Jan 17, 2023
b94bee8
Merge branch 'add/alternate_stores' into add/sqlstore
oruebel Jan 17, 2023
c14ca94
Fix merge error in docs
oruebel Jan 17, 2023
d629f9d
Update integrate new store docs
oruebel Jan 17, 2023
6298a67
Merge branch 'add/alternate_stores' into add/sqlstore
oruebel Jan 17, 2023
92ee3e0
Fix bad documentation of class members of mixins
oruebel Jan 17, 2023
10c983f
Remove references to SQLite store
oruebel Jan 18, 2023
077875d
Add missing message to assert in MixinTestCaseConvert
oruebel Jan 18, 2023
f373727
Consistenlty close file in test when explicitly opened
oruebel Jan 18, 2023
2bce016
Simplify test to reuse IO object
oruebel Jan 18, 2023
5e8fb0a
Fix bugs from bad merge
oruebel Jan 18, 2023
ec3adf2
Fix bugs from bad merge
oruebel Jan 18, 2023
c518569
Fix error from bad merge in CHANGELOG
oruebel Jan 18, 2023
94c7aef
Merge branch 'dev' into add/sqlstore
oruebel Jan 18, 2023
c9faccc
Fix flake8 error due to merge conflict resolve
oruebel Jan 18, 2023
5414adf
Remove unused warning filter
oruebel Jan 18, 2023
d10e03f
Remove unused warning filter
oruebel Jan 18, 2023
6eef41a
Changed SUPPORTED_ZARR_STORES back to tuple
oruebel Jan 18, 2023
d63d439
Updated changelog
oruebel Jan 18, 2023
ebdca10
Fix flake8 for tests
oruebel Jan 18, 2023
63c2035
Merge branch 'dev' into add/sqlstore
oruebel Jan 18, 2023
ad56654
Close EXPORT_PATHS and WRITE_PATHS stores to try and fix Windows tests
oruebel Jan 18, 2023
aac135b
Fix flake8
oruebel Jan 18, 2023
f0d86d0
Attempt to close stores to fix Windows tests
oruebel Jan 18, 2023
d36e738
Do full tearDown and setUp in iteration of conversion tests
oruebel Jan 18, 2023
47d0a90
Remove unnecessary test_stimple test case
oruebel Jan 18, 2023
cf3c571
Added note on closing stores to load_namespaces
oruebel Jan 18, 2023
595c9cb
Attempt to close store in test_cache_spec
oruebel Jan 18, 2023
64525fe
Merge branch 'dev' into add/sqlstore
oruebel Jan 18, 2023
fb601ba
Merge branch 'dev' into add/sqlstore
oruebel Aug 30, 2023
2ad1394
Merge branch 'dev' into add/sqlstore
oruebel Oct 1, 2023
4a29f9a
Update CHANGELOG.md
oruebel Oct 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Attempt to fix Permission issues on Windows
oruebel committed Jan 8, 2023
commit 998409c8110eb54ceb68b6435743c4d41d1757a3
2 changes: 2 additions & 0 deletions tests/unit/base_tests_zarrio.py
Original file line number Diff line number Diff line change
@@ -848,6 +848,7 @@ def test_link_dataset_zarrdataio_input(self):
'source': os.path.abspath(self.store_path)}
self.assertEqual(len(tempf.attrs['zarr_link']), 1)
self.assertDictEqual(tempf.attrs['zarr_link'][0], expected_link)
tempf.store.close()

def test_copy_dataset_zarrdataio_input(self):
tempIO = ZarrIO(self.store, mode='w')
@@ -1013,6 +1014,7 @@ def test_cache_spec_enabled(self):

with zarr.open(reopen_store(self.store[1]), mode='r') as zarr_io:
self.assertTrue('specifications' in zarr_io.keys())
zarr_io.store.close()

def test_soft_link_group(self):
"""