Skip to content

Commit

Permalink
Rework ZarrIO backend/Remove python3.7/Update HDMF and PyNWB min/Upda…
Browse files Browse the repository at this point in the history
…te workflows (#120)

* Both table and dataset of references work

* checkpoint

* all tests pass

* tests pass

* ignore warning

* clean up

* versions

* 3.8 min

* 3.8 min

* remove 3.7

* remove 3.7

* remove 3.7

* remove 3.7

* remove

* chckpoint

* Update .readthedocs.yaml

* Update .readthedocs.yaml

* doc build test

* sanity check

* sanity check doc build

* Update .readthedocs.yaml

* doc build

* build doc

* clean up

* docs

* Update src/hdmf_zarr/backend.py

Co-authored-by: Oliver Ruebel <[email protected]>

* Update src/hdmf_zarr/backend.py

Co-authored-by: Oliver Ruebel <[email protected]>

* Update src/hdmf_zarr/backend.py

Co-authored-by: Oliver Ruebel <[email protected]>

* Update src/hdmf_zarr/backend.py

Co-authored-by: Oliver Ruebel <[email protected]>

* Update src/hdmf_zarr/zarr_utils.py

Co-authored-by: Oliver Ruebel <[email protected]>

* class docstrings

* flake8/slight change to TableRef

* first round clean up for review

* Update CHANGELOG.md

* clean up

* Update requirements.txt

* Update setup.py

* clean up spacing

* source

* default

* Update src/hdmf_zarr/backend.py

Co-authored-by: Oliver Ruebel <[email protected]>

* Update CHANGELOG.md

Co-authored-by: Oliver Ruebel <[email protected]>

* export_source

* Update src/hdmf_zarr/backend.py

Co-authored-by: Oliver Ruebel <[email protected]>

* clean up/adjust resolve_ref

* Update storage.rst

* note

* Update run_all_tests.yml

Co-authored-by: Ryan Ly <[email protected]>

* Update backend.py

Co-authored-by: Ryan Ly <[email protected]>

* test format

* Update CHANGELOG.md

Co-authored-by: Oliver Ruebel <[email protected]>

* Update src/hdmf_zarr/backend.py

Co-authored-by: Oliver Ruebel <[email protected]>

* feedback fix

* tests

* Update tests/unit/utils.py

Co-authored-by: Oliver Ruebel <[email protected]>

* flake

---------

Co-authored-by: Oliver Ruebel <[email protected]>
Co-authored-by: Ryan Ly <[email protected]>
  • Loading branch information
3 people authored Sep 29, 2023
1 parent 6c13e14 commit c262481
Show file tree
Hide file tree
Showing 17 changed files with 719 additions and 184 deletions.
1 change: 0 additions & 1 deletion .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ body:
attributes:
label: Python Version
options:
- "3.7"
- "3.8"
- "3.9"
- "3.10"
Expand Down
21 changes: 8 additions & 13 deletions .github/workflows/run_all_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,24 +22,21 @@ jobs:
fail-fast: false
matrix:
include:
- { name: linux-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: ubuntu-latest }
- { name: linux-python3.8 , test-tox-env: py38 , build-tox-env: build-py38 , python-ver: "3.8" , os: ubuntu-latest }
- { name: linux-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: ubuntu-latest }
- { name: linux-python3.9 , test-tox-env: py39 , build-tox-env: build-py39 , python-ver: "3.9" , os: ubuntu-latest }
- { name: linux-python3.10 , test-tox-env: py310 , build-tox-env: build-py310 , python-ver: "3.10", os: ubuntu-latest }
- { name: linux-python3.11 , test-tox-env: py311 , build-tox-env: build-py311 , python-ver: "3.11", os: ubuntu-latest }
- { name: linux-python3.11-optional , test-tox-env: py311-optional , build-tox-env: build-py311-optional , python-ver: "3.11", os: ubuntu-latest }
- { name: linux-python3.11-upgraded , test-tox-env: py311-upgraded , build-tox-env: build-py311-upgraded , python-ver: "3.11", os: ubuntu-latest }
- { name: linux-python3.11-prerelease , test-tox-env: py311-prerelease, build-tox-env: build-py311-prerelease, python-ver: "3.11", os: ubuntu-latest }
- { name: windows-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: windows-latest }
- { name: windows-python3.8 , test-tox-env: py38 , build-tox-env: build-py38 , python-ver: "3.8" , os: windows-latest }
- { name: windows-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: windows-latest }
- { name: windows-python3.9 , test-tox-env: py39 , build-tox-env: build-py39 , python-ver: "3.9" , os: windows-latest }
- { name: windows-python3.10 , test-tox-env: py310 , build-tox-env: build-py310 , python-ver: "3.10", os: windows-latest }
- { name: windows-python3.11 , test-tox-env: py311 , build-tox-env: build-py311 , python-ver: "3.11", os: windows-latest }
- { name: windows-python3.11-optional , test-tox-env: py311-optional , build-tox-env: build-py311-optional , python-ver: "3.11", os: windows-latest }
- { name: windows-python3.11-upgraded , test-tox-env: py311-upgraded , build-tox-env: build-py311-upgraded , python-ver: "3.11", os: windows-latest }
- { name: windows-python3.11-prerelease, test-tox-env: py311-prerelease, build-tox-env: build-py311-prerelease, python-ver: "3.11", os: windows-latest }
- { name: macos-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: macos-latest }
- { name: macos-python3.8 , test-tox-env: py38 , build-tox-env: build-py38 , python-ver: "3.8" , os: macos-latest }
- { name: macos-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: macos-latest }
- { name: macos-python3.9 , test-tox-env: py39 , build-tox-env: build-py39 , python-ver: "3.9" , os: macos-latest }
- { name: macos-python3.10 , test-tox-env: py310 , build-tox-env: build-py310 , python-ver: "3.10", os: macos-latest }
- { name: macos-python3.11 , test-tox-env: py311 , build-tox-env: build-py311 , python-ver: "3.11", os: macos-latest }
Expand Down Expand Up @@ -88,13 +85,13 @@ jobs:
fail-fast: false
matrix:
include:
- { name: linux-gallery-python3.7-minimum , test-tox-env: gallery-py37-minimum , python-ver: "3.7" , os: ubuntu-latest }
- { name: linux-gallery-python3.8-minimum , test-tox-env: gallery-py38-minimum , python-ver: "3.8" , os: ubuntu-latest }
- { name: linux-gallery-python3.11-upgraded , test-tox-env: gallery-py311-upgraded , python-ver: "3.11", os: ubuntu-latest }
- { name: linux-gallery-python3.11-prerelease , test-tox-env: gallery-py311-prerelease, python-ver: "3.11", os: ubuntu-latest }
- { name: windows-gallery-python3.7-minimum , test-tox-env: gallery-py37-minimum , python-ver: "3.7" , os: windows-latest }
- { name: windows-gallery-python3.8-minimum , test-tox-env: gallery-py38-minimum , python-ver: "3.8" , os: windows-latest }
- { name: windows-gallery-python3.11-upgraded , test-tox-env: gallery-py311-upgraded , python-ver: "3.11", os: windows-latest }
- { name: windows-gallery-python3.11-prerelease, test-tox-env: gallery-py311-prerelease, python-ver: "3.11", os: windows-latest }
- { name: macos-gallery-python3.7-minimum , test-tox-env: gallery-py37-minimum , python-ver: "3.7" , os: macos-latest }
- { name: macos-gallery-python3.8-minimum , test-tox-env: gallery-py38-minimum , python-ver: "3.8" , os: macos-latest }
- { name: macos-gallery-python3.11-upgraded , test-tox-env: gallery-py311-upgraded , python-ver: "3.11", os: macos-latest }
- { name: macos-gallery-python3.11-prerelease , test-tox-env: gallery-py311-prerelease, python-ver: "3.11", os: macos-latest }
steps:
Expand Down Expand Up @@ -132,8 +129,7 @@ jobs:
fail-fast: false
matrix:
include:
- { name: conda-linux-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: ubuntu-latest }
- { name: conda-linux-python3.8 , test-tox-env: py38 , build-tox-env: build-py38 , python-ver: "3.8" , os: ubuntu-latest }
- { name: conda-linux-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: ubuntu-latest }
- { name: conda-linux-python3.9 , test-tox-env: py39 , build-tox-env: build-py39 , python-ver: "3.9" , os: ubuntu-latest }
- { name: conda-linux-python3.10 , test-tox-env: py310 , build-tox-env: build-py310 , python-ver: "3.10", os: ubuntu-latest }
- { name: conda-linux-python3.11 , test-tox-env: py311 , build-tox-env: build-py311 , python-ver: "3.11", os: ubuntu-latest }
Expand Down Expand Up @@ -162,8 +158,7 @@ jobs:
run: |
conda config --set always_yes yes --set changeps1 no
conda info
# the conda dependency resolution for tox under python 3.7 can install the wrong importlib_metadata
conda install -c conda-forge tox "importlib_metadata>4"
conda install -c conda-forge tox
- name: Conda reporting
run: |
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/run_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ jobs:
fail-fast: false
matrix:
include:
- { name: linux-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: ubuntu-latest }
- { name: linux-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: ubuntu-latest }
- { name: linux-python3.11 , test-tox-env: py311 , build-tox-env: build-py311 , python-ver: "3.11", os: ubuntu-latest }
# NOTE config below with "upload-wheels: true" specifies that wheels should be uploaded as an artifact
- { name: linux-python3.11-upgraded , test-tox-env: py311-upgraded , build-tox-env: build-py311-upgraded , python-ver: "3.11", os: ubuntu-latest , upload-wheels: true }
- { name: windows-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: windows-latest }
- { name: windows-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: windows-latest }
- { name: windows-python3.11-upgraded , test-tox-env: py311-upgraded , build-tox-env: build-py311-upgraded , python-ver: "3.11", os: windows-latest }
- { name: macos-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: macos-latest }
- { name: macos-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: macos-latest }
- { name: macos-python3.11-upgraded , test-tox-env: py311-upgraded , build-tox-env: build-py311-upgraded , python-ver: "3.11", os: macos-latest }
steps:
- name: Cancel non-latest runs
Expand Down Expand Up @@ -76,9 +76,9 @@ jobs:
fail-fast: false
matrix:
include:
- { name: linux-gallery-python3.7-minimum , test-tox-env: gallery-py37-minimum , python-ver: "3.7" , os: ubuntu-latest }
- { name: linux-gallery-python3.8-minimum , test-tox-env: gallery-py38-minimum , python-ver: "3.8" , os: ubuntu-latest }
- { name: linux-gallery-python3.11-upgraded , test-tox-env: gallery-py311-upgraded, python-ver: "3.11", os: ubuntu-latest }
- { name: windows-gallery-python3.7-minimum , test-tox-env: gallery-py37-minimum , python-ver: "3.7" , os: windows-latest }
- { name: windows-gallery-python3.8-minimum , test-tox-env: gallery-py38-minimum , python-ver: "3.8" , os: windows-latest }
- { name: windows-gallery-python3.11-upgraded, test-tox-env: gallery-py311-upgraded, python-ver: "3.11", os: windows-latest }
steps:
- name: Cancel non-latest runs
Expand Down Expand Up @@ -114,7 +114,7 @@ jobs:
fail-fast: false
matrix:
include:
- { name: conda-linux-python3.7-minimum , test-tox-env: py37-minimum , build-tox-env: build-py37-minimum , python-ver: "3.7" , os: ubuntu-latest }
- { name: conda-linux-python3.8-minimum , test-tox-env: py38-minimum , build-tox-env: build-py38-minimum , python-ver: "3.8" , os: ubuntu-latest }
- { name: conda-linux-python3.11-upgraded , test-tox-env: py311-upgraded , build-tox-env: build-py311-upgraded , python-ver: "3.11", os: ubuntu-latest }
steps:
- name: Cancel non-latest runs
Expand Down
6 changes: 6 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@
# Required
version: 2

build:
os: ubuntu-20.04
tools:
python: '3.9'

# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/source/conf.py
Expand All @@ -21,6 +26,7 @@ python:
install:
- requirements: requirements-doc.txt
- requirements: requirements.txt
- path: . # path to the package relative to the root

# Optionally include all submodules
submodules:
Expand Down
9 changes: 8 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
# HDMF-ZARR Changelog

## 0.3.1 (Upcoming)
## 0.4.0 (Upcoming)

### Enhancements
* Enhanced ZarrIO to resolve object references lazily on read similar to HDMF's `HDF5IO` backend @mavaylon1 [#120](https://github.com/hdmf-dev/hdmf-zarr/pull/120)

### Dependencies
* Updated HDMF and PyNWB version to the most recent release @mavaylon1 [#120](https://github.com/hdmf-dev/hdmf-zarr/pull/120)
* Updated minimum Python version from 3.7 to 3.8 @mavaylon1 [#120](https://github.com/hdmf-dev/hdmf-zarr/pull/120)

### Bug fixes
* Fixed error in deploy workflow. @mavaylon1 [#109](https://github.com/hdmf-dev/hdmf-zarr/pull/109)
Expand Down
22 changes: 11 additions & 11 deletions docs/gallery/plot_convert_nwb_hdf5.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
:py:class:`~ hdmf.backends.hdf5.h5tools.HDF5IO` HDF5 backend from HDMF for storage.
"""


###############################################################################
# Setup
# -----
Expand All @@ -31,11 +30,14 @@
# asset.download(filename)
#
# We here use a local copy of a small file from this DANDIset as an example:
#


# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnail_plot_convert_nwb.png'
import os
import shutil
from pynwb import NWBHDF5IO
from hdmf_zarr.nwb import NWBZarrIO
from contextlib import suppress

# Input file to convert
basedir = "resources"
Expand All @@ -62,9 +64,6 @@
# As this is an NWB file, we here use the :py:class:`pynwb.NWBHDF5IO` backend for reading the file from
# from HDF5 and use the :py:class:`~hdmf_zarr.nwb.NWBZarrIO` backend to export the file to Zarr.

from pynwb import NWBHDF5IO
from hdmf_zarr.nwb import NWBZarrIO

with NWBHDF5IO(filename, 'r', load_namespaces=False) as read_io: # Create HDF5 IO object for read
with NWBZarrIO(zarr_filename, mode='w') as export_io: # Create Zarr IO object for write
export_io.export(src_io=read_io, write_args=dict(link_data=False)) # Export from HDF5 to Zarr
Expand All @@ -77,7 +76,6 @@
#
# Read the Zarr file back in
# --------------------------
#

zr = NWBZarrIO(zarr_filename, 'r')
zf = zr.read()
Expand Down Expand Up @@ -107,9 +105,10 @@
#
# Using the same approach as above, we can now convert our Zarr file back to HDF5.

with NWBZarrIO(zarr_filename, mode='r') as read_io: # Create Zarr IO object for read
with NWBHDF5IO(hdf_filename, 'w') as export_io: # Create HDF5 IO object for write
export_io.export(src_io=read_io, write_args=dict(link_data=False)) # Export from Zarr to HDF5
with suppress(Exception): # TODO: This is a temporary ignore on the convert_dtype exception.
with NWBZarrIO(zarr_filename, mode='r') as read_io: # Create Zarr IO object for read
with NWBHDF5IO(hdf_filename, 'w') as export_io: # Create HDF5 IO object for write
export_io.export(src_io=read_io, write_args=dict(link_data=False)) # Export from Zarr to HDF5

###############################################################################
# Read the new HDF5 file back
Expand All @@ -118,5 +117,6 @@
# Now our file has been converted from HDF5 to Zarr and back again to HDF5.
# Here we check that we can still read that file.

with NWBHDF5IO(hdf_filename, 'r') as hr:
hf = hr.read()
with suppress(Exception): # TODO: This is a temporary ignore on the convert_dtype exception.
with NWBHDF5IO(hdf_filename, 'r') as hr:
hf = hr.read()
10 changes: 5 additions & 5 deletions docs/source/storage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,11 @@ store the definition of the ``region`` that is being referenced, e.g., a slice o
4) :py:meth:`~hdmf_zarr.backend.ZarrIO.__read_dataset` to support reading region references,
which may also require updates to :py:meth:`~hdmf_zarr.backend.ZarrIO.__parse_ref` and
:py:meth:`~hdmf_zarr.backend.ZarrIO.__resolve_ref`, and
5) and possibly other parts of :py:class:`~hdmf_zarr.backend.ZarrIO`
5) and possibly other parts of :py:class:`~hdmf_zarr.backend.ZarrIO`.
6) The py:class:`~hdmf_zarr.zarr_utils.ContainerZarrRegionDataset` and
py:class:`~hdmf_zarr.zarr_utils.ContainerZarrRegionDataset` classes will also need to be finalized
to support region references.


.. _sec-zarr-storage-dtypes:

Expand Down Expand Up @@ -379,7 +383,3 @@ data type. The specification of the namespace is stored in
``/specifications/<namespace-name>/<version>/<source-filename>``. Here ``<source-filename>`` refers to the main name
of the source-file without file extension (e.g., the core namespace defines ``nwb.ephys.yaml`` as source which would
be stored in ``/specifications/core/2.0.1/nwb.ecephys``).




2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.black]
line-length = 120
target-version = ['py37']
target-version = ['py38']
include = '\.pyi?$'
extend-exclude = '''
/(
Expand Down
4 changes: 2 additions & 2 deletions requirements-min.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
hdmf==3.5.4
hdmf==3.9.0
zarr==2.11.0
numcodecs==0.9.1
pynwb==2.3.2
pynwb==2.5.0
setuptools
importlib_resources;python_version<'3.9' # Remove when python 3.9 becomes the new minimum
10 changes: 4 additions & 6 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# pinned dependencies to reproduce an entire development environment to use HDMF-ZARR
hdmf==3.5.4
hdmf==3.9.0
zarr==2.11.0
pynwb==2.3.2
numpy==1.21; python_version < "3.8"
numpy==1.23; python_version >= "3.8"
numcodecs==0.10.2; python_version < "3.8"
numcodecs==0.11.0; python_version >= "3.8"
pynwb==2.5.0
numpy==1.24
numcodecs==0.11.0
13 changes: 5 additions & 8 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,12 @@


reqs = [
'hdmf==3.5.4', # temporary
'hdmf>=3.9.0',
'zarr>=2.11.0',
'numpy<1.22; python_version < "3.8"',
'numpy>=1.22; python_version >= "3.8"',
'numpy>=1.24',
'numcodecs>=0.9.1',
'numcodecs==0.10.2; python_version < "3.8"',
'numcodecs==0.11.0; python_version >= "3.8"',
'pynwb>=2.3.2',
'numcodecs==0.11.0',
'pynwb>=2.5.0',
'setuptools',
]

Expand All @@ -45,10 +43,9 @@
'packages': pkgs,
'package_dir': {'': 'src'},
'package_data': {},
'python_requires': '>=3.7',
'python_requires': '>=3.8',
'classifiers': [
"Programming Language :: Python",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
Expand Down
Loading

0 comments on commit c262481

Please sign in to comment.