Enable zarrita by default (#950)

* enable zarrita by default * deactivate zarr-python * upgrade zarrita * changelog * organize imports and add ome-metadata for zarr3 aswell * more relative imports * Zarr volume annotations (#952) * Adds support for Zarr3-based volume annotations * adds test * defaults for Volume * changelog * adds test zip * merge * merge * test_annotation with id * Update webknossos/webknossos/dataset/dataset.py Co-authored-by: Florian M <[email protected]> * Update webknossos/webknossos/annotation/annotation.py Co-authored-by: Florian M <[email protected]> * pr feedback * ci
scalableminds · Oct 12, 2023 · 2a5d1c0 · 2a5d1c0
1 parent 2e989d6
commit 2a5d1c0
Show file tree

Hide file tree

Showing 63 changed files with 540 additions and 394 deletions.
diff --git a/docs/generate.sh b/docs/generate.sh
@@ -17,10 +17,10 @@ fi
 
 export PDOC_CLASS_MODULES="$(poetry run python get_keyword_mapping.py)"
 if [ $# -eq 1 ] && [ "$1" = "--api" ]; then
-    poetry run pdoc ../webknossos/webknossos -t pdoc_templates/pure_pdoc -h 0.0.0.0 -p 8196
+    poetry run pdoc ../webknossos/webknossos !webknossos.dataset._utils -t pdoc_templates/pure_pdoc -h 0.0.0.0 -p 8196
 else
     rm -rf src/api
-    poetry run pdoc ../webknossos/webknossos -t pdoc_templates/with_mkdocs -o src/api
+    poetry run pdoc ../webknossos/webknossos !webknossos.dataset._utils -t pdoc_templates/with_mkdocs -o src/api
     # rename .html files to .md
     find src/api -iname "*.html" -exec sh -c 'mv "$0" "${0%.html}.md"' {} \;
     # assert that API docs are written

diff --git a/webknossos/Changelog.md b/webknossos/Changelog.md
@@ -10,99 +10,115 @@ and this project adheres to [Semantic Versioning](http://semver.org/) `MAJOR.MIN
 For upgrade instructions, please check the respective _Breaking Changes_ sections.
 
 ## Unreleased
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.0...HEAD)
 
 ### Breaking Changes
 
 ### Added
 
+- Adds support for Zarr3-based volume annotations as introduced in [webknossos#7288](https://github.com/scalableminds/webknossos/pull/7288). [#952](https://github.com/scalableminds/webknossos-libs/pull/952)
+
 ### Changed
 
-### Fixed
+- The `WK_USE_ZARRITA` env variable is no longer required. [`zarrita`](https://github.com/scalableminds/zarrita) is always installed and now the default for Zarr and Zarr3 datasets. [#950](https://github.com/scalableminds/webknossos-libs/issues/950)
+- Updates various dependencies. [#943](https://github.com/scalableminds/webknossos-libs/pull/943)
 
+### Fixed
 
 ## [0.14.0](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.0) - 2023-10-11
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.7...v0.14.0)
 
 ### Breaking Changes
+
 - `wait_and_ensure_success` from `webknossos.utils` now requires an `executor` argument. [#943](https://github.com/scalableminds/webknossos-libs/pull/943)
 
 ### Changed
-- Updates various dependencies. [#943](https://github.com/scalableminds/webknossos-libs/pull/943)
 
+- Updates various dependencies. [#943](https://github.com/scalableminds/webknossos-libs/pull/943)
 
 ## [0.13.7](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.7) - 2023-10-07
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.6...v0.13.7)
 
 ### Fixed
-- Fixed a bug in writing compressed data. [#942](https://github.com/scalableminds/webknossos-libs/pull/942)
-
 
+- Fixed a bug in writing compressed data. [#942](https://github.com/scalableminds/webknossos-libs/pull/942)
 
 ## [0.13.6](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.6) - 2023-08-17
-[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.5...v0.13.6)
 
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.5...v0.13.6)
 
 ## [0.13.5](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.5) - 2023-08-15
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.4...v0.13.5)
 
 ### Added
+
 - Added `task_type` property to `Task` class. [#938](https://github.com/scalableminds/webknossos-libs/pull/938)
 
 ### Fixed
-- Fixed a bug where parallel access to the properties json leads to an JsonDecodeError in the webknossos CLI [#919](https://github.com/scalableminds/webknossos-libs/issues/919)
-
 
+- Fixed a bug where parallel access to the properties json leads to an JsonDecodeError in the webknossos CLI [#919](https://github.com/scalableminds/webknossos-libs/issues/919)
 
 ## [0.13.4](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.4) - 2023-08-14
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.3...v0.13.4)
 
 ### Breaking Changes
+
 - Task/Project management: `open` tasks have been renamed to `pending`. Use `Task.status.pending_instance_count` instead of `Task.status.open_instance_count`. [#930](https://github.com/scalableminds/webknossos-libs/pull/930)
 
 ### Fixed
-- Fixed an infinite loop in the mag calculation during anisotropic downsampling in situations where the target mag cannot possibly be reached while adhering to the anisotropic downsampling scheme. [#934](https://github.com/scalableminds/webknossos-libs/pull/934)
-
 
+- Fixed an infinite loop in the mag calculation during anisotropic downsampling in situations where the target mag cannot possibly be reached while adhering to the anisotropic downsampling scheme. [#934](https://github.com/scalableminds/webknossos-libs/pull/934)
 
 ## [0.13.3](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.3) - 2023-08-08
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.2...v0.13.3)
 
 ### Added
+
 - `View` has a `map_chunk` method now to run a function on each chunk and collect the results in a list.
 
 ### Changed
+
 - As WEBKNOSSOS does not require the largest segment id. It is also not mandatory in the WEBKNOSSOS libs anymore. [#917](https://github.com/scalableminds/webknossos-libs/issues/917) The method `SegmentationLayer.refresh_largest_segment_id` was added to lookup the highest value in segmentation data and set `largest_segment_id` accordingly.
 - The `convert` command of the cli now has a `--category` flag, to select the LayerCategoryType.
 
-
 ## [0.13.2](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.2) - 2023-07-26
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.1...v0.13.2)
 
 ### Changed
-- The `convert` command of the cli now has a `--category` flag, to select the LayerCategoryType.
 
+- The `convert` command of the cli now has a `--category` flag, to select the LayerCategoryType.
 
 ## [0.13.1](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.1) - 2023-07-17
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.13.0...v0.13.1)
 
 ### Changed
+
 - The conversion folder structures to layer names does not allow slashes in the layer name anymore. [#918](https://github.com/scalableminds/webknossos-libs/issues/918)
 
 ### Fixed
-- Fixed a bug where compression in add_layer_from_images uses too much memory [#900](https://github.com/scalableminds/webknossos-libs/issues/900)
-
 
+- Fixed a bug where compression in add_layer_from_images uses too much memory [#900](https://github.com/scalableminds/webknossos-libs/issues/900)
 
 ## [0.13.0](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.13.0) - 2023-06-21
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.6...v0.13.0)
 
 ### Added
+
 - Added `duration_in_seconds` and `modified` to `AnnotationInfo`. [#914](https://github.com/scalableminds/webknossos-libs/pull/914)
 - Added [`zarrita`](https://github.com/scalableminds/zarrita) storage backend for arrays. zarrita supports Zarr v2 and v3 including sharding. To activate zarrita, the environment variable `WK_USE_ZARRITA` must be set. [#912](https://github.com/scalableminds/webknossos-libs/pull/912)
 - Added a `Zarr3` data format which supports sharding. [#912](https://github.com/scalableminds/webknossos-libs/pull/912)
 
 ### Changed
+
 - Integrated the `wkcuber` CLI tool into `webknossos` package. [#903](https://github.com/scalableminds/webknossos-libs/pull/903)
   - To get an overview of all webknossos subcommands type `webknossos --help`. If the usage of a single subcommand is of interest type `webknossos <subcommand> --help`
   - These commands were changed:
@@ -123,63 +139,68 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
     - `python -m wkcuber.recubing`
 
 ### Fixed
-- Fixed a bug where upsampling of a layer would fail, if the layer had a bounding box that doesn't align with the from_mag mag. [#915](https://github.com/scalableminds/webknossos-libs/pull/915)
-
 
+- Fixed a bug where upsampling of a layer would fail, if the layer had a bounding box that doesn't align with the from_mag mag. [#915](https://github.com/scalableminds/webknossos-libs/pull/915)
 
 ## [0.12.6](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.12.6) - 2023-06-09
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.5...v0.12.6)
 
 ### Changed
-- Upgrades `wkw`. [#911](https://github.com/scalableminds/webknossos-libs/pull/911)
 
+- Upgrades `wkw`. [#911](https://github.com/scalableminds/webknossos-libs/pull/911)
 
 ## [0.12.5](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.12.5) - 2023-06-01
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.4...v0.12.5)
 
 ### Added
-- Added support for Python 3.11. [#843](https://github.com/scalableminds/webknossos-libs/pull/843)
 
+- Added support for Python 3.11. [#843](https://github.com/scalableminds/webknossos-libs/pull/843)
 
 ## [0.12.4](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.12.4) - 2023-05-25
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.3...v0.12.4)
 
 ### Added
+
 - `Group.add_tree` now also accepts a tree object as a first parameter (instead of only a string). [#891](https://github.com/scalableminds/webknossos-libs/pull/891)
 - `Group.remove_tree_by_id` was added. [#891](https://github.com/scalableminds/webknossos-libs/pull/891)
 
 ### Changed
+
 - Upgrades `black`, `mypy`, `pylint`, `pytest`. [#873](https://github.com/scalableminds/webknossos-libs/pull/873)
 
 ### Fixed
+
 - Fixed poetry build backend for new versions of Poetry. [#899](https://github.com/scalableminds/webknossos-libs/pull/899)
 - Added axis_order fields for Zarr data format. [#902](https://github.com/scalableminds/webknossos-libs/issues/902)
 
-
-
 ## [0.12.3](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.12.3) - 2023-02-22
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.2...v0.12.3)
 
 ### Added
+
 - Added support to import ImageJ Hyperstack tiff files via `Dataset.from_images` and `dataset.add_layer_from_images`. [#877](https://github.com/scalableminds/webknossos-libs/pull/877)
 
 ### Changed
+
 - `Dataset.from_images` and `dataset.add_layer_from_images` now automatically convert big endian dtypes to their little endian counterparts by default. [#877](https://github.com/scalableminds/webknossos-libs/pull/877)
 
 ### Fixed
-- Fixed reading czi files with non-zero axis offsets. [#876](https://github.com/scalableminds/webknossos-libs/pull/876)
-
 
+- Fixed reading czi files with non-zero axis offsets. [#876](https://github.com/scalableminds/webknossos-libs/pull/876)
 
 ## [0.12.2](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.12.2) - 2023-02-20
+
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.1...v0.12.2)
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.1...HEAD)
 
 ### Added
 
 - Added `RemoteFolder` for assigning remote datasets to a WEBKNOSSOS folder. [#868](https://github.com/scalableminds/webknossos-libs/pull/868)
 
-
 ## [0.12.1](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.12.1) - 2023-02-16
 
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.12.0...v0.12.1)

diff --git a/webknossos/poetry.lock b/webknossos/poetry.lock
diff --git a/webknossos/pyproject.toml b/webknossos/pyproject.toml
@@ -54,6 +54,7 @@ typing-extensions = "^4.0"
 universal-pathlib = "0.1.3"
 wkw = "1.1.22"
 zarr = "^2.16.0"
+zarrita = "0.2.1"
 zipp = "^3.5.0"
 
 # A list of all of the optional dependencies, some of which are included in the
@@ -63,15 +64,14 @@ JPype1 = { version = "^1.3.0", optional = true }
 pims = { version = "^0.6.0", optional = true }
 tifffile = { version = ">=2021.11.2", optional = true }
 pylibCZIrw = { version = "3.5.1", source = "scm", optional = true }
-zarrita = "0.2.1"
 
 [tool.poetry.extras]
 pims = ["pims"]
 tifffile = ["pims", "tifffile"]
 imagecodecs = ["pims", "imagecodecs"]
 bioformats = ["pims","JPype1"]
 czi = ["pims","pylibCZIrw"]
-all = ["pims","tifffile","imagecodecs","JPype1","pylibCZIrw","zarrita"]
+all = ["pims","tifffile","imagecodecs","JPype1","pylibCZIrw"]
 
 [tool.poetry.dev-dependencies]
 # autoflake

diff --git a/webknossos/test.sh b/webknossos/test.sh
@@ -11,7 +11,6 @@ export_vars
 # (which is standard python behavior). This is necessary so that the imports
 # refer to the checked out (and potentially modified) code.
 PYTEST="poetry run python -m pytest --suppress-no-test-exit-code"
-export WK_USE_ZARRITA=True
 
 
 if [ $# -gt 0 ] && [ "$1" = "--refresh-snapshots" ]; then

diff --git a/webknossos/testdata/annotations/l4_sample__explorational__suser__94b271.zip b/webknossos/testdata/annotations/l4_sample__explorational__suser__94b271.zip
diff --git a/webknossos/tests/test_annotation.py b/webknossos/tests/test_annotation.py
@@ -1,17 +1,19 @@
 import tempfile
 from pathlib import Path
 
+import numpy as np
 import pytest
 
 import webknossos as wk
+from webknossos.dataset import DataFormat
 from webknossos.geometry import BoundingBox, Vec3Int
 
 from .constants import TESTDATA_DIR, TESTOUTPUT_DIR
 
 pytestmark = [pytest.mark.with_vcr]
 
 
-def test_annotation_from_zip_file() -> None:
+def test_annotation_from_wkw_zip_file() -> None:
     annotation = wk.Annotation.load(
         TESTDATA_DIR
         / "annotations"
@@ -57,6 +59,24 @@ def test_annotation_from_zip_file() -> None:
         assert voxel_id == 2504698
 
 
+def test_annotation_from_zarr3_zip_file() -> None:
+    annotation = wk.Annotation.load(
+        TESTDATA_DIR / "annotations" / "l4_sample__explorational__suser__94b271.zip"
+    )
+
+    with annotation.temporary_volume_layer_copy() as volume_layer:
+        assert volume_layer.data_format == DataFormat.Zarr3
+        assert volume_layer.bounding_box == BoundingBox(
+            (3072, 3072, 512), (1024, 1024, 1024)
+        )
+        input_annotation_mag = volume_layer.get_mag("2-2-1")
+        voxel_id = input_annotation_mag.read(
+            absolute_offset=Vec3Int(3630, 3502, 1024), size=Vec3Int(2, 2, 1)
+        )
+
+        assert np.array_equiv(voxel_id, 1)
+
+
 def test_annotation_from_nml_file() -> None:
     snapshot_path = TESTDATA_DIR / "nmls" / "generated_annotation_snapshot.nml"
 

diff --git a/webknossos/webknossos/__init__.py b/webknossos/webknossos/__init__.py
@@ -24,9 +24,9 @@
 
 # The table above contains zero-width spaces in the code examples after each dot to enforce correct line-breaks.
 
-from webknossos.administration import *
-from webknossos.annotation import *
-from webknossos.client import *
-from webknossos.dataset import *
-from webknossos.geometry import *
-from webknossos.skeleton import *
+from .administration import *
+from .annotation import *
+from .client import *
+from .dataset import *
+from .geometry import *
+from .skeleton import *
diff --git a/webknossos/webknossos/_nml/parameters.py b/webknossos/webknossos/_nml/parameters.py
@@ -3,9 +3,8 @@
 
 from loxun import XmlWriter
 
-from webknossos.geometry import BoundingBox
-from webknossos.geometry.bounding_box import _DEFAULT_BBOX_NAME
-
+from ..geometry import BoundingBox
+from ..geometry.bounding_box import _DEFAULT_BBOX_NAME
 from .utils import Vector3, enforce_not_null, filter_none_values
 
 DEFAULT_BOUNDING_BOX_COLOR = [0.2, 0.5, 0.1, 1]

diff --git a/webknossos/webknossos/_nml/segment.py b/webknossos/webknossos/_nml/segment.py
@@ -3,8 +3,7 @@
 
 from loxun import XmlWriter
 
-from webknossos.geometry import Vec3Int
-
+from ..geometry import Vec3Int
 from .utils import Vector4, enforce_not_null, filter_none_values
 
 

diff --git a/webknossos/webknossos/_nml/volume.py b/webknossos/webknossos/_nml/volume.py
@@ -17,6 +17,8 @@ class Volume(NamedTuple):
     # older wk versions did not serialize the name which is why the name is optional:
     name: Optional[str]
     segments: List[Segment]
+    format: Optional[str] = None
+    largest_segment_id: Optional[int] = None
 
     def _dump(self, xf: XmlWriter) -> None:
         xf.startTag(
@@ -27,6 +29,10 @@ def _dump(self, xf: XmlWriter) -> None:
                     "location": self.location,
                     "fallbackLayer": self.fallback_layer,
                     "name": self.name,
+                    "format": self.format,
+                    "largestSegmentId": str(self.largest_segment_id)
+                    if self.largest_segment_id
+                    else None,
                 }
             ),
         )
@@ -39,10 +45,15 @@ def _dump(self, xf: XmlWriter) -> None:
 
     @classmethod
     def _parse(cls, nml_volume: Element) -> "Volume":
+        largest_segment_id_str = nml_volume.get("largestSegmentId", default=None)
         return cls(
             id=int(enforce_not_null(nml_volume.get("id"))),
             location=nml_volume.get("location"),
             fallback_layer=nml_volume.get("fallbackLayer", default=None),
             name=nml_volume.get("name", default=None),
+            format=nml_volume.get("format", default=None),
             segments=[],
+            largest_segment_id=None
+            if largest_segment_id_str is None
+            else int(largest_segment_id_str),
         )
diff --git a/webknossos/webknossos/administration/__init__.py b/webknossos/webknossos/administration/__init__.py
@@ -1,3 +1,3 @@
-from webknossos.administration.project import Project
-from webknossos.administration.task import Task, TaskType
-from webknossos.administration.user import Team, User
+from .project import Project
+from .task import Task, TaskType
+from .user import Team, User