Merge branch 'master' into fix_setup_mag_in_case_of_non_public

scalableminds · Dec 18, 2024 · b8cafbd · b8cafbd
2 parents 28d94ef + 6a3ade5
commit b8cafbd
Show file tree

Hide file tree

Showing 98 changed files with 11,533 additions and 70,644 deletions.
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
@@ -25,6 +25,9 @@ jobs:
       uses: astral-sh/setup-uv@v3
       with:
         version: "0.4.19"
+
+    - name: Install proxay
+      run: npm install -g proxay
 
     - name: Set up Python ${{ matrix.python-version }}
       run: uv python install ${{ matrix.python-version }}
@@ -34,6 +37,10 @@ jobs:
         git diff --no-ext-diff --quiet --exit-code
         [[ -z $(git status -s) ]]
 
+    - name: Use currently deployed Webknossos version
+      run: |
+        sed -i "s/export DOCKER_TAG=.*$/export DOCKER_TAG=master__$(curl https://webknossos.org/api/buildinfo | jq -r .webknossos.version)/" local_wk_setup.sh
+
     - name: Python tests, refreshing the network snapshots
       env:
         WK_TOKEN: ${{ secrets.WK_TOKEN }}

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -140,6 +140,11 @@ npm install --global proxay
 # or if you're using yarn
 yarn global add proxay
 ```
+When running the `./test.sh` script on MacOS, it is also necessary to install `minio` using the following command:
+
+```bash
+brew install minio
+```
 
 The tests also contain functionality for the WEBKNOSSOS client. There a two modes to run the tests:
 

diff --git a/cluster_tools/Changelog.md b/cluster_tools/Changelog.md
@@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/) `MAJOR.MIN
 For upgrade instructions, please check the respective *Breaking Changes* sections.
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.16.0...HEAD)
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.16.1...HEAD)
 
 ### Breaking Changes
 
@@ -18,6 +18,10 @@ For upgrade instructions, please check the respective *Breaking Changes* section
 ### Fixed
 
 
+## [0.16.1](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.16.1) - 2024-12-05
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.16.0...v0.16.1)
+
+
 ## [0.16.0](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.16.0) - 2024-11-27
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.15.11...v0.16.0)
 

diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
@@ -93,7 +93,6 @@ plugins:
                     show_if_no_docstring: true
                     signature_crossrefs: true
                     parameter_headings: false
-                    summary: true
                     unwrap_annotated: true
                     line_length: 60
                     filters:
@@ -235,7 +234,6 @@ nav:
         - webknossos-py/examples/download_image_data.md
         - webknossos-py/examples/download_tiff_stack.md
         - webknossos-py/examples/remote_datasets.md
-        - webknossos-py/examples/zarr_and_dask.md
         - webknossos-py/examples/convert_4d_tiff.md
         - webknossos-py/examples/announce_dataset_upload.md
         - webknossos-py/examples/accessing_metadata.md

diff --git a/docs/src/webknossos-py/examples/zarr_and_dask.md b/docs/src/webknossos-py/examples/zarr_and_dask.md
diff --git a/docs/src/webknossos-py/index.md b/docs/src/webknossos-py/index.md
@@ -22,12 +22,13 @@ To get started, check out the [installation instructions](installation.md) and t
     - Up/downsample layers
     - Compress layers
     - Add/remove magnifications
-    - Execute any of the `wkCuber` operations from your code
+    - Execute any of the `webknossos` CLI operations from your code
 - Manipulation of WEBKNOSSOS skeleton annotations (*.nml) as Python objects
     - Access to nodes, comments, trees, bounding boxes, metadata, etc.
     - Create new skeleton annotation from Graph structures or Python objects
 - Interaction, connection & scripting with your WEBKNOSSOS instance over the REST API
     - Up- & downloading annotations and datasets
+    - Read datasets from the WEBKNOSSOS server in a streaming fashion 
 
 ## Source Code
 

diff --git a/docs/uv.lock b/docs/uv.lock
diff --git a/webknossos/Changelog.md b/webknossos/Changelog.md
@@ -10,20 +10,37 @@ and this project adheres to [Semantic Versioning](http://semver.org/) `MAJOR.MIN
 For upgrade instructions, please check the respective _Breaking Changes_ sections.
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.16.0...HEAD)
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.16.1...HEAD)
 
 ### Breaking Changes
+- `MagView.get_zarr_array` now returns a `tensorstore` array instead of a `zarr-python` array. [#1174](https://github.com/scalableminds/webknossos-libs/pull/1174) 
 
 ### Added
 
+### Changed
+- Updated to WEBKNOSSOS API version 8. [#1185](https://github.com/scalableminds/webknossos-libs/pull/1185)
+- Using tensorstore for reading and writing zarr 2 and 3 arrays. Removed `zarrita` and `zarr` dependency. [#1174](https://github.com/scalableminds/webknossos-libs/pull/1174) 
+
+### Fixed
+- Fixed Mag setup for non-public datasets. [#1222](https://github.com/scalableminds/webknossos-libs/pull/1222)
+
+
+## [0.16.1](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.16.1) - 2024-12-05
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.16.0...v0.16.1)
+
+### Added
+- Added .nrrd and .nhdr to supported suffixes. [#1228](https://github.com/scalableminds/webknossos-libs/pull/1228)
+- Added more docstrings for many public classes and methods. [#1225](https://github.com/scalableminds/webknossos-libs/pull/1225)
+
 ### Changed
 - Removes vcr-py from developer dependencies for testing and adds proxay for recording and replaying API requests. [#1198](https://github.com/scalableminds/webknossos-libs/pull/1198)
 - Removed the CZI installation extra from `pip install webknossos[all]` by default. Users need to manually install it with `pip install --extra-index-url https://pypi.scm.io/simple/ webknossos[czi]`. [#1219](https://github.com/scalableminds/webknossos-libs/pull/1219)
+- Refactored the PimsTiffReader to read the data directly from the tiff file without creating a memmap-able copy first. This greatly reduces the time and storage requirements for converting large tiff files. [#1212](https://github.com/scalableminds/webknossos-libs/pull/1212)
 
 ### Fixed
-
+- Fixed an issue where adding existing trees to an annotation fails. [#1201](https://github.com/scalableminds/webknossos-libs/pull/1201)
 - Fixed unpickling of the SSL_Context to allow for a second or third pickling. [#1223](https://github.com/scalableminds/webknossos-libs/pull/1223)
-- Fixed Mag setup for non-public datasets. [#1222](https://github.com/scalableminds/webknossos-libs/pull/1222)
+- Fixed offset error in upsample_cube job [#1209](https://github.com/scalableminds/webknossos-libs/pull/1209)
 
 
 ## [0.16.0](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.16.0) - 2024-11-27
@@ -37,15 +54,13 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
 - Fixed pickling issue that has been introduced in 0.15.9. [#1218](https://github.com/scalableminds/webknossos-libs/pull/1218)
 
 
-
 ## [0.15.10](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.15.10) - 2024-11-25
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.15.9...v0.15.10)
 
 ### Fixed
 - Fixed pickling issue that has been introduced in 0.15.9. [#1218](https://github.com/scalableminds/webknossos-libs/pull/1218)
 
 
-
 ## [0.15.9](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.15.9) - 2024-11-25
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.15.8...v0.15.9)
 
@@ -68,6 +83,7 @@ Removed the CZI installation extra from `pip install webknossos[all]` by default
 
 ### Fixed
 - Fixed an issue with merging annotations with compressed fallback layers.
+- Fixed an issue where adding a Zarr array with other axes than `cxyz` leads to an error. [#1204](https://github.com/scalableminds/webknossos-libs/pull/1204)
 
 
 

diff --git a/webknossos/examples/apply_merger_mode.py b/webknossos/examples/apply_merger_mode.py
@@ -1,4 +1,3 @@
-from os import environ
 from pathlib import Path
 from typing import Tuple, cast
 
@@ -84,19 +83,18 @@ def apply_mapping_for_chunk(args: Tuple[wk.View, wk.View, int]) -> None:
     # Optionally, downsample and re-upload #
     ########################################
 
-    if "PYTEST_CURRENT_TEST" not in environ:
-        out_layer.downsample()
-        dataset.delete_layer("segmentation")
-        dataset.upload(
-            "l4_sample_remapped",
-            layers_to_link=[
-                wk.LayerToLink(
-                    dataset_name="l4_sample",
-                    layer_name="color",
-                    organization_id="scalable_minds",
-                )
-            ],
-        )
+    out_layer.downsample()
+    dataset.delete_layer("segmentation")
+    dataset.upload(
+        "l4_sample_remapped",
+        layers_to_link=[
+            wk.LayerToLink(
+                dataset_name="l4_sample",
+                layer_name="color",
+                organization_id="scalable_minds",
+            )
+        ],
+    )
 
 
 if __name__ == "__main__":

diff --git a/webknossos/examples/zarr_and_dask.py b/webknossos/examples/zarr_and_dask.py
diff --git a/webknossos/local_wk_setup.sh b/webknossos/local_wk_setup.sh
@@ -1,7 +1,7 @@
 function export_vars {
     export WK_TOKEN=1b88db86331a38c21a0b235794b9e459856490d70408bcffb767f64ade0f83d2bdb4c4e181b9a9a30cdece7cb7c65208cc43b6c1bb5987f5ece00d348b1a905502a266f8fc64f0371cd6559393d72e031d0c2d0cabad58cccf957bb258bc86f05b5dc3d4fff3d5e3d9c0389a6027d861a21e78e3222fb6c5b7944520ef21761e
     export WK_URL=http://localhost:9000
-    export DOCKER_TAG=master__30776
+    export DOCKER_TAG=master__31116
 }
 
 function ensure_local_test_wk {

diff --git a/webknossos/pyproject.toml b/webknossos/pyproject.toml
@@ -38,25 +38,22 @@ dependencies = [
     "cattrs >=22.0.0",
     "certifi>=2023",
     "cluster-tools",
-    "fsspec ~=2024.6.0",
     "httpx ~=0.27.0",
     "loxun ~=2.0.0",
     "natsort ~=8.4.0",
     "networkx ~=3.2.1",
-    "numcodecs ~=0.12",
     "numpy ~=2.0.0",
     "pims ~=0.7.0",
     "psutil ~=6.0.0",
     "python-dateutil ~=2.8.0",
     "python-dotenv ~=1.0.1",
     "rich ~=13.8.0",
     "scipy ~=1.13",
+    "tensorstore~=0.1.69",
     "typer ~=0.12.5",
     "typing-extensions ~=4.0",
     "universal-pathlib ~=0.2",
     "wkw ==1.1.24",
-    "zarr ~=2.18.0",
-    "zarrita ==0.2.7",
     "zipp ~=3.5.0",
 ]
 
@@ -91,7 +88,7 @@ all = [
 [tool.uv]
 extra-index-url = ["https://pypi.scm.io/simple"]
 dev-dependencies = [
-    "dask[distributed] ~=2023.9.1; python_version >='3.9'",
+    "fsspec ~=2024.6.0",
     "hypothesis ~=6.35.0",
     "icecream ~=2.1.1",
     "inducoapi ~=2.0.2",
@@ -160,7 +157,7 @@ target-version = "py39"
 # PERF = Perflint https://docs.astral.sh/ruff/rules/#perflint-perf
 # FIX = Flake8 fixme https://docs.astral.sh/ruff/rules/#flake8-fixme-fix
 # ARG = Flake8 unused-arguments https://docs.astral.sh/ruff/rules/#flake8-unused-arguments-arg
-select = ["F", "E", "W", "I", "A", "PERF", "FIX", "ARG"]
+select = ["F", "E", "W", "I", "A", "PERF", "FIX", "ARG", "T201"]
 
 # Allow fix for all enabled rules (when `--fix`) is provided.
 fixable = ["ALL"]
@@ -169,6 +166,9 @@ fixable = ["ALL"]
 # E712 true-false-comparison https://docs.astral.sh/ruff/rules/true-false-comparison/
 ignore = ["E501", "E712"]
 
+[tool.ruff.lint.per-file-ignores]
+"{examples,tests,script_collection,webknossos/cli}/*" = ["T201"]
+
 [tool.ruff.format]
 # Like Black, use double quotes for strings.
 quote-style = "double"

diff --git a/webknossos/test.sh b/webknossos/test.sh
@@ -31,6 +31,21 @@ if [ $# -gt 0 ] && [ "$1" = "--refresh-snapshots" ]; then
 
     stop_local_test_wk
 
+    exit $PYTEST_EXIT_CODE
+elif [ $# -gt 0 ] && [ "$1" = "--add-snapshots" ]; then
+    ensure_local_test_wk
+
+    # Starts a proxy server in record mode on port 3000 and sets the HTTP_PROXY env var
+    proxay --mode record --host http://localhost:9000 --tapes-dir tests/cassettes &
+    PROXAY_PID=$!
+
+    shift
+    $PYTEST "-m" "use_proxay" "$@"
+    PYTEST_EXIT_CODE=$?
+    trap 'kill $PROXAY_PID' EXIT
+
+    stop_local_test_wk
+
     exit $PYTEST_EXIT_CODE
 elif [ $# -gt 0 ] && [ "$1" = "--debug-cassettes" ]; then
     # This will start a proxay server in replay mode so that the stored cassettes can be used for debugging tests.

diff --git a/webknossos/testdata/annotations/l4_sample__explorational__suser__94b271.zip b/webknossos/testdata/annotations/l4_sample__explorational__suser__94b271.zip
diff --git a/webknossos/testdata/simple_zarr3_dataset/color/1/zarr.json b/webknossos/testdata/simple_zarr3_dataset/color/1/zarr.json
@@ -16,17 +16,17 @@
       "configuration": {
         "chunk_shape": [3, 16, 16, 16],
         "codecs": [
-          { "name": "endian", "configuration": { "endian": "little" } }
+          { "name": "bytes", "configuration": { "endian": "little" } }
         ],
         "index_codecs": [
-          { "name": "endian", "configuration": { "endian": "little" } },
+          { "name": "bytes", "configuration": { "endian": "little" } },
           { "name": "crc32c" }
         ]
       },
       "name": "sharding_indexed"
     }
   ],
-  "dimension_names": null,
+  "dimension_names": ["c", "x", "y", "z"],
   "zarr_format": 3,
   "node_type": "array"
 }