Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading nd datasets #966

Merged
merged 50 commits into from
Apr 2, 2024
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
6269a11
Started debuging for reading nd datasets.
markbader Nov 7, 2023
f891606
Implementation of VecNInt and NDBoundingBox.
markbader Nov 23, 2023
0681252
Merge branch 'master' into reading_nd_datasets
markbader Nov 23, 2023
7cb395e
Rename VecInt and fix some issues.
markbader Nov 27, 2023
b3c972e
Work on import of existing zarr dataset.
markbader Dec 21, 2023
fdf9c03
Merge branch 'master' into reading_nd_datasets
markbader Dec 21, 2023
89a38e6
Add nd_bounding_box to properties of Layer.
markbader Dec 21, 2023
5580631
Update hooks in properties.py to make import of 4d zarr datasets poss…
markbader Jan 4, 2024
c4d2838
Add axis_order and index of additional axes to nd_bounding box creation.
markbader Jan 8, 2024
cf36c58
Working on reading nd data with pims images.
markbader Jan 11, 2024
4033eb0
Modify pims images to support more _iter_dims.
markbader Jan 16, 2024
ae29105
Add method for expected bounding box instead of expected shape in pim…
markbader Jan 18, 2024
f7ffb6b
Adding functions for editing ndboundingbox in 3d and update import fr…
markbader Jan 18, 2024
21bb0ea
Propagade nd-bounding box to different methods that take care of writ…
markbader Jan 22, 2024
51631c0
Update object unstructuring for json and start implementing nd array …
markbader Jan 23, 2024
2a755d3
Adapt array classes and add axes information to ArrayInfo.
markbader Jan 25, 2024
58c728b
Updated zarr writing for nd arrays. Axes order still get mixed up bet…
markbader Jan 29, 2024
af249e7
Adapted buffered_slice_reader and test behaviour with different tif i…
markbader Jan 30, 2024
ad3b22e
Adding testdata and fix bugs with axes operation for writing zarr array.
markbader Jan 31, 2024
e975f08
Fixing issues with axis order.
markbader Feb 1, 2024
61146c9
Rewrite buffered_slice_writer and fix bugs to get old tests working.
markbader Feb 5, 2024
6fff638
Fix chunking in buffered slice writer.
markbader Feb 5, 2024
6e9efd6
Fix issues with old tests.
markbader Feb 6, 2024
0cb8241
Working on fixing all tests.
markbader Feb 8, 2024
bb16e83
Merge branch 'master' into reading_nd_datasets
markbader Feb 12, 2024
0d8a4f7
Fixing issues with alignment, writing with offsets and different mags.
markbader Feb 15, 2024
9fc0253
Fix reading without parameters for nd datasets and addint test.
markbader Feb 15, 2024
48f9586
Fix issue with empty additionalAxes in json and some minor issues.
markbader Feb 16, 2024
51f173c
Move script reading_nd_data to examples in docs and implement some fe…
markbader Feb 20, 2024
7340073
Do some formatting and implement requested changes.
markbader Feb 20, 2024
1a94714
Update naming for accessing attributes of nd bounding boxes.
markbader Feb 21, 2024
e1d522f
Fix buffered_slice_writer for different axis and typechecking.
markbader Feb 26, 2024
e614525
Fix issue with ensure_size in initialization.
markbader Feb 26, 2024
23b992a
Merge branch 'master' into reading_nd_datasets
markbader Feb 26, 2024
27d3a3d
Adapt pims_images to support xyz images with channels and timepoint.
markbader Feb 26, 2024
8a1ca27
run formatter
markbader Feb 26, 2024
4167b9b
Fix issues with failed tests and add comments.
markbader Mar 7, 2024
2f69c30
Fix statement with wrong VecInt initialization.
markbader Mar 8, 2024
31255e3
Insert previously deleted assertions.
markbader Mar 8, 2024
8bedec5
Merge branch 'master' into reading_nd_datasets
markbader Mar 11, 2024
c354dd6
Add docstrings and use bounding box for read in nd case instead of Ve…
markbader Mar 11, 2024
ffbe7f7
Implement requested changes.
markbader Mar 18, 2024
36fef79
Merge branch 'master' into reading_nd_datasets
markbader Mar 18, 2024
2bfe961
Changes init of NDBoundingBoxes.
markbader Mar 18, 2024
0575eb9
Add converter for VecInt attributes of nd bounding box to pass typech…
markbader Mar 18, 2024
2ce18df
Merge branch 'master' into reading_nd_datasets
markbader Mar 19, 2024
e6781d7
Merge branch 'master' into reading_nd_datasets
markbader Mar 28, 2024
1a51e9b
Enhance documentation.
markbader Mar 28, 2024
f21ac09
Update Changelog.md
markbader Mar 28, 2024
092af26
Merge branch 'master' into reading_nd_datasets
markbader Apr 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion webknossos/examples/download_segments.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ def main() -> None:
)
mag_view = dataset.get_segmentation_layers()[0].get_mag(MAG)

z = mag_view.bounding_box.topleft.z
z = mag_view.bounding_box.get_bounds("z")[0]
markbader marked this conversation as resolved.
Show resolved Hide resolved
with mag_view.get_buffered_slice_reader() as reader:
for slice_data in reader:
slice_data = slice_data[0] # First channel only
Expand Down
2 changes: 1 addition & 1 deletion webknossos/examples/download_tiff_stack.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ def main() -> None:
)
mag_view = dataset.get_layer(LAYER_NAME).get_mag(MAG)

z = mag_view.bounding_box.topleft.z
z = mag_view.bounding_box.get_bounds("z")[0]
markbader marked this conversation as resolved.
Show resolved Hide resolved
with mag_view.get_buffered_slice_reader() as reader:
for slice_data in reader:
slice_data = slice_data[0] # First channel only
Expand Down
65 changes: 65 additions & 0 deletions webknossos/script_collection/reading_nd_data.py
markbader marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
from pathlib import Path

from tifffile import imwrite

import webknossos as wk
from webknossos.geometry.nd_bounding_box import NDBoundingBox

TIF_PATH = Path(".") / "webknossos" / "testdata" / "4D" / "4D_series"
OUTPUT = Path(".") / "testoutput" / "4D_series"


def from_images_import():
dataset = wk.Dataset.from_images(
TIF_PATH,
OUTPUT,
voxel_size=(10, 10, 10),
data_format="zarr3",
compress=True,
use_bioformats=True,
)
layer = dataset.get_color_layers()[0]
mag_view = layer.get_finest_mag()
# data = mag_view.read()[0, 0, 0, :, :]
markbader marked this conversation as resolved.
Show resolved Hide resolved

for bbox in layer.bounding_box.chunk((439, 167, 5)):
with mag_view.get_buffered_slice_reader(absolute_bounding_box=bbox) as reader:
for i, slice_data in enumerate(reader):
imwrite(
f"l4_sample_tiff/tiff_{i}_from_bbox{bbox}.tiff",
slice_data,
)


def open_existing_dataset():
ds = wk.Dataset.open(OUTPUT)
layer = ds.get_color_layers()[0]
mag_view = layer.get_finest_mag()
read_bbox = NDBoundingBox(
topleft=(0, 0, 0, 0),
size=(1, 1, 439, 167),
axes=("t", "z", "y", "x"),
index=(1, 2, 3, 4),
)
data = mag_view.read(absolute_bounding_box=read_bbox)
assert data.shape == (1,) + read_bbox.size
data = mag_view.read(
absolute_bounding_box=NDBoundingBox(
topleft=(0, 0, 0, 0),
size=(1, 1, 439, 167),
axes=("t", "z", "y", "x"),
index=(1, 2, 3, 4),
)
)

imwrite("l4_sample_tiff/test.tiff", data)


def main() -> None:
"""Imports a dataset with more than 3 dimensions."""
from_images_import()
# open_existing_dataset()
markbader marked this conversation as resolved.
Show resolved Hide resolved


if __name__ == "__main__":
main()
Binary file not shown.
Binary file not shown.
Binary file not shown.
20 changes: 20 additions & 0 deletions webknossos/tests/dataset/test_add_layer_from_images.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,26 @@ def test_compare_tifffile(tmp_path: Path) -> None:
assert np.array_equal(data[:, :, z_index], comparison_slice)


def test_compare_nd_tifffile(tmp_path: Path) -> None:
ds = wk.Dataset(tmp_path, (1, 1, 1))
l = ds.add_layer_from_images(
"testdata/4D/4D_series/4D-series.ome.tif",
layer_name="color",
category="color",
topleft=(100, 100, 55),
use_bioformats=True,
data_format="zarr3",
chunk_shape=(8, 8, 8),
chunks_per_shard=(8, 8, 8),
)
assert l.bounding_box.topleft == wk.VecInt(0, 55, 100, 100)
assert l.bounding_box.size == wk.VecInt(7, 5, 167, 439)
assert np.array_equal(
l.get_finest_mag().read()[0],
markbader marked this conversation as resolved.
Show resolved Hide resolved
TiffFile("testdata/4D/4D_series/4D-series.ome.tif").asarray(),
)


REPO_IMAGES_ARGS: List[
Tuple[Union[str, List[Path]], Dict[str, Any], str, int, Tuple[int, int, int]]
] = [
Expand Down
4 changes: 2 additions & 2 deletions webknossos/tests/test_examples.py
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,7 @@ def test_download_segments() -> None:

assert (
len(list(output_path.iterdir()))
== 2 * mag_view.layer.bounding_box.size.z / mag_view.mag.z
== 2 * mag_view.layer.bounding_box.get_3d("size").z / mag_view.mag.z
markbader marked this conversation as resolved.
Show resolved Hide resolved
)


Expand All @@ -329,7 +329,7 @@ def test_download_tiff_stack() -> None:

assert (
len(list(output_path.iterdir()))
== mag_view.bounding_box.size.z / mag_view.mag.z
== mag_view.bounding_box.get_3d("size").z / mag_view.mag.z
)


Expand Down
2 changes: 1 addition & 1 deletion webknossos/webknossos/annotation/annotation.py
Original file line number Diff line number Diff line change
Expand Up @@ -476,7 +476,7 @@ def _load_from_zip(cls, content: Union[str, PathLike, BinaryIO]) -> "Annotation"
assert len(nml_paths) > 0, "Couldn't find an nml file in the supplied zip-file."
assert (
len(nml_paths) == 1
), f"There must be exactly one nml file in the zip-file, buf found {len(nml_paths)}."
), f"There must be exactly one nml file in the zip-file, but found {len(nml_paths)}."
with nml_paths[0].open(mode="rb") as f:
return cls._load_from_nml(nml_paths[0].stem, f, possible_volume_paths=paths)

Expand Down
Loading
Loading