76 sharding in zarr

CHANGELOG.md

@@ -7,9 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+### Added
+
+- Support for [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html).
+- Support for
+  the [sharding storage transformer](https://web.archive.org/web/20230213221154/https://zarr-specs.readthedocs.io/en/latest/extensions/storage-transformers/sharding/v1.0.html)
+  in Zarr v3.
+- Ship debug libs for C-Blosc on Linux and Mac.
+
+### Changed
+
+- Upgrades C-Blosc from v1.21.4 to v1.21.5.
+
 ### Fixed
 
 - A bug where enabling multiscale without specifying the tile size would cause an error.
+- Exceptions thrown off the main thread are now caught and logged, and Zarr throws an error in `append`.
+- Job queue is now cleared after every operation.
 
 ## [0.1.4](https://github.com/acquire-project/acquire-driver-zarr/compare/v0.1.3...v0.1.4) - 2023-08-11
 

README.md

@@ -13,6 +13,9 @@ This is an Acquire Driver that supports chunked streaming to [zarr][].
 - **Zarr**
 - **ZarrBlosc1ZstdByteShuffle**
 - **ZarrBlosc1Lz4ByteShuffle**
+- **ZarrV3**
+- **ZarrV3Blosc1ZstdByteShuffle**
+- **ZarrV3Blosc1Lz4ByteShuffle**
 
 ## Using the Zarr storage device
 
@@ -24,6 +27,10 @@ Chunking is configured using `storage_properties_set_chunking_props()` when conf
 Multiscale storage can be enabled or disabled by calling `storage_properties_set_enable_multiscale()` when configuring
 the video stream.
 
+For the [Zarr v3] version of each device, you can use the `ZarrV3*` devices.
+**Note:** Zarr v3 is not [yet](https://github.com/ome/ngff/pull/206) supported by the Python OME-Zarr library, so you
+will not be able to read multiscale metadata from the resulting dataset.
+
 ### Configuring chunking
 
 You can configure chunking by calling `storage_properties_set_chunking_props()` on your `StorageProperties` object
@@ -41,21 +48,21 @@ storage_properties_set_chunking_props(struct StorageProperties* out,
 ```
 
 | ![frames](https://github.com/aliddell/acquire-driver-zarr/assets/844464/3510d468-4751-4fa0-b2bf-0e29a5f3ea1c) |
-|:--:|
-| A collection of frames. |
+|:-------------------------------------------------------------------------------------------------------------:|
+|                                            A collection of frames.                                            |
 
 A _tile_ is a contiguous section, or region of interest, of a _frame_.
 
 | ![tiles](https://github.com/aliddell/acquire-driver-zarr/assets/844464/f8d16139-e0ac-44db-855f-2f5ef305c98b) |
-|:--:|
-| A collection of frames, divided into tiles. |
+|:------------------------------------------------------------------------------------------------------------:|
+|                                 A collection of frames, divided into tiles.                                  |
 
 A _chunk_ is nothing more than some number of stacked tiles from subsequent frames, with each tile in a chunk having
 the same ROI in its respective frame.
 
-|  ![chunks](https://github.com/aliddell/acquire-driver-zarr/assets/844464/653e4d82-363e-4e04-9a42-927b052fb6e7) |
-|:--:|
-| A collection of frames, divided into tiles. A single chunk has been highlighted in red. |
+| ![chunks](https://github.com/aliddell/acquire-driver-zarr/assets/844464/653e4d82-363e-4e04-9a42-927b052fb6e7) |
+|:-------------------------------------------------------------------------------------------------------------:|
+|            A collection of frames, divided into tiles. A single chunk has been highlighted in red.            |
 
 You can specify the width and height, in pixels, of each tile, and if your frame size has more than one plane, you can
 specify the number of planes you want per tile as well.
@@ -120,3 +127,5 @@ Then the sequence of levels will have dimensions 1920 x 1080, 960 x 540, 480 x 2
 [Blosc]: https://github.com/Blosc/c-blosc
 
 [Blosc docs]: https://www.blosc.org/
+
+[Zarr v3]: https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html

src/3rdParty/cblosc/CMakeLists.txt

@@ -7,10 +7,10 @@ if(WIN32)
     set(libd "${pwd}/lib/win64/libblosc-debug.lib")
 elseif(APPLE)
     set(lib "${pwd}/lib/osx/libblosc.a")
-    set(libd ${lib})
+    set(libd "${pwd}/lib/osx/libblosc-debug.a")
 elseif(LINUX)
     set(lib "${pwd}/lib/linux-amd64/libblosc.a")
-    set(libd ${lib})
+    set(libd "${pwd}/lib/linux-amd64/libblosc-debug.a")
 endif()
 
 message(STATUS "C-Blosc: ${pwd}")

src/3rdParty/cblosc/lib/osx/README.md

@@ -1,6 +1,6 @@
 The library here was built as follows.
 
-Against v1.21.4 (2c2f9bd) of the c-blosc source hosted at:
+Against v1.21.5 (d306135) of the c-blosc source hosted at:
 https://github.com/Blosc/c-blosc
 
 It's a universal binary compiled for `arm64` and `x86_64`.

src/CMakeLists.txt

@@ -1,26 +1,28 @@
 add_subdirectory(3rdParty)
 
-if(NOT TARGET acquire-core-logger)
-        add_subdirectory(acquire-core-libs)
-endif()
+if (NOT TARGET acquire-core-logger)
+    add_subdirectory(acquire-core-libs)
+endif ()
 
 set(tgt acquire-driver-zarr)
 add_library(${tgt} MODULE
         prelude.h
-        tiled.frame.hh
-        tiled.frame.cpp
-        chunk.writer.hh
-        chunk.writer.cpp
-        frame.scaler.hh
-        frame.scaler.cpp
+        common.hh
+        common.cpp
+        writers/writer.hh
+        writers/writer.cpp
+        writers/chunk.writer.hh
+        writers/chunk.writer.cpp
+        writers/shard.writer.hh
+        writers/shard.writer.cpp
+        writers/blosc.compressor.hh
+        writers/blosc.compressor.cpp
         zarr.hh
         zarr.cpp
-        zarr.encoder.hh
-        zarr.encoder.cpp
-        zarr.raw.hh
-        zarr.raw.cpp
-        zarr.blosc.hh
-        zarr.blosc.cpp
+        zarr.v2.hh
+        zarr.v2.cpp
+        zarr.v3.hh
+        zarr.v3.cpp
         zarr.driver.c
 )
 target_enable_simd(${tgt})