Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vectorized chunk writes #23

Draft
wants to merge 32 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
5f2795e
Add benchmark to test different chunk/shard/compression/storage confi…
aliddell Nov 20, 2024
1cf276e
Add ShardBuffer class to keep track of ready-to-write chunks.
aliddell Nov 21, 2024
48fe1a6
Increase number of runs/config in benchmark.
aliddell Nov 21, 2024
1d8c605
Add ChunkIndexBuffer to ZarrV2ArrayWriter.
aliddell Nov 21, 2024
ee1dac4
Merge remote-tracking branch 'upstream/main' into vector-writes
aliddell Nov 27, 2024
f0a97e8
wip: Vectorized file write (Windows).
aliddell Nov 27, 2024
6aadf65
Changes for Linux
aliddell Nov 27, 2024
97e93c1
Vectorized file write on OSX
aliddell Dec 2, 2024
d70ca8e
Test vectorized file write with multiple writes to the same file.
aliddell Dec 2, 2024
1f95b05
Minor edits
aliddell Dec 2, 2024
d5e4b37
Remove ArrayWriter::version_()
aliddell Dec 2, 2024
f0bf009
Do-nothing compress_and_flush_() method.
aliddell Dec 2, 2024
7559cbf
Compress right into flush (wip: Zarr V2 only).
aliddell Dec 3, 2024
d0bc1d3
wip
aliddell Dec 4, 2024
5717867
Use a vector of spans to avoid copies.
aliddell Dec 4, 2024
c3c1e1c
Merge branch 'vector-writes' into vectorized-chunk-writes
aliddell Dec 5, 2024
0f71dd0
Remove dead code.
aliddell Dec 5, 2024
ed89a98
Remove benchmark.
aliddell Dec 5, 2024
ee842cb
Merge branch 'vector-writes' into vectorized-chunk-writes
aliddell Dec 5, 2024
dfc3228
Merge remote-tracking branch 'upstream/main' into vectorized-chunk-wr…
aliddell Dec 6, 2024
dbed6e7
Update and test shard writer.
aliddell Dec 10, 2024
a3e2aca
Implement and test make_dirs.
aliddell Dec 10, 2024
5d6ce8d
Rename SinkCreator methods to be more explicit.
aliddell Dec 10, 2024
7d7d399
Make `ArrayWriter::make_data_sinks_()` a pure virtual method.
aliddell Dec 10, 2024
bc4ff3b
Add missing EXPECT error message to make-dirs.cpp.
aliddell Dec 11, 2024
97ede7c
Remove `ArrayWriter::flush_impl_()`
aliddell Dec 11, 2024
72508ce
Everything except ZarrV3 compressed to filesystem working again.
aliddell Dec 11, 2024
1fc703f
We did it.
aliddell Dec 13, 2024
99a553f
Fix ragged internal dim size check on OSX.
aliddell Dec 13, 2024
f1bb9e9
Merge remote-tracking branch 'upstream/main' into vectorized-chunk-wr…
aliddell Dec 13, 2024
b6fd05d
Respond to PR comments.
aliddell Dec 18, 2024
1c26a5c
Merge remote-tracking branch 'upstream/main' into vectorized-chunk-wr…
aliddell Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove benchmark.
aliddell committed Dec 5, 2024

Verified

This commit was signed with the committer’s verified signature.
aliddell Alan Liddell
commit ed89a98c9f9aa02e2881342cfd951ab29d70df7b
7 changes: 0 additions & 7 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -21,7 +21,6 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

option(BUILD_PYTHON "Build Python bindings" OFF)
option(BUILD_BENCHMARK "Build benchmarks" OFF)

if(CMAKE_PROJECT_NAME STREQUAL PROJECT_NAME)
include(CTest)
@@ -34,12 +33,6 @@ else ()
message(STATUS "Skipping test targets")
endif ()

if (BUILD_BENCHMARK)
add_subdirectory(benchmarks)
else ()
message(STATUS "Skipping benchmarks")
endif ()

if (${BUILD_PYTHON})
add_subdirectory(python)
else ()
18 changes: 0 additions & 18 deletions benchmarks/CMakeLists.txt

This file was deleted.

233 changes: 0 additions & 233 deletions benchmarks/benchmark.cpp

This file was deleted.