Vectorized chunk writes

[aliddell]

• Add a few virtual methods to ArrayWriter
  • data_root_() constructs the path to the location where chunk or shard files are currently going to be written
  • metadata_path_() constructs the path to the array metadata file
  • Remove version_()
  • Replace compress_buffers_() with compress_chunk_buffer_(), which compresses one buffer at a time
  • Make make_data_sinks_() and close_sinks_() pure virtual methods
• Rename SinkCreator methods to be explicit about which type of sink (S3 or filesystem).
• Zarr V2: Enqueue each chunk to be written immediately after compression, rather than waiting until all chunks are compressed before writing them
• Zarr V3: Write chunks to shards using FileWriteGather (Windows) or pwritev (POSIX).

[jeskesen]

Compared to the old functionality, rollover_() would now be called if should_rollover==false but is_finalizing==true, whereas it wouldn't have before. I just want to make sure that is intended.

[aliddell]

Correct. I previously didn't want to call rollover_() when finalizing because it would uselessly increment a counter, but the alternative was to add another check on is_finalizing_ to call close_files_() -- but only when we had a ragged append dimension. This is simpler, and the increment doesn't matter.

[jeskesen]

Probably doesn't need to queue a new job?