Skip to content
This repository has been archived by the owner on Oct 1, 2024. It is now read-only.

Configurable compression #66

Closed
clbarnes opened this issue Jul 16, 2019 · 2 comments
Closed

Configurable compression #66

clbarnes opened this issue Jul 16, 2019 · 2 comments

Comments

@clbarnes
Copy link
Collaborator

As mentioned in #65.

Getting arbitrary compression options across the pyo3 interface seems like it could be a pain. Also, choosing an API may require a bit of thought:

The h5py way

Takes a compression: Optional[str]=None, and a compression_opts: Optional[Any]=None. For level-5 gzip, these would be "gzip" and 5. h5py doesn't support that many compression types, but n5 may want to support things which fairly complicated configurations, as the config is just stored as a JSON object. Zarr supports compression layers. Splitting across multiple arguments is a bit unfortunate.

the z5 way

**compression_opts are kwargs. This is nice because it means the type of each option can be consistent, but it means that you prevent yourself adding any other **kwargs in the future.

The strongly-typed way

As rust (and presumably java) does: take a single object whose type describes the type of compression, and whose members configure that compression. This would be discoverable and hopefully easy to map to rust (namedtuple -> struct). But it means duplicating some of the compression-configuration machinery from rust into python.

The sketchy MVP way

Whenever you create a dataset (writing its attributes.json), python then re-opens the attributes file and dumps an arbitrary compression JSON object in there, hopefully before any data is written.

@aschampion
Copy link
Collaborator

rust-n5's compression is serde deserializable, so a sketchy MVP way already exists just passing JSON through ({"type": "gzip", "level": 5}).

@clbarnes
Copy link
Collaborator Author

clbarnes commented Sep 18, 2019

Released the h5py way in v1.0.0 (implemented using the sketchy MVP way)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants