Working with remote data

[LucaMarconato]

Working with remote data is currently only partially supported. Here is a collections of related issues and pull requests that constitute a good point to get started from.

Any user request/feedback from users is highly welcomed! 😊

• Support for consolidated remote zarr #278
• Support remote private storage by consistent use of substore #442

[vjcitn]

What is meant by "consolidated" here in #278 ?

[LucaMarconato]

Hi @vjcitn, it refers to putting all the metadata of a Zarr store into a single file at the root level. This is done for two reasons: 1) reduce the number of requests to the remote data; 2) actually be able to discover which data is available in a store for those file systems for which ls is not available, such as how it's often the case for S3 object storages.

You can find more details and info in the docs from the Zarr-Python package.