[WIP] adding an option to read file from s3

[djarecka]

Recently I've been testing an option to access files directly from s3 using remfile. remfile provides a file-like object for reading a remote file over HTTP, optimized for use with h5py.

I've added a simple change to the code and a simple test that access a big file directly from s3 and is able to check the file size without reading entire content. I'd be happy to expand the tests and changes if the package maintainers say that are interested in the addition.

• Closes HDF5 Cloud Storage #634
• Tests added
• Release note added (or unnecessary)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c1afc3e) 85.70% compared to head (18ddcd0) 83.50%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1322      +/-   ##
==========================================
- Coverage   85.70%   83.50%   -2.20%     
==========================================
  Files          34       34              
  Lines        5450     5451       +1     
==========================================
- Hits         4671     4552     -119     
- Misses        779      899     +120     

Flag	Coverage Δ
gpu-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
anndata/_core/file_backing.py	90.32% <100.00%> (+0.10%)	⬆️

... and 8 files with indirect coverage changes

[flying-sheep]

Hi, I think enabling HDF5-on-S3 through such a small change is pretty cool. But I think it would make sense to make an issue first, to talk about support for other formats and how to tackle them.

[djarecka]

@flying-sheep - this is a bit related to #634, I've commented there at some point, but didn't get answer. I will open a fresh issue to get some new feedback

[Zethson]

Maybe @Koncopd has a comment here, too?

[Koncopd]

Yes, we use s3fs for this and it works without any problems and any changes in anndata code.

[djarecka]

Do you have integration of s3fs and anndata?

How can I use ad.read_h5ad with backed="r"?

[Koncopd]

@djarecka https://github.com/laminlabs/lamindb/blob/bf1c81e7ae8c177a7338d560521cc5b0c1cdc1aa/lamindb/dev/storage/file.py#L84
This loads an AnnData object directly into memory. fs there is an s3fs filesystem (or any fsspec filesystem actually).
you can also try with backed="r".
We also have fully backed functionality (not only for .X as in anndata) here https://github.com/laminlabs/lamindb/blob/bf1c81e7ae8c177a7338d560521cc5b0c1cdc1aa/lamindb/dev/storage/_backed_access.py#L609