Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce code duplication with funlib.persistence #206

Open
4 tasks
adjavon opened this issue Apr 16, 2024 · 3 comments
Open
4 tasks

Reduce code duplication with funlib.persistence #206

adjavon opened this issue Apr 16, 2024 · 3 comments

Comments

@adjavon
Copy link
Contributor

adjavon commented Apr 16, 2024

Many of the specifics of reading Zarr files are already handled in fun lib.persistence. It also already generalizes to other sources of array data.
Goal is to generalize the ZarrSource by making it use an Array object.

  • Change ZarrSource into an ArraySource that uses persistence in the background
  • Have the ArraySource read N5, numpy arrays, etc.
  • Deprecate H5LikeSource
  • Deprecate KLBSource
@pattonw
Copy link
Collaborator

pattonw commented Apr 18, 2024

There is a little ArraySource in the tests/helpers.py. Fairly simple to make with any gunpowder Array class that can be built from numpy arrays and I think most things that resemble numpy arrays? I haven't tested that.
It might be worth while to depricate the gunpowder Array class and just use the funlib.persistence Array class, but they are a bit different with the whole ArraySpec for gunpowder.

@adjavon
Copy link
Contributor Author

adjavon commented Apr 18, 2024

What we were thinking about was more specifically related to the out-of-memory arrays like zarr datasets (or numpy saved files, rather than arrays). So for things like ROI-based slicing etc.

This is more of a self-reminder for something we're thinking of working on next week which you should probably be a part of @pattonw! I'll reach out

@pattonw
Copy link
Collaborator

pattonw commented Aug 30, 2024

This is now available via the ArraySource. It takes a funlib.persistence.Array and an ArrayKey.
old HDF5LikeSource not yet depricated. I'll depricate in the next patch release and remove in the next minor release.
old HDF5LikeWrite nodes should still be depricated and replaced with an ArrayWrite node. TODO for next patch release

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants