Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HDF5 loader #36

Open
adamslc opened this issue Oct 9, 2023 · 1 comment
Open

Add HDF5 loader #36

adamslc opened this issue Oct 9, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@adamslc
Copy link

adamslc commented Oct 9, 2023

Some complications that this loader might introduce:

  • a single .h5 file frequently corresponds to several Datasets, and it would be nice if the specification in Data.toml could refer directly to a single DataSet, but preserve the hierarchal relationship between multiple DataSets.
    • More generally, I frequently work with a directory full of .h5 files, and it would be nice to preserve the relationship between all of these files as well.
  • In HDF5, a dataset can have attributes, and it is important that these be accessible from DataToolkit
  • The HDF5.jl package is relatively heavyweight, and so it would be good to not take on an explicit dependency of this package.
@tecosaur tecosaur transferred this issue from tecosaur/DataToolkit.jl Oct 10, 2023
@tecosaur
Copy link
Owner

Thanks for the interest. A few things:

  • You'd have to either access the whole hdf5 object as usual, or create new data sets for the subdatasets (which is actually fairly easy)
  • What sort of relationship between .h5 files exists?
  • If we just use h5open you'd still be able to access the attributes
  • I don't know of any other appropriate packages besides HDF5, but don't worry about it being a bit big to have as a dependency, that problem's already been sorted. See https://tecosaur.github.io/DataToolkitDocs/base/stable/packages/ :)

@tecosaur tecosaur added the enhancement New feature or request label May 16, 2024
@tecosaur tecosaur transferred this issue from tecosaur/DataToolkitCommon.jl May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants