Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve GridLAB-D data parsing for datastore #126

Open
trevorhardy opened this issue Oct 13, 2023 · 0 comments
Open

Improve GridLAB-D data parsing for datastore #126

trevorhardy opened this issue Oct 13, 2023 · 0 comments

Comments

@trevorhardy
Copy link
Collaborator

Though HDF5 files can have arbitrary schemas, since we use GridLAB-D a lot its probably worth setting up a dedicated parsing path that totally re-works the DataFrame before giving it to the user. The name of the files produced by GridLAB-D are hard-coded so we should be able to easily detect when an HDF5 file is of that format. Features we should probably include:

  • Concatenating data from all days in the file into one table. Data is currently split up by day into unique tables "index1", "index2, ...
  • Making unique tables for each object in the table. Right now all objects are aggregated into one table which creates non-unique values in the table (all of the objects have a row with a given timestamp). Separating out each object into a table allows for indexing which is a good thing.
  • Index each table by the timestamp
  • Define data types for each column in the table. These are known as the output from GridLAB-D that creates this HDF5 is static (takes a source code change to adjust).

This will result in a lot of unique tables BUT they will all be easy to work with.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant