Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EIA 176 wide tables - follow-up fixes #3978

Merged
merged 20 commits into from
Nov 30, 2024
Merged

EIA 176 wide tables - follow-up fixes #3978

merged 20 commits into from
Nov 30, 2024

Conversation

jdangerx
Copy link
Member

@jdangerx jdangerx commented Nov 28, 2024

Overview

Fixes a few things from the original PR (#3590):

  • the _core_eia176__data multi-asset now has named Outputs - not sure how Dagster was associating these correctly before
  • made all the assets in the _core_eia176_ namespace so that it's clear we don't persist them yet
  • asset checks now succeed: work around some "expected" diffs

My changes on top of the original PR are all in this commit

TODO in future follow-ups:

  • refactor so we only use the aggregate data to cross-check the company data - it doesn't seem that useful to have both the aggregated & disaggregated tables around. Especially if the aggregated data has a bunch of incorrect data floating around in it.
  • add the tables to the database, by adding them to db schema and hooking them up to the pudl_io_manager.
  • rename the columns to something more useful than the alphanumeric soup they are now
  • Update EIA-176 data source documentation

Documentation

Make sure to update relevant aspects of the documentation.

Tasks

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Copy link
Member

@zaneselvans zaneselvans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Materialized fine locally. I did notice that there's no 2023 data for EIA-176 and then found catalyst-cooperative/pudl-archiver#484.

Just one tiny naming thing -- I don't think we've ever applied the "not ready for primetime" underscore prefix to the asset group names. I get the analogy, but it feels like an extra complication that we might forget to undo, when the assets in the group are finally ready.

@@ -64,6 +64,7 @@


core_module_groups = {
"_core_eia176": [pudl.transform.eia176],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I know we haven't used the underscore prefix for any of the other asset groups. I think just having the prefix on the assets themselves is probably sufficient to indicate that they are not yet ready for use? Then the asset group will show up in a more expected place in the list of groups, alphabetically.

@zaneselvans zaneselvans self-requested a review November 30, 2024 18:40
@@ -64,6 +64,7 @@


core_module_groups = {
"core_eia176": [pudl.transform.eia176],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went ahead and made this 1-character change in order to get this merged in before I leave.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(and also re-materialized the eia176 assets locally just to make sure nothing weird happened)

@zaneselvans zaneselvans added the eia176 Issues related to the EIA Form 176 natural gas supply and disposition dataset. label Nov 30, 2024
@zaneselvans zaneselvans added this pull request to the merge queue Nov 30, 2024
Merged via the queue into main with commit 7490fd9 Nov 30, 2024
17 checks passed
@zaneselvans zaneselvans deleted the eia176_wide_table branch November 30, 2024 20:14
@zaneselvans zaneselvans added the new-data Requests for integration of new data. label Nov 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
eia176 Issues related to the EIA Form 176 natural gas supply and disposition dataset. new-data Requests for integration of new data.
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants