Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proof of concept for GH bug #509 #2169

Closed
wants to merge 2 commits into from
Closed

Proof of concept for GH bug #509 #2169

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
24e8710
Proof of concept for GH bug #509. This particular field may or may no…
knordback Dec 30, 2022
5fa5356
Merge branch 'dev' into 509-poc
zaneselvans Jan 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions src/pudl/metadata/fields.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,10 @@
"description": "Average monthly coincident peak (CP) demand (for requirements purchases, and any transactions involving demand charges). Monthly CP demand is the metered demand during the hour (60-minute integration) in which the supplier's system reaches its monthly peak. In megawatts.",
"unit": "MW",
},
"combined_heat_power": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really a different field than associated_combined_heat_power or do they represent the same characteristic applied to generators vs. boilers?

"type": "boolean",
"description": "Indicates whether the generator is a combined heat and power system",
},
"conductor_size_and_material": {
"type": "string",
"description": "Size of transmission conductor and material of the transmission line.",
Expand Down
1 change: 1 addition & 0 deletions src/pudl/metadata/resources/eia923.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"fuel_mmbtu_per_unit",
"sulfur_content_pct",
"ash_content_pct",
"combined_heat_power",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is truly a new field that we were inappropriately dropping (and losing entirely) before, and we want to have it independently associated with the boilers (in addition to the associated_combined_heat_power field that is applied to generators) then this seems appropriate.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I think the point is that #509 is specific to generators. I hadn't been thinking about generators vs. boilers specifically. I'll do the analogous thing just for generators.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't sound right to me. #509 (the issue) is very generic, even if it provides an example from the generators.

We're doing dropping potentially informational columns across a bunch of the EIA table transforms, and would prefer to feed them into the harvesting process so they can be used to more appropriately set the boiler / generator / utility / plant / balancing authority attributes (static or annual).

Addressing #509 should not have any effect on the database schema -- if there are columns in the wrong place, or columns we have failed to integrate entirely, those should probably be their own separate issues. This issue should just stop dropping columns prior to the harvesting process, so they can be used within it.

],
"primary_key": [
"plant_id_eia",
Expand Down
9 changes: 7 additions & 2 deletions src/pudl/transform/eia923.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -759,7 +759,13 @@ def _aggregate_duplicate_boiler_fuel_keys(boiler_fuel_df: pd.DataFrame) -> pd.Da
relative_cols = ["ash_content_pct", "sulfur_content_pct", "fuel_mmbtu_per_unit"]
key_cols = ["boiler_id", "energy_source_code", "plant_id_eia", "report_date"]

expected_cols = set(quantity_cols + relative_cols + key_cols + ["prime_mover_code"])
expected_cols = set(
quantity_cols
+ relative_cols
+ key_cols
+ ["prime_mover_code"]
+ ["combined_heat_power"]
)
actual_cols = set(boiler_fuel_df.columns)
difference = actual_cols.symmetric_difference(expected_cols)

Expand Down Expand Up @@ -829,7 +835,6 @@ def boiler_fuel(eia923_dfs, eia923_transformed_dfs):
# Need to stop dropping fields that contain harvestable entity attributes.
# See https://github.com/catalyst-cooperative/pudl/issues/509
cols_to_drop = [
"combined_heat_power",
"plant_name_eia",
"operator_name",
"operator_id",
Expand Down