Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate Eia923 Q2 2024 Data #3768

Merged
merged 7 commits into from
Aug 12, 2024
Merged

Integrate Eia923 Q2 2024 Data #3768

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b4d92b7
Add DOI for new 923
aesharpe Aug 6, 2024
fd6c53f
Update release notes
aesharpe Aug 6, 2024
d900ac5
Update 923 package data
aesharpe Aug 6, 2024
a0d739c
Map unmapped plants and utilities
aesharpe Aug 7, 2024
c8e4fde
update minmax row validation test
aesharpe Aug 7, 2024
522a94b
Add description of duplicate rows to docs and docstrings
aesharpe Aug 9, 2024
5cbae58
Merge branch 'main' into eia923-q2-24
zaneselvans Aug 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/release_notes.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,8 @@ EIA 860
EIA 923
~~~~~~~
* Added EIA 923 early release data from 2023. See :issue:`3719` and PR :pr:`3721`.
* Added EIA 923 monthly data through May as part of the Q2 quarterly release. See
:issue:`3760` and :pr:`3768`.

FERC 714
~~~~~~~~
Expand Down
20 changes: 10 additions & 10 deletions src/pudl/package_data/eia923/file_map.csv
View file
Open in desktop

Large diffs are not rendered by default.

Binary file modified src/pudl/package_data/glue/pudl_id_mapping.xlsx
View file
Open in desktop
Binary file not shown.
33 changes: 33 additions & 0 deletions src/pudl/package_data/glue/utility_id_pudl.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16342,3 +16342,36 @@ utility_id_pudl,utility_id_ferc1,utility_name_ferc1,utility_id_eia,utility_name_
16384,,,66290,NSF Energy One LLC
16385,,,66291,Portage Solar Plant
16386,,,66292,Desert Willow Energy Storage
16387,,,66317,"Kola Energy Storage, LLC"
16388,,,66336,"Wild Plains Wind Project, LLC"
16389,,,66352,SMT Ironman BESS LLC
16390,,,66345,"Sebree Solar, LLC"
16391,,,66354,"Anole Energy Storage, LLC"
16392,,,66351,Citadel BESS LLC
16393,,,66348,"Silver State South Storage, LLC"
16394,,,65860,"Madison Fields Solar Project, LLC"
16395,,,66346,"Silver Peak Solar, LLC"
16396,,,66314,JGT2 Energy LLC
16397,,,66318,"Zeta Solar, LLC"
16398,,,66334,Twin Lakes Solar LLC
16399,,,66319,"Heartwood Solar, LLC"
16400,,,66360,Reliability Design & Development LLC
16401,,,66320,"White Tail Solar, LLC"
16402,,,66350,Wigeon Whistle BESS LLC
16403,,,66338,Al Pastor BESS LLC
16404,,,66316,"Northumberland Solar I, LLC"
16405,,,66331,"Birch Creek Power, LLC"
16406,,,66347,"Placid Solar II, LLC"
16407,,,66335,REV Renewables LLC
16408,,,66294,"NSF Torrey Site 2, LLC"
16409,,,66300,NY CDG Genesee 1 LLC
16410,,,66321,NY CDG Montgomery 1 LLC
16411,,,66293,"NSF Torrey Site 3, LLC"
16412,,,66295,"NSF Torrey Site 1, LLC"
16413,,,66301,NY CDG Genesee 4 LLC
16414,,,66342,"Catalyze Joliet 1101 Cherry Hill Road Microgrid, LLC"
16415,,,66305,"Rio Vista Executive Boat & RV Storage, LLC"
16416,,,66304,PFMD LL Baltimore LLC
16417,,,66303,PFMD LL Jessup LLC
16418,,,66306,Town Of Cary
16419,,,66343,"Catalyze Rochelle Wiscold Drive Microgrid, LLC"
2 changes: 1 addition & 1 deletion src/pudl/workspace/datastore.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ class ZenodoDoiSettings(BaseSettings):
eia860: ZenodoDoi = "10.5281/zenodo.11662381"
eia860m: ZenodoDoi = "10.5281/zenodo.11110602"
eia861: ZenodoDoi = "10.5281/zenodo.10204708"
eia923: ZenodoDoi = "10.5281/zenodo.12656894"
eia923: ZenodoDoi = "10.5281/zenodo.12721286"
eia930: ZenodoDoi = "10.5281/zenodo.10840078"
eiawater: ZenodoDoi = "10.5281/zenodo.10806016"
eiaaeo: ZenodoDoi = "10.5281/zenodo.10838488"
Expand Down
14 changes: 7 additions & 7 deletions test/validate/eia_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,17 @@ def test_no_null_cols_eia(pudl_out_eia, live_dbs, cols, df_name):
@pytest.mark.parametrize(
"df_name,raw_rows,monthly_rows,annual_rows",
[
("bf_eia923", 1_642_829, 1_642_829, 135_980),
("bf_eia923", 1_642_806, 1_642_806, 135_980),
("bga_eia860", 153_487, 153_487, 153_487),
("boil_eia860", 89_051, 89_051, 89_051),
("boil_eia860", 89_050, 89_050, 89_050),
Comment on lines -49 to +51
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you have a sense of why these went down? the bf in particular here seems wrong because there should be a few more months of data in there for the raw and monthly tables

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new months of 923 data show up as data inputs into existing columns. Once we hit January there will be rows (transformed from the raw table columns) for every month, regardless of whether there is data in them. Row fluctuations are more likely due to retroactive data changes which can lead to positive and negative changes.

I dug a little deeper in to the bf table and uncovered some unexpected cruft that explains some of the row count changes. See my comment below!

("frc_eia923", 673_343, 274_479, 26_709),
("gen_eia923", None, 5_494_932, 459_711),
("gens_eia860", 590_881, 590_881, 590_881),
("gf_eia923", 3_064_042, 3_064_042, 260_842),
("gens_eia860", 591_256, 591_256, 591_256),
("gf_eia923", 3_064_045, 3_064_045, 260_842),
Comment on lines -54 to +55
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would also assume that this gf rows would go up by a few months of data

("own_eia860", 95_104, 95_104, 95_104),
("plants_eia860", 215_884, 215_884, 215_884),
("pu_eia860", 214_965, 214_965, 214_965),
("utils_eia860", 147_877, 147_877, 147_877),
("plants_eia860", 216_206, 216_206, 216_206),
("pu_eia860", 215_288, 215_288, 215_288),
("utils_eia860", 147_922, 147_922, 147_922),
("emissions_control_equipment_eia860", 62_102, 62_102, 62_102),
("denorm_emissions_control_equipment_eia860", 62_102, 62_102, 62_102),
("boiler_emissions_control_equipment_assn_eia860", 83_977, 83_977, 83_977),
Expand Down