In many of our older EIA transformation functions, we preemptively drop columns from the tables that are being processed, in order to produce normalized tables. However, many of these columns contain information about the entities (plants, generators, utilities) that should be integrated into the entity harvesting and resolution process, which happens after the transform step.

Discarded Columns

• Check whether the column name is defined in pudl.metadata.fields
• If it is not defined but does correspond to an existing column, change the name in the appropriate column_map.csv under src/pudl/package_data/{data source}/ so that it matches the DB schema.
• If the column does not correspond to any existing defined field, it may be appropriate to discard it. E.g. total_fuel_consumption_mmbtu is an annual total of monthly values that are retained, and so we don't need it.
• If the column corresponds to a defined field (either before or after the name has been fixed) then retain it and debug any issues that keeping it around results in later on in the transform process. It should make it to the harvesting step and go into the process of informing plant/generator/boiler/utility attributes.

EIA-860

pudl.transform.eia860.ownership()

• None

pudl.transform.eia860.generators()

• None

pudl.transform.eia860.plants()

• None

pudl.transform.eia860.utilities()

• None

EIA-923

pudl.transform.eia923.plants()

• None

pudl.transform.eia923.generation_fuel()

• combined_heat_power
• plant_name_eia
• operator_name (probably utility_name_eia)
• operator_id (probably utility_id_eia)
• plant_state
• census_region
• nerc_region
• naics_code
• fuel_unit (should probably be dropped, since unit is implied by fuel type)
• total_fuel_consumption_quantity (annual total?)
• electric_fuel_consumption_quantity (annual total?)
• total_fuel_consumption_mmbtu (annual total?)
• elec_fuel_consumption_mmbtu (annual total?)
• net_generation_megawatthours (annual total?)
• early_release

pudl.transform.eia923.boiler_fuel()

This one may give you trouble. See #1847 and #1836.

• combined_heat_power
• plant_name_eia
• operator_name (probably utility_name_eia)
• operator_id (probably utility_id_eia)
• plant_state
• census_region
• nerc_region
• naics_code
• fuel_unit (should probably be dropped, since unit is implied by fuel type)
• total_fuel_consumption_quantity (annual total?)
• balancing_authority_code_eia
• early_release
• reporting_frequency_code
• data_maturity (WE add this field in the extraction... getting dropped b/c of aggregations. See enable non-data columns in aggregated boiler_fuel_eia923 table #1847)

pudl.transform.eia923.generation()

• combined_heat_power
• plant_name_eia
• operator_name (probably utility_name_eia)
• operator_id (probably utility_id_eia)
• plant_state
• census_region
• nerc_region
• naics_code
• early_release

pudl.transform.eia923.coalmine()

• None -- we really do just want the very small set of columns retained here, as we're stripping them out to create a new table, normalizing the Fuel Receipts & Costs table.

pudl.transform.eia923.fuel_receipts_costs()

• plant_name_eia
• plant_state
• operator_name (probably utility_name_eia)
• operator_id (probably utility_id_eia)
• mine_id_msha (should be dropped)
• mine_type_code (should be dropped)
• state (of the mine?)
• county_id_fips (of the mine?)
• state_id_fips (of the mine?)
• mine_name (should be dropped)
• regulated (mine or plant?)
• early_release