only fix a reporting_frequency_code when the column exists

[cmgosnell]

PR Overview

fixing weird "n" reporting_frequency_code encoder error.

alternatively, this filling in or frequency could happen in transform. but it would need to be applied in many of the tables' transforms.

still need to run the validation tests!

these are the validation errors I found. some are related to the EIA923m update (#2936). the 714 ones are related to (#2999).

Why why the FERC1 errors?!?!

#2936

FAILED test/validate/mcoe_test.py::test_minmax_rows_mcoe[eia_monthly-capacity_factor-5178892-433286] - ValueError: capacity_factor: found 5178828 rows, expected 5178892. Off by -0.001%, allowed margin of 0.000%
FAILED test/validate/mcoe_test.py::test_minmax_rows_mcoe[eia_monthly-mcoe-5179300-433320] - ValueError: mcoe: found 5179236 rows, expected 5179300. Off by -0.001%, allowed margin of 0.000%

WTFERC?

FAILED test/validate/fbp_ferc1_test.py::test_fbp_ferc1_mismatched_fuels[ferc1_annual] - AssertionError: Too many records (9.53%) have mismatched primary fuel types.
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-fbp_ferc1-26188] - ValueError: fbp_ferc1: found 25437 rows, expected 26188. Off by -2.868%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-fuel_ferc1-50039] - ValueError: fuel_ferc1: found 48887 rows, expected 50039. Off by -2.302%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-plant_in_service_ferc1-335750] - ValueError: plant_in_service_ferc1: found 315301 rows, expected 335750. Off by -6.091%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-plants_all_ferc1-56409] - ValueError: plants_all_ferc1: found 54535 rows, expected 56409. Off by -3.322%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-plants_hydro_ferc1-6979] - ValueError: plants_hydro_ferc1: found 6792 rows, expected 6979. Off by -2.679%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-plants_pumped_storage_ferc1-562] - ValueError: plants_pumped_storage_ferc1: found 544 rows, expected 562. Off by -3.203%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-plants_small_ferc1-16989] - ValueError: plants_small_ferc1: found 16269 rows, expected 16989. Off by -4.238%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-plants_steam_ferc1-31879] - ValueError: plants_steam_ferc1: found 30930 rows, expected 31879. Off by -2.977%, allowed margin of 0.000%
FAILED test/validate/ferc1_test.py::test_minmax_rows[ferc1_annual-purchased_power_ferc1-204720] - ValueError: purchased_power_ferc1: found 197829 rows, expected 204720. Off by -3.366%, allowed margin of 0.000%

#2999

FAILED test/validate/service_territory_test.py::test_minmax_rows[summarized_demand_ferc714-3195] - ValueError:  summarized_demand_ferc714: found 3105 rows, expected 3195. Off by -2.817%, allowed margin of 0.000%
FAILED test/validate/service_territory_test.py::test_minmax_rows[fipsified_respondents_ferc714-135627] - ValueError: fipsified_respondents_ferc714: found 135537 rows, expected 135627. Off by -0.066%, allowed margin of 0.000%

PR Checklist

• Merge the most recent version of the branch you are merging into (probably dev).
• All CI checks are passing. Run tests locally to debug failures
• Make sure you've included good docstrings.
• For major data coverage & analysis changes, run data validation tests
• Include unit tests for new functions and classes.
• Defensive data quality/sanity checks in analyses & data processing functions.
• Update the release notes and reference reference the PR and related issues.
• Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

[cmgosnell]

okay so what i've learned is that across the board we're currently loosing almost all of the FERC1 2021 records.

[cmgosnell]

okay @jdangerx I think i need your eyeballs on this.

it seems like whatever has changed in the xbrl extractor land is causing basically all of the 2021 data to go bye bye sad times.

[cmgosnell]

at this point i think we should bubble these FERC1 and FERC714 validation errors off into their own separate issues.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (aea60b9) 88.7% compared to head (5a96335) 88.7%.
Report is 1 commits behind head on dev.

Additional details and impacted files

@@           Coverage Diff           @@
##             dev   #3013     +/-   ##
=======================================
- Coverage   88.7%   88.7%   -0.1%     
=======================================
  Files         91      91             
  Lines      11009   11010      +1     
=======================================
  Hits        9768    9768             
- Misses      1241    1242      +1     

Files	Coverage Δ
src/pudl/extract/eia923.py	100.0% <100.0%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[cmgosnell]

@jdangerx perhaps ignore my previous comments.. it seems the nightly builds last night did not include the missing 2021 FERC1 issue. So this is almost certainly a local to me problem. Which is great for us and only low-key concerning for me :-)