Merge dev into main for 2023-11-09

[zaneselvans]

• First pass of integrating the monthly EIA923 data into the rest of the EIA data. This includes updating the package data to account for the 2023 year and updating the way to assign data maturities to 923 data. This also updates some of the expected row counts for the data. It should still fail on the gen_eia923 table because the row count was going down which doesn't seem right. There are also some failures related to check_date_freq as there are now less than 12 months expected in a given round of updates. Will handle those errors in another commit.
• remove breakpoint
• Add function to drop ytd records for annual tables
• Adjust monthly row expectations for gf and frc tables after dropping ytd values for annual tables
• Tweak the way we add data maturity to the eia923 monthly files and remove double returns from the drop_ytd_for_annual_tables function
• Litle updates: - Add a note about how the plants are getting dropped in the gen_eia923 output table and link to the issue.
• For now, comment out the checks that make sure we have the same years of EIA923 and EIA860 data. This is causing issues for the monthly EIA923 data that gets integrated ahead of any available 860 data. This might cause issues elsewhere which is why I haven't committed to fully deleting it yet.
• Update min max rows
• Add data_maturity field to harvested EIA tables so that we can drop ytd records from annual EIA tables
• Address PR comments: - Restructure the way that the data_maturity field is dropped from certain tables when merging multiple tables together that each have that field. Previously it was ad-hoc, now it just gets dropped in the denorm_by_plant function.
• Fix release note trailing whitespace error
• Update test_eia923_dependency function to make sure some 860 and 923 years overlap but don't need to be the same
• Only generate alphanumeric entity IDs in test - non-printable characters seem to break groupby. (Only generate alphanumeric entity IDs in test #2993)
• Set up Cloud SQL postgres database for dagster storage
• Copy dagster.yaml after DAGSTER_HOME is created
• Add proper quoting rules to DAGSTER_PG_PASSWORD secret
• Use max cpus for nightly builds and spin dagster-storage SQL instance up and down
• Create and delete Cloud SQL db during nightly builds
• Set PUDL_SETTINGS_YML to etl_full.yml and add git sha to Cloud SQL database name
• Add short github ref to database name
• Update DAGSTER_PG_DB with short git sha
• Update date range for nightly build links to include 2022
• Update 923 settings files to accomodate 2023 data and update settings tests so that they aren't dependent on having the same years of EIA923 and EIA860 data
• Fix calculating the report_date in demand_hourly_pa_ferc714
• Require non-null report_date in FERC 714 hourly demand table.
• Update date validation function to only look at instances where data_maturity is not ytd_incremental
• Remove Cloud SQL lifecycle management from gcp_pudl_etl.sh script
• Update data contributors, add zenodo role and doi field, update US copyright link
• Update to ZenodoDoi class, update to https
• Remove leftover string
• Switch regex strategy to sampling strategy to improve performance (Switch regex strategy to sampling strategy to improve performance #2998)
• add alembic schema changes for the recent constraint.
• only fix a reporting_frequency_code when the column exists
• Update responses requirement from <0.24,>=0.14 to >=0.14,<0.25
• Update pyarrow requirement from <14,>=13 to >=13,<15
• Update dagster-postgres requirement
• [pre-commit.ci] pre-commit autoupdate
• update tox and eia923 rows
• update excepted rows for no-fips id-ed respondents but keep annualized demand
• add report year validation test
• add minmax rows into validation test for chonky table
• idk exactly why the "nan"s began existing but this fixes it
• revert the replace of "nan" by stopping introducing them! plus some light clean up
• REALLY REALLY its a nullable string

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (bbd82ba) 88.6% compared to head (a2bdffa) 88.7%.
Report is 18 commits behind head on main.

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #3031   +/-   ##
=====================================
  Coverage   88.6%   88.7%           
=====================================
  Files         91      91           
  Lines      10991   11011   +20     
=====================================
+ Hits        9749    9769   +20     
  Misses      1242    1242           

Files	Coverage Δ
src/pudl/analysis/allocate_gen_fuel.py	91.3% <ø> (ø)
src/pudl/analysis/classify_plants_ferc1.py	92.5% <100.0%> (+<0.1%)	⬆️
src/pudl/extract/eia923.py	100.0% <100.0%> (ø)
src/pudl/extract/excel.py	96.3% <100.0%> (-0.5%)	⬇️
src/pudl/metadata/classes.py	86.5% <100.0%> (+<0.1%)	⬆️
src/pudl/metadata/constants.py	100.0% <ø> (ø)
src/pudl/metadata/fields.py	100.0% <ø> (ø)
src/pudl/metadata/resources/eia.py	100.0% <ø> (ø)
src/pudl/metadata/resources/eia923.py	100.0% <ø> (ø)
src/pudl/metadata/sources.py	100.0% <ø> (ø)
... and 9 more

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.