Extract 714 xbrl

[aesharpe]

Overview

closes #3813
closes #3814
closes #3815

What problem does this address?

We don't currently extract XBRL tables for the FERC 714 data. This code adds the capability to do so.

What did you change?

pudl/extract/ferc714.py

• Add _csv suffix to csv-specific extraction functions/variables.
• Rename csv-specific encoding dictionary to FERC714_CSV_ENCODING
• Add TABLE_NAME_MAP_FERC714 dictionary
• Add a function called create_raw_ferc714_xbrl_assets to reference the raw xbrl tables in the FERC 714 SQLite db and call it.
• Add a function called raw_ferc714_xbrl__metadata_json to make an asset for the metadata table

pudl/io_managers.py

• Upgrade the FercXBRLSQLiteIOManager to accommodate more than just ferc1
• Add a ferc714_xbrl_sqlite_io_manager function

pudl/settings.py

• Add xbrl_years functino to the Ferc714Settings class to be able to access XBRL years.
• Remove comments about not processing the XBRL data.

pudl/metadata/sources.py

• Update the FERC714 working partitions.
• Remove comments about not processing XBRL data.

Testing

How did you make sure this worked? How can a reviewer verify this?

Load each of the raw assets via and make sure they contain the expected years:

defs.load_asset_value("raw_ferc714_xbrl__table_name")

To-do list

Beta Give feedback

1. Update the release notes: reference the PR and related issues.
2. Run make pytest-coverage locally to ensure that the merge queue will accept your PR.
3. Review the PR yourself and call out any questions or issues you have
4. For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
5. For bigger ETL or data changes run the full ETL locally and then run the data validations using make pytest-validate.
6. Alternatively, run the build-deploy-pudl GitHub Action manually.

[aesharpe]

Unit test failures have something to do with this snipit I changed in io_managers.py:

ferc_settings = context.resources.dataset_settings.get_datasets()[
    self.db_name.replace("_xbrl", "")
]

The pytest test/unit/io_managers_test.py::test_ferc_xbrl_sqlite_io_manager_dedupes function does not like this.

UPDATE:

I fixed this by changing this snipit to:

ferc_settings = getattr(
    context.resources.dataset_settings, self.db_name.replace("_xbrl", "")
)

[cmgosnell]

two requests for changes in comments, plus the settings yml files need to updated! but those seem like minor tweaks overall this is looking great imo and I was able to materialize these xbrl raw assets locally

[cmgosnell]

two blocking needs here:

to change
years = ", ".join(map(str, ferc714_settings.years))
to
years = ", ".join(map(str, ferc714_settings.csv_years))

in _extract_raw_ferc714_csv

and again to add the new years in the settings yml files.

[aesharpe]

and again to add the new years in the settings yml files.

The reason I didn't do this is because I think it will break the transforms and I didn't want to merge it into main that way.

[cmgosnell]

The reason I didn't do this is because I think it will break the transforms and I didn't want to merge it into main that way.

hm this is a good point... but I think in this context because the only existing transforms are based on the csv extracted tables they will be fine. but still i hadn't put it in context that this was only the extraction so i will renege on this suggestion.

[zaneselvans]

I think the integration test failures are probably because the tests don't know they need to run the FERC-714 XBRL to SQLite conversion before the main PUDL ETL can happen. You can follow the pattern that's used for the FERC Form 1 XBRL to SQLite conversion in test/conftest.py. These are the analogous fixtures:

@pytest.fixture(scope="session", name="ferc1_engine_xbrl")
def ferc1_xbrl_sql_engine(ferc1_xbrl_extract, dataset_settings_config) -> sa.Engine:
    """Grab a connection to the FERC Form 1 DB clone."""
    context = build_init_resource_context(
        resources={"dataset_settings": dataset_settings_config}
    )
    return ferc1_xbrl_sqlite_io_manager(context).engine


@pytest.fixture(scope="session", name="ferc1_xbrl_taxonomy_metadata")
def ferc1_xbrl_taxonomy_metadata(ferc1_engine_xbrl: sa.Engine):
    """Read the FERC 1 XBRL taxonomy metadata from JSON."""
    result = materialize_to_memory([raw_ferc1_xbrl__metadata_json])
    assert result.success

    return result.output_for_node("raw_ferc1_xbrl__metadata_json")

And then the PUDL IO Manager has a declared dependency on the FERC1 XBRL engine:

@pytest.fixture(scope="session")
def pudl_io_manager(
    ferc1_engine_dbf: sa.Engine,  # Implicit dependency
    ferc1_engine_xbrl: sa.Engine,  # Implicit dependency
    live_dbs: bool,
    pudl_datastore_config,
    dataset_settings_config,
    request,
) -> PudlMixedFormatIOManager:

[aesharpe]

@zaneselvans thank you for that suggestion! I just had to add one more fixture for ferc714_xbrl_extract and it was good to go.

[aesharpe]

mmm getting a unit test failure seemingly related to timeseries. Is timeseries cleaning part of the extraction or transformation phase @zaneselvans?

        for method in "tubal", "tnn":
            # Impute null values
            imputed0 = s.impute(mask=mask, method=method, rho0=1, maxiter=1)
            imputed = s.impute(mask=mask, method=method, rho0=1, maxiter=10)
            # Deviations between original and imputed values
            fit0 = s.summarize_imputed(imputed0, mask)
            fit = s.summarize_imputed(imputed, mask)
            # Mean MAPE (mean absolute percent error) is converging
>           assert fit["mape"].mean() < fit0["mape"].mean()
E           assert 0.030191300830775987 < 0.024084802339886784
E            +  where 0.030191300830775987 = mean()
E            +    where mean = 0    0.002769\n1    0.001304\n2    0.011850\n3    0.002106\n4    0.008777\n5    0.007833\n6    0.135690\n7    0.001992\n8    0.118109\n9    0.011482\nName: mape, dtype: float64.mean
E            +  and   0.024084802339886784 = mean()
E            +    where mean = 0    0.010499\n1    0.011460\n2    0.010525\n3    0.005897\n4    0.007852\n5    0.015753\n6    0.054820\n7    0.022598\n8    0.045421\n9    0.056023\nName: mape, dtype: float64.mean

test/unit/analysis/timeseries_cleaning_test.py:109: AssertionError

FAILED test/unit/analysis/timeseries_cleaning_test.py::test_flags_and_imputes_anomalies[5248964137-8991153078] - assert 0.030191300830775987 < 0.024084802339886784
 +  where 0.030191300830775987 = mean()
 +    where mean = 0    0.002769\n1    0.001304\n2    0.011850\n3    0.002106\n4    0.008777\n5    0.007833\n6    0.135690\n7    0.001992\n8    0.118109\n9    0.011482\nName: mape, dtype: float64.mean
 +  and   0.024084802339886784 = mean()
 +    where mean = 0    0.010499\n1    0.011460\n2    0.010525\n3    0.005897\n4    0.007852\n5    0.015753\n6    0.054820\n7    0.022598\n8    0.045421\n9    0.056023\nName: mape, dtype: float64.mean
===== 1 failed, 1645 passed, 1 skipped, 9 xfailed, 328 warnings in 40.74s ======
make: *** [Makefile:118: pytest-unit] Error 1

[zaneselvans]

Oh, the timeseries issue is probably a rare stochastic failure -- it happens once in a blue moon, probably having something to do with the seed being used for the random number generator. If you re-run the unit tests I suspect they'll pass just fine.

[aesharpe]

Oh, the timeseries issue is probably a rare stochastic failure -- it happens once in a blue moon, probably having something to do with the seed being used for the random number generator. If you re-run the unit tests I suspect they'll pass just fine.

That is wild and also strangely comforting. I'm glad I didn't try and preemptively figure this out...haha

[zaneselvans]

I spent a bunch of time at some point hunting down all the RNG initializations and tried to give them fixed seeds in the tests so we always get the same outputs, but I must have missed one somewhere, and it bites us every so often. It just didn't seem like it was worth continuing the hunt because it's so infrequent.

[zaneselvans]

Especially in the unit tests, whenever there's something like this that seems totally out of left field and unrelated to what's being changed my first instinct is just to re-run the tests to see if it's real. Doesn't take much time or effort, and often it works.