Should the EPACAMD-EIA Crosswalk have an output table?

[aesharpe]

Historically we have not given glue tables their own output table. This is because the output or "analysis" table generally represents the denormalized version of the tables in the pudl db and combines info from glue tables with other tables.

I don't really want to integrate CEMS ids into all EIA data or vice versa, so I think it makes most sense to just have an accessible crosswalk that you can merge with whatever data you want whenever you want. My question is, Should this be an output table?

I think this ties into larger conversations surrounding PUDL access. Are we going to encourage people to tap into the SQL db more often? Are we going to save and catalog (for intake) the normalized or denormalized versions of the tables? Are we going to treat the output tables as a caching mechanism more than a denormalization mechanism?

I want to make sure we aren't confusing users by giving them access to some tables this way and some tables that way.

[grgmiller]

I think that there should be an output table for the crosswalk, since this is a table that other data users may want to use independently (for example, we access it in our eGRID project, and it would be great to have a standardized version available from pudl).

Other ideas:

• Should the crosswalk just be scraped and transformed as part of the ETL, since it is updated by EPA on an ongoing basis, rather than living as a static table in glue (how do the glue tables get updated?).
• The crosswalk itself depends a bit on some manual crosswalks, but otherwise uses EIA and CMAD input files and takes a programmatic approach to crosswalking the data. In a way, this isn't that different than the bga table, its just that the code to do this lives in a different repository and not pudl (although long run, maybe it could be integrated into pudl (in Python!)).

[aesharpe]

Should the crosswalk just be scraped and transformed as part of the ETL, since it is updated by EPA on an ongoing basis, rather than living as a static table in glue (how do the glue tables get updated?).

Luckily this is what I've been up to in #1692! It's still a PR, but now the crosswalk is scraped, extracted, and has its own glue ETL module instead of being stored as a static file.

[aesharpe]

Ok @grgmiller I just tacked an output table onto that same PR! #1692 getting very close to done!! 🎉 That is, until the next crosswalk release... :)