Dagster Housekeeping

[zaneselvans]

Description

It's been ~1 year since we've worked intensively on orchestration stuff, and Dagster has evolved since then. There are deprecation and design changes that we probably need to accommodate as the project continues to grow in complexity. Is there anything we want to bring in from #2386? Or can that be closed?

Tasks

Beta Give feedback

1. Remove pudl_etl and ferc_to_sqlite commands in favor of dagster job execute #3161
   0 of 4
   cli dagster +2
2. Switch from using the deprecated SourceAsset to AssetSpec #3977
   0 of 2
   dagster +1
3. Remove "experimental" warning from Parquet IO Manager
4. Look at using Dagster's tags and hierarchical key prefixes to better organize our DAG
5. Switch to using Postgres instead of SQLite for local development.
6. Make asset checks run during integration tests #3928
   0 of 4
   dagster data-validation testing +3
7. Add a Dagster RetryPolicy to PUDL's ETL #3913
   dagster good-first-issue nightly-builds +3
8. Investigate whether we can use Dagster's built-in partitioning to split up big jobs like EPA CEMS more simply.
9. Create a sensor that checks foreign keys when a pudl.etl job succeeds #2468
   0 of 2
   dagster inframundo sqlite +3
10. Use test-only $DAGSTER_HOME and default IOManager in tests #2470
    dagster testing +2