Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelize the creation of plant_parts_eia #2920

Open
katie-lamb opened this issue Oct 5, 2023 · 0 comments
Open

Parallelize the creation of plant_parts_eia #2920

katie-lamb opened this issue Oct 5, 2023 · 0 comments
Labels
ccai Tasks related to CCAI grant for entity matching ppe Plant Parts EIA (formerly the EIA plant parts list)

Comments

@katie-lamb
Copy link
Member

Now that plant_parts_eia is dagsterized and part of the ETL, it's a bit of a bummer that it takes so long to generate. Currently, there's a loop in the creation of the table here that loops through all the different parts and concatenates a bunch of dataframes together. This could be done concurrently instead of looping to speed things up.

@katie-lamb katie-lamb added ppe Plant Parts EIA (formerly the EIA plant parts list) ccai Tasks related to CCAI grant for entity matching labels Oct 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ccai Tasks related to CCAI grant for entity matching ppe Plant Parts EIA (formerly the EIA plant parts list)
Projects
Status: New
Development

No branches or pull requests

1 participant