Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance with TI data load scripts for HydroTable load #997

Open
RobHanna-NOAA opened this issue Nov 22, 2024 · 0 comments
Open
Assignees
Labels
enhancement New feature or request Request
Milestone

Comments

@RobHanna-NOAA
Copy link
Contributor

RobHanna-NOAA commented Nov 22, 2024

For the current 2.1.8 Code/Manual_Workflows/(data load script specific to a version), I am actively working on loading all data.

The script is/has been very in-efficient in areas for many versions. Almost all parts are relatively quick, ie < 35 mins, and are easy to keep an eye on and validate its completion and results, easily and quickly.

However, the loading of the Hydrotables is brutal. It is run entirely in a Jupyter kernal loading 1000's of records and take 5 plus hours to complete. Even though the kernal stays running and the job continues running even after logout, it does not appear to complete updating the output panel, so I am unable to tell if it finished.

I recommend we move this to some sort of Step function system, possibly grouping them up by huc or something. We can likely use other tools like lambda or something.

Note: For learning purposes, I would like to the one to do this task but will need a lot of guidance when we get there in the new year.

@RobHanna-NOAA RobHanna-NOAA added enhancement New feature or request Request labels Nov 22, 2024
@RobHanna-NOAA RobHanna-NOAA self-assigned this Nov 22, 2024
@RobHanna-NOAA RobHanna-NOAA added this to the V2.1.x milestone Nov 22, 2024
@RobHanna-NOAA RobHanna-NOAA changed the title Improve performance with TI data load scripts for HydroTable data load Improve performance with TI data load scripts for HydroTable load Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Request
Projects
None yet
Development

No branches or pull requests

1 participant