You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For the current 2.1.8 Code/Manual_Workflows/(data load script specific to a version), I am actively working on loading all data.
The script is/has been very in-efficient in areas for many versions. Almost all parts are relatively quick, ie < 35 mins, and are easy to keep an eye on and validate its completion and results, easily and quickly.
However, the loading of the Hydrotables is brutal. It is run entirely in a Jupyter kernal loading 1000's of records and take 5 plus hours to complete. Even though the kernal stays running and the job continues running even after logout, it does not appear to complete updating the output panel, so I am unable to tell if it finished.
I recommend we move this to some sort of Step function system, possibly grouping them up by huc or something. We can likely use other tools like lambda or something.
Note: For learning purposes, I would like to the one to do this task but will need a lot of guidance when we get there in the new year.
The text was updated successfully, but these errors were encountered:
RobHanna-NOAA
changed the title
Improve performance with TI data load scripts for HydroTable data load
Improve performance with TI data load scripts for HydroTable load
Nov 22, 2024
For the current 2.1.8 Code/Manual_Workflows/(data load script specific to a version), I am actively working on loading all data.
The script is/has been very in-efficient in areas for many versions. Almost all parts are relatively quick, ie < 35 mins, and are easy to keep an eye on and validate its completion and results, easily and quickly.
However, the loading of the Hydrotables is brutal. It is run entirely in a Jupyter kernal loading 1000's of records and take 5 plus hours to complete. Even though the kernal stays running and the job continues running even after logout, it does not appear to complete updating the output panel, so I am unable to tell if it finished.
I recommend we move this to some sort of Step function system, possibly grouping them up by huc or something. We can likely use other tools like lambda or something.
Note: For learning purposes, I would like to the one to do this task but will need a lot of guidance when we get there in the new year.
The text was updated successfully, but these errors were encountered: