Integration of nextsim.dg module in NEDAS workflow #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
myying
commented
Dec 4, 2024
•
myying
commented
- added a JobSubmitter class to handle different ways to run jobs; replacing the job_submit_cmd approach (@yumengch could you help test this?)
- moved nextsimdg restart and forcing file logic from top-level config file to its own default.yml config file. Adding a namelist() function.
- moved perturb.py under models/nextsim/dg, keeping it as a nextsimdg specific perturbation scheme, along with the standalone scheme in scripts/perturb.py
- added the missing bits in nextsim/dg/model.py methods, to integrate with core assimilate algorithm
…ing yml file. The perturbation follows the original HYCOM approach.
…perturb function;
…ction needs attention
… time units instead of giving it explicitly.
Adding random perturbation to forcing - an initial implementation
…g job arrays for the forecast phase
…ll perturbation variance for cycled testing;updating model configuration files for each cycle; put generated restart file to next cycle;
saved the old copy of config.py moved perturb/randoim_perturb.py to ./perturb.py temporarily, will try to merge with utils/random_perturb.py later output_dir is no longer needed, makedir function using os.makedirs but catching FileExistsError from race conditions, adding nens to the job_opts for batch mode generate_init_ensemble.py is removed and the function is merged into preprocess.py as time==time_start the restart_dir is where the initial ensemble will be obtained run_exp.py renamed into run_expt.py and made more clean
replacing the job_submit_cmd approach in the config file # Changes not staged for commit:
implemented OAR/SLURM submitters updated nextsim.dg.model.run to use the submitter
previously I have the perturb settings under the model_def section, defined for each model; but now the perturb section is made stand alone. The reason doing so is that I can now run perturbation scheme involving multiple model components. I'm keeping the model-specific perturb settings so that the code is still compatible. Note that model.preprocess method will run the model-specific perturb settings while the scripts/perturb.py will run the standalone perturb settings. Moving the perturb.py inside nextsim.dg module as well prepare_forcing and prepare_restart are included in the preprocess step
combined prepare_restart and prepare_forcing into preprocess since nextsim.dg config_file is holding the files and perturb options, they are initialized directly in model class, instead of passing through kwargs everytime.
-initialize grid inside __init__, can we define model grid parameters in default.yml? -include restart forcing modules read_var and write_var in model class -in preprocess, there is a bit issue with copying previous cycle restart files to the current cycle. In config file files:restart:format, there seems to be no ensemble member index?
since only ensemble forecast gets to use this job array functionality, moving it out of job_submit section, so that other scripts like assimilate.py and perturb.py will not get accidently assigned use_job_array=True. Note; if not specified, use_job_array is defaulted to False
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.