Implement FB-LTS Time-Stepping Scheme #6224
Conversation
|
@mark-petersen please review. |
|
Thanks. I was out but I'm back now. The history has some pulls from master. I'm working with Jeremy to clean it up so the history is less confusing. |
jeremy-lilly
force-pushed
the
mpaso/fb-lts
branch
from
a2a3548
to
5263736
Compare
|
Just squashed all the commits for FB-LTS and rebased onto the current head of master. Thanks for the help with that @mark-petersen! |
|
History looks good. Thanks for your work on that! Compiles on chicoma and chrysalis in stand-alone for both gnu and intel. This does not compile with E3SM because there is a new file to compile and it was not added to cmake. We will not add the flags to the E3SM namelist scripts because LTS is not used in E3SM coupled mode yet. |
|
Added fblts to cmake. E3SM now compiles. You can test with: but alter it for your machine. |
|
Passes nightly stand-alone suite BFB against master branch-point on chicoma with gnu optimized, with default flags (lts off). Also passes nightly on chicoma with gnu debug with default flags. |
mark-petersen
added
BFB
There was a problem hiding this comment.
This is adding to a current MPAS-Ocean standalone feature, as Local Time Stepping is not used in E3SM testing or mainline simulations. My testing showed that this PR is BFB with master using default flags (LTS off).
I also ran the two additional tests provided by @jeremy-lilly in E3SM-Project/polaris#187 and MPAS-Dev/compass#771. These show that the new fb-lts is working correctly.
Thanks @jeremy-lilly for your attention to detail and additional testing on this PR!
mark-petersen
removed
the
Stealth
xylar
added
the
MPAS-Ocean standalone
|
Closing and opening to force a rerun of circle-ci |
|
@rljacob, are you able to restart circleci? Or do you know why it's refusing to run? |
|
The circleci build error is:
The test duration is 0 seconds, so it did not run. It looks like a problem with the testing configuration. |
|
ignore the circle ci fail. |
| @@ -202,7 +202,7 @@ | |||
| /> | |||
| <nml_option name="config_time_integrator" type="character" default_value="split_explicit_ab2" | |||
| description="Time integration method." | |||
| possible_values="'split_explicit', 'RK4', 'unsplit_explicit', 'split_implicit', 'LTS', 'split_explicit_ab2'" | |||
| possible_values="'split_explicit', 'RK4', 'unsplit_explicit', 'split_implicit', 'LTS', 'FB_LTS', 'split_explicit_ab2'" | |||
There was a problem hiding this comment.
I think we need a comment in these possible_values about which are useable by E3SM and which are currently limited to the standalone model. The Registry file gets used to make bld files in E3SM and this having these options there could cause confusion
There was a problem hiding this comment.
Good point. I added that information to the description, because I didn't want to add text to the list in possible_values.
|
@jeremy-lilly and @mark-petersen -- I'll need to run the scripts to (semi-) automatically make the E3SM bld files consistent with the changes to Registry. I can either point you at the required updates or push them to this branch, though I think Jeremy will have to give me permission to do so |
|
@jonbob I've just invited you as a collaborator on the repo. Feel free to push the changes yourself, or to point me in the right direction, whatever will be easier. Thanks! |
|
Thanks @jeremy-lilly -- invitation is already accepted. I can show you what we do if you're interested, but otherwise I'll handle it and push back to your fork/branch |
|
I'd be interested! |
|
@jonbob, could you put these instructions on confluence? I would also be interested in a refresher because it's been too long. I tried running generate_e3sm_namelist_files.py and it still seems like there are tons of manual edits to be done that require expertise in what E3SM uses and what it doesn't. But perhaps I'm just doing it wrong. |
|
@jeremy-lilly and @xylar -- there is a confluence page at Updating MPAS Framework and Components in ACME, but it is woefully out-of date (even the name). I'll update it as I go, but also happy to sit down and walk through it |
|
Running the scripts picked up more changes than I expected, but nothing critical. I'll start testing this PR and hope to merge it today or tomorrow |
Implement FB-LTS Time-Stepping Scheme This PR implements a new local time-stepping (LTS) scheme in MPAS-Ocean for single layer configurations. This new scheme, called FB-LTS, is similar to to the LTS3 scheme already implemented by MPAS-O [1], but boasts an increased maximal CFL number. This results in FB-LTS being able to use time-steps between ~2 times larger than time-steps for LTS3 while costing the same number of tendency evaluations. The FB-LTS scheme is based on the forward-backward, three-stage, second- order Runge-Kutta (FB-RK(3,2)) scheme. The existing LTS3 scheme is based on a three-stage, third-order, strong stability preserving Runge-Kutta scheme (SSPRK3), so we expect FB-LTS to outperform LTS3 in admittable time-step by factors of between ~1.6-2.2. In preliminary performance experiments, FB-LTS provides a 8x speedup over RK4 (and 2x over LTS3) in the hurricane Sandy test case, on a variable resolution mesh going to resolutions as high as 2km in Delaware Bay. [BFB] - only for mpaso-standalone
|
passes:
merged to next |
|
merged to master |
|
Awesome! Thanks so much for the help all! |
This merge updates the E3SM-Project submodule from [93e511d](https://github.com/E3SM-Project/E3SM/tree/93e511d) to [31e0924](https://github.com/E3SM-Project/E3SM/tree/31e0924). This update includes the following MPAS-Ocean and MPAS-Frameworks PRs (check mark indicates bit-for-bit with previous PR in the list): - [ ] (ocn) E3SM-Project/E3SM#6256 - [ ] (ocn) E3SM-Project/E3SM#6224 - [ ] (ocn) E3SM-Project/E3SM#6270 - [ ] (ocn) E3SM-Project/E3SM#6293 - [ ] (ocn) E3SM-Project/E3SM#6321 - [ ] (ocn) E3SM-Project/E3SM#6262 - [ ] (ocn) E3SM-Project/E3SM#6300 - [ ] (ocn) E3SM-Project/E3SM#6334 - [ ] (ocn) E3SM-Project/E3SM#6371 - [ ] (ocn) E3SM-Project/E3SM#6288
This PR implements a new local time-stepping (LTS) scheme in MPAS-Ocean for single layer configurations. This new scheme, called FB-LTS, is similar to to the LTS3 scheme already implemented by MPAS-O [1], but boasts an increased maximal CFL number. This results in FB-LTS being able to use time-steps between ~2 times larger than time-steps for LTS3 while costing the same number of tendency evaluations.
The FB-LTS scheme is based on the forward-backward, three-stage, second-order Runge-Kutta (FB-RK(3,2)) scheme from [2]. The existing LTS3 scheme is based on a three-stage, third-order, strong stability preserving Runge-Kutta scheme (SSPRK3), so we expect FB-LTS to outperform LTS3 in admittable time-step by factors of between ~1.6-2.2 [2].
In preliminary performance experiments, FB-LTS provides a 8x speedup over RK4 (and 2x over LTS3) in the hurricane Sandy test case described by [3], on a variable resolution mesh going to resolutions as high as 2km in Delaware Bay.
Also, see corresponding PRs to Polaris and Compass with relevant automated tests:
[1]: Capodaglio and Petersen
[2]: Lilly, Engwirda, Capodaglio, et al.
[3]: Lilly, Capodaglio, Petersen, et al.
[BFB] - only for mpaso-standalone