Instability with the latest version of model/nh_util.F90

[tanyasmirnova]

The version of the nh_util.F90 that was committed by Jun Wang in October 2020 (https://github.com/NOAA-EMC/GFDL_atmos_cubed_sphere/pull/41/files) is changed back to the old version in the latest version of FV3. This old version of nh_utils.F90 together with the default value of dz_min=2. are causing some instability crashes in our tests at GSL. It appears that the sensitivity is especially large to the value of dz_min.

[junwang-noaa]

@tanyasmirnova Please see the PR#54 where Kate made the dz_min a namelist variable. Did you try dz_min=6 ( the hardcoded number in PR#41 )?

[tanyasmirnova]

@junwang-noaa Thank you for your response. When I discovered that dz_min is moved to the namelist, I used the value dz_min=6. that you recommended to use in October 2020, and just this change fixed some crash cases. When in addition to dz_min=6. I used your code from PR#41, all crash cases ran successfully. Therefore, I think that the combination of these two changes gives the most stability.

[junwang-noaa]

@XiaqiongZhou-NOAA Do you have any recommendations for tanyasmirnova ?

[XiaqiongZhou-NOAA]

@tanyasmirnova @yangfanglin
Glad this fix is working.
https://github.com/NOAA-EMC/GFDL_atmos_cubed_sphere/pull/41/files also included the way to adjust delz by borrowing geopotential height from the lower interface when delz<dz_min while the original one is to borrow from the upper interface. This change is reversed back to the original in PR#54 as GFDL has different consideration. In PR#54 , you can try psm_bc=1 to see if this also fix your all crashed cases. We think the dz_min fix is a little artificial but psm_bc can really solve the instability.
By the way, what kind of configurations/cases are you running?

[tanyasmirnova]

@XiaqiongZhou-NOAA Thank you for your suggestions. My tests are global with 127 levels and GSL physics suite.
If I set the psm_bc=1, should I go back to dz_min=2 value and the original nh_utils.F90? Thank you!

[XiaqiongZhou-NOAA]

Yes, we assume that psm_bc=1 can avoid the issue of extreme small delz at the model lowest level, thus avoid the model crash. dz_min=2 is as default in PR #54 @tanyasmirnova

[tanyasmirnova]

@XiaqiongZhou-NOAA The crashed case ran successfully with psm_bc=1. Thank you very much for this clue.

[tanyasmirnova]

@XiaqiongZhou-NOAA Could you, please, give us some advice on the use of fv_sg_adj in C768 resolution: 450 or 1800 or something else? Also, will you recommend setting inline_q to true for the global runs so that nort_tr=2 could be used?

[XiaqiongZhou-NOAA]

It seems GSL is discussing about fv_sg_adj et al parameters. For C768, EMC is using fv_sg_adj=450. Both GFDL and EMC seems not using inline_q, but It is better to confirm with GFDL.

[tanyasmirnova]

@XiaqiongZhou-NOAA Thank you!

[junwang-noaa]

@tanyasmirnova @XiaqiongZhou-NOAA Is this issue resolved?

[XiaqiongZhou-NOAA]

I think this issue can be closed.

[junwang-noaa]

Thanks, Kate. Closing issue.