Failing forecast jobs with new HAFS config

[jak5808]

Hello,

I'm currently running the most recent version of HAFS on Orion and I'm having trouble with my forecast jobs. For both experiments I'm running, the 2nd cycle forecast job fails immediately, and I can't figure out why. For reference, I'll link the system and one of the experiment's working directory below:

/work/noaa/aoml-hafsda/knisely/HAFS_Feb_22

/work2/noaa/aoml-hafsda/knisely/HAFS_Feb_22_h2db_01online_6km/2020081106/00L

I was concerned changes I made to the system were to blame (running on a 6 km analysis grid and with online satbias turned on), but I instituted the same changes to a version of HAFS I grabbed back in Jan and it's working fine. I'm wondering if this issue has been encountered by other groups running this newest version or if it's unique to my system due to the options I changed.

Thanks,
Joey

[evankalina]

Hi @jak5808,

Thanks for posting this discussion thread. It looks like this is a warm-start run with HYCOM ocean coupling turned on. There is an error in your /work2/noaa/aoml-hafsda/knisely/HAFS_Feb_22_h2db_01online_6km/2020081106/00L/forecast/err.forecast:

fcst_initialize ERROR: date_init /= date_init_res
date_init = 2020 8 11 6 0 0
date_init_res = 2020 8 11 0 0 0

It is happening when the fcst_initialize subroutine in ./sorc/hafs_forecast.fd/FV3/module_fcst_grid_comp.F90 opens the ./forecast/INPUT/coupler.res file and sees that date_init and date_init_res are not the same, at which point the code is purposefully written to exit.

One thing that strikes me as strange is that in /work2/noaa/aoml-hafsda/knisely/HAFS_Feb_22_h2db_01online_6km/com/2020081106/00L/storm1.holdvars.txt, it says export warm_start_opt=0, which would indicate a cold start. This might be an indication that the model has been misconfigured. Can you compare against your older version of HAFS and see what the value of warm_start_opt is for the second cycle and beyond?

Cheers,
evan

[evankalina]

./sorc/hafs_forecast.fd/FV3/module_fcst_grid_comp.F90 was changed between your Jan. 22 and Feb. 22 code checkouts. In particular, PR #450 in NOAA-EMC/fv3atm breaks the DA capability for HAFS by adding this check to module_fcst_grid_comp.F90 that is run on the coupler.res file:

    if (.not. ALL(date_init.EQ.date_init_res)) then
      write(0,'(A)')      'fcst_initialize ERROR: date_init /= date_init_res'
      write(0,'(A,6(I4))')'                       date_init     = ', date_init
      write(0,'(A,6(I4))')'                       date_init_res = ', date_init_res
      call ESMF_Finalize(endflag=ESMF_END_ABORT)
    endif

Since date_init and date_init_res are not the same, the new code will error out. In the short term, you could try to simply comment out this block, recompile, and see if the second cycle will run? @BinLiu-NOAA, do you have any advice on how to fix this? Should we try to change the dates printed to the coupler.res file, or should we modify module_fcst_grid_comp.F90?

[BinLiu-NOAA]

@evankalina, Thanks for looking into this!

Also, in terms of fixing this, I would prefer to not changing the source code. Instead we can modify coupler.res when running the warm-start forecast. For example, in the feature/hafs_vida branch, here is how we change the coupler.res with correct time info.
https://github.com/hafs-community/HAFS/blob/feature/hafs_vida/scripts/exhafs_forecast.sh#L1314

Thanks!

Bin

[jak5808]

Hi Bin,

Thanks for the fix. I'll give it a try and report back with results.

Cheers,
Joey

[jak5808]

Hi again,

This fix failed. I realized afterwards it's because the HAFS system doesn't use the variable "run_init," which is a part of the if function in that fix. Bin, what does this variable do, and would it be acceptable to simply remove this if function so that the script always performs the sed?

Cheers,
Joey

[BinLiu-NOAA]

Hi @jak5808, You definitely need to adapt/change and make it working for the develop branch version exhafs_forecast.sh. The feature/hafs_vida branch is ahead of HAFS develop branch, for vortex initialization and data assimilation developments. But, for your failed case here, you can just change your script to update the coupler.res with a correct/consistent model initial time stamp.