CICE does not restart correctly in wave/ice coupling

[mvertens]

The following field from CICE is not bfb on restart:

 iceImp_Si_floediam   (iceImp_nx,iceImp_ny,time)  t_index =      1     1
        311   138240  (   325,   359,     1) (     1,     1,     1) (    31,    66,     1) (    31,    66,     1)
              138240   1.674731125838850E+03   0.000000000000000E+00 4.3E+01  3.103339245842001E+01 1.5E-05  3.103339245842001E+01
              138240   1.674731125838850E+03   0.000000000000000E+00          7.371746970397571E+01          7.371746970397571E+01
              138240  (   325,   359,     1) (     1,     1,     1)
          avg abs field values:    3.306170191991240E+01    rms diff: 1.2E-01   avg rel diff(npos):  1.5E-05
                                   3.306208736543892E+01                        avg decimal digits(ndif):  6.9 worst:  0.2
 RMS iceImp_Si_floediam               1.1756E-01            NORMALIZED  3.5558E-03

The above was obtained as follows:

Create a case with the following command (modify for your own username and srcroot):

> ./create_newcase --case /cluster/home/mvertens/noresm/test_blom_ww3_debug2 --res T62_tn14_wtn14 --compset 2000_DATM%NYF_SLND_CICE_BLOM_DROF%NYF_SGLC_WW3DEV --project nn9560k --user-mods-dir /cluster/home/mvertens/src/noresm_develop_v6_base/cime_config/usermods_dir
s/wav_ice_coupling --run-unsupported --mach betzy

1. Run 1 - do a 30 step initial run - writing coupler and ww3 history files every time step and restart files only at the end of the first day

> ./xmlchange STOP_N=30
> ./xmlchange STOP_OPTION=nsteps
> ./xmlchange HIST_N=1
> ./xmlchange HIST_OPTION=nsteps
> ./xmlchange REST_N=1
> ./xmlchange REST_OPTION=ndays

Move the hi history files to a temporary directory

2. Run 2 - do a 6 time step restart run from the restart files produced in 1.
   > ./xmlchange CONTINUE_RUN=TRUE

Now compare the first history file after restart with the history file from the initial run. In my case this was:
cprnc test_blom_ww3_debug2.cpl.hi.0001-01-02-03600.nc debug_rest2/test_blom_ww3_debug2.cpl.hi.0001-01-02-03600.nc

Note that cprnc is in /cluster/shared/noresm/tools/cprnc/cprnc
You will load to do the following first
> module load netCDF/4.8.1-iompi-2021b

NOTE: You will need the code for the ww3 PR NorESMhub/WW3#8 in order to be able to duplicate these results.

[mvertens]

@JensBDebernard - I have assigned this PR to you. @Mats @anacarraso I believe that simulations that require wave/cice coupling cannot occur until this restart issue is resolved. I am happy to bring Dave Bailey in on discussions if that would help.

[mvertens]

@ dabail10 - It would be extremely helpful to get your input on this. I'm not sure what needs to be added to the cice6 restart file - but I suspect that would fix the issue.

[dabail10]

I don't know what version you have of CICE, but the FSD is still under development and may have some issues. I suggest you update to CICE Consortium main here:

https://github.com/CICE-Consortium/CICE

If there is still a restart issue then you should open an issue on the CICE Consortium website.

[mvertens]

@dabail10 - thanks. I confirmed that the problem still occurred with the latest version of CICE in ESCOMP. Thanks for the idea of looking at the consortium code.

[JensBDebernard]

I think I have found the reason behind the problem.

The wave fracturing of ice utilizes a random number generator
(components/cice/src/icepack/columnphysics/icepack_wavefracspec.F90)

When I manipulate the rand_array result to give the same result every time step, the floe-diameter field is constant after restart identical in the restart test described by Mariana.
This underpins the point of 'work in progress' when it comes to wave-ice interaction.

We could look for a wave fracture method not using random numbers, or introduce the random_seed functionality, storing the seeds to the restart file.

[mvertens]

@JensBDebernard - great work in finding this! For now I am wondering if storing the seed to the restart file is not the simples way to move forwards. What do you think?

[JensBDebernard]

@mvertens - I think the random_seed should be introduced sooner or later in any case to evaluate the impact of this function.
I know about another methods that we can implement and test, but that might require an different set of wave parameters.
In any way, I think the random_seed approach is the natural next step.

[dabail10]

I have linked this to an issue in the CICE Consortium. There was work planned to remove the random number generator here. Not sure if this has been resolved. What is wave_spec_type here?

[eclare108213]

The random number generator is what the original FSD developer used in her publications, and so we provide it. We also have other options which are intended only for testing (restarts in particular); they aren't expected to produce good results, physically. Alternative approaches would be most welcome!

[lettie-roach]

Hi all,

If wave_spec_type is not set to 'random', the code does not use a random phase. Physically this isn't ideal, but then recent work has suggested that this scheme is over-estimating wave fracture, so it probably doesn't hurt to switch to this while setting up new configurations.

Bruno Tremblay has been working on a new, improved wave fracture scheme. I hope to work with him to implement it in CICE over the next few months.

I'll post updates on CICE-Consortium#293 (comment) when available.

[mvertens]

@lettie-roach @eclare108213 @dbailey - thanks so much for your input. That is super helpful.
@JensBDebernard - how do you want to proceed for updating cice to get a new noresm snapshot for Making Waves?

[JensBDebernard]

Yes, thank you, this is very useful.

@mvertens I suggest that we put wave_spec_type = "constant" for the ice model for now (namelist-variable). This will ensure exact restart and some wave fraction. Then you have something to work with during summer, and we will reconsider the holiday.