PLASIM fails with T31 resolution and parallel mode

[ValerioLembo]

Plasim is unstable after a few years when running in parallel mode (possibly even in sequential mode).

• With 8 processors

The atmospheric diagnostics seem ok, but LSG does not seem to print out anything, as it can be seen in the *.DIAG containing in the Dropbox folder below.

• With 16 processors

"Classic" instabiity arises, already in the first simulated year. Run goes on but nothing meaningful is written (see Dropbox folder);

Link to PLASIM folder containing namelist, bash scripts and diagnostics

[jhardenberg]

Hi, this looks serious and I will give a look. Did you try in sequential mode to confirm if it is a general T31 issue ?

[ValerioLembo]

@jhardenberg I have just finished a 10yr run in sequential mode. So far, it does not seem to have any issue...

[ValerioLembo]

Hi Jost,

I was wondering if you managed to have a look at this issue with T31 resolution. Apparently, the new compiler at G100 (CINECA) does not seem to improve the situation.

I am trying to bring up again that idea of the multiresolution step forcing experiments and T31 is key in bridging the gap between the reasonably stable T21 and T42 resolutions.

I played around with the MPSTEP but it is not of much help. I would like to play around with the ocean diffusion, but I did not find any info about how to do that...

[jhardenberg]

Where do thos " WARNING: abs(zeta) > du(1) detected at " messages in the log come from?

[jhardenberg]

So basically SSH becomes larger than the first ocean layer ... hmmm

[jhardenberg]

Can we somehow really exclude that this problem appears also in serial then? So it is something which only occurs in parallel ? Is a small problema ccumulating over time or does it explode 'suddenly' ?

[ValerioLembo]

Hi Jost,

from what I can see, it does not seem to accumulate some sort of problem and then exploding. It looks as if it suddendly crashes because of some numerical error.

Not sure we can rule out that the sequential run might crash at some point. I ran it for 10 years and it was ok, but it took a while before completing the task, that is why I did not proceed further.

Here is a new DIAG file that shows the model crashing a few months after the start of the run.

The run was compiled with NOCEAN=1, NLSG=1 and NICE=1 on CINECA's G100 machine with 8 processors using gfortran.

most_test_t31_p8_DIAG.0001.txt