My Bering Sea coupled ice-ocean runs don't reproduce

[kshedstrom]

My Bering Sea runs now get different stats result for each run, it seems. The differences creep in through " u/v WindStr_[xy] before SIS_C_dynamics" and then "str_d/str_t/str_s in SIS_C_dynamics".

This is followed by pages of stuff like:

f[xy]ic in SIS_C_dynamics redundant v-comps  0.0000E+00  2.3488E-24 differ by ...

This didn't used to be the case, but it has been nagging me for over a month now.

[Hallberg-NOAA]

Obviously something unfortunate has crept into the code somewhere. What versions of MOM6 and SIS2 are you using, and do you have the versions of the MOM6 and SIS2 code that did reproduce across successive runs?

[kshedstrom]

I went back to some older codes and had to edit my MOM_input to get things to reproduce. Still, the current code with the updated MOM_input gets me back to the trouble again. The change I made to MOM_input is to go back to the original OBC where the entire domain has open boundaries, including all of the land masks.

The weird thing is my Arctic does reproduce from one run to the next and I've tried to make the Bering as close to it as possible!

[kshedstrom]

It wasn't SIS2 or MOM6 versions changing, it was the change to FMS2 that introduced the oddity. With FMS2 I got two consecutive runs to match, but not the third, just random. When compiled for debugging, things do reproduce. I noted at the top of the thread that the diffs are coming in via the wind stress.

Note that going to FMS2 implies new versions of the coupler, atmos_null, land_null, and maybe ice_params as well.

[marshallward]

AFAIK we are still using FMS1 interpolation operations in the FMS2 infra layer, so I'd say its more likely to be coming from the other components.

[kshedstrom]

If I add a print statement (I know, I know) to set_wind_stresses_C, I recover the old FMS1 answer. This is with gfortran 8.3.0.

gfortran 11.3.0 compiling with FMS1 gives this:

//import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS1/mpp/mpp_parameter.F90:120:42:

  120 |   integer(LONG_KIND), parameter :: DOMAIN_ID_BASE=Z'0000000100000000' ! Workaround for 64bit init problem
      |                                          1
Error: BOZ literal constant at (1) is neither a data-stmt-constant nor an actual argument to INT, REAL, DBLE, or CMPLX intrinsic function [see ‘-fno-allow-invalid-boz’]
//import/c1/AKWATERS/kate/ESMG/ESMG-configs/src/FMS1/mpp/mpp_parameter.F90:52:49:

   52 |   public :: AGRID, GLOBAL, CYCLIC, DOMAIN_ID_BASE, CENTER, CORNER
      |                                                 1
Error: Symbol ‘domain_id_base’ at (1) has no IMPLICIT type
make[1]: *** [Makefile:133: mpp_parameter.o] Error 1

[marshallward]

Add -fallow-invalid-boz to your compiler flags, older FMS needs this with the newer GFortran compilers. Unfortunately I think your problem runs deeper.