cannot run om5 in debug mode because of division by zero in coupler

[nikizadehgfdl]

I get the following crash and traceback when trying to run om5 in debug mode.

forrtl: error (73): floating divide by zero                                                                             
Image              PC                Routine            Line        Source                                              
libpthread-2.31.s  000014F37935F910  Unknown               Unknown  Unknown                                             
fms_OM5_compile.x  0000000006926600  monin_obukhov_int         131  monin_obukhov_inter.inc                             
fms_OM5_compile.x  00000000057A6199  monin_obukhov_mod          50  monin_obukhov.inc                                   
fms_OM5_compile.x  000000000055C90F  surface_flux_mod_         386  surface_flux.F90                                    
fms_OM5_compile.x  000000000046CB3A  atm_land_ice_flux        1242  atm_land_ice_flux_exchange.F90                      
fms_OM5_compile.x  00000000005A4744  full_coupler_mod_        2006  full_coupler_mod.F90                                
fms_OM5_compile.x  0000000000412A32  MAIN__                    481  coupler_main.F90      

[nikizadehgfdl]

Other users have also seen this issue and suggeted: "coupler does not work when you trap division by zero. My workaround is to hack the Makefile to compile coupler without fpe and mom6 with fpe".

I tried the user suggestion by taking out "-fpe0 -ftrapuv" from compiler switches for coupler in making the debug exec and it worked, no more crash! (Taking out "-fpe0" by itself, does not seem to bypass the issue).

Anyone knows what is going on?

[bensonr]

@nikizadehgfdl - Based on old knowledge that may or may not still be valid for Intel, the -fpe flag used at any point in a compile or link phase flips a switch that applies to the entirety of the executable. Once that flag is seen, floating point exception trapping will occur. The use of the -ftrapuv on a per file basis adds in the ability to preset the heap or stack (I don't remember which) memory space with a specific value (malloc + set to 0xffffffffffffff) to aid in finding references to variables that don't have explicit values given to them.

[rem1776]

My understanding is -fpe0 turns on the floating point exceptions, while -ftrapuv is for catching uninitialized local variables in the code by setting them to odd values. If -fpe0 still works, then its likely an issue with unintialized values somewhere.

[nikizadehgfdl]

Thanks. Leaving -fpe0 in and removing just -ftrapuv cause the same crash.
So the trick to run in debug mode is to remove them both when compiling the coupler.

BTW, note that monin_obukhov_inter.inc is in FMS which compiles with both switches in debug mode! And line 131 is:

fm(i)     = log(zz(i)/z0(i))

where i goes over the exchange grid cells.

[J-Lentz]

I'm pretty sure this is the same problem that Justin Perket encountered previously: NOAA-GFDL/FMS#1256 (comment)

Is FMS being built with optimization in the OM5 debug build? If so, there are two options to fix the crash without losing FP exception trapping: build FMS without optimization (if the performance is tolerable), or add -fp-speculation=safe to the FMS build flags.

[JosephMouallem]

I encounter the same issue for the coupled SHiELD-MOM6 model as in here: NOAA-GFDL/FMS#1614
I ended up removing both -fpe0 -ftrapuv when compiling in debug mode to make it work. FMS is still compiled in repro mode.

[nikizadehgfdl]

By "debug mode" here I mean "fremake -t debug" where all components (inclduing FMS) are built with

-O0 -fpe0 -ftrapuv  -traceback -g  -check -check noarg_temp_created -check nopointer -warn -warn noerrors

@J-Lentz are you suggesting to add -fp-speculation=safe to the above without removing -fpe0 -ftrapuv?

[nikizadehgfdl]

Unfortunately adding either -fp-speculation=safe or -fp-speculation=strict to all componets compilation does not help with the issue, the model crashed the same way.

[J-Lentz]

By "debug mode" here I mean "fremake -t debug" where all components (inclduing FMS) are built with

-O0 -fpe0 -ftrapuv  -traceback -g  -check -check noarg_temp_created -check nopointer -warn -warn noerrors

@J-Lentz are you suggesting to add -fp-speculation=safe to the above without removing -fpe0 -ftrapuv?

Hmm, if FMS is being built with -O0, then this is probably not the issue that I'd assumed. The issue that I've seen in the past is due to speculative execution in monin_obukhov_mod, which should only occur when FMS is built with optimizations.