Axis size error introduced by recent changes to the FMS infra layer in the MOM6 framework

[alex-huth]

Commit 9e48380 ("Decouple FMS infra from framework") is causing crashes when trying to apply sponges.

Traceback:

FATAL from PE   0: get_axis_data called with too small of an output data array for lon.

Image              PC                Routine            Line        Source             
MOM6_SIS2_Alex_be  0000000000B66C6B  mpp_mod_mp_mpp_er          79  mpp_util_mpi.inc
MOM6_SIS2_Alex_be  00000000006A5A63  mom_io_infra_mp_g         730  MOM_io_infra.F90
MOM6_SIS2_Alex_be  0000000000CAA01C  mom_interpolate_m         316  MOM_interpolate.F90
MOM6_SIS2_Alex_be  0000000000DA3C2D  mom_horizontal_re         752  MOM_horizontal_regridding.F90
MOM6_SIS2_Alex_be  0000000001147CFF  mom_ale_sponge_mp         992  MOM_ALE_sponge.F90
MOM6_SIS2_Alex_be  0000000000CD8365  mom_diabatic_driv        1839  MOM_diabatic_driver.F90
MOM6_SIS2_Alex_be  0000000000CC86FF  mom_diabatic_driv         410  MOM_diabatic_driver.F90
MOM6_SIS2_Alex_be  00000000011985CF  mom_mp_step_mom_t        1667  MOM.F90
MOM6_SIS2_Alex_be  0000000001188532  mom_mp_step_mom_         1002  MOM.F90
MOM6_SIS2_Alex_be  00000000018D9B11  ocean_model_mod_m         610  ocean_model_MOM.F90
MOM6_SIS2_Alex_be  00000000009D349A  full_coupler_mod_        2440  full_coupler_mod.F90
MOM6_SIS2_Alex_be  0000000001438E9D  MAIN__                    665  coupler_main.F90
MOM6_SIS2_Alex_be  000000000041195D  Unknown               Unknown  Unknown
libc.so.6          00007F94D4099E6C  Unknown               Unknown  Unknown
libc.so.6          00007F94D4099F35  __libc_start_main     Unknown  Unknown
MOM6_SIS2_Alex_be  0000000000411861  Unknown               Unknown  Unknown

I think that at line L730 in MOM_io_infra.F90, size(axis%ax_data) has the axis size in the data file, while size(axis_data) has the (smaller) axis size on the current PE only

[marshallward]

Do you know what the sizes of axis_data and axis%ax_data are for lon? Is there something that I can run?

[alex-huth]

Here is an experiment to reproduce the error: /gpfs/f5/gfdl_o/scratch/Alexander.Huth/mom6runs/amundsen_axis_issue. Output/error is in slurm.135644312.out and slurm.135644312.err respectively. Batch script to run is mom.sub. Build script for the executable is /ncrc/home1/Alexander.Huth/cryo-examples/ios_fms2_c5_sym_icepack_axis.sh.

On the root PE, size(axis_data) is 28 and size(axis%ax_data) is 600.

[marshallward]

Looks like this is the error:

MOM6/src/framework/MOM_interpolate.F90

Lines 307 to 322 in 7ee13cd

	if (present(axes)) then
	call get_external_field_info_infra(field, size=size, axes=axes_infra, &
	missing=missing)
	! TODO: Most of these methods were written to expect four dimensions.
	! I would not expect a generic field to be well-behaved, but I am unsure
	! how to validate such a field.
	do n=1,4
	! Convert axistype to axis_info
	allocate(ax_data(size(n)))
	call get_axis_data(axes_infra(n), axis_name, ax_data)
	call set_axis_info(axes(n), trim(axis_name), ax_data=ax_data)
	deallocate(ax_data)
	enddo
	else
	call get_external_field_info_infra(field, size=size, missing=missing)
	endif

ax_data (passed as axis_data to get_axis_data()) gets its size from get_external_field_info(), but this was initialized as a domain-decomposed field so the size it returns is also domain-decomposed. So this was the mistake.

I am testing a version of get_axis_data() which internally allocates axis_data:

allocate(axis_data, source=axis%ax_data)

but this will create a slightly different API between FMS1 and FMS2. Maybe there's a better way.

But the main mistake here is that I should have not relied on get_external_field_info() to get axis size.

[marshallward]

I have a possible fix on this branch:

https://github.com/marshallward/MOM6/tree/interp-get-axis-size

I added a new function (get_axis_size()) to fetch the axis size for pre-allocation. If this works for you then I can send it over to dev/gfdl.

[alex-huth]

@marshallward This seems to have fixed the issue -- thank you for the quick fix