Potential make_coupler_mosaic bug when using `--land_mosaic`

[JosephMouallem]

A strange behavior was encountered when using make_coupler_mosaic for a coarse C48 atmosphere and 1 degree ocean configuration. Some fields on the coupler level are not populated correctly with null values in certain locations leading to a model crash.
The issue occurs when adding --land_mosaic to the call. It works fine without --land_mosaic.
This behavior was observed on C6 using both serial and parallel versions of make_coupler_mosaic of fre-nctools-parallel/2024.04.

Below is an example:

1. Fails (with --land_mosaic)
   srun --ntasks=$npes make_coupler_mosaic_parallel --atmos_mosaic atmos_mosaic.nc --land_mosaic land_mosaic.nc --ocean_mosaic ocean_mosaic.nc --ocean_topog topog.nc --mosaic_name grid_spec --area_ratio_thresh 0.0001 --check --print_memory --verbose

data location: /gpfs/f6/bil-coastal-gfdl/proj-shared/Joseph.Mouallem/shiemom_pdata/GLOBAL/C48_1deg_mask_issue

2. Works (without --land_mosaic):
   srun --ntasks=$npes make_coupler_mosaic_parallel --atmos_mosaic atmos_mosaic.nc --ocean_mosaic ocean_mosaic.nc --ocean_topog topog.nc --mosaic_name grid_spec --area_ratio_thresh 0.0001 --check --print_memory --verbose

data location: /gpfs/f6/bil-coastal-gfdl/proj-shared/Joseph.Mouallem/shiemom_pdata/GLOBAL/C48_1deg_ok

For more info check this slide deck.

[nikizadehgfdl]

@mlee03 , have you ever tried passing --land_mosaic as well as --atmos_mosaic to make_coupler_mosaic?
It gives a big tiling error even when I pass essentially the same mosaic file to both arguments (see below), and the land mask gets wierd line features. Do you know what might be going on? Thanks.

        $(FRE_NCTOOLS)/make_solo_mosaic --num_tiles 6 --dir ./ --mosaic_name C48_mosaic --tile_file C48_grid.tile1.nc,C48_grid.tile2.nc,C48_grid.tile3.nc,C48_grid.tile4.nc,C48_grid.tile5.nc,C48_grid.tile6.nc
        #use same grid files and make another mosaic
        $(FRE_NCTOOLS)/make_solo_mosaic --num_tiles 6 --dir ./ --mosaic_name land_mosaic --tile_file C48_grid.tile1.nc,C48_grid.tile2.nc,C48_grid.tile3.nc,C48_grid.tile4.nc,C48_grid.tile5.nc,C48_grid.tile6.nc
       
/home/fms/local/opt/fre-nctools/2022.02/gfdl/bin/make_coupler_mosaic --atmos_mosaic C48_mosaic.nc **--land_mosaic land_mosaic.nc** --ocean_mosaic ocean_mosaic.nc --mosaic_name mosaic_c48_om4_SPEAR --ocean_topog ocean_topog.nc --verbose --check

[nikizadehgfdl]

@JosephMouallem , @bensonr I have a fix that addresses this issue. I plan to run more tests and revamp/refactor this make_coupler_mosaic tool further. In the mean while please see if this fix makes sense and you can try my build on PAN to test it:

/work/Niki.Zadeh/exchange_grid_toolset/tools/FRE-NCtools_main/builds/gfdlpan/bin/make_coupler_mosaic_parallel