Remove FMS from UFS MPAS Atmosphere + More MPAS infrastructure in UFS

[dustinswales]

Description

This PR removes all FMS dependencies in the MPAS enabled UFS.

For logging and error handling, we now use the MPAS native logging functionality.

ESMF has replaced FMS for clock management. Most of the FMS functionality is included with ESMF, so this was straightforward.

With the FMS removal, we had to create some new infrastructure to provide MPAS configurability. This was necessary since we relied on FMS for runtime configurability.
A new routine, ufs_mpas_read_stream_lists, allows for runtime control over which fields are included in the various MPAS output streams (e.g., output, restart, diag, etc...). This provides the same functionality of standalone MPAS.

Initial tracer/constituent handling. This replaces the FMS tracer_tables. This may be revamped in the future to handle constituents differently.

New, non-FMS, handling of the atmospheric namelist.

Other things (non-FMS related) included here:

Added new interval to write restart files at different interval than output files. These are controlled by the UFS model_configure file.

Adopted shared UFS tool to control restart, and output, file frequency, shr_is_restart_fh_mod. This is used by other UFS components NUOPC caps. Thanks @NickSzapiro-NOAA

Output/restart file format now matches with standalone MPAS-A. Timestamp in file name was HH:MM:SS, changed to HH.MM.SS

Issue(s) addressed

• fixes UWM #3181

Testing

The MPAS output file format has changed, which results in error during the comparison step.

[NickSzapiro-NOAA]

Hi @dustinswales . As you're making changes around restart_fh, may I ask if you would be okay for MPAS in UFSATM also using restart_fh via UFS configuration and share code like some other components (ufs-community/ufs-weather-model#2419)?

The restart times are independent of settings for output

[dustinswales]

@NickSzapiro-NOAA FYI

[NickSzapiro-NOAA]

I wouldn't mix outputfh with restart_fh @dustinswales

[dustinswales]

Thanks. This is a bug.

[dustinswales]

Hi @dustinswales . As you're making changes around restart_fh, may I ask if you would be okay for MPAS in UFSATM also using restart_fh via UFS configuration and share code like some other components (ufs-community/ufs-weather-model#2419)?

The restart times are independent of settings for output

@NickSzapiro-NOAA I actually just added the restart_fh piece today.

I generalized some routines in the ATM NUOPC Cap that were used to compute output_fh to compute output_fh or restart_fh. Both output_fh and restart_fh are still part of the UFS configurations (i.e. model_configure file).

Is there common code to compute output_fh sitting somewhere that I should use instead?

[dustinswales]

@NickSzapiro-NOAA I think I understand now.
I will try replacing my recent changes to the atm nuopc cap to use shr_is_restart_fh_mod.F90 instead of the homebaked thing I added.

[NickSzapiro-NOAA]

Thanks @dustinswales . Please let me know if I can help

output_fh is just for ATM. We don't have shared output triggers but could and maybe should...

[dustinswales]

@NickSzapiro-NOAA ufs-community/ufs-weather-model#3211

[DusanJovic-NOAA]

@NickSzapiro-NOAA I think I understand now. I will try replacing my recent changes to the atm nuopc cap to use shr_is_restart_fh_mod.F90 instead of the homebaked thing I added.

Will this change (using shr_is_restart_fh_mod.F90 from CDEPS) make the code in this repository depend on the code from CDEPS repo or wherever else it is located? It it is, then I do not like that. Yet another build dependency.

[NickSzapiro-NOAA]

Will this change (using shr_is_restart_fh_mod.F90 from CDEPS) make the code in this repository depend on the code from CDEPS repo or wherever else it is located?

It's ufs-weather-model code @DusanJovic-NOAA , currently at https://github.com/ufs-community/ufs-weather-model/tree/develop/CDEPS-interface/ufs/cdeps_share for lack of current share code

It's a follow-up to #978 , that ATM should also accept triggering restarts via restart_fh like other components

[NickSzapiro-NOAA]

Continuing, the ESMF team was open to adding some support for this (https://github.com/orgs/esmf-org/discussions/291#discussioncomment-11102390) through the UFS driver's clock that is passed to the subcomponents

But neither ESMF nor global-workflow changes were going to be ready for GFS.v17. The shared restart_fh is the current working solution

We can follow up at infrastructure meeting. For now, I don't think separate/duplicate restart_fh code in each component is the way

[DusanJovic-NOAA]

OK. I see now. The 'CDEPS' shared code is not actually in CDEPS repository but in ufs-wm repository in CDEPS-interface directory, which is confusing. I looked at that 'restart' module file, and see nothing CDEPS specific, it looks like just a utility subroutine that depends only on the ESMF module. So keeping that file in CDEPS-interface is definitely not the ideal place. Maybe we should move it to a top-level ufs (in some ufs_utils, or ufs_shared directory).

But, regardless of the file location it still introduces a build dependency. How will that impact, for example, building the ufsatm code in a CI workflow? The CI workflow will need access to that file somehow. Am I missing something here?

[NickSzapiro-NOAA]

I agree. I think it's mainly a tension between separate components and a cohesive, unified coupled model.

UFSATM is particular as it's not shared with other models (like CESM, E3SM, SPEAR,...), afaik. Arguably,

• Does it have a standalone way of running? Then it's natural to checkout UFSATM within ufs-weather-model as part of its CI too. That would avoid some "customizations" to include stochastic_physics in CI too

• Model restarts are so fundamental/required that a unified way of coordinating them outweighs component particularities and is worth the dependencies

[dustinswales]

Continuing, the ESMF team was open to adding some support for this (https://github.com/orgs/esmf-org/discussions/291#discussioncomment-11102390) through the UFS driver's clock that is passed to the subcomponents

But neither ESMF nor global-workflow changes were going to be ready for GFS.v17. The shared restart_fh is the current working solution

We can follow up at infrastructure meeting. For now, I don't think separate/duplicate restart_fh code in each component is the way

All we need is a simple utility to create an array of ESMF Times given restart_fh and a start time. No?

In UFS-MPAS we use this array of ESMF times during integration to trigger writing output. I assume this is similar, if not identical, to how other components use it?

If this will ultimately be a ESMF feature, I agree with @DusanJovic-NOAA to keep this current functionality in the atmosphere, rather than introducing cross-component dependencies.
This is with the goal of eventually using ESMF to create the array of restart(output/da/etc..) times within the atmosphere's NUOPC cap.

[NickSzapiro-NOAA]

I think history output and restart writes are separate and independent concerns and can be handled similarly or not.

At least currently, history seems more component-native as it's tied to customizable fields, instantaneous vs. averages vs. extrema, internal timestepping,...

Restart writes are more checkpoints so that each component and coupled model can restart and reproduce bit-for-bit (as operational requirement)

All we need is a simple utility to create an array of ESMF Times given restart_fh and a start time. No?

Yes, so let's have a shared restart_fh utility (whatever it's called and wherever it lives) that carries details (like referencing from startTime or setting it in model_configure because reading a list from ufs.configure was buggy...). Especially if the details can change, it's cleaner for different components to share it.

imo, it's ATM that insists on being particular with a separate "encoded" restart_interval in model_configure where "3 6 9" are hours, "3 -1" is a frequency, and "time" changes if ATM is doing IAU.

[dustinswales]

@NickSzapiro-NOAA Sounds good.
Just so we are on the same page.
The plan is that I will modify the restart_fh in the ATM cap to use the common code in CDEPS-interface?
Then within the MPAS atmosphere, we can use this utility to control all of the MPAS output streams (history, restart, diag, da_state)?

One comment though. output_fh is specific to the ATM, but how it's translated to ESMF times is identical to restart_fh. So there is no reason to not use this common utility for all output files?

[NickSzapiro-NOAA]

Maybe we're in the process of writing the page...

Do you want all of the MPAS streams (like history, restart, diag, da_state) to have to write at the same times? I suspect not.
ufsatm_cap keeps separate output_fh and restart_fh

I think it's important that restart_fh be unified across UFS. output_fh not so much.

If you wanted it all to work similarly, the current restart_fh utility would need to be more flexible.
Currently, the shared restart_fh utility has this underlying data structure:

type :: is_restart_fh_type
    logical :: write_restartfh = .false.
    type(ESMF_Time), allocatable :: restartFhTimes(:)
  end type is_restart_fh_type

and components then do some flavor of

! initialize from restart_fh attribute in model_configure:
#ifndef CESMCOUPLED
       type(is_restart_fh_type)     :: restartfh_info     ! For flexible restarts in UFS
       call ESMF_TimeIntervalGet( dtimestep, s=dtime, rc=rc )
       if (ChkErr(rc,__LINE__,u_FILE_u)) return
       call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restartfh_info)

! check every timestep:
call is_restart_fh(clock, restartfh_info, write_restartfh)
if (write_restartfh) force_restart_now = .true.

One option would be for ATM to have seperate objects type(is_restart_fh_type) :: history_info, restart_info, diag_info, da_state_info and change init_is_restart_fh to accept an optional ESMF_ConfigGetAttribute label to read so each can be set and checked independently. Like:

call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, history_info, label='history_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restart_info, label='restart_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, diag_info, label='diag_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, da_state_info, label='da_state_fh')

[NickSzapiro-NOAA]

Wouldn't you rather use the native MPAS streams to handle output?
https://www2.mmm.ucar.edu/projects/mpas/site/documentation/users_guide/configuring_io.html

[dustinswales]

Maybe we're in the process of writing the page...

Do you want all of the MPAS streams (like history, restart, diag, da_state) to have to write at the same times? I suspect not. ufsatm_cap keeps separate output_fh and restart_fh

I think it's important that restart_fh be unified across UFS. output_fh not so much.

If you wanted it all to work similarly, the current restart_fh utility would need to be more flexible. Currently, the shared restart_fh utility has this underlying data structure:

type :: is_restart_fh_type
    logical :: write_restartfh = .false.
    type(ESMF_Time), allocatable :: restartFhTimes(:)
  end type is_restart_fh_type

and components then do some flavor of

! initialize from restart_fh attribute in model_configure:
#ifndef CESMCOUPLED
       type(is_restart_fh_type)     :: restartfh_info     ! For flexible restarts in UFS
       call ESMF_TimeIntervalGet( dtimestep, s=dtime, rc=rc )
       if (ChkErr(rc,__LINE__,u_FILE_u)) return
       call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restartfh_info)

! check every timestep:
call is_restart_fh(clock, restartfh_info, write_restartfh)
if (write_restartfh) force_restart_now = .true.

One option would be for ATM to have seperate objects type(is_restart_fh_type) :: history_info, restart_info, diag_info, da_state_info and change init_is_restart_fh to accept an optional ESMF_ConfigGetAttribute label to read so each can be set and checked independently. Like:

call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, history_info, label='history_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, restart_info, label='restart_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, diag_info, label='diag_fh')
call init_is_restart_fh(mcurrTime, dtime, my_task == master_task, da_state_info, label='da_state_fh')

We are on the same page.
My plan was to have an instance of type(is_restart_fh_type) for each MPAS stream.

[dustinswales]

Wouldn't you rather use the native MPAS streams to handle output? https://www2.mmm.ucar.edu/projects/mpas/site/documentation/users_guide/configuring_io.html

We do...

The only thing I need output_fh, restart_fg, xyz_fh, is to derive the ESMF Time used to trigger the output calls.

Currently, I take output_fh and restart_fh and translate them to ESMF time arrays.
Then these are used to trigger output into the MPAS streams after forward integration. These output calls use the MPAS native IO.
Each MPAS stream should have its own trigger.

What I don't like about this is that I have to convert from and a real array into the ESMF times. This could be set in the NUOPC cap using the utility you suggested, by having multiple instances of is_restart_fh_type for each MPAS stream.

[dustinswales]

@NickSzapiro-NOAA Here's some more history and how this fits into this effort.

When bringing the MPAS dynamical core into the UFS, we had to draw a line of responsibility between the host (UFS) and the dycore (MPAS-A). MPAS-A is a standalone model with its own infrastructure, and we only want to use its dynamical core. MPAS-A has infrastructure for I/O (i.e, mpas_write_stream and mpas_read_stream) and additional infrastructure for controlling the I/O (i.e. MPAS_stream_manager). We drew the responsibility line in-between. So we are using MPAS native I/O, but the control of I/O is the hosts (UFS) responsibility.

This responsibility on the host is modest. We need to tell the MPAS atmosphere (a) when to write which stream (e.g. output_fh, restart_fh, etc...) and (b) which fields to include in these streams (e.g. stream_list.atmosphere.output)

The PR merged yesterday added the MPAS output stream, using output_fh the derive the ESMF times needed to trigger the output stream. No functionality for a stream_list included here, so all available fields are included in the output file.

This PR does the same thing, but for the MPAS restart stream. Additionally, this PR adds infrastructure to use the MPAS stream_list files to control which fields are included in the stream. If no stream_list file is provided, all available fields in the stream are written (previous basic functionality).

Our next PR will include the physics-to-dynamics coupling, which will introduce another MPAS stream (diag) to the UFS. So having a common way for the host (UFS) to communicate the requested output file times for the various MPAS streams is much needed.

[dustinswales]

@NickSzapiro-NOAA In the last commit I reverted my changes and adopted using shr_is_restart_fh_mod for determining the MPAS output stream times.
I had to extend shr_is_restart_fh_mod to use it for other stream types, but the default is to use restart_fh, so it is backward compatible.