Swapping from JRA55 to ERA5 atmospheric forcing

[helenmacdonald]

There is a community need to run MOM6 on nuopc using ERA5 forcing so I am opening this issue for us to work on it.
At the moment, the configurations use JRA55 atmospheric focing

@ezhilsabareesh8 has made a start on this and says:

no code changes are required in CDEPS to use ERA5 forcing. However, datamode and the variables names should be changed in datm.streams.xml file. I am in the process of modifying this code, to generate datm.streams.xml with ERA5 forcing.

@anton-seaice says:

It's` possible no code changes are needed (https://github.com/ESCOMP/CDEPS/blob/9e57b278de18cf69351bc693fe1988143e859fa7/datm/datm_datamode_era5_mod.F90#L248-L261 includes ERA5).

You need to make a datm.xml file for the ERA5 forcing (and decide what to do with runoff).

Ezhil wrote this to make the JRA55 xml files - https://github.com/COSIMA/om3-scripts/blob/main/data_stream_xml_generation/generate_xml_datm.py, it could be adapted I think

[anton-seaice]

I think you know these things:

• we use jra55do rather than jra55, which has some corrections to reduce drift
• there's no run-off field easily available in ERA5, but presumably not important for regional
• NCAR were working on a datm based on ERA5 but it seems delayed.
• ERA5 chunking in time might be troublesome!

There's some results from OM2 with ERA5 on the hive forum, and this mega-thread. Note the comments about some fields being a bit different

[mmr0]

Some notes from meeting with @helenmacdonald + @ezhilsabareesh8 last week.
Steps are:

• Ezhil working on a python script to generate the datm.streams.xml file
• Change data mode flag to ERA5 in datm_in (ERA is already a CDEPS supported “data mode” )
• Generate a mesh file for the data atmosphere using https://github.com/ACCESS-NRI/om3-scripts/blob/main/mesh_generation/generate_mesh.py. Equivalent JRAdo file is INPUT/JRA55do-datm-ESMFmesh.nc
• Some changes to config.yaml (tbd)
• Regional model specific: no need to generate/point to the ERA forcing files produced by the regional mom6 package. Instead, will just point to the same ERA5 files used in the global config
• Madi to look into flux calculation and potential impacts of using 2m vs 10m temperature.

[mmr0]

I want to to create the equivalent of this file
/g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-datm-ESMFmesh.nc created from /g/data/ik11/inputs/JRA-55/RYF/v1-4/RYF.prsn.1990_1991.nc using the generate_mesh.py
but for an ERA5 atmosphere. Any idea if an equivalent ERA5 grid file lives on gadi aready? @helenmacdonald @ezhilsabareesh8 @anton-seaice ?

Update - I think I misunderstood what RYF.prsn.1990_1991.nc is (i.e. it's just a normal output file containing a variable but also has grid info). Can I just give generate_mesh any ERA5 file e.g., /g/data/rt52/era5/single-levels/reanalysis/100u/...?

[anton-seaice]

Can I just give generate_mesh any ERA5 file e.g., /g/data/rt52/era5/single-levels/reanalysis/100u/...?

Yes this should work fine as the only required information is latitude and longitude. It looks like you'll need to rename lat to latitude (and rename lon) in the generate_mesh.py script (If this and any other changes are needed, please make an issue in the https://github.com/ACCESS-NRI/om3-scripts/ repo :-) ).

p.s. - at some point we can add the mesh to /g/data/vk83 also

[dougiesquire]

Can I just give generate_mesh any ERA5 file e.g., /g/data/rt52/era5/single-levels/reanalysis/100u/...?

Yes, except that unfortunately generate_mesh.py currently assumes that the latitude and longitude coordinates are called "lat" and "lon". I'll fix that up today, but if you need a workaround in the meantime you could first rename the coordinates:

module load nco
module load payu

ncrename -v longitude,lon -v latitude,lat /g/data/rt52/era5/single-levels/reanalysis/100u/2025/100u_era5_oper_sfc_20250101-20250131.nc tmp.nc

python3 /g/data/vk83/apps/om3-scripts/mesh_generation/generate_mesh.py --grid-type=latlon --grid-filename=tmp.nc --mesh-filename=ERA5-datm-ESMFmesh.nc

We don't want to be doing this is the long run, as the intermediate file makes the provenance of the mesh unclear.

[dougiesquire]

Oops snap @anton-seaice

[mmr0]

Thanks @dougiesquire and @anton-seaice - that worked a charm!

[mmr0]

@helenmacdonald @ezhilsabareesh8 I've forgotten what we decided about runoff - will I keep the JRA55 runoff for the moment?

i.e., leave
/g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-datm-ESMFmesh.nc
/g/data/qv56/replicas/input4MIPs/CMIP6/OMIP/MRI/MRI-JRA55-do-1-4-0
in config.yaml

[ezhilsabareesh8]

will I keep the JRA55 runoff for the moment?

Yes, I think so

[dougiesquire]

generate_mesh.py currently assumes that the latitude and longitude coordinates are called "lat" and "lon". I'll fix that up today

@mmr0 this fix has now been merged, so you can do:

module use /g/data/xp65/public/modules
module load conda/analysis3-25.04

python3 /g/data/vk83/apps/om3-scripts/mesh_generation/generate_mesh.py --grid-type=latlon --grid-filename=/g/data/rt52/era5/single-levels/reanalysis/100u/2025/100u_era5_oper_sfc_20250101-20250131.nc --mesh-filename=ERA5-datm-ESMFmesh.nc

You'll get a warning like:

UserWarning: Cannot automatically determine the name of the area variable. If area is available in the input grid file and it needs to be included in the mesh, please pass --area-name

Don't worry about this. The area is not needed.

[mmr0]

Thanks @dougiesquire! Have done this successfully

[ezhilsabareesh8]

Summary of Progress on ERA5 Forcing:

Following discussions with @aekiss, the following points and conclusions were reached:

1. The inconsistent chunking in ERA5 data at NCI has been fixed. This is no longer a concern.

2. ERA5 does not include runoff data. We will continue to use runoff from JRA55-do.

3. ERA5 provides t2m (temperature at 2 meters), while wind variables u and v are at 10 meters. For initial testing, we will use t2m. In UFS models, tbot is computed externally, which is not directly available in ERA5.

4. Specific humidity (Sa_shum) is not available in ERA5. It is computed from pbot and tbot as shown in this code snippet).

Implementation Progress

1. Created a branch in access3-share called ERA5-forcing here. This includes a modified version of datm_datamode_era5_mod_UFS.F90, here

2. Modified the generate_xml_datm.py script to support ERA5 forcing, script is here
   The resulting datm.streams.xml file is here

3. Created an access-om3 executable with the modified ERA5 datamode via this draft PR 109

To use it, the following module config can be added to config.yaml:

modules:
    use:
        - /g/data/vk83/modules
    load:
        - access-om3/pr109-2
        - nco/5.0.5

At runtime, I encountered the following error in the atmosphere NUOPC cap:

[gadi-cpu-clx-2022.gadi.nci.org.au:3795224] PMIX ERROR: UNREACHABLE in file /jobfs/129486601.gadi-pbs/0/openmpi/4.1.7/source/openmpi-4.1.7/opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2198
forrtl: error (78): process killed (SIGTERM)
Image              PC                Routine            Line        Source
libpthread-2.28.s  0000145EAE8E7990  Unknown               Unknown  Unknown
libpthread-2.28.s  0000145EAE8E336F  pthread_cond_wait     Unknown  Unknown
libopen-pal.so.40  0000145EABCED53D  PMIx_Abort            Unknown  Unknown
libopen-pal.so.40  0000145EABD5A780  pmix3x_abort          Unknown  Unknown
libopen-rte.so.40  0000145EAC482ED7  Unknown               Unknown  Unknown
libopen-rte.so.40  0000145EAC49AC66  orte_errmgr_base_     Unknown  Unknown
libopen-rte.so.40  0000145EAC48AA1B  Unknown               Unknown  Unknown
libmpi.so.40.30.7  0000145EB13D119A  ompi_mpi_abort        Unknown  Unknown
libpioc.so         0000145EB0CE815C  piodie                Unknown  Unknown
libpioc.so         0000145EB0CE851A  check_netcdf2         Unknown  Unknown
libpioc.so         0000145EB0CEC143  PIOc_openfile_ret     Unknown  Unknown
libpioc.so         0000145EB0CE7373  PIOc_openfile         Unknown  Unknown
libpiof.so         0000145EB0F36F8C  piolib_mod_mp_pio     Unknown  Unknown
access-om3-MOM6-C  000000000232E3FA  shr_stream_getcal        1539  dshr_stream_mod.F90
access-om3-MOM6-C  000000000232687D  shr_stream_init_f         427  dshr_stream_mod.F90
access-om3-MOM6-C  0000000002310EE1  shr_strdata_init_         221  dshr_strdata_mod.F90
access-om3-MOM6-C  00000000022F1F88  initializerealize         458  atm_comp_nuopc.F90

Further debugging is ongoing to resolve the issue. The 1deg MOM6-CICE6 configuration modified for ERA5 forcing is here, /g/data/tm70/ek4684/dev-MC_1deg_jra_iaf

[ezhilsabareesh8]

Since the previous update, I’ve resolved the initial runtime error in the NUOPC atmosphere cap.

• I updated the generate_xml_datm.py script here, and regenerated the datm.streams.xml file here.
• The dev-MC_100km_era_iaf now initializes and runs for one coupling timestep, but crashes with the following new error:

20250618 144047.196 ERROR            PET001 med_methods_mod.F90:2629 ERROR: 16 nans found in Fioi_meltw
20250618 144047.211 ERROR            PET001 med_methods_mod.F90:2629 ERROR: 2851 nans found in Foxx_evap
20250618 144047.212 ERROR            PET001 med_methods_mod.F90:2629 ERROR: 2851 nans found in Foxx_sen
20250618 144047.212 ERROR            PET001 med_methods_mod.F90:2629 ERROR: 2851 nans found in Foxx_taux
20250618 144047.212 ERROR            PET001 med_methods_mod.F90:2629 ERROR: 2851 nans found in Foxx_tauy

This likely indicates missing or uninitialized values in the atmosphere-ocean flux fields at the initial coupling step.

To allow the model to start despite the ERA5 forcing data beginning at rDatelvd = 19400101.2916667 (i.e., 1940-01-01 07:00), I set the following in datm.streams.xml:

<taxmode>extend</taxmode>
<dtlimit>1.0e30</dtlimit>

In addition, I’ve created a dedicated branch for the ERA5-forced 100 km MOM6–CICE6 configuration here, which includes the updated datm.streams.xml and other modifications to enable the use of ERA5 forcing. You can clone this branch and run the configuration directly, provided you have access to the rt52and ik11 projects.

The configuration runs up to the first coupling step, but the NaN issue still needs to be resolved.