Skip to content

Remove path_exists Calls from stage_ic YAMLs or Move them to link_opt Sections #4492

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
AntonMFernando-NOAA wants to merge 30 commits into
base: develop
Choose a base branch
from
+95 −65
Draft

Remove path_exists Calls from stage_ic YAMLs or Move them to link_opt Sections #4492

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
15cc4b2
remove path_exists
AntonMFernando-NOAA Jan 29, 2026
1c26d83
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Jan 29, 2026
f0a325b
update parm/stage/analysis.yaml.j2
AntonMFernando-NOAA Jan 29, 2026
f4fca24
it typo
AntonMFernando-NOAA Jan 29, 2026
08acf92
update parm/stage/analysis.yaml.j2
AntonMFernando-NOAA Jan 29, 2026
d2b8cce
update parm/stage/analysis.yaml.j2
AntonMFernando-NOAA Jan 29, 2026
01b3e9f
update parm/stage/analysis.yaml.j2
AntonMFernando-NOAA Jan 29, 2026
358ffd7
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Jan 30, 2026
a01ce5f
Merge branch 'develop' into maint/path
AntonMFernando-NOAA Jan 30, 2026
059e17e
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Jan 30, 2026
7244661
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Feb 3, 2026
0b043c0
update new flags
AntonMFernando-NOAA Feb 3, 2026
e443f6b
update new flags 2
AntonMFernando-NOAA Feb 3, 2026
57f720f
add path_exists back
AntonMFernando-NOAA Feb 3, 2026
9709b35
update atmosphere_warm.yaml.j2
AntonMFernando-NOAA Feb 3, 2026
3e9376c
update gdas.cd
AntonMFernando-NOAA Feb 4, 2026
a9419f1
update wxflow
AntonMFernando-NOAA Feb 4, 2026
27db87e
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Feb 5, 2026
cc11648
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Feb 6, 2026
1f16262
Merge branch 'develop' into maint/path
AntonMFernando-NOAA Feb 6, 2026
8b6a0d0
update parm/stage/analysis.yaml.j2
AntonMFernando-NOAA Feb 6, 2026
69e9635
update yamls
AntonMFernando-NOAA Feb 6, 2026
530be8e
update analysis.yaml
AntonMFernando-NOAA Feb 6, 2026
3ee9e4c
update analysis.yaml
AntonMFernando-NOAA Feb 7, 2026
0b917fa
update analysis.yaml
AntonMFernando-NOAA Feb 7, 2026
1af29a9
update ocean_mdeiator
AntonMFernando-NOAA Feb 7, 2026
879d6e0
update ocean_mediator
AntonMFernando-NOAA Feb 7, 2026
5fb3dcc
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Feb 9, 2026
ffac271
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Feb 10, 2026
87201f6
Merge branch 'NOAA-EMC:develop' into maint/path
AntonMFernando-NOAA Feb 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions parm/stage/aero.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ aero:
- "{{ COMOUT_CHEM_ANALYSIS }}"

link_req:
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_CHEM_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %}
{% if DO_AERO_ANL %}
- ["{{ ICSDIR }}/{{ COMOUT_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.aero_varbc_params.tar",
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bbakernoaa @CoryMartin-NOAA is there a case where the aero_varbc_params.tar file will not be present? Is there a flag available to identify that case?

Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Jan 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually aren't using VarBc for aerosols for GCAFSv1, we thought we might but decided against it. When cycling it makes an empty file I believe. Can we check if it's the first half cycle or not and only expect to copy it when not the first cycle?

"{{ COMOUT_CHEM_ANALYSIS }}"]
{% endif %}
{% endif %}
45 changes: 23 additions & 22 deletions parm/stage/analysis.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
{# ============================================================================ #}
{# Analysis Staging Configuration #}
{# ============================================================================ #}
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR)) %}
analysis:
mkdir:
- "{{ COMOUT_ATMOS_ANALYSIS }}"
Expand All @@ -10,7 +9,7 @@ analysis:
{# -------------------------------------------------------------------- #}
{# JEDI Atmospheric Variational Data Assimilation #}
{# -------------------------------------------------------------------- #}
{% if DO_JEDIATMVAR %}
{% if DO_JEDIATMVAR or DO_JEDIATMENS %}

{# Cubed Sphere Grid Configuration #}
{% if ATMINC_GRID == "cubed_sphere_grid" %}
Expand All @@ -35,9 +34,7 @@ analysis:
{% endif %}

{% for ftype in ftypes %}
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype ~ ".tile" ~ (itile+1) ~ ".nc") %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}.tile{{ itile+1 }}.nc", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endif %}
{% endfor %}

{% endfor %}
Expand All @@ -48,28 +45,34 @@ analysis:
{# Atmospheric Increments #}
{% for ftype in ["increment.atm.i006.nc", "increment.atm.i009.nc", "increment.atm.i003.nc",
"recentered_increment.atm.i006.nc", "recentered_increment.atm.i009.nc", "recentered_increment.atm.i003.nc"] %}
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endif %}
{% endfor %}

{% endif %}

{# Bias Correction Files #}
{% for ftype in ["abias.txt", "abias_air.txt", "abias_int.txt", "abias_pc.txt", "rad_varbc_params.tar"] %}
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endif %}
{% endfor %}

{% endif %}

{# -------------------------------------------------------------------- #}
{# GSI Data Assimilation (Non-JEDI) #}
{# -------------------------------------------------------------------- #}
{% else %}

{% if not DO_JEDIATMVAR %}
{% if DOIAU or DOIAU_ENKF %}
{% for ftype in ["increment.atm.i003.nc", "increment.atm.i009.nc",
"recentered_increment.atm.i003.nc", "recentered_increment.atm.i009.nc"] %}
Copy link
Contributor

@TravisElless-NOAA TravisElless-NOAA Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are going to be files present only during warm starts, but the cold start tests are trying to access these without line 67. Probably going to need a warm start flag here.

{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endif %}
{% endfor %}
{% endif %}
{# All GSI Analysis Files #}
{% for ftype in ["abias.txt", "abias_air.txt", "abias_int.txt", "abias_pc.txt",
"increment.atm.i006.nc", "increment.atm.i009.nc", "increment.atm.i003.nc", "radstat.tar",
"recentered_increment.atm.i006.nc", "recentered_increment.atm.i009.nc", "recentered_increment.atm.i003.nc", "rad_varbc_params.tar"] %}
"increment.atm.i006.nc", "radstat.tar",
"recentered_increment.atm.i006.nc", "rad_varbc_params.tar"] %}
Copy link
Contributor

@TravisElless-NOAA TravisElless-NOAA Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Increment files are again warm starts. the abias and radstat files will be needed by both cold and warm. rad_varbc_params is a JEDI file and will need a DO_JEDIVAR=YES flag

{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endif %}
Expand All @@ -81,21 +84,19 @@ analysis:
{# GSI Soil Analysis (Land DA) #}
{# -------------------------------------------------------------------- #}
{% if DO_GSISOILDA %}
Copy link
Contributor

@TravisElless-NOAA TravisElless-NOAA Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably another warm start only file


{# Surface Increments #}
{% for ftype in ["increment.sfc.i003.nc", "increment.sfc.i006.nc", "increment.sfc.i009.nc"] %}
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endif %}
{% endfor %}

{# Surface Increments #}
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z.increment.sfc.i006.nc") %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.increment.sfc.i006.nc", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endif %}
{# Land IAU Increments (Tiled) #}
{% if DO_LAND_IAU %}
{# Surface Increments #}
{% for ftype in ["increment.sfc.i003.nc", "increment.sfc.i009.nc"] %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.{{ ftype }}", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endfor %}
{% for itile in range(1,7) %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) }}/increment.sfc.tile{{ itile }}.nc", "{{ COMOUT_ATMOS_ANALYSIS }}"]
{% endfor %}
{% endif %}

{% endif %}

{% endif %}
17 changes: 9 additions & 8 deletions parm/stage/atmosphere_warm.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,23 @@ atmosphere_warm:
- "{{ COMOUT_ATMOS_RESTART_PREV }}"
- "{{ COMOUT_ATMOS_RESTART }}"
link_req:
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_RESTART_PREV | relpath(ROTDIR) ~ "/" ~ m_prefix ~ ".atm_stoch.res.nc") %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_RESTART_PREV | relpath(ROTDIR) }}/{{ m_prefix }}.atm_stoch.res.nc", "{{ COMOUT_ATMOS_RESTART_PREV }}"]
{% endif %} # path_exists

{% for ftype in ["coupler.res", "fv_core.res.nc"] %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_RESTART_PREV | relpath(ROTDIR) }}/{{ m_prefix }}.{{ ftype }}", "{{ COMOUT_ATMOS_RESTART_PREV }}"]
{% endfor %}

{% for ntile in range(1, ntiles + 1) %}
{% for ftype in ["fv_core.res", "fv_srf_wnd.res", "fv_tracer.res", "phy_data", "sfc_data"] %}
{% for ftype in ["fv_core.res", "fv_srf_wnd.res", "fv_tracer.res", "phy_data", "sfc_data"] %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_RESTART_PREV | relpath(ROTDIR) }}/{{ m_prefix }}.{{ ftype }}.tile{{ ntile }}.nc", "{{ COMOUT_ATMOS_RESTART_PREV }}"]
{% endfor %} # ftype
{% if DO_CA %}
{% endfor %} # ftype
{% if DO_CA %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_RESTART_PREV | relpath(ROTDIR) }}/{{ m_prefix }}.ca_data.tile{{ ntile }}.nc", "{{ COMOUT_ATMOS_RESTART_PREV }}"]
{% endif %}
{% endif %}
{% endfor %} # ntile

{% for ntile in range(1, ntiles + 1) %}
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_RESTART | relpath(ROTDIR) ~ "/" ~ m_prefix ~ ".sfcanl_data.tile" ~ ntile ~ ".nc") %}
{% if member > 0 %}
- ["{{ ICSDIR }}/{{ COMOUT_ATMOS_RESTART | relpath(ROTDIR) }}/{{ m_prefix }}.sfcanl_data.tile{{ ntile }}.nc", "{{ COMOUT_ATMOS_RESTART }}"]
{% endif %} # path_exists
{% endif %}
{% endfor %} # ntile
19 changes: 9 additions & 10 deletions parm/stage/atmosphere_warm_RT.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,16 @@ atmosphere_warm_RT:
- ["{{ file }}", "{{ increment_destination_path }}"]
{% endfor %}

{% if path_exists(ICSDIR) %}
{% set file = restart_mem_path ~ 'coupler.res' %}
{% set file = restart_mem_path ~ 'coupler.res' %}
- ["{{ file }}", "{{ COMOUT_ATMOS_RESTART_PREV }}"]
{% set file = restart_mem_path ~ 'fv_core.res.nc' %}
{% set file = restart_mem_path ~ 'fv_core.res.nc' %}
- ["{{ file }}", "{{ COMOUT_ATMOS_RESTART_PREV }}"]
{% set ntiles = 6 %}
{% for ftype in ["fv_core.res", "fv_srf_wnd.res", "phy_data", "sfc_data"] %}
# Include restart tile files (e.g., .tile1 to .tile6)
{% for tile in range(1, ntiles + 1) %}
{% set file = restart_mem_path ~ ftype ~ '.tile' ~ tile ~ '.nc' %}

{% set ntiles = 6 %}
{% for ftype in ["fv_core.res", "fv_srf_wnd.res", "phy_data", "sfc_data"] %}
# Include restart tile files (e.g., .tile1 to .tile6)
{% for tile in range(1, ntiles + 1) %}
{% set file = restart_mem_path ~ ftype ~ '.tile' ~ tile ~ '.nc' %}
- ["{{ file }}", "{{ COMOUT_ATMOS_RESTART_PREV }}"]
{% endfor %}
{% endfor %}
{% endif %}
{% endfor %}
12 changes: 8 additions & 4 deletions parm/stage/ocean_RT.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,14 @@ ocean_RT:
{% set file = increment_mem_path %}
- ["{{ file }}", "{{ COMOUT_OCEAN_ANALYSIS }}"]

{% if path_exists(ICSDIR) %}
{% for ftype in ["MOM.res.nc", "MOM.res_1.nc", "MOM.res_2.nc", "MOM.res_3.nc"] %}
{% if DO_OCN %}
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, DO_OCN is implied.

{% set file = restart_mem_path ~ 'MOM.res.nc' %}
- ["{{ file }}", "{{ COMOUT_OCEAN_RESTART_PREV }}"]
{% if OCNRES > 25 %}
{% for ftype in ["MOM.res_1.nc", "MOM.res_2.nc", "MOM.res_3.nc"] %}
# Include base file for restart files
{% set file = restart_mem_path ~ ftype %}
{% set file = restart_mem_path ~ ftype %}
- ["{{ file }}", "{{ COMOUT_OCEAN_RESTART_PREV }}"]
{% endfor %}
{% endfor %}
{% endif %}
{% endif %}
4 changes: 2 additions & 2 deletions parm/stage/ocean_mediator.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{% if path_exists(ICSDIR ~ "/" ~ COMOUT_MED_RESTART_PREV | relpath(ROTDIR) ~ "/" ~ m_prefix ~ ".ufs.cpld.cpl.r.nc") %}
ocean_mediator:
mkdir:
- "{{ COMOUT_MED_RESTART_PREV }}"
link_req:
{% if DO_OCN && EXP_WARM_START %}
- ["{{ ICSDIR }}/{{ COMOUT_MED_RESTART_PREV | relpath(ROTDIR) }}/{{ m_prefix }}.ufs.cpld.cpl.r.nc", "{{ COMOUT_MED_RESTART_PREV }}"]
{% endif %} # path exists
{% endif %}
9 changes: 3 additions & 6 deletions parm/stage/wave.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
wave:
mkdir:
- "{{ COMOUT_WAVE_RESTART_PREV }}"
link_req:
link_opt:
{% if DO_WAVE %}
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The DO_WAVE here is implied, so I think you can remove it.

{% set ww3_file = ICSDIR ~ "/" ~ COMOUT_WAVE_RESTART_PREV | relpath(ROTDIR) ~ "/" ~ m_prefix ~ ".restart.ww3" %}
{% if path_exists(ww3_file ~ ".nc") %}
- ["{{ ww3_file }}.nc" , "{{ COMOUT_WAVE_RESTART_PREV }}/{{ m_prefix }}.restart.ww3.nc"]
{% else %}
{% if path_exists(ww3_file) %}
- ["{{ ww3_file }}" , "{{ COMOUT_WAVE_RESTART_PREV }}/{{ m_prefix }}.restart.ww3"]
Comment on lines 7 to 8
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AntonMFernando-NOAA I think this is one location where path_exists would be fine to keep. We still need to support the binary and netcdf versions.

What I would suggest is

pseudocode

link_opt:
   if path_exists(ww3_file ~ ".nc")
       - [ ..., ...ww3.nc ]
   else
       - [ ..., ...ww3 ]

In other words, only add one path_exists. If both files are absent, then it should raise a warning.

{% endif %}
{% endif %}
{% endif %}
2 changes: 1 addition & 1 deletion ush/python/pygfs/task/stage_ic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def _copy_base_config(self) -> Dict[str, Any]:
'assim_freq', 'current_cycle', 'previous_cycle',
'ROTDIR', 'ICSDIR', 'STAGE_IC_YAML_TMPL', 'DO_JEDIATMVAR',
'OCNRES', 'waveGRD', 'ntiles', 'DOIAU', 'ATMINC_GRID',
'DO_JEDIOCNVAR', 'DO_STARTMEM_FROM_JEDIICE',
'DO_JEDIOCNVAR', 'DO_STARTMEM_FROM_JEDIICE', 'DO_JEDIATMENS',
'DO_WAVE', 'DO_OCN', 'DO_ICE', 'DO_NEST', 'DO_CA', 'DO_AERO_ANL', 'MOM6_INTERP_ICS',
'USE_ATM_ENS_PERTURB_FILES', 'USE_OCN_ENS_PERTURB_FILES', 'DO_GSISOILDA', 'DO_LAND_IAU'
]
Expand Down
Loading