Remove path_exists Calls from stage_ic YAMLs or Move them to link_opt Sections#4492
Remove path_exists Calls from stage_ic YAMLs or Move them to link_opt Sections#4492AntonMFernando-NOAA wants to merge 30 commits intoNOAA-EMC:developfrom
path_exists Calls from stage_ic YAMLs or Move them to link_opt Sections#4492Conversation
| link_req: | ||
| {% if path_exists(ICSDIR ~ "/" ~ COMOUT_CHEM_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %} | ||
| link_opt: | ||
| - ["{{ ICSDIR }}/{{ COMOUT_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ RUN }}.t{{ current_cycle_HH }}z.aero_varbc_params.tar", |
There was a problem hiding this comment.
@bbakernoaa @CoryMartin-NOAA is there a case where the aero_varbc_params.tar file will not be present? Is there a flag available to identify that case?
There was a problem hiding this comment.
We actually aren't using VarBc for aerosols for GCAFSv1, we thought we might but decided against it. When cycling it makes an empty file I believe. Can we check if it's the first half cycle or not and only expect to copy it when not the first cycle?
|
I haven't looked closely at this, but in general the cold vs warm start needs are significantly different, so might want to keep that in mind |
parm/stage/atmosphere_warm.yaml.j2
Outdated
| {% endif %} | ||
| {% endfor %} # ntile | ||
| link_opt: | ||
| - ["{{ ICSDIR }}/{{ COMOUT_ATMOS_RESTART_PREV | relpath(ROTDIR) }}/{{ m_prefix }}.atm_stoch.res.nc", "{{ COMOUT_ATMOS_RESTART_PREV }}"] |
There was a problem hiding this comment.
I think this is a GEFS-required (and maybe SFS) file. Is that correct @EricSinsky-NOAA?
TravisElless-NOAA
left a comment
TravisElless-NOAA
left a comment
There was a problem hiding this comment.
I tried running the C96C48_hybatmDA ci test on WCOSS. I removed a if path_exists line that are still included in the analysis.yaml.j2 file (line 67). The test failed because the cold start test case was looking for files that would only be present during a warm start. In order to remove all of the path_exists calls there will likely need to be some warm start and cold start flags. I highlighted just a few files in that one block, but should likely be applied throughout the rest of the files.
parm/stage/analysis.yaml.j2
Outdated
| {% for ftype in ["increment.atm.i003.nc", "increment.atm.i009.nc", | ||
| "recentered_increment.atm.i003.nc", "recentered_increment.atm.i009.nc"] %} |
There was a problem hiding this comment.
these are going to be files present only during warm starts, but the cold start tests are trying to access these without line 67. Probably going to need a warm start flag here.
parm/stage/analysis.yaml.j2
Outdated
| "increment.atm.i006.nc", "radstat.tar", | ||
| "recentered_increment.atm.i006.nc", "rad_varbc_params.tar"] %} |
There was a problem hiding this comment.
Increment files are again warm starts. the abias and radstat files will be needed by both cold and warm. rad_varbc_params is a JEDI file and will need a DO_JEDIVAR=YES flag
| @@ -81,21 +84,19 @@ analysis: | |||
| {# GSI Soil Analysis (Land DA) #} | |||
| {# -------------------------------------------------------------------- #} | |||
| {% if DO_GSISOILDA %} | |||
There was a problem hiding this comment.
probably another warm start only file
| - ["{{ ww3_file }}.nc" , "{{ COMOUT_WAVE_RESTART_PREV }}/{{ m_prefix }}.restart.ww3.nc"] | ||
| {% else %} | ||
| {% if path_exists(ww3_file) %} | ||
| - ["{{ ww3_file }}" , "{{ COMOUT_WAVE_RESTART_PREV }}/{{ m_prefix }}.restart.ww3"] |
There was a problem hiding this comment.
@AntonMFernando-NOAA I think this is one location where path_exists would be fine to keep. We still need to support the binary and netcdf versions.
What I would suggest is
pseudocode
link_opt:
if path_exists(ww3_file ~ ".nc")
- [ ..., ...ww3.nc ]
else
- [ ..., ...ww3 ]
In other words, only add one path_exists. If both files are absent, then it should raise a warning.
| - "{{ COMOUT_WAVE_RESTART_PREV }}" | ||
| link_req: | ||
| link_opt: | ||
| {% if DO_WAVE %} |
There was a problem hiding this comment.
The DO_WAVE here is implied, so I think you can remove it.
|
|
||
| {% if path_exists(ICSDIR) %} | ||
| {% for ftype in ["MOM.res.nc", "MOM.res_1.nc", "MOM.res_2.nc", "MOM.res_3.nc"] %} | ||
| {% if DO_OCN %} |
There was a problem hiding this comment.
Same, DO_OCN is implied.
| mkdir: | ||
| - "{{ COMOUT_MED_RESTART_PREV }}" | ||
| link_req: | ||
| {% if DO_OCN and EXP_WARM_START %} |
There was a problem hiding this comment.
Only needs to be EXP_WARM_START as DO_OCN is implied.
| "increment.atm.i006.nc", "increment.atm.i009.nc", "increment.atm.i003.nc", "radstat.tar", | ||
| "recentered_increment.atm.i006.nc", "recentered_increment.atm.i009.nc", "recentered_increment.atm.i003.nc", "rad_varbc_params.tar"] %} | ||
| {% if path_exists(ICSDIR ~ "/" ~ COMOUT_ATMOS_ANALYSIS | relpath(ROTDIR) ~ "/" ~ RUN ~ ".t" ~ current_cycle_HH ~ "z." ~ ftype) %} | ||
| {% if DOIAU or DOIAU_ENKF %} |
There was a problem hiding this comment.
Maybe it would be worth it to check the RUN in stage_ic.py and set doiau = DOIAU for deterministic and doiau = DOIAU_ENKF for ensemble RUNs. Then you could just check for doiau here. Not sure if that is easier/harder.
| {# Cubed Sphere Grid Configuration #} | ||
| {% if ATMINC_GRID == "cubed_sphere_grid" %} | ||
|
|
||
| {% if (DO_JEDIATMVAR or DO_JEDIATMENS) and EXP_WARM_START %} |
There was a problem hiding this comment.
Similarly, perhaps setting do_jediatm in stage_ic.py based on the RUN. Again, not sure that is easier/harder.
4 participants
Description
This PR removes all uses of
path_existsfrom the templatedstage_icYAMLs in the workflow and moved tolink_optsections.Resolves Remove
path_existscalls fromstage_icYAMLs or move them tolink_optsections #4477Type of change
Change characteristics
How has this been tested?
stage_icjob onheraandwcoss2and make sure all necessary files are stagedChecklist