Update of dev/gfsv17 branch #4324
Merged
DavidHuber-NOAA merged 31 commits intoNOAA-EMC:dev/gfs.v17from Dec 12, 2025
Merged
Conversation
This contains changes needed to run JEDI atmospheric variational DA with a GSI-based EnKF ensemble. This configuration is currently being used in the near realtime global parallel running on WCOSS2. The changes in this PR do not alter the behavior of GDASApp ctests or g-w CI.
…OAA-EMC#4276) # Description This PR removes obsolete GDASApp logic from build_compute.sh Resolves NOAA-EMC#4275
Updates the ex-scripts to be compliant with both `shellcheck` and `shfmt`. This does _not_ necessarily enforce some standards not checked by `shellcheck`. Export statements are added to some variables to avoid 'not used' errors. In the future, a more careful analysis should be done to see if these variables are used downstream by executables or if they are truly unused, in which case they should be removed. Scripts employing MPMD are also updated to use `run_mpmd.sh`. This resulted in some retooling of those scripts.
This adds support for GCAFS cycling on MSU HPC (that do not have access to HPSS) by allowing for the extraction of GDAS files from prestaged tarballs on disk. This includes a shell script to run on Mercury/Ursa/other machines to prepare the files that will then need to be copied to the MSU HPC disks.
…aster Templates to Python Scripts (NOAA-EMC#3984) - Move Jinja logic, com, and cycle vars in stage_ic YAML into Python; simplify YAML to component includes and switches only. - This PR: 1. Implements per‑member COM path computation for gfs, gefs_offline, gefs_rt, gcafs, and enkfgdas cases 2. Move logic, member com paths and general variables to `stage_ic.py`
Updates the GitHub action for static shell code analysis. The shellcheck is switched over to the reviewdog ecosystem and now checks all files in directories that have been made compliant, rather than just checking changes. A second pass on just changes is temporarily included (also using reviewdog, as that mode is supported) to ensure any new changes are compliant until the entire workflow is brough up to spec. There is also a new check against `shfmt`. The reviewdog version does not allow for path restrictions for shfmt, so for now this is being done manually on compliant directories. The reviewdog version is included to be uncommented when the entire repo is compliant. Additionally, the bash and python GH workflows are renamed to better represent their contents and updated to have the same set of triggers. The push trigger is restricted to develop to avoid double-testing of PRs, and a workflow dispatch trigger is added. Finally, a couple scripts that became non-compliant due to a lack of a shfmt check have been corrected. Refs: NOAA-EMC#397 Resolves NOAA-EMC#4239
…dates (NOAA-EMC#4267) - Tar up the gsidiags into a gsidiag.tar instead of the GSI writing out to COM via symlinks. The tarred GSI diags are then untarred where they are needed. - provides a number of EE2 related updates (replacing link with copy), `COMIN/COMOUT`, etc. During the course of this , it was noticed that the EnKF was not assimilating any satellite obs. This hotfix was issued by @CatherineThomas-NOAA in a different PR already. Several variables in the analysis-related scripts are unnecessary and seem to come from earlier copy/paste versions. They can lead to confusion. This PR tries to eliminate some, though more work is recommended before code hand-off to expedite code acceptance into ops. There are also several deprecated "options" that are either no longer functional or not used.
Reducing the I/O burden by reducing the number of variables, layers, and adding compression. Also adds archiving for native subsets
This updates the PR template to include checklist items for expected changes.
Explicitly list the marine yamls to archive during the gfsocean_analysis archiving task. This replaces the * wildcard which was grabbing all of the available yamls in the conf directory, including yaml files that do not need to be archived.
This moves `jobs/` to `dev/jobs`, `dev/jobs` to `dev/job_cards/rocoto` and `scripts` to `dev/scripts` (but symlinks the latter to the original location for now. This is in anticipation for T2O where we will only want to put in `jobs/` and `scripts/` the files needed for each application from the total global-workflow develop set.
NOAA-EMC#4303) # Description This PR generalizes the AWS GFS `config.resources.AWSPW` file for the default compute partition and specifies `mem_node_max` for this partition in the AWS machine block for GFS `config.resources`. Resolves NOAA-EMC#4213
# Description This fixes the METplus dependencies by checking backwards for up to (`GFS_INTERVAL / assim_freq`) cycles for completed `gfs_arch_vrfy` jobs. It also does a separate check for the last cycle. Resolves NOAA-EMC#4281 # Type of change - [x] Bug fix (fixes something broken) - [ ] New feature (adds functionality) - [ ] Maintenance (code refactor, clean-up, new CI test, etc.) # Change characteristics - Is this a breaking change (a change in existing functionality)? NO - Does this change require a documentation update? NO - Does this change require an update to any of the following submodules? NO # How has this been tested? Generated XMLs for several different cases. # Checklist - [x] Any dependent changes have been merged and published - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] My changes generate no new warnings - [ ] New and existing tests pass with my changes - [ ] This change is covered by an existing CI test or a new one has been added
# Description This reduces the memory request for WCOSS2 HPSS jobs to 3GB to (hopefully) speed up queue time. Resolves NOAA-EMC#4288 # Type of change - [ ] Bug fix (fixes something broken) - [ ] New feature (adds functionality) - [x] Maintenance (code refactor, clean-up, new CI test, etc.) # Change characteristics - Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO - Is this a breaking change (a change in existing functionality)? NO - Does this change require a documentation update? NO - Does this change require an update to any of the following submodules? NO # How has this been tested? Full suite of tests on WCOSS2 has started. # Checklist - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my own code - [x] My changes generate no new warnings - [ ] New and existing tests pass with my changes - [x] This change is covered by an existing CI test or a new one has been added
This adds more specific checks for gempak files and makes an attempt at making the cleanup job less disk-intensive by running a single `find -f` command.
…) (NOAA-EMC#4304) This updates the GSI hash to include an updated version of build_gsinfo which turns on abi_g19 assimilaton.
…#4292) Updates the ush scripts (including dev/ush) to be compliant with the static bash code analysis checkers. Also updates the reporter for the GH action so that it should be able to post recommendations on PRs.
If the cpfs calls within the ush/minmon* scripts have been modified to ensure any error code is reported, both in the script log and via error code returned to the calling ex-script.
…le Ensemble/Deterministic Archiving (NOAA-EMC#4298) - The archiving process includes two separate components: local archiving (arch_vrfy) and external tar creation (arch_tar). This issue focuses only on local archiving. This will handle: - Archiving of ensemble output files - Archiving of deterministic (control) output files - Logic, COM paths, and cycle-dependent variables currently defined in the YAML files ( `***_arcdir.yaml.j2`) will be moved into archive_vars.py. The YAML files will retain only the file-specific archiving definitions.
…ycle (NOAA-EMC#4305) # Description This adds <OR> dependencies on the `gdas_cleanup` and `enkfgdas_cleanup` jobs when `SDATE_gfs` several cycles beyond `SDATE` to prevent the cleanup jobs from never executing. Resolves NOAA-EMC#4262 # Type of change - [x] Bug fix (fixes something broken) # Change characteristics - Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO - Is this a breaking change (a change in existing functionality)? NO - Does this change require a documentation update? NO - Does this change require an update to any of the following submodules? NO # How has this been tested? Generated a full suite of XMLs and checked the output databases. Generated an additional XML with `SDATE=2021122018`, `SDATE_gfs=2021122500` and `EDATE=2021123100` and verified the dependency matrix. # Checklist - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] New and existing tests pass with my changes - [x] This change is covered by an existing CI test or a new one has been added
…nws-global-pds bucket (NOAA-EMC#4301) This updates the noaacloud data paths to come from an FSx for Lustre mount provisioned with a data repository association to the S3 bucket [noaa-nws-global-pds](https://noaa-nws-global-pds.s3.amazonaws.com/index.html).
Updating ufs-weather-model, in particular for two updates for GFSv17.
Contributor
Author
|
The easier comparison is with develop: develop...JessicaMeixner-NOAA:global-workflow:gfsv17/updateplusgdas which just shows that gdasapp hash is changed. |
DavidHuber-NOAA
added
CI-Wcoss2-Running
DavidHuber-NOAA
approved these changes
Dec 12, 2025
Contributor
Author
The dead job is here: /gpfs/f6/ira-sti/world-shared/Jessica.Meixner/updategfsv17branch/RUNTESTS/COMROOT/C96_gcafs_cycled_v17o1/logs/2021122018/gcdas_aeroanlvar.log @CoryMartin-NOAA is this perhaps related to the fact that we're out of sync w/develop of GDASApp? |
Contributor
Author
The SFS error /lfs/h2/emc/couple/noscrub/jessica.meixner/updateforv17branch/RUNTESTS/COMROOT/C96mx100_S2S_v17o1/logs/1994050100/sfs_fcst_mem000_seg1.log is something that I see occasionally on WCOSS2. I don't think this is of concern. I'm assuming that the errors for C96C48_ufsgsi_hybatmDA and C96C48_ufs_hybatmDA are because of GDAS App hash. Same for the GCAFS error, which was also seen on GAEA. @CatherineThomas-NOAA @CoryMartin-NOAA @DavidHuber-NOAA @RuiyuSun - I'm not concerned about these errors, perhaps in the future just -G is sufficient for future testing b/c of the divergence of GDASApp -- Please let me know your thoughts though. |
Contributor
Author
|
@CatherineThomas-NOAA @CoryMartin-NOAA - Is this the correct GDASApp hash or is there a further update that should be needed? |
Contributor
|
The complication for the GCDAS test is that the staged observations have changed name, and we do not version the observation dump, so this will fail in perpetuity with an older version of the workflow + GDASApp. I suggest you don't worry about it. @CatherineThomas-NOAA @danholdaway this does raise our first big EE2 JEDI issue... JEDI should probably just return an increment of 0 if there are no obs? |
Contributor
|
@JessicaMeixner-NOAA - The GDASApp release branch still needs NOAA-EMC/GDASApp#1973 to be included. Working on merging that now with @CoryMartin-NOAA... |
Contributor
Author
|
Is it more important to get the GDAS App #1973 in here or is it more important to get the snow update in runs? I don't think we'll be able to update things until MOnday unless someone works over the weekend at this point if we wait. |
Contributor
Author
|
Also I had done a full resolution test with the old version of GDAS App hash - should this be repeated before we include the new update -- (the reason we thought we needed the test is NOAA-EMC/GDASApp#1996 ) |
Contributor
Author
|
@DavidHuber-NOAA -- I just updated the gdas app hash - can you restart the CI tests. Also - if the CI tests I run are not sufficient - please let me know and I'll prioritize getting a PR in over waiting until after I run CI tests. |
Contributor
Author
|
Also - if we run by hand we can just run the -G which is likely sufficient here. |
Contributor
|
@JessicaMeixner-NOAA Yes, I will launch them now. |
Contributor
Author
|
Given that C96C48mx500_S2SW_cyc_gfs has passed on Gaea and WCOSS2 (minus metp jobs still running), let's go ahead and merge this PR. |
DavidHuber-NOAA
added
CI-Wcoss2-Passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR brings dev/gfsv17 branch up to date with develop + points the GDASApp to https://github.com/NOAA-EMC/GDASApp/tree/release/gfs.v17 which is slightly behind develop's version of GDASApp.
Type of change
Change characteristics
Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES/NO (If YES, please indicate to which system(s))
Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES - none pending, changing gdasapp hash.
How has this been tested?
CI on gaea and wcoss2 - - results will be put in comments below.
Checklist