Skip to content

Ensures any failure in piped output is captured and reported.#4328

Merged
DavidHuber-NOAA merged 28 commits intoNOAA-EMC:developfrom
aerorahul:feature/pipedown
Dec 19, 2025
Merged

Ensures any failure in piped output is captured and reported.#4328
DavidHuber-NOAA merged 28 commits intoNOAA-EMC:developfrom
aerorahul:feature/pipedown

Conversation

@aerorahul
Copy link
Contributor

@aerorahul aerorahul commented Dec 12, 2025

Description

This PR:

This PR also provides a hotfix for #4334

Type of change

  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO
  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

  • GFS tests on GaeaC6 and WCOSS
  • More testing is in progress

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried that sometimes we want to ignore the pipefail (particularly when using grep with wgrib2).

Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple of suggestions with find speedups. Otherwise, it looks good. I'm expecting a few more bugs to resolve as testing continues.

@aerorahul
Copy link
Contributor Author

I'm worried that sometimes we want to ignore the pipefail (particularly when using grep with wgrib2).

Thanks @WalterKolczynski-NOAA
Do you know where that is the case presently? If we know, then we can wrap that call with unset_strict such that"

unset_strict
wgrib2 ....
set_strict

What do you think?

Co-authored-by: David Huber <[email protected]>
Co-authored-by: Travis Elless <[email protected]>
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96C48_hybatmDA FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96C48_hybatmDA_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96C48_hybatmDA_5a190987-6509/logs/2021122018/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96C48_hybatmsnowDA FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96C48_hybatmsnowDA_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96C48_hybatmsnowDA_5a190987-6509/logs/2021122012/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96C48_ufsgsi_hybatmDA FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96C48_ufsgsi_hybatmDA_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96C48_ufsgsi_hybatmDA_5a190987-6509/logs/2024022318/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96C48_hybatmsoilDA FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96C48_hybatmsoilDA_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96C48_hybatmsoilDA_5a190987-6509/logs/2022051506/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C48mx500_hybAOWCDA FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_5a190987-6509/logs/2021032418/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96_atm3DVar FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96_atm3DVar_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96_atm3DVar_5a190987-6509/logs/2021122018/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96C48_ufs_hybatmDA FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_5a190987-6509/logs/2024022318/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C48_S2SW FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C48_S2SW_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SW_5a190987-6509/logs/2021032312/gfs_atmos_prod_f000-f002.log

View Error Logs: (gfs_atmos_prod_f000-f002.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C48mx500_3DVarAOWCDA FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_5a190987-6509/logs/2021032418/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96C48mx500_S2SW_cyc_gfs FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_5a190987-6509/logs/2021122012/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96mx100_S2S FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96mx100_S2S_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96mx100_S2S_5a190987-6509/logs/1994050100/sfs_atmos_prod_mem000_f000.log
/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96mx100_S2S_5a190987-6509/logs/1994050100/sfs_atmos_prod_mem001_f000.log
/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96mx100_S2S_5a190987-6509/logs/1994050100/sfs_atmos_prod_mem002_f000.log

View Error Logs: (sfs_atmos_prod_mem000_f000.log) (sfs_atmos_prod_mem001_f000.log) (sfs_atmos_prod_mem002_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C48_S2SW FAILED on Hercules (pipeline ID: 6509)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C48_S2SW_5a190987-6509

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SW_5a190987-6509/logs/2021032312/gfs_atmos_prod_f000-f002.log

View Error Logs: (gfs_atmos_prod_f000-f002.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C48mx500_3DVarAOWCDA FAILED on Hercules (pipeline ID: 6509)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_5a190987-6509

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_5a190987-6509/logs/2021032418/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C48_S2SWA_gefs FAILED on Ursa (pipeline ID: 6509)

In directory: /scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C48_S2SWA_gefs_5a190987-6509

Error Log Files:


/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SWA_gefs_5a190987-6509/logs/2021032312/gefs_atmos_prod_mem000_f000.log
/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SWA_gefs_5a190987-6509/logs/2021032312/gefs_atmos_prod_mem001_f000.log
/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SWA_gefs_5a190987-6509/logs/2021032312/gefs_atmos_prod_mem002_f000.log

View Error Logs: (gefs_atmos_prod_mem000_f000.log) (gefs_atmos_prod_mem001_f000.log) (gefs_atmos_prod_mem002_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96C48mx500_S2SW_cyc_gfs FAILED on Hercules (pipeline ID: 6509)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_5a190987-6509

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_5a190987-6509/logs/2021122012/gdas_atmos_prod_f000.log

View Error Logs: (gdas_atmos_prod_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C96mx100_S2S FAILED on Hercules (pipeline ID: 6509)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C96mx100_S2S_5a190987-6509

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96mx100_S2S_5a190987-6509/logs/1994050100/sfs_atmos_prod_mem000_f000.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96mx100_S2S_5a190987-6509/logs/1994050100/sfs_atmos_prod_mem001_f000.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C96mx100_S2S_5a190987-6509/logs/1994050100/sfs_atmos_prod_mem002_f000.log

View Error Logs: (sfs_atmos_prod_mem000_f000.log) (sfs_atmos_prod_mem001_f000.log) (sfs_atmos_prod_mem002_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot
Copy link

emcbot commented Dec 18, 2025

C48_S2SWA_gefs FAILED on Hercules (pipeline ID: 6509)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/EXPDIR/C48_S2SWA_gefs_5a190987-6509

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SWA_gefs_5a190987-6509/logs/2021032312/gefs_atmos_prod_mem000_f000.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SWA_gefs_5a190987-6509/logs/2021032312/gefs_atmos_prod_mem001_f000.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4328_5a190987_6509/RUNTESTS/COMROOT/C48_S2SWA_gefs_5a190987-6509/logs/2021032312/gefs_atmos_prod_mem002_f000.log

View Error Logs: (gefs_atmos_prod_mem000_f000.log) (gefs_atmos_prod_mem001_f000.log) (gefs_atmos_prod_mem002_f000.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@aerorahul
Copy link
Contributor Author

The failures in the gfs|gdas|sfs|gefs_atmos_products_f000 jobs are arising from #4347. Specifically,

export paramlistb_f000="${PARMgfs}/product/gcafs.f000.paramlist.b.txt"
introduced in gcafs but not in gfs|gefs|sfs configs, and required for
paramlistb="${paramlistb_f000}"
.

@aerorahul aerorahul mentioned this pull request Dec 18, 2025
10 tasks
@DavidHuber-NOAA
Copy link
Contributor

Launching on Hercules and Ursa.

@CoryMartin-NOAA
Copy link
Contributor

@aerorahul @bbakernoaa is on AL today. Do you need me to open a bugfix PR for this or can those changes be incorporated here?

@CoryMartin-NOAA
Copy link
Contributor

Never mind I see that @DavidHuber-NOAA reverted the PR. @bbakernoaa let's get this fixed in develop first then when we are back,

@aerorahul
Copy link
Contributor Author

@DavidHuber-NOAA
I applied your suggestion to comment out the gcafs pres_b archiving. With that, hopefully develop will not fail on that account.

@DavidHuber-NOAA
Copy link
Contributor

Turns out that Hercules never actually ran. The GitLab runner was killed at some point. Merging based on success on Ursa and WCOSS2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GCAFS Attempts to archive non-existent pres_b products Add set -o pipefail to set_strict

6 participants