Skip to content

Increase thread counts for eupd on Hercules#4376

Merged
DavidHuber-NOAA merged 10 commits intoNOAA-EMC:developfrom
DavidHuber-NOAA:fix/hercules_eupd
Dec 30, 2025
Merged

Increase thread counts for eupd on Hercules#4376
DavidHuber-NOAA merged 10 commits intoNOAA-EMC:developfrom
DavidHuber-NOAA:fix/hercules_eupd

Conversation

@DavidHuber-NOAA
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA commented Dec 23, 2025 •
edited
Loading

Description

This should fix the occasional EUPD NaN production on Hercules related to the enkf.x executable running out of memory.
Resolves #4348

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this change expected to change outputs NO
  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

All tests pass on Hercules.

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added

@emcbot emcbot added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Dec 23, 2025
@DavidHuber-NOAA DavidHuber-NOAA removed the CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules label Dec 23, 2025
@emcbot emcbot added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Dec 23, 2025
@emcbot
Copy link

emcbot commented Dec 23, 2025

C96_gcafs_cycled FAILED on Hercules (pipeline ID: 6608)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4376_ab1b7dc7_6608/RUNTESTS/EXPDIR/C96_gcafs_cycled_ab1b7dc7-6608

Error Log Files:


/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4376_ab1b7dc7_6608/RUNTESTS/COMROOT/C96_gcafs_cycled_ab1b7dc7-6608/logs/2021122012/gcdas_aeroanlgenb.log

View Error Logs: (gcdas_aeroanlgenb.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

@emcbot emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Dec 23, 2025
@emcbot
Copy link

emcbot commented Dec 23, 2025

C96C48mx500_S2SW_cyc_gfs FAILED on Hercules (pipeline ID: 6608)

In directory: /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/BUILDS/GITLAB/pr_cases_4376_ab1b7dc7_6608/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_ab1b7dc7-6608

This failure was detected automatically by global-workflow's CI/CD Pipeline

DavidHuber-NOAA
DavidHuber-NOAA commented Dec 24, 2025
View reviewed changes
DavidHuber-NOAA
DavidHuber-NOAA commented Dec 24, 2025
View reviewed changes
@DavidHuber-NOAA DavidHuber-NOAA removed the CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed label Dec 24, 2025
@DavidHuber-NOAA DavidHuber-NOAA marked this pull request as ready for review December 24, 2025 17:55
@DavidHuber-NOAA
Copy link
Contributor Author

DavidHuber-NOAA commented Dec 24, 2025

All tests passed on Hercules (except GCAFS, which is a known issue) following the fixes in this PR. Opening for review. Noting that this is not a high-priority. Please review as you are able.

TravisElless-NOAA
TravisElless-NOAA reviewed Dec 29, 2025
View reviewed changes
@DavidHuber-NOAA @TravisElless-NOAA
Update dev/parm/config/gfs/config.resources.HERCULES
c5b4b49 
Co-authored-by: Travis Elless <113720457+TravisElless-NOAA@users.noreply.github.com>
TravisElless-NOAA
TravisElless-NOAA approved these changes Dec 29, 2025
View reviewed changes
Copy link
Contributor

@TravisElless-NOAA TravisElless-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have Hercules access yet to test but things look fine otherwise

@DavidHuber-NOAA
Copy link
Contributor Author

DavidHuber-NOAA commented Dec 29, 2025

Launching CI on Hercules.

@emcbot emcbot added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Dec 29, 2025
@emcbot emcbot added CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Dec 29, 2025
@DavidHuber-NOAA
Copy link
Contributor Author

DavidHuber-NOAA commented Dec 30, 2025

Tests passed on Hercules. Merging.

@DavidHuber-NOAA DavidHuber-NOAA merged commit e7cdf50 into NOAA-EMC:develop Dec 30, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TravisElless-NOAA TravisElless-NOAA TravisElless-NOAA approved these changes

@aerorahul aerorahul Awaiting requested review from aerorahul

@RussTreadon-NOAA RussTreadon-NOAA Awaiting requested review from RussTreadon-NOAA

Assignees

No one assigned

Labels

CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Transient instability on Hercules

3 participants

@DavidHuber-NOAA @emcbot @TravisElless-NOAA