Skip to content

Weekly update of JEDI hashes (20260223)#2072

Merged
RussTreadon-NOAA merged 10 commits intodevelopfrom
feature/stable-nightly
Feb 23, 2026
Merged

Weekly update of JEDI hashes (20260223)#2072
RussTreadon-NOAA merged 10 commits intodevelopfrom
feature/stable-nightly

Conversation

@RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Feb 20, 2026 •
edited
Loading

Description

This PR commits two sets of changes

  • weekly update of select JEDI hashes
  • add C48_ufsenkf_atmDA to ctest and label based testing

Companion PRs

None

Issues

None

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmsnowDA
  • C96_gcafs_cycled
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_ufsgsi_hybatmDA
  • C48_ufsenkf_atmDA
  • C96C48_hybatmDA

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Feb 20, 2026
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 20, 2026

Test label based testing of C48_ufsenkf_atmDA on Ursa.

@emcbot
Copy link

emcbot commented Feb 20, 2026

Automated GW-GDASApp Testing Results:
Machine: ursa

Start: Fri Feb 20 20:01:52 UTC 2026 on ufe03
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Feb 20 20:14:40 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Fri Feb 20 20:49:31 UTC 2026
Tests: 99% tests passed, 1 tests failed out of 158
	256 - test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000 (Failed)
Tests: see output at /scratch3/NCEPDEV/da/role.jedipara/CI/ursa/GDASApp/workflow/PR/2072/global-workflow/sorc/gdas.cd/build/log.ctest

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 20, 2026

Manually rerun failed C48_ufsenkf_atmDA on Ursa after updating to f0b61fe.

Test project /scratch3/NCEPDEV/da/role.jedipara/CI/ursa/GDASApp/workflow/PR/2072/global-workflow/sorc/gdas.cd/build
      Start 302: test_gdasapp_C48_ufsenkf_atmDA
 1/14 Test #302: test_gdasapp_C48_ufsenkf_atmDA .........................................   Passed    4.11 sec
      Start 303: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_stage_ic_202402231800
 2/14 Test #303: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_stage_ic_202402231800 ..........   Passed   15.66 sec
      Start 304: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402231800
 3/14 Test #304: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402231800 ..............   Passed  179.63 sec
      Start 305: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_epmn_202402231800
 4/14 Test #305: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_epmn_202402231800 ..............   Passed   57.11 sec
      Start 307: test_gdasapp_C48_ufsenkf_atmDA_gdas_prep_202402240000
      Start 306: test_gdasapp_C48_ufsenkf_atmDA_gdas_fetchatmanlbias_202402240000
 5/14 Test #306: test_gdasapp_C48_ufsenkf_atmDA_gdas_fetchatmanlbias_202402240000 .......   Passed   26.54 sec
      Start 308: test_gdasapp_C48_ufsenkf_atmDA_gdas_prepatmanlbias_202402240000
 6/14 Test #308: test_gdasapp_C48_ufsenkf_atmDA_gdas_prepatmanlbias_202402240000 ........   Passed   61.83 sec
 7/14 Test #307: test_gdasapp_C48_ufsenkf_atmDA_gdas_prep_202402240000 ..................   Passed  403.53 sec
      Start 309: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlinit_202402240000
 8/14 Test #309: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlinit_202402240000 .....   Passed   38.23 sec
      Start 310: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlobs_202402240000
 9/14 Test #310: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlobs_202402240000 ......   Passed  250.47 sec
      Start 311: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlsol_202402240000
10/14 Test #311: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlsol_202402240000 ......   Passed  993.87 sec
      Start 312: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfv3inc_202402240000
11/14 Test #312: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfv3inc_202402240000 ...   Passed   38.17 sec
      Start 313: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfinal_202402240000
12/14 Test #313: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfinal_202402240000 ....   Passed   50.00 sec
      Start 314: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_esfc_202402240000
13/14 Test #314: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_esfc_202402240000 ..............   Passed   61.68 sec
      Start 315: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000
14/14 Test #315: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000 ..............   Passed  444.08 sec

100% tests passed, 0 tests failed out of 14

Label Time Summary:
gdasapp    = 2624.91 sec*proc (14 tests)
script     = 2624.91 sec*proc (14 tests)

Total Test time (real) = 2536.74 sec

@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion hercules-GW-RT Queue for automated testing with global-workflow on Hercules ursa-GW-RT gaeac6-GW-RT labels Feb 21, 2026
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules orion-GW-RT-Running Automated testing with global-workflow running on Orion ursa-GW-RT-Running hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion ursa-GW-RT hera-GW-RT Queue for automated testing with global-workflow on Hera gaeac6-GW-RT labels Feb 21, 2026
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 21, 2026

Ursa build failure
The Ursa build failed with an OOM kill

make[1]: Leaving directory '/scratch3/NCEPDEV/da/role.jedipara/CI/ursa/GDASApp/workflow/PR/2072/global-workflow/sorc/gdas.cd/build'
make: *** [Makefile:166: all] Error 2
[2026-02-21T16:30:12.705] error: Detected 2 oom_kill events in StepId=8919670.batch. Some of the step tasks have been OOM Killed.

g-w build.xml specifies ./build_gdas.sh -j 40 with

        <account>da-cpu</account>
        <queue>batch</queue>
        <partition>u1-compute</partition>
        <walltime>01:45:00</walltime>
        <cores>40</cores>
        <native>--export=NONE</native>

I don't recall seeing this error in the past. Will retry the Ursa build and test later.

@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Sat Feb 21 09:03:16 AM CST 2026 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 09:29:14 AM CST 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E C48_ufsenkf_atmDA
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 10:49:27 AM CST 2026
Tests: 100% tests passed, 0 tests failed out of 144

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Feb 21, 2026
@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: gaeac6

Start: Sat Feb 21 11:23:17 AM EST 2026 on gaea65
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 11:42:55 AM EST 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 12:14:02 PM EST 2026
Tests: 100% tests passed, 0 tests failed out of 158

@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Sat Feb 21 16:22:43 UTC 2026 on hfe05
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 16:43:20 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 17:23:01 UTC 2026
Tests: 100% tests passed, 0 tests failed out of 158

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Feb 21, 2026
@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: ursa

Start: Sat Feb 21 17:32:25 UTC 2026 on ufe01
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 17:45:12 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 18:15:48 UTC 2026
Tests: 100% tests passed, 0 tests failed out of 158

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 21, 2026

Though it took a few tries, GDASApp ctests include select g-w CI Passed on Gaea C6, Hera, Hercules, Orion, Ursa, and WCOSS2 (Cactus). This PR is ready for review.

@RussTreadon-NOAA RussTreadon-NOAA marked this pull request as ready for review February 21, 2026 18:42
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 23, 2026

Thank you @DavidNew-NOAA . Merging now.

@RussTreadon-NOAA RussTreadon-NOAA merged commit 6b356ac into develop Feb 23, 2026
6 checks passed
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/stable-nightly branch February 23, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidNew-NOAA DavidNew-NOAA DavidNew-NOAA approved these changes

@bhuang95 bhuang95 Awaiting requested review from bhuang95

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels

gaeac6-GW-RT-Passed hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules orion-GW-RT-Passed Automated testing with global-workflow successful on Orion ursa-GW-RT-Passed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@RussTreadon-NOAA @emcbot @DavidNew-NOAA