Skip to content

Weekly update of JEDI hashes (20260223)#2072

Merged
RussTreadon-NOAA merged 10 commits intodevelopfrom
feature/stable-nightly
Feb 23, 2026
Merged

Weekly update of JEDI hashes (20260223)#2072
RussTreadon-NOAA merged 10 commits intodevelopfrom
feature/stable-nightly

Conversation

@RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Feb 20, 2026 •
edited
Loading

Description

This PR commits two sets of changes

  • weekly update of select JEDI hashes
  • add C48_ufsenkf_atmDA to ctest and label based testing

Companion PRs

None

Issues

None

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmsnowDA
  • C96_gcafs_cycled
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_ufsgsi_hybatmDA
  • C48_ufsenkf_atmDA
  • C96C48_hybatmDA

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Feb 20, 2026
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 20, 2026

Test label based testing of C48_ufsenkf_atmDA on Ursa.

@emcbot
Copy link

emcbot commented Feb 20, 2026

Automated GW-GDASApp Testing Results:
Machine: ursa

Start: Fri Feb 20 20:01:52 UTC 2026 on ufe03
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Feb 20 20:14:40 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Fri Feb 20 20:49:31 UTC 2026
Tests: 99% tests passed, 1 tests failed out of 158
	256 - test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000 (Failed)
Tests: see output at /scratch3/NCEPDEV/da/role.jedipara/CI/ursa/GDASApp/workflow/PR/2072/global-workflow/sorc/gdas.cd/build/log.ctest

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 20, 2026

Manually rerun failed C48_ufsenkf_atmDA on Ursa after updating to f0b61fe.

Test project /scratch3/NCEPDEV/da/role.jedipara/CI/ursa/GDASApp/workflow/PR/2072/global-workflow/sorc/gdas.cd/build
      Start 302: test_gdasapp_C48_ufsenkf_atmDA
 1/14 Test #302: test_gdasapp_C48_ufsenkf_atmDA .........................................   Passed    4.11 sec
      Start 303: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_stage_ic_202402231800
 2/14 Test #303: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_stage_ic_202402231800 ..........   Passed   15.66 sec
      Start 304: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402231800
 3/14 Test #304: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402231800 ..............   Passed  179.63 sec
      Start 305: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_epmn_202402231800
 4/14 Test #305: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_epmn_202402231800 ..............   Passed   57.11 sec
      Start 307: test_gdasapp_C48_ufsenkf_atmDA_gdas_prep_202402240000
      Start 306: test_gdasapp_C48_ufsenkf_atmDA_gdas_fetchatmanlbias_202402240000
 5/14 Test #306: test_gdasapp_C48_ufsenkf_atmDA_gdas_fetchatmanlbias_202402240000 .......   Passed   26.54 sec
      Start 308: test_gdasapp_C48_ufsenkf_atmDA_gdas_prepatmanlbias_202402240000
 6/14 Test #308: test_gdasapp_C48_ufsenkf_atmDA_gdas_prepatmanlbias_202402240000 ........   Passed   61.83 sec
 7/14 Test #307: test_gdasapp_C48_ufsenkf_atmDA_gdas_prep_202402240000 ..................   Passed  403.53 sec
      Start 309: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlinit_202402240000
 8/14 Test #309: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlinit_202402240000 .....   Passed   38.23 sec
      Start 310: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlobs_202402240000
 9/14 Test #310: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlobs_202402240000 ......   Passed  250.47 sec
      Start 311: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlsol_202402240000
10/14 Test #311: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlsol_202402240000 ......   Passed  993.87 sec
      Start 312: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfv3inc_202402240000
11/14 Test #312: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfv3inc_202402240000 ...   Passed   38.17 sec
      Start 313: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfinal_202402240000
12/14 Test #313: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfinal_202402240000 ....   Passed   50.00 sec
      Start 314: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_esfc_202402240000
13/14 Test #314: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_esfc_202402240000 ..............   Passed   61.68 sec
      Start 315: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000
14/14 Test #315: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000 ..............   Passed  444.08 sec

100% tests passed, 0 tests failed out of 14

Label Time Summary:
gdasapp    = 2624.91 sec*proc (14 tests)
script     = 2624.91 sec*proc (14 tests)

Total Test time (real) = 2536.74 sec

@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion hercules-GW-RT Queue for automated testing with global-workflow on Hercules ursa-GW-RT gaeac6-GW-RT labels Feb 21, 2026
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules orion-GW-RT-Running Automated testing with global-workflow running on Orion ursa-GW-RT-Running hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion ursa-GW-RT hera-GW-RT Queue for automated testing with global-workflow on Hera gaeac6-GW-RT labels Feb 21, 2026
@emcbot emcbot added hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Feb 21, 2026
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 21, 2026

WCOSS2 GDASApp CI

Install feature/stable-nightly at 1d35802 inside g-w develop at 7bd7b7e on Cactus. Run GDASApp ctests including select g-w CI with the following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build
        Start 221: test_gdasapp_C96C48_ufsgsi_hybatmDA
        Start 243: test_gdasapp_C48_ufsenkf_atmDA
        Start 179: test_gdasapp_C96C48_hybatmDA
        Start 198: test_gdasapp_C96C48_ufs_hybatmDA
        Start 257: test_gdasapp_C96_gcafs_cycled
        Start 273: test_gdasapp_C96C48_hybatmsnowDA
        Start 294: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 305: test_gdasapp_C48mx500_hybAOWCDA
        Start 320: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 175: test_gdasapp_check_python_norms
        Start 176: test_gdasapp_check_yaml_keys
        Start 177: test_gdasapp_jcb_installation
  1/158 Test #176: test_gdasapp_check_yaml_keys .............................................   Passed    1.32 sec
        Start 178: test_gdasapp_snow_create_ens
  2/158 Test #177: test_gdasapp_jcb_installation ............................................   Passed    2.40 sec
...
157/158 Test #330: test_gdasapp_atm_jjob_ens_inc ............................................   Passed   69.05 sec
        Start 331: test_gdasapp_atm_jjob_ens_final
158/158 Test #331: test_gdasapp_atm_jjob_ens_final ..........................................   Passed   37.97 sec

100% tests passed, 0 tests failed out of 158

Label Time Summary:
gdasapp    = 25268.43 sec*proc (152 tests)
script     = 25268.43 sec*proc (152 tests)

Total Test time (real) = 2623.75 sec

All GDASApp ctests including select g-w CI _Passed on Cactus. Note that the g-w CI now include C48_ufsenkf_atmDA

russ.treadon@clogin04:/lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build> grep C48_ufsenkf log.ctest 
        Start 243: test_gdasapp_C48_ufsenkf_atmDA
  9/158 Test #243: test_gdasapp_C48_ufsenkf_atmDA ...........................................   Passed    9.62 sec
        Start 244: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_stage_ic_202402231800
 17/158 Test #244: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_stage_ic_202402231800 ............   Passed   36.31 sec
        Start 245: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402231800
 33/158 Test #245: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402231800 ................   Passed  253.30 sec
        Start 246: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_epmn_202402231800
 37/158 Test #246: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_epmn_202402231800 ................   Passed   65.09 sec
        Start 247: test_gdasapp_C48_ufsenkf_atmDA_gdas_fetchatmanlbias_202402240000
        Start 248: test_gdasapp_C48_ufsenkf_atmDA_gdas_prep_202402240000
 42/158 Test #247: test_gdasapp_C48_ufsenkf_atmDA_gdas_fetchatmanlbias_202402240000 .........   Passed   60.44 sec
        Start 249: test_gdasapp_C48_ufsenkf_atmDA_gdas_prepatmanlbias_202402240000
 50/158 Test #249: test_gdasapp_C48_ufsenkf_atmDA_gdas_prepatmanlbias_202402240000 ..........   Passed   87.57 sec
 64/158 Test #248: test_gdasapp_C48_ufsenkf_atmDA_gdas_prep_202402240000 ....................   Passed  285.13 sec
        Start 250: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlinit_202402240000
 67/158 Test #250: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlinit_202402240000 .......   Passed   63.54 sec
        Start 251: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlobs_202402240000
 78/158 Test #251: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlobs_202402240000 ........   Passed  130.01 sec
        Start 252: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlsol_202402240000
 85/158 Test #252: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlsol_202402240000 ........   Passed   95.38 sec
        Start 253: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfv3inc_202402240000
 89/158 Test #253: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfv3inc_202402240000 .....   Passed   79.51 sec
        Start 254: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfinal_202402240000
 92/158 Test #254: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_atmensanlfinal_202402240000 ......   Passed   60.89 sec
        Start 255: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_esfc_202402240000
107/158 Test #255: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_esfc_202402240000 ................   Passed   79.74 sec
        Start 256: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000
129/158 Test #256: test_gdasapp_C48_ufsenkf_atmDA_enkfgdas_fcst_202402240000 ................   Passed  247.61 sec

@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera ursa-GW-RT gaeac6-GW-RT labels Feb 21, 2026
@emcbot emcbot added ursa-GW-RT-Running hera-GW-RT-Running Automated testing with global-workflow running on Hera gaeac6-GW-RT-Running and removed ursa-GW-RT hera-GW-RT Queue for automated testing with global-workflow on Hera gaeac6-GW-RT labels Feb 21, 2026
@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: ursa

Start: Sat Feb 21 16:17:19 UTC 2026 on ufe04
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Sat Feb 21 16:31:23 UTC 2026
Build: see output at /scratch3/NCEPDEV/da/role.jedipara/CI/ursa/GDASApp/workflow/PR/2072/global-workflow/sorc/log.build

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 21, 2026

Ursa build failure
The Ursa build failed with an OOM kill

make[1]: Leaving directory '/scratch3/NCEPDEV/da/role.jedipara/CI/ursa/GDASApp/workflow/PR/2072/global-workflow/sorc/gdas.cd/build'
make: *** [Makefile:166: all] Error 2
[2026-02-21T16:30:12.705] error: Detected 2 oom_kill events in StepId=8919670.batch. Some of the step tasks have been OOM Killed.

g-w build.xml specifies ./build_gdas.sh -j 40 with

        <account>da-cpu</account>
        <queue>batch</queue>
        <partition>u1-compute</partition>
        <walltime>01:45:00</walltime>
        <cores>40</cores>
        <native>--export=NONE</native>

I don't recall seeing this error in the past. Will retry the Ursa build and test later.

@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Sat Feb 21 09:03:16 AM CST 2026 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 09:29:14 AM CST 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E C48_ufsenkf_atmDA
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 10:49:27 AM CST 2026
Tests: 100% tests passed, 0 tests failed out of 144

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Feb 21, 2026
@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: gaeac6

Start: Sat Feb 21 11:23:17 AM EST 2026 on gaea65
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 11:42:55 AM EST 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 12:14:02 PM EST 2026
Tests: 100% tests passed, 0 tests failed out of 158

@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Sat Feb 21 16:22:43 UTC 2026 on hfe05
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 16:43:20 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 17:23:01 UTC 2026
Tests: 100% tests passed, 0 tests failed out of 158

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Feb 21, 2026
@emcbot
Copy link

emcbot commented Feb 21, 2026

Automated GW-GDASApp Testing Results:
Machine: ursa

Start: Sat Feb 21 17:32:25 UTC 2026 on ufe01
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Feb 21 17:45:12 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Feb 21 18:15:48 UTC 2026
Tests: 100% tests passed, 0 tests failed out of 158

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 21, 2026

Though it took a few tries, GDASApp ctests include select g-w CI Passed on Gaea C6, Hera, Hercules, Orion, Ursa, and WCOSS2 (Cactus). This PR is ready for review.

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Feb 23, 2026

Thank you @DavidNew-NOAA . Merging now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidNew-NOAA DavidNew-NOAA DavidNew-NOAA approved these changes

@bhuang95 bhuang95 Awaiting requested review from bhuang95

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels

gaeac6-GW-RT-Passed hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules orion-GW-RT-Passed Automated testing with global-workflow successful on Orion ursa-GW-RT-Passed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@RussTreadon-NOAA @emcbot @DavidNew-NOAA