Skip to content

Weekly update of JEDI hashes (20260119)#2037

Merged
RussTreadon-NOAA merged 5 commits intodevelopfrom
feature/stable-nightly
Jan 20, 2026
Merged

Weekly update of JEDI hashes (20260119)#2037
RussTreadon-NOAA merged 5 commits intodevelopfrom
feature/stable-nightly

Conversation

@RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Jan 16, 2026 •
edited
Loading

Description

This PR contains two sets of changes:

  • weekly update of select JEDI hashes
  • update the g-w build command in ci/run_ci.sh to be consistent with changes in g-w PR #4380

Companion PRs

None

Issues

None

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmsnowDA
  • C96_gcafs_cycled
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_ufsgsi_hybatmDA
  • C96C48_hybatmDA

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Jan 16, 2026
@RussTreadon-NOAA RussTreadon-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion hercules-GW-RT Queue for automated testing with global-workflow on Hercules ursa-GW-RT gaeac6-GW-RT labels Jan 17, 2026
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules orion-GW-RT-Running Automated testing with global-workflow running on Orion ursa-GW-RT-Running hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules orion-GW-RT Queue for automated testing with global-workflow on Orion ursa-GW-RT hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jan 17, 2026
@emcbot
Copy link

emcbot commented Jan 17, 2026

Automated GW-GDASApp Testing Results:
Machine: ursa

Start: Sat Jan 17 11:16:43 UTC 2026 on ufe04
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Jan 17 11:30:47 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Jan 17 12:03:07 UTC 2026
Tests: 100% tests passed, 0 tests failed out of 144

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jan 17, 2026

Automated GW-GDASApp Testing Results:
Machine: gaeac6

Start: Sat Jan 17 06:17:56 AM EST 2026 on gaea62
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Jan 17 06:38:49 AM EST 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Jan 17 07:08:41 AM EST 2026
Tests: 100% tests passed, 0 tests failed out of 144

@emcbot
Copy link

emcbot commented Jan 17, 2026

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Sat Jan 17 11:18:59 UTC 2026 on hfe11
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Jan 17 11:39:53 UTC 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Jan 17 12:20:27 UTC 2026
Tests: 100% tests passed, 0 tests failed out of 144

@emcbot emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jan 17, 2026
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jan 17, 2026

WCOSS2 ctests

Install feature/stable-nightly at c4c3c3e inside g-w develop at 7fae1ac on Dogwood. Run GDASApp ctests with the following results:

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/global-workflow/stable-nightly/sorc/gdas.cd/build
        Start 221: test_gdasapp_C96C48_ufsgsi_hybatmDA
        Start 179: test_gdasapp_C96C48_hybatmDA
        Start 198: test_gdasapp_C96C48_ufs_hybatmDA
        Start 243: test_gdasapp_C96_gcafs_cycled
        Start 259: test_gdasapp_C96C48_hybatmsnowDA
        Start 280: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 291: test_gdasapp_C48mx500_hybAOWCDA
        Start 306: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 175: test_gdasapp_check_python_norms
        Start 176: test_gdasapp_check_yaml_keys
        Start 177: test_gdasapp_jcb_installation
        Start 178: test_gdasapp_snow_create_ens
  1/144 Test #176: test_gdasapp_check_yaml_keys .............................................   Passed    1.67 sec
        Start 305: test_gdasapp_convert_gsi_satbias
  2/144 Test #177: test_gdasapp_jcb_installation ............................................   Passed    3.09 sec
...
143/144 Test #279: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201800 ..............   Passed  307.62 sec
144/144 Test #272: test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201800 ..................   Passed  492.23 sec

100% tests passed, 0 tests failed out of 144

Label Time Summary:
gdasapp    = 23756.48 sec*proc (138 tests)
script     = 23756.48 sec*proc (138 tests)

Total Test time (real) = 2559.85 sec

All tests, including select g-w CI, Passed.

@emcbot
Copy link

emcbot commented Jan 17, 2026

Automated GW-GDASApp Testing Results:
Machine: orion

Start: Sat Jan 17 05:18:03 AM CST 2026 on orion-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Jan 17 05:42:42 AM CST 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                 *SUCCESS*
Tests: Completed at Sat Jan 17 06:53:01 AM CST 2026
Tests: 100% tests passed, 0 tests failed out of 144

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jan 17, 2026
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jan 17, 2026

Hercules problem

GDASApp CI on Hercules is encountering many jobs stuck in Pending mode with reason launch failed requeued held. For example, the Hercules queue has the following jobs for this PR either running, pending, or held

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           7701548  hercules C96C48_u  role-da PD       0:00      1 (BeginTime)
           7701513  hercules C96C48_h  role-da PD       0:00      3 (launch failed requeued held)
           7701535  hercules C96C48_u  role-da PD       0:00      1 (launch failed requeued held)
           7701542  hercules C96C48_u  role-da PD       0:00      1 (launch failed requeued held)
           7701546  hercules C96C48_u  role-da PD       0:00      3 (launch failed requeued held)
           7701545  hercules C96C48_h  role-da  R       0:00      3 hercules-03-[33-35]
           7701517  hercules C48mx500  role-da  R       0:00      1 hercules-03-45
           7701529  hercules C96C48_h  role-da  R       0:00      1 hercules-03-46
           7701531  hercules C96C48_h  role-da  R       0:00      1 hercules-03-47
           7701547  hercules C96C48_u  role-da  R       0:00      1 hercules-03-48
           7701506  hercules C96C48_h  role-da  R       0:00      1 hercules-03-49
           7701538  hercules C96C48_u  role-da  R       0:00      1 hercules-03-51
           7701476  hercules C96C48_h  role-da RH       0:00      1 (launch failure limit exceeded requeued held)
           7701478  hercules C96C48_h  role-da RH       0:00      1 (launch failure limit exceeded requeued held)
           7701274  hercules C48mx500  role-da RH       0:00      1 (launch failure limit exceeded requeued held)
           7701250  hercules C96_gcaf  role-da RH       0:00      1 (launch failure limit exceeded requeued held)

This appears to be a system issue. MSU had fileset problems yesterday (1/16/2025). Perhaps the above behavior is related to these issues. I'll let the Hercules CI run until it hits the specified ctest --timeout and fails. GDASApp CI successfully ran on Orion. An attempt to rerun on Hercules will be made later.

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jan 17, 2026

Hercules update

Problems on Hercules persist. Ticket RDHPCS#2026011754000013 has been opened reporting Hercules queue problems.

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jan 17, 2026

GDASApp ctests, including select g-w CI, Passed on Gaea C6, Hera, Orion, Ursa, and WCOSS2 (Dogwood). This PR is ready for review.

@RussTreadon-NOAA RussTreadon-NOAA marked this pull request as ready for review January 17, 2026 15:50
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jan 19, 2026

Hercules update (20260119)

As of Mon Jan 19 19:45:11 UTC 2026, 715 jobs are in the Hercules queue with the following breakdown

  • 644 PENDING due to (launch failed requeued held)
  • 56 PENDING due to (JobHeldAdmin)
  • 8 PENDING due to (BeginTime)
  • 4 REQUEUE_HOLD due to (launch failure limit exceeded requeued held)
  • 3 RUN

These job cover 22 users. Something is not right on Hercules.

No response yet to Hercules ticket RDHPCS#2026011754000013.

Remove Hercules Running label.

@RussTreadon-NOAA RussTreadon-NOAA removed the hercules-GW-RT-Running Automated testing with global-workflow running on Hercules label Jan 19, 2026
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Jan 20, 2026

Thank you @DavidNew-NOAA . Merging now.

@RussTreadon-NOAA RussTreadon-NOAA changed the title Weekly update of JEDI hashes (20260120) Weekly update of JEDI hashes (20260119) Jan 20, 2026
@RussTreadon-NOAA RussTreadon-NOAA merged commit a9a6d12 into develop Jan 20, 2026
33 checks passed
@RussTreadon-NOAA RussTreadon-NOAA deleted the feature/stable-nightly branch January 20, 2026 00:55
@emcbot
Copy link

emcbot commented Jan 20, 2026

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Sat Jan 17 05:17:40 CST 2026 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Sat Jan 17 05:34:16 CST 2026
---------------------------------------------------
Tests: ctest -j12 -R gdasapp
Tests:                                  *Failed*
Tests: Failed at Tue Jan 20 07:44:33 CST 2026
Tests: 47% tests passed, 77 tests failed out of 144
	193 - test_gdasapp_C96C48_hybatmDA_enkfgdas_ediag_202112210000 (Failed)
	196 - test_gdasapp_C96C48_hybatmDA_enkfgdas_esfc_202112210000 (Failed)
	197 - test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000 (Failed)
	207 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000 (Failed)
	208 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000 (Failed)
	209 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000 (Failed)
	211 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_fv3jedi_202402240000 (Failed)
	212 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000 (Failed)
	216 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfv3inc_202402240000 (Failed)
	217 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlfinal_202402240000 (Failed)
	218 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecen_fv3jedi_202402240000 (Failed)
	220 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 (Failed)
	233 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_atmanlfinal_202402240000 (Failed)
	236 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_fcst_202402240000 (Failed)
	238 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_ediag_202402240000 (Failed)
	242 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_fcst_202402240000 (Failed)
	246 - test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201200 (Failed)
	247 - test_gdasapp_C96_gcafs_cycled_gcdas_atmos_prod_202112201200 (Failed)
	250 - test_gdasapp_C96_gcafs_cycled_gcdas_offlineanl_202112201800 (Failed)
	251 - test_gdasapp_C96_gcafs_cycled_gcdas_sfcanl_202112201800 (Failed)
	255 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlfinal_202112201800 (Failed)
	258 - test_gdasapp_C96_gcafs_cycled_gcdas_fcst_202112201800 (Failed)
	273 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eobs_202112201800 (Failed)
	274 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ediag_202112201800 (Failed)
	278 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_esfc_202112201800 (Failed)
	286 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmat_202103250000 (Failed)
	287 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	288 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	289 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	290 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	298 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmat_202103250000 (Failed)
	299 - test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_marineanlletkf_202103250000 (Failed)
	300 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	301 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	302 - test_gdasapp_C48mx500_hybAOWCDA_enkfgdas_marineanlecen_202103250000 (Failed)
	303 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	304 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	316 - test_gdasapp_atm_jjob_ens_inc (Failed)
	317 - test_gdasapp_atm_jjob_ens_final (Failed)
	188 - test_gdasapp_C96C48_hybatmDA_gdas_anal_202112210000 (Timeout)
	189 - test_gdasapp_C96C48_hybatmDA_gdas_sfcanl_202112210000 (Timeout)
	190 - test_gdasapp_C96C48_hybatmDA_gdas_analcalc_202112210000 (Timeout)
	191 - test_gdasapp_C96C48_hybatmDA_gdas_fcst_202112210000 (Timeout)
	192 - test_gdasapp_C96C48_hybatmDA_enkfgdas_eobs_202112210000 (Timeout)
	194 - test_gdasapp_C96C48_hybatmDA_enkfgdas_eupd_202112210000 (Timeout)
	195 - test_gdasapp_C96C48_hybatmDA_enkfgdas_ecmn_202112210000 (Timeout)
	201 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmos_prod_202402231800 (Timeout)
	206 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlinit_202402240000 (Timeout)
	214 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlobs_202402240000 (Timeout)
	215 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_atmensanlsol_202402240000 (Timeout)
	227 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_echgres_202402231800 (Timeout)
	231 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_atmanlvar_202402240000 (Timeout)
	232 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_atmanlfv3inc_202402240000 (Timeout)
	235 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_analcalc_202402240000 (Timeout)
	237 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_eobs_202402240000 (Timeout)
	239 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_eupd_202402240000 (Timeout)
	240 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_ecmn_202402240000 (Timeout)
	245 - test_gdasapp_C96_gcafs_cycled_gcdas_prep_emissions_202112201200 (Timeout)
	248 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201200 (Timeout)
	252 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlgenb_202112201800 (Timeout)
	253 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlinit_202112201800 (Timeout)
	254 - test_gdasapp_C96_gcafs_cycled_gcdas_aeroanlvar_202112201800 (Timeout)
	265 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_echgres_202112201200 (Timeout)
	266 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_epmn_202112201200 (Timeout)
	267 - test_gdasapp_C96C48_hybatmsnowDA_gdas_prep_202112201800 (Timeout)
	268 - test_gdasapp_C96C48_hybatmsnowDA_gdas_anal_202112201800 (Timeout)
	269 - test_gdasapp_C96C48_hybatmsnowDA_gdas_snowanl_202112201800 (Timeout)
	270 - test_gdasapp_C96C48_hybatmsnowDA_gdas_sfcanl_202112201800 (Timeout)
	271 - test_gdasapp_C96C48_hybatmsnowDA_gdas_analcalc_202112201800 (Timeout)
	272 - test_gdasapp_C96C48_hybatmsnowDA_gdas_fcst_202112201800 (Timeout)
	275 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_eupd_202112201800 (Timeout)
	276 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_ecmn_202112201800 (Timeout)
	279 - test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201800 (Timeout)
	285 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marinebmatinit_202103250000 (Timeout)
	293 - test_gdasapp_C48mx500_hybAOWCDA_gdas_fcst_202103241800 (Timeout)
	297 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marinebmatinit_202103250000 (Timeout)
	315 - test_gdasapp_atm_jjob_ens_sol (Timeout)
Tests: see output at /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/2037/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added the hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera label Jan 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidNew-NOAA DavidNew-NOAA DavidNew-NOAA approved these changes

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels

gaeac6-GW-RT-Passed hera-GW-RT-Passed Automated testing with global-workflow successful on Hera hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Passed Automated testing with global-workflow successful on Orion ursa-GW-RT-Passed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@RussTreadon-NOAA @emcbot @DavidNew-NOAA