Upgrade to spack-stack 2.0.0

[RussTreadon-NOAA]

spack-stack/2.0.0 was released January 6, 2026. spack-stack/2.0.0 has been installed on the following NOAA RDHPCS machines: Gaea C6, Hercules, and Ursa. The GDASApp modulefiles and builds on these machines need to updated to spack-stack/2.0.0.

This issue is opened to document the update to spack-stack/2.0.0.

[RussTreadon-NOAA]

Work for this issue will be done in feature/ss200

[RussTreadon-NOAA]

Attempts to build GDASApp on Ursa from 18eee4df failed when compiling saber with the message

/usr/bin/ld: /contrib/spack-stack/spack-stack-2.0.0/envs/ue-oneapi-2025.2.1/install/intel-oneapi-compilers/2025.2.1/ecmwf-atlas-0.44.1-f26wo7d/lib64/libatlas.so.0.44: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
icpx: error: linker command failed with exit code 1 (use -v to see invocation)

We should cross check with JCSDA to see how they build various JEDI components with spack-stack/2.0.0.

[RussTreadon-NOAA]

Initial run of GDASApp ctests on Gaea C6 using spack-stack/2.0.0 build encountered the following failures

Test project /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build
        Start 271: test_gdasapp_C96C48_ufsgsi_hybatmDA
        Start 229: test_gdasapp_C96C48_hybatmDA
        Start 248: test_gdasapp_C96C48_ufs_hybatmDA
        Start 293: test_gdasapp_C96_gcafs_cycled
        Start 309: test_gdasapp_C96C48_hybatmsnowDA
        Start 330: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 341: test_gdasapp_C48mx500_hybAOWCDA
        Start 356: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 225: test_gdasapp_check_python_norms
        Start 226: test_gdasapp_check_yaml_keys
        Start 227: test_gdasapp_jcb_installation
        Start 228: test_gdasapp_snow_create_ens
  1/144 Test #226: test_gdasapp_check_yaml_keys .............................................   Passed    0.21 sec
        Start 355: test_gdasapp_convert_gsi_satbias
  2/144 Test #227: test_gdasapp_jcb_installation ............................................   Passed    0.52 sec
...
143/144 Test #291: test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_esfc_202402240000 ...........   Passed   99.89 sec
        Start 292: test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_fcst_202402240000
144/144 Test #292: test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_fcst_202402240000 ...........***Failed   70.13 sec

90% tests passed, 14 tests failed out of 144

Label Time Summary:
gdasapp    = 23437.95 sec*proc (138 tests)
script     = 23437.95 sec*proc (138 tests)

Total Test time (real) = 4812.41 sec

The following tests FAILED:
        257 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlvar_202402240000 (Failed) gdasapp script
        258 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfv3inc_202402240000 (Failed) gdasapp script
        259 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_atmanlfinal_202402240000 (Failed) gdasapp script
        261 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_analcalc_fv3jedi_202402240000 (Failed) gdasapp script
        262 - test_gdasapp_C96C48_ufs_hybatmDA_gdas_fcst_202402240000 (Failed) gdasapp script
        268 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_ecen_fv3jedi_202402240000 (Failed) gdasapp script
        270 - test_gdasapp_C96C48_ufs_hybatmDA_enkfgdas_fcst_202402240000 (Failed) gdasapp script
        281 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_atmanlvar_202402240000 (Failed) gdasapp script
        282 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_atmanlfv3inc_202402240000 (Failed) gdasapp script
        283 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_atmanlfinal_202402240000 (Failed) gdasapp script
        285 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_analcalc_202402240000 (Failed) gdasapp script
        286 - test_gdasapp_C96C48_ufsgsi_hybatmDA_gdas_fcst_202402240000 (Failed) gdasapp script
        290 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_ecmn_202402240000 (Failed) gdasapp script
        292 - test_gdasapp_C96C48_ufsgsi_hybatmDA_enkfgdas_fcst_202402240000 (Failed) gdasapp script

Both atmanlvar jobs failed with

 0:  *** Finish reading file: ./fv3jedi/akbk.nc4
 0:  ERROR connecting GSI-block to JEDI -- inconsistent lat with gsi, atlas =
 0:   0.000000000000000E+000  -88.5721685140070
10:  ERROR connecting GSI-block to JEDI -- inconsistent lat with gsi, atlas =
10:   0.000000000000000E+000  -69.9460806469830
10:  ERROR connecting GSI-block to JEDI -- inconsistent lat with gsi, atlas =
10:   0.000000000000000E+000  -68.0809909856510
10: gsi_covariance_mod*create: GSI and JEDI grids are inconsistent!
10: MPICH ERROR [Rank 10] [job id 215510579.0] [Tue Feb  3 13:42:01 2026] [c6n1407] - Abort(1) (rank 10 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 10

GSIBEC is not working well with ecmwf-atlas/0.44.1.

[RussTreadon-NOAA]

Repeat Ursa spack-stack/2.0.0 build with updated ursa.intel.lua. Build fails with

3.31.8-lowbhqn/bin/cmake -E cmake_link_script CMakeFiles/saber_quench_error_covariance_toolbox.x.dir/link.txt --verbose=YES
/usr/bin/ld: /contrib/spack-stack/spack-stack-2.0.0/envs/ue-oneapi-2025.2.1/install/intel-oneapi-compilers/2025.2.1/ecmwf-atlas-0.44.1-f26wo7d/lib64/libatlas.so.0.44: undefined reference to `std::condition_variable::wait(std::unique_lock<std::mutex>&)@GLIBCXX_3.4.30'
icpx: error: linker command failed with exit code 1 (use -v to see invocation)

[ADCollard]

@RussTreadon-NOAA I don't know if this helps but I was able to compile GSI with spack-stack 2.0.0 with the following changes:

ADCollard/GSI@5903fb1

Having said that, my version is seg faulting, but I am trying to use the netCDF files. It might be best to use the binaries.

[RussTreadon-NOAA]

Gaea C6: 20261112 update

Install feature/ss200 at 4be7761 inside g-w develop at 4a26a69 on Gaea C6. Run GDASApp ctests including select g-w CI with the following results

Test project /gpfs/f6/ira-sti/scratch/Russ.Treadon/git/global-workflow/ss200/sorc/gdas.cd/build
        Start 343: test_gdasapp_C96C48_ufsgsi_hybatmDA
        Start 301: test_gdasapp_C96C48_hybatmDA
        Start 320: test_gdasapp_C96C48_ufs_hybatmDA
        Start 365: test_gdasapp_C96_gcafs_cycled
        Start 381: test_gdasapp_C96C48_hybatmsnowDA
        Start 402: test_gdasapp_C48mx500_3DVarAOWCDA
        Start 413: test_gdasapp_C48mx500_hybAOWCDA
        Start 428: test_gdasapp_setup_atm_jjob_cycled_exp
        Start 297: test_gdasapp_check_python_norms
        Start 298: test_gdasapp_check_yaml_keys
        Start 299: test_gdasapp_jcb_installation
        Start 300: test_gdasapp_snow_create_ens
  1/144 Test #298: test_gdasapp_check_yaml_keys .............................................   Passed    0.15 sec
        Start 427: test_gdasapp_convert_gsi_satbias
  2/144 Test #299: test_gdasapp_jcb_installation ............................................   Passed    0.32 sec
...
143/144 Test #319: test_gdasapp_C96C48_hybatmDA_enkfgdas_fcst_202112210000 ..................   Passed  266.86 sec
144/144 Test #401: test_gdasapp_C96C48_hybatmsnowDA_enkfgdas_fcst_202112201800 ..............   Passed  187.89 sec

100% tests passed, 0 tests failed out of 144

Label Time Summary:
gdasapp    = 26988.86 sec*proc (138 tests)
script     = 26988.86 sec*proc (138 tests)

Total Test time (real) = 2757.60 sec

All tests Passed.