Re-enable and upgrade verif-global to use spack-stack 2.0.0, metplus 6, met 12

[DavidHuber-NOAA]

Description

This reenables verif-global on most platforms (except Orion and Hera). The upgrades to verif-global include an overhaul to port the system to MET/METplus v12.0.1/6.0.0 and new support for the satellite suite of products. This also applies fixes to the JGFS_ATMOS_VERIFICATION and verif-global scripts so that the metp* jobs can be run in parallel.

Caveats:

• Currently, v16 statistics are not compatible with this version of verif-global. New .stat files will need to be generated from v16 products in standalone mode to work with this.
• Relatedly, .stat files generated by older versions of EMC_verif-global are not compatible with this version. Users will need to generate new .stat files to compare against.
• Satellite step 1 verification is now enabled in verif-global, but is not (yet) part of the workflow. At this time, there are not plans to incorporate this feature into the workflow. Additionally, this feature (AFAIK) is only supported on Gaea C6.
• VSDB validation is no longer supported. All validation should be performed using the VALID method. Standalone config files will need to be updated accordingly.

Resolves #4127

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this change expected to change outputs YES (new GFS verification products)
• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? YES (updated HPC support checklist)
• Does this change require an update to any of the following submodules? YES
  • EMC verif-global Update Ursa, Hercules to spack-stack 2.0.0, remove support for S4, Jet, C5, and update WCOSS2 support to MET/METplus 12/6 EMC_verif-global#200

How has this been tested?

Will generate .stat files for metpg2g1, metppcp1, and metpg2o1 jobs for both C48_ATM and C96C48_hybatmDA cases (with the latter being extended 48 hours to verify pcp1 statistics) on

• Ursa
• Hercules
• Gaea C6
• WCOSS2

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidHuber-NOAA]

I have successfully tested this PR on WCOSS2 and Gaea C6. Plots were generated on both platforms and align with those generated by the VPPPG team in standalone mode. Opening this PR for review.

[RussTreadon-NOAA]

Thank you @DavidHuber-NOAA for testing and confirming correct results on Gaea C6 and WCOSS2. Approve on the basis of your successful tests.

Question: After this PR enters g-w develop, developers will no longer be able to use g-w develop and have METplus support on Hera. Is this correct?

I'm aware of DA developers using Hera for global parallels testing new development. They will either need to stick with older versions of g-w for their work or rerun METplus on other machines, perhaps, Ursa, to create verification plots.

I noticed that docs/source/hpc.rst lists Hera as a Tier 1 machine. If we can't run METplus verification on Hera after this PR enters develop, should we designate Hera as a tier 2 machine?

[DavidHuber-NOAA]

@RussTreadon-NOAA Yes, that is correct, verif-global support will end on Hera when this PR is merged. And yes, I think it makes sense to downgrade Hera to tier 2.

As a side note, the background on Hera support is based on the decision by the spack-stack team not to install v2.0.0 on that platform given that it will be retiring soon. At least in theory, someone could install spack-stack v2.0.0 on that platform to enable verif-global support on that platform. However, we are actively being encouraged to migrate to other machines, so I don't think the effort would be justified, especially given that users with ongoing experiments on Hera should be able to run standalone step 1 and step 2 statistics generation/plotting on Ursa.

[RussTreadon-NOAA]

Agreed. We'll need to be prepared to answer questions from developers who use Hera after this PR is merged. They may not realize the implications of this PR.