Skip to content

Fix eobs diags being left on the floor and a number of EE2 related updates#4267

Merged
DavidHuber-NOAA merged 23 commits intoNOAA-EMC:developfrom
aerorahul:feature/issue-2092-lean
Dec 4, 2025
Merged

Fix eobs diags being left on the floor and a number of EE2 related updates#4267
DavidHuber-NOAA merged 23 commits intoNOAA-EMC:developfrom
aerorahul:feature/issue-2092-lean

Conversation

@aerorahul
Copy link
Contributor

@aerorahul aerorahul commented Nov 24, 2025

Description

This PR:

During the course of this , it was noticed that the EnKF was not assimilating any satellite obs. This hotfix was issued by @CatherineThomas-NOAA in a different PR already.
Several variables in the analysis-related scripts are unnecessary and seem to come from earlier copy/paste versions. They can lead to confusion. This PR tries to eliminate some, though more work is recommended before code hand-off to expedite code acceptance into ops. There are also several deprecated "options" that are either no longer functional or not used.

Type of change

  • Bug fix (fixes something broken)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

Tested with the C96C48_hybatmDA test on Gaea.
Full testing is in progress on GaeaC6 and can be viewed at:
/gpfs/f6/drsa-precip4/world-shared/Rahul.Mahajan/RUNTESTS/gw-2092

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@aerorahul
Copy link
Contributor Author

Testing commenced on Gaea C6 at /gpfs/f6/drsa-precip4/world-shared/Rahul.Mahajan/RUNTESTS/gw-2092

@aerorahul
Copy link
Contributor Author

Testing commenced on Gaea C6 at /gpfs/f6/drsa-precip4/world-shared/Rahul.Mahajan/RUNTESTS/gw-2092

Testing has concluded on GaeaC6. All tests have passed. The results are below:

========== C48_ATM ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:20:08

========== C48mx500_3DVarAOWCDA ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Nov 24 2025 21:30:06    Nov 24 2025 21:55:07
202103250000        Done    Nov 24 2025 21:30:06    Nov 25 2025 00:05:06

========== C48mx500_hybAOWCDA ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Nov 24 2025 21:30:06    Nov 24 2025 21:55:07
202103250000        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:40:07

========== C48_S2SW ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Nov 24 2025 21:30:05    Nov 24 2025 23:15:07

========== C48_S2SWA_gefs ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:40:08

========== C96_atm3DVar ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Nov 24 2025 21:30:07    Nov 24 2025 21:55:05
202112210000        Done    Nov 24 2025 21:30:07    Nov 25 2025 00:05:06
202112210600        Done    Nov 24 2025 21:30:07    Nov 25 2025 00:30:08

========== C96C48_hybatmDA ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Nov 24 2025 21:30:06    Nov 24 2025 21:55:05
202112210000        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:50:07
202112210600        Done    Nov 24 2025 21:30:06    Nov 25 2025 00:30:08

========== C96C48_hybatmsnowDA ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Nov 24 2025 21:30:08    Nov 24 2025 21:55:06
202112201800        Done    Nov 24 2025 21:30:08    Nov 24 2025 23:50:09
202112210000        Done    Nov 24 2025 21:30:08    Nov 25 2025 00:25:11

========== C96C48_hybatmsoilDA ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Nov 24 2025 21:30:06    Nov 24 2025 21:55:06
202205151200        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:50:07
202205151800        Done    Nov 24 2025 21:30:06    Nov 25 2025 00:40:07

========== C96C48mx500_S2SW_cyc_gfs ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Nov 24 2025 21:30:07    Nov 24 2025 21:55:06
202112201800        Done    Nov 24 2025 21:30:07    Nov 25 2025 01:00:08
202112210000        Done    Nov 24 2025 21:30:07    Nov 25 2025 01:30:09

========== C96_gcafs_cycled ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Nov 24 2025 21:30:05    Nov 24 2025 21:55:05
202112201800        Done    Nov 24 2025 21:30:05    Nov 24 2025 23:40:07
202112210000        Done    Nov 24 2025 21:30:05    Nov 24 2025 23:45:06

========== C96_gcafs_cycled_noDA ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Nov 24 2025 21:30:06    Nov 24 2025 21:50:08
202112201800        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:40:07
202112210000        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:20:08

========== C96mx100_S2S ==========
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Nov 24 2025 21:30:06    Nov 24 2025 23:30:07

Marking the PR as ready for review.

Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the plan is to test through the full suite and then remove the commented out lines?

/
/
EOF
cat fort.41
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this cat necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added this so that the stdout can show what the input namelist to this program was. It helps with debugging when things don't go as planned.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EE2 won't like any debugging statements left in the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not considered debugging.

@DavidHuber-NOAA
Copy link
Contributor

@CatherineThomas-NOAA can you please review this PR?

@CatherineThomas-NOAA
Copy link
Contributor

Except for @CoryMartin-NOAA's comment about the commented out unused variables, this PR looks good to me. Thanks @aerorahul for giving the GSI scripts some much needed TLC.

I'd like to poke around the CI test directories a bit before approving. Can you share a path for a currently running/recently run CI test? Any machine is fine.

@aerorahul
Copy link
Contributor Author

Except for @CoryMartin-NOAA's comment about the commented out unused variables, this PR looks good to me. Thanks @aerorahul for giving the GSI scripts some much needed TLC.

I'd like to poke around the CI test directories a bit before approving. Can you share a path for a currently running/recently run CI test? Any machine is fine.

Gaea C6
/gpfs/f6/drsa-precip4/world-shared/Rahul.Mahajan/RUNTESTS/gw-2092

Note some tests have failed due to the libfabric error, but the COMs should be good enough for you to be able to see the changes.

I can remove the unused variables, but I want to keep them as a reminder that a cleanup is needed GSI-scripting-wide, and as @CoryMartin-NOAA suggests a 1 day sprint can take care of all of them.

@JessicaMeixner-NOAA
Copy link
Contributor

Rewinds of the failed jobs, should succeed with the libfabric error. (Sometimes it does take multiple tries). A helpdesk ticket is open.

@aerorahul
Copy link
Contributor Author

Rewinds of the failed jobs, should succeed with the libfabric error. (Sometimes it does take multiple tries). A helpdesk ticket is open.

Yes. That does work sometimes and with multiple tries. I did try it a few times this AM, and some attempts were successful.
Thanks for opening a helpdesk ticket.

@CatherineThomas-NOAA
Copy link
Contributor

Gaea C6
/gpfs/f6/drsa-precip4/world-shared/Rahul.Mahajan/RUNTESTS/gw-2092

Thanks! I've poked around the COM and logs and didn't find anything unexpected.

I can remove the unused variables, but I want to keep them as a reminder that a cleanup is needed GSI-scripting-wide, and as @CoryMartin-NOAA suggests a 1 day sprint can take care of all of them.

That works for me.

Copy link
Contributor

@CatherineThomas-NOAA CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved pending CI

@emcbot emcbot added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Ursa-Ready **CM use only** PR is ready for CI testing on Ursa CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully and removed CI-Ursa-Ready **CM use only** PR is ready for CI testing on Ursa CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress labels Dec 3, 2025
@DavidHuber-NOAA DavidHuber-NOAA merged commit c76804b into NOAA-EMC:develop Dec 4, 2025
6 checks passed
@DavidHuber-NOAA
Copy link
Contributor

Thank you for working on this very tricky issue @aerorahul!

@aerorahul aerorahul deleted the feature/issue-2092-lean branch December 4, 2025 20:48
JessicaMeixner-NOAA pushed a commit to JessicaMeixner-NOAA/global-workflow that referenced this pull request Dec 9, 2025
…dates (NOAA-EMC#4267)

- Tar up the gsidiags into a
gsidiag.tar instead of the GSI writing out to COM via symlinks. The
tarred GSI diags are then untarred where they are needed.
- provides a number of EE2 related updates (replacing link with copy),
`COMIN/COMOUT`, etc.

During the course of this , it was noticed that the EnKF was not
assimilating any satellite obs. This hotfix was issued by
@CatherineThomas-NOAA in a different PR already.
Several variables in the analysis-related scripts are unnecessary and
seem to come from earlier copy/paste versions. They can lead to
confusion. This PR tries to eliminate some, though more work is
recommended before code hand-off to expedite code acceptance into ops.
There are also several deprecated "options" that are either no longer
functional or not used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add additional logging to staging job The enkfgdaseobs job can fail to collect all necessary data

8 participants