Skip to content

Enable WW3Tools to run across multiple machines#85

Merged
sbanihash merged 19 commits intoNOAA-EMC:developfrom
mingchen-NOAA:feature/runOnURSA
Feb 25, 2026
Merged

Enable WW3Tools to run across multiple machines#85
sbanihash merged 19 commits intoNOAA-EMC:developfrom
mingchen-NOAA:feature/runOnURSA

Conversation

@mingchen-NOAA
Copy link
Contributor

Pull Request Summary

This PR extends WW3Tools to support execution on multiple machines by introducing machine-aware configuration and logic. The update improves portability across different HPC systems.

Description

  1. Process satellite data: makeprocsatsubmit.py
  • Move all user-editable configurations to the top of the script for easier customization and readability
  • Added a MACHINE option (ursa, orion, hercules) to support multiple HPC systems
  • Enabled user-defined input and output directories
  • Added configurable SLURM settings to make job submission more flexible across platforms
  • Relocated the definition of the processed output directory from the YAML file to the Python script for clearer workflow control
  1. Combine processed satellite data into monthly: makecombinemonthly.sh
    Note: script from Jessica

  2. Interpolate model results to satellite track: makesubmitinterpgfsv16.py and makesubmitinterpretrotest.py

  • Moved all user-editable configurations to the top of each script for improved readability and easier customization
  • Added a MACHINE option (ursa, orion, hercules) to support multiple HPC systems
  • Added configurable SLURM settings to make job submission more flexible across platforms
  • Enhanced jobcard generation to be data-aware, creating jobs only when required input data are present, preventing crashes caused by missing files
  1. Created combined NetCDF files for easier post processing: CombineSatInterpOut.py
  • Moved model definitions and output directory settings from input arguments to a user-editable configuration block at the top of the script
  • Consolidated all user-editable configurations at the top of the script to improve readability and ease of customization
  • Enabled user-defined date range, processing interval (hours), and forecast days
  • Added support for user-defined seasons
  • When multiple years of data are available, enabled selection of specific year or multiple years to combine
  • Added flexible output options to allow: combining all data into a single NetCDF file, generating outputs separated by forecast days, or producing both formats
  1. Add test scripts and plotting scripts working on ursa

Issue(s) addressed

fixes #83

Commit Message

Enable WW3Tools to run across multiple machines

Check list

  • Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.

Testing

  • How were these changes tested?

MODULE_USE_PATH = "/scratch4/NCEPDEV/marine/Saeideh.Banihashemi/installs/python-modules/"
MODULE_LOAD = "Ursa_ENV"
elif MACHINE in ("orion", "hercules"):
MODULE_USE_PATH = "/work2/noaa/marine/jmeixner/general/modulefiles"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

orion and hercules probably shouldn't be sharing a module.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works on Orion and Ursa. After maintenance, I will try Hercules.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure i have a specific hercules module, i'll check after maintenance.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The module works on Hercules

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question about this script.

@mingchen-NOAA
Copy link
Contributor Author

@sbanihash This PR is ready for review. TY

Copy link
Collaborator

@sbanihash sbanihash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather have all paths in a config file, but that could be left for another PR. Tests passed. Approving PR

@sbanihash sbanihash merged commit 26cc007 into NOAA-EMC:develop Feb 25, 2026
@mingchen-NOAA
Copy link
Contributor Author

Thank you @sbanihash!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

update wave-eval to work for multiple machines from one script

3 participants