Reconciling ngwpc-3.1.2.3.0 PR Work, NWMv3.1 Updates for Short & Medium Range Configurations#30
Draft
jduckerOWP wants to merge 719 commits into
Draft
Reconciling ngwpc-3.1.2.3.0 PR Work, NWMv3.1 Updates for Short & Medium Range Configurations#30jduckerOWP wants to merge 719 commits into
jduckerOWP wants to merge 719 commits into
Conversation
…ing to bypass forcing_extraction
IForecast validation root dir
…emoving scratch files.
ngwpc 7354 - Fixing unstructured mesh variables node/element locations
… to the current time
fixing logic for supplemental radar forcing_extraction time handling
…-same-time Reduce unnecessary model runs (NGWPC-8671)
Klarkin ana times
…id_variable values. 2) fixed the tpxo error in the CONUS domain.
… Engine BMI can work in standalone mode and BMI implementation. Also patch configuration bug issue with implementing downscaling and bias correction methods
…CDF I/O module to produce different formats for netcdf4 forcing output supported by the NGWPC and NOAA-OWP NextGen model engine
…ions to properly utilize the ESMF output
…wnloading multiple RAP operational output datasets to enable new liquid fraction calculations
…bric and to ensure we can properly handle multi-polygons, self-intersections, and clockwise geometries for ESMF mesh compliance
…n partitioning for HRRR (hourly, 15 minute) and RAP products, masking method for liquid fraction to force temperature partitioning, inclusion of temperature dynamic lapse rate from NCAR for WRF-Hydro domain, change of NDFD precipitation variable, and inclusion of liquid fraction of precipitation for Alaska extended AnA
…uirement in production of ESMF mesh
…ructure Updated repository overview and directory descriptions for clarity and added details about workflows and capabilities.
Updated README to provide an overview of the NextGen Forcings Engine BMI and detailed installation steps.
Added installation instructions for the NextGen Forcings Engine, including dependencies and steps for setting up the environment.
Updated the README.md to enhance the clarity and detail of the NextGen Forcings Engine BMI workflow. Added information about configuration files, logging, and operational procedures based on NGWPC work and reconciliation of that work.
…and regridding for AWS s3 bucket Updated example execution code and clarified input descriptions in the README.
Added new sections for Geopackage and project options, and clarified existing configurations.
…odel engine netcdf provider standards
Update the forcing file extraction method to ensure extraction of the "pgrb" and "bgrb" operational model outputs so we can implement the liquid fraction partitioning scheme for NWMv3.1
…I Initilization phase. Removed ESMF mesh creation and forcing extraction calls from the BMI model initialization process.
Patch code to ensure ana flag will force the forecast hour to be defaulted to 1 for RAP with the given NWMv3.1 code updates.
NOAA-OWP -> (catchment-id, time) NGWPC -> (time, catchment-id)
…sistent functionality with BMI execution of Forcing Engine
Removed unnecessary fill value handling for precipitation data and updated conditions for valid indices that reflect instances when data fields do not overlap the target domain and thus, spit out fill values.
If regridded supplementary precipitation data is out of the valid range expected by the Forcing Engine, then raise an error instead of attempting to implement a nearest neighbor gap fill. There is either some wrong with the input precipitation data itself, or the methodology being implemented for the supplementary precipitation regridding and should be further investigated.
christophertubbs
requested changes
May 18, 2026
christophertubbs
left a comment
christophertubbs
left a comment
There was a problem hiding this comment.
This is not ready for submission and probably needs to be set as a draft.
| - test | ||
| - deploy | ||
|
|
||
| image: registry.sh.nextgenwaterprediction.com/infrastructure/docker/docker:latest |
There was a problem hiding this comment.
This is not a valid NOAA OWP image registry
| - docker push $DOCKER_IMAGE_NAME_BMI_FORCING:$DOCKER_IMAGE_TAG | ||
| - docker build --file=Dockerfile.lumped-forcings --build-arg CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME} --tag "$DOCKER_IMAGE_NAME_LUMPED_FORCING:$DOCKER_IMAGE_TAG" . | ||
| - docker push $DOCKER_IMAGE_NAME_LUMPED_FORCING:$DOCKER_IMAGE_TAG | ||
| - docker build --secret id=GITLAB_TOKEN,env=GITLAB_TOKEN --file=Dockerfile.ngencoastal --build-arg CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME} --tag "$DOCKER_IMAGE_NAME_COASTAL:$DOCKER_IMAGE_TAG" . |
There was a problem hiding this comment.
Where is Dockerfile.ngencoastal? Is this intentional?
|
|
||
| - name: Unit Test BMI | ||
| run: | | ||
| docker run --rm ${{ needs.setup.outputs.image_bmi }}:${{ needs.setup.outputs.test_image_tag }} \ |
There was a problem hiding this comment.
I'm not sure that this actually runs tests - it may just echo "Running BMI Forcing Unit Tests..." due to that trailing \.
There was a problem hiding this comment.
- What is this?
- Was this accidental?
jduckerOWP
marked this pull request as draft
May 18, 2026 20:01
…r Python versions beyond current 3.10.0 constraint.
…e finished regridding
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This work here attempts to reconcile PR #27 work with the scope of what needs to be harbored in the
NOAA-OWP/ngen-forcingrepository for the benefit of the National Water Model (NWM) v4 project. This includes the removal of the data assimilation and coastal scripts repository, restructuring of the NextGen Forcing Engine BMI module dependencies, and ensuring standalone execution can be completed on personal laptops or supercomputer environments, and updating documentation across the board to reflect new functionality completed by the NGWPC project. We've also updated the NextGen ESMF Mesh file production workflow to ensure ESMF compatibility is checked across the board and to improve computational performance time. This work also includes theNCAR/WrfHydroForcingenhancements completed for the NWMv3.1 project related to liquid fraction partitioning of NWM short and medium range operational configurations.Additions
Removals
Changes
Testing
ngen-forcing/NextGen_Forcings_Engine_BMIrepository.sr_config.ymlfile within thengen-forcing/NextGen_Forcings_Engine_BMIrepository to replace the pathways or forcing file output, the scratch directory to point to where you want your output, the downscaling params directory (to just a empty directory), specify the geopackage file you want to regrid forcings for, and then specify the output ESMF mesh file name based on the geopackage input file you're transforming into an ESMF mesh.module loadcommand if your system has it already preloaded.ngen-forcing/NextGen_Forcings_Engine_BMIrepository, execute the NextGen Forcing Engine BMI workflow to produce a NextGen netcdf forcing file output in your prescribed directory for a short-range operational configuration:mpirun -n 1 python bmi_wrapper.py ./sr_config.yml -output_path ./Pathway/to/Scratch/Directory -np 2