Explicit Subcanopy Effect in UFSATM/ccpp-physics PBL-SATMEDMF Scheme

[drnimbusrain]

• This PR addresses a relevant WM issue (if not, create an issue).
• All subcomponent pull requests (if any) have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines), preferably on Ursa (Derecho or Hercules are acceptable alternatives). Exceptions: documentation-only PRs, CI-only PRs, etc.
  • Commit log file w/full results from RT suite run (if applicable).
  • Verify that test_changes.list indicates which tests, if any, are changed by this PR. Commit test_changes.list, even if it is empty.
• Fill out all sections of this template.

---

Description:

This PR facilitates using updates to the current integrated subcanopy effect to use a more robust, explicit, 3-layer subcanopy effect on chemical tracers (ONLY) in the ufsatm/ccpp-physics PBL-SATMEDMF scheme. The vegetative canopy modulates vertical diffusion. We split the diffusion on 3 sub-canopy layers with variable height located at 1, 0.5 and 0.2 canopy height (based on Makar et al. 2017), while ensuring mass conservation before returning to the original model layers (for advection, deposition, other physical processes). This leads to a more pronounced effect on the transport of chemical tracers, particularly those as precursors to near-surface ozone formation in the UFS-AQM configuration.

Commit Message:

* UFSWM - Update for Explicit Subcanopy Effects on Chemical Tracers in the UFSATM/ccpp-physics PBL-SATMEDMF Scheme
  * UFSATM - Physics (PBL-SATMEDF) Update for Explicit Subcanopy Effects on Chemical Tracers 
    * ccpp-physics - Update for Explicit Subcanopy Effects on Chemical Tracers in the PBL-SATMEDMF Scheme

Priority:

• Critical Bugfix: Reason
• High: Reason
• Normal

Git Tracking

UFSWM:

• Closes Physics (PBL-SATMEDF) Update for Explicit Subcanopy Effects on Chemical Tracers #3059

Sub component Pull Requests:

• UFSATM: Physics (PBL-SATMEDF) Update for Explicit Subcanopy Effects on Chemical Tracers NOAA-EMC/ufsatm#1056
  • ccpp-physics: Update for Explicit Subcanopy Effects on Chemical Tracers in the PBL-SATMEDMF Scheme ccpp-physics#350

UFSWM Blocking Dependencies:

• Blocked by #
• None

Documentation:

• Documentation update required.
  • Relevant updates are included with this PR.
  • A WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
• Documentation update NOT required.
  • Explanation: Unsure on how well this canopy version of the SRW-App AQM configuration has been documented initially.

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.
• PR Updates/Changes Baselines.
  Will change atmaq_canopy baseline RT.
• No Baseline Changes.

Input data Changes:

• None.
• PR adds input data.
• PR changes existing input data.

Library Changes/Upgrades:

• Required
  • Library names w/versions:
  • Git Stack Issue (JCSDA/spack-stack#)
• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[gspetro-NOAA]

@drnimbusrain Have you had a chance to run tests on this PR? Before we can schedule it, we'll need you to run the full RT suite (preferably on Ursa, but Hercules or another system would do). The hashes for stochastic_physics, fire_behavior, WW3, CICE, CMEPS, GOCART, and MOM6 are out of date and should be updated before running the RTs. You'll also need to fill out the PR template fully; currently, it's missing some information including a commit message for each submodule.

[drnimbusrain]

@drnimbusrain Have you had a chance to run tests on this PR? Before we can schedule it, we'll need you to run the full RT suite (preferably on Ursa, but Hercules or another system would do). The hashes for stochastic_physics, fire_behavior, WW3, CICE, CMEPS, GOCART, and MOM6 are out of date and should be updated before running the RTs. You'll also need to fill out the PR template fully; currently, it's missing some information including a commit message for each submodule.

@gspetro-NOAA Thank you for the comments. I have updated all hashes you mention to develop, so they should match now. I will try and run the full RT suite and commit the logs back here. We have been developing/testing on GaeaC6, but will do this on Ursa. Any support or guidance you can help with on running the full RT suite is helpful. Seems to be running the RT now though... Thank you!

For the PR template commit messages for each submodule, is that reported by the RT suite, or is what I have there now what's expected for ONLY the submodules we are modifying here, i.e., UFSATM/ccpp-physics?

[gspetro-NOAA]

I have updated all hashes you mention to develop, so they should match now. I will try and run the full RT suite and commit the logs back here. We have been developing/testing on GaeaC6, but will do this on Ursa. Any support or guidance you can help with on running the full RT suite is helpful. Seems to be running the RT now though... Thank you!

For the PR template commit messages for each submodule, is that reported by the RT suite, or is what I have there now what's expected for ONLY the submodules we are modifying here, i.e., UFSATM/ccpp-physics?

Thanks for the updates and for running the RTs! I generally just run nohup ./rt.sh -a epic -e & (you'd switch epic to your own account name), so if you did something like that, it should be sufficient. When you commit the logs, you should also commit the test_changes.list file, even if it's empty. (You can use the --allow-empty argument if there are no changes.)

For the commit messages, it looks like what you have there now is sufficient. The idea is that instead of a full history of all the commit messages, the squash and merge should just contain the summary of what this PR does in each submodule that is changed. No need to include messages for untouched submodules.

[drnimbusrain]

Hi @gspetro-NOAA Thank you for the guidance. I ran the RT and submitted the Ursa log results, as well as the test_changes.list. Actually, we do expect the atmaq_canopy baseline RT since these updates impact results there. So, with all else passing, this seems OK, correct? Thanks again!

Yes, this looks great! We merged another PR today, so you'll have to update the CMEPS hash. Other than that, we just need to get the sub-PRs approved, and we can move it into the commit queue.

Great, thank you @gspetro-NOAA for your help. I have updated CMEPS hash; however, I still see a difference in my PR related to a CMEPS submodule file, https://github.com/ufs-community/ufs-weather-model/pull/3060/files, should this be?

[gspetro-NOAA]

Great, thank you @gspetro-NOAA for your help. I have updated CMEPS hash; however, I still see a difference in my PR related to a CMEPS submodule file, https://github.com/ufs-community/ufs-weather-model/pull/3060/files, should this be?

It looks like you updated the submodule hash but didn't sync your overall branch with WM develop. See if that fixes the issue. You shouldn't see a diff for any components that your PR doesn't intend to change.

[drnimbusrain]

Great, thank you @gspetro-NOAA for your help. I have updated CMEPS hash; however, I still see a difference in my PR related to a CMEPS submodule file, https://github.com/ufs-community/ufs-weather-model/pull/3060/files, should this be?

It looks like you updated the submodule hash but didn't sync your overall branch with WM develop. See if that fixes the issue. You shouldn't see a diff for any components that your PR doesn't intend to change.

Thanks again @gspetro-NOAA. I have synced up my branch, so now it should be up-to-date, and only shows our PR changes.

[drnimbusrain]

@gspetro-NOAA OK, so we had to make some updates to our ccpp codes, and I have since updated our submodule, merged with upstream develop, and re-ran the RTs. Same results, which is good. Hopefully this can move forward soon. Thank you

[drnimbusrain]

@gspetro-NOAA I have updated again to latest develop and tried to run RTs again on Ursa to replace the Ursa RT logs. However, I am getting a lot of strange failures now in even compiling different RTs not related to atmaq canopy. We recently were over quota on Ursa scratch4 stmp where the RT rundirs are, but we cleaned up, and I am still getting the errors.

For example: /scratch3/NAGAPE/gpu-arl-gpu/Patrick.C.Campbell/ufs-weather-model_aqm_canopy_vdf_20260211_0/tests/fail_compile_wam_debug_intel

Any suggestions on why these compiles are failing?

[DeniseWorthen]

@drnimbusrain Look in your actual run directory, in the compile job's err log

/scratch4/NCEPDEV/stmp/Patrick.C.Campbell/RT_RUNDIRS/Patrick.C.Campbell/FV3_RT/rt_1645633/compile_wam_debug_intel

[BrianCurtis-NOAA]

@drnimbusrain Look in your actual run directory, in the compile job's err log

/scratch4/NCEPDEV/stmp/Patrick.C.Campbell/RT_RUNDIRS/Patrick.C.Campbell/FV3_RT/rt_1645633/compile_wam_debug_intel

Possibly an incomplete git submodule update --init --recursive that didn't bring in something?

[drnimbusrain]

Thank you @DeniseWorthen and @BrianCurtis-NOAA Seems we have a new compile error in our ccpp canopy_driver that crept in after latest update for some cleaning that we need to fix. Will correct this ASAP. Thanks!

[drnimbusrain]

@gspetro-NOAA @DeniseWorthen OK, we have fixed the new bug that was in our branch of ccpp-physics, and all submodules are updated to match upstream develop. Everything passes the RT (except expected changes in atmaq_canopy baseline).

However, FYI, now I see one additional test that has memory failure seemingly related to new p8 ugwpv1 physics and TEMPO MP/aerosol RT (nothing should be related to canopy here):

control_p8_ugwpv1_tempo_aerosol_hail intel

Failed Tests:

• TEST control_p8_ugwpv1_tempo_aerosol_hail_intel: FAILED: RUN DID NOT COMPLETE
  -- LOG: /scratch3/NAGAPE/gpu-arl-gpu/Patrick.C.Campbell/ufs-weather-model_aqm_canopy_vdf_20260211_3/tests/logs/log_ursa/run_control_p8_ugwpv1_tempo_aerosol_hail_intel.log

'[' NO = WHEN_RUNNING ']'
srun --label --distribution=block:block -n 384 ./fv3.exe
0: [2026-02-11T19:13:10.008] error: Detected 1 oom_kill event in StepId=8487681.0. Some of the step tasks have been OOM Killed.
srun: error: u17c23: task 116: Out Of Memory
srun: Terminating StepId=8487681.0
0: [2026-02-11T19:13:19.570] error: *** STEP 8487681.0 ON u17c23 CANCELLED AT 2026-02-11T19:13:19 DUE TO TASK FAILURE ***
244: forrtl: error (78): process killed (SIGTERM)
244: Image PC Routine Line Source
244: libc.so.6 0000150A2F940BF0 Unknown Unknown Unknown
244: libmpi.so.12.0.0 0000150A300986B6 Unknown Unknown Unknown
244: libmpi.so.12.0.0 0000150A30099922 Unknown Unknown Unknown
244: libmpi.so.12.0.0 0000150A2FFCBDB3 Unknown Unknown Unknown
244: libmpi.so.12.0.0 0000150A2FFAA779 Unknown Unknown Unknown
244: libmpi.so.12.0.0 0000150A300AF3ED Unknown Unknown Unknown
244: libmpi.so.12.0.0 0000150A2FE8FD34 PMPI_Allreduce Unknown Unknown
244: libmpifort.so.12. 0000150A39C7C3C0 Unknown Unknown Unknown
244: libmpifort.so.12. 0000150A39BE44FA mpi_allreduce_f08 Unknown Unknown
244: fv3.exe 00000000016C393F mpi_wrapper_mp_mp 406 mpi_wrapper.F90

When comparing our UWM ufsatm/ccpp-physics submodules we seem fully up-to-date in our branch. Thank you.

[drnimbusrain]

I also checked the latest authoritative UWM develop and ran same RTs, and get the same error with control_p8_ugwpv1_tempo_aerosol_hail intel

/scratch4/NCEPDEV/stmp/Patrick.C.Campbell/RT_RUNDIRS/Patrick.C.Campbell/FV3_RT/rt_1327997/control_p8_ugwpv1_tempo_aerosol_hail_intel/err

'[' NO = WHEN_RUNNING ']'
srun --label --distribution=block:block -n 384 ./fv3.exe
0: [2026-02-11T23:07:39.265] error: Detected 2 oom_kill events in StepId=8494401.0. Some of the step tasks have been OOM Killed.
srun: error: u13c23: tasks 33,114: Out Of Memory
srun: Terminating StepId=8494401.0
0: [2026-02-11T23:07:39.571] error: *** STEP 8494401.0 ON u13c23 CANCELLED AT 2026-02-11T23:07:39 DUE TO TASK FAILURE ***
327: forrtl: error (78): process killed (SIGTERM)

[drnimbusrain]

I also checked the latest authoritative UWM develop and ran same RTs, and get the same error with control_p8_ugwpv1_tempo_aerosol_hail intel

/scratch4/NCEPDEV/stmp/Patrick.C.Campbell/RT_RUNDIRS/Patrick.C.Campbell/FV3_RT/rt_1327997/control_p8_ugwpv1_tempo_aerosol_hail_intel/err

'[' NO = WHEN_RUNNING ']' srun --label --distribution=block:block -n 384 ./fv3.exe 0: [2026-02-11T23:07:39.265] error: Detected 2 oom_kill events in StepId=8494401.0. Some of the step tasks have been OOM Killed. srun: error: u13c23: tasks 33,114: Out Of Memory srun: Terminating StepId=8494401.0 0: [2026-02-11T23:07:39.571] error: *** STEP 8494401.0 ON u13c23 CANCELLED AT 2026-02-11T23:07:39 DUE TO TASK FAILURE *** 327: forrtl: error (78): process killed (SIGTERM)

OK, upon another RT on Ursa, I do not see the memory issue for the control_p8_ugwpv1_tempo_aerosol_hail intel this time, which is good but seems intermittent.

[gspetro-NOAA]

I also checked the latest authoritative UWM develop and ran same RTs, and get the same error with control_p8_ugwpv1_tempo_aerosol_hail intel
/scratch4/NCEPDEV/stmp/Patrick.C.Campbell/RT_RUNDIRS/Patrick.C.Campbell/FV3_RT/rt_1327997/control_p8_ugwpv1_tempo_aerosol_hail_intel/err
'[' NO = WHEN_RUNNING ']' srun --label --distribution=block:block -n 384 ./fv3.exe 0: [2026-02-11T23:07:39.265] error: Detected 2 oom_kill events in StepId=8494401.0. Some of the step tasks have been OOM Killed. srun: error: u13c23: tasks 33,114: Out Of Memory srun: Terminating StepId=8494401.0 0: [2026-02-11T23:07:39.571] error: *** STEP 8494401.0 ON u13c23 CANCELLED AT 2026-02-11T23:07:39 DUE TO TASK FAILURE *** 327: forrtl: error (78): process killed (SIGTERM)

OK, upon another RT on Ursa, I do not see the memory issue for the control_p8_ugwpv1_tempo_aerosol_hail intel this time, which is good but seems intermittent.

Thanks for pointing this out (and sorry for my delayed reply!). @mingchen-NOAA recently reported a failure on that test, and I believe @FernandoAndrade-NOAA was experiencing out of memory errors for it as well, although it eventually passed on Ursa. We may need to create an issue for it. 🤔