Changes in the FV3 dycore to support 3DTKE EDMF PBL physic scheme #382

BinLiu-NOAA · 2025-05-09T17:41:14Z

Description

Enable the scale-aware 3DTKE capability for the GFS TKE EDMF PBL scheme developed by Prof. Ping Zhu (@zhup01) from FIU and his group (e.g., @samuelkyfung).

Related publication: https://www.nature.com/articles/s41612-025-01117-6

Fixes ufs-community/ufs-weather-model/issues/2732

Related pull requests:

3DTKE related changes for GFS PBL scheme and add an option to use liquid potential temperature in local mixing of GFSPBL #977 NOAA-EMC/ufsatm#963
3DTKE EDMF GFS PBL scheme related changes and add an option to use liquid potential temperature in local mixing for GFSPBL PR #288 ufs-community/ccpp-physics#279
Bring in scale-aware 3DTKE related changes for GFS TKE EDMF PBL scheme #2734; Add option in GFS PBL to use liquid/ice potential temperature in local mixing #2763; Update MOM6 to its main repo. 20250527 commit #2757; Mom6 025 warmstart option #2761 ufs-community/ufs-weather-model#2734

How Has This Been Tested?

Tests conducted with the UFS HAFS application. Regression tests within ufs-weather-model passed.

Checklist:

Please check all whether they apply or not

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules

…hu@noaa.gov, and Kwun Y. Fung).

lharris4 · 2025-05-09T18:50:50Z

Hi @BinLiu-NOAA . A few comments:

So far, the 3D TKE scheme is only implemented for divergence damping; am I correct that damping on scalars and the vorticial flow component is not included in this PR?
We should consider putting the additional quantities (deform_*, dku3d_*) into a data structure to avoid cluttering up the subroutine statements, and so that these only need to be allocated when the 3D TKE scheme is active.
Should "EMDF" be "EDMF"?
It would also be good to have a reference to @zhup01's paper, even if it is in revision and not yet published.

@gaokun227 please take a close look. Ultimately we would like the code to seamlessly work with future developments in the 3D TKE scheme.

bensonr · 2025-05-09T18:53:37Z

@BinLiu-NOAA & @lharris4 - I have done a preliminary review and have some questions regarding the logic in some places. I was going to wait until the draft status was changed, but can add them in as a review if desired.

zhup01 · 2025-05-10T01:44:15Z

Hi @lharris4, Thanks for the comments. Your first comment is right. The current version only modifies the second-order damping of horizontal winds based on the predicted TKE. For your second comment, could you provide a suggestion so that I can work on it. For your last comment, a paper entitled "Toward a Unified Parameterization of Three-Dimensional Turbulent Transport in High-Resolution Numerical Weather Prediction Models" has been submitted to npj Climate and Atmospheric Science". Currently, the manuscript is under revision. I'll update the status. Thanks!

lharris4 · 2025-05-12T15:42:23Z

@zhup01 we have a couple of ways of handling arrays that are only used with specific configurations. Here, we declare nonhydrostatic arrays to be full-size only when the nonhydrostatic dynamics is active, and change the dummy array shapes as necessary:

https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1612-L1626
https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_dynamics.F90#L116

Alternately, you can put arrays in a data structure and allocate the arrays only when needed; then the data structure can be passed to routines regardless of whether the arrays in the structure are allocated or not:

https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1023
https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1568-L1579

zhup01 · 2025-05-12T17:06:50Z

@lharris4, Thank you so much for the message. Let me try it. Ping

gaokun227 · 2025-05-12T17:43:04Z

Hi @BinLiu-NOAA . A few comments:

So far, the 3D TKE scheme is only implemented for divergence damping; am I correct that damping on scalars and the vorticial flow component is not included in this PR?

We should consider putting the additional quantities (deform_*, dku3d_*) into a data structure to avoid cluttering up the subroutine statements, and so that these only need to be allocated when the 3D TKE scheme is active.

Should "EMDF" be "EDMF"?

It would also be good to have a reference to @zhup01's paper, even if it is in revision and not yet published.

@gaokun227 please take a close look. Ultimately we would like the code to seamlessly work with future developments in the 3D TKE scheme.

My general comment is that the handling of grid staggering and numerical discretization in this PR could be improved. The calculation of the TKE budget terms (deform_1 and deform_2) would be better performed using FV3's native D-grid winds. Additionally, the TKE-informed damping coefficient applied to the horizontal divergence should be defined at cell corners, where the divergence itself is computed. However, these aspects can be addressed in future development.

zhup01 · 2025-05-12T20:52:16Z

@gaokun227 Thanks for reviewing the codes and the comments. We will work on these improvements. I agree with you that we can upgrade them in the future developement.

BinLiu-NOAA · 2025-05-28T18:23:35Z

@lharris4, @gaokun227, @bensonr, thanks for the comments/suggestions!

@zhup01 and @samuelkyfung have made some changes accordingly to use a data structure for the added 3dtke arrays.

Meanwhile, please feel free to provide further review comments/suggestions!

Thanks!

driver/UFS/atmosphere.F90

model/dyn_core.F90

bensonr · 2025-05-29T13:16:03Z

driver/UFS/atmosphere.F90

+            Atm(mygrid)%sa3dtke_var%dku3d_h(i,j,k) = IPD_Tbd%dku3d_h(im,k1)
+            Atm(mygrid)%sa3dtke_var%dku3d_e(i,j,k) = IPD_Tbd%dku3d_e(im,k1)


This logic is copying the dku3d_h and dku3d_e from the physics state (IPD_tbd) into the dynamics memory space (Atm) after the dycore has run, but before the atmospheric physics are computed. Is this seeming lagging of data for dku3d*_ the intended behavior? If so, please add some comments explaining why this logic is the way it is here.

Thanks for the comments! This is a great question. dku3d_h and dku3d_e are the horizontal eddy diffusivities for momentum and scalars, which are used for calculating damping coefficient and horizontal TKE transport in the dycore. dku3d_h and dku3d_e are the functions of TKE that this determined by the TKE budget equation in the atmospheric physics. So, here is the thing. If they are put before the atmospheric physics, it means that dku3d_h and dku3d_e determined by the previous timestep TKE is used for calculating TKE transport that feeds into the atmospheric physics for updating the next timestep TKE. The updated TKE is then used for updating dku3d_h and dku3d_e for the next timestep. If they are put after the dku3d_h and dku3d_e, the TKE is updated but dku3d_h and dku3d_e are still the ones determined by the last timestep TKE since they are caclulated before TKE updating. So, the difference would be negligible. Of course, we may transfer dku3d_h and dku3d_e after the atmospheric phyics. But we don't know how to do this in the dycore.

bensonr · 2025-05-29T13:17:01Z

model/dyn_core.F90

+       call mpp_update_domains(q(:,:,:,ntke), domain)
+       call mpp_update_domains(sa3dtke_var%dku3d_e(:,:,:), domain) ! update dku3d_e at halo


These are inefficient halo updates as two MPI communications are done - one for each variable. I suggest using the optional COMPLETE argument with proper flags to buffer the updates and have them occur in a single communication.

Thank you for the comments! We have added the 'complete' argument in the recent push.

bensonr · 2025-05-29T13:17:20Z

model/dyn_core.F90

+       call mpp_update_domains(sa3dtke_var%deform_1, domain)
+       call mpp_update_domains(sa3dtke_var%deform_2, domain)
+       call mpp_update_domains(sa3dtke_var%deform_3, domain)


These are inefficient halo updates as three MPI communications are done - one for each variable. I suggest using the optional COMPLETE argument with proper flags to buffer the updates and have them occur in a single communication.

These variables are being updated within haloes, but I can't find anywhere where they are actually referenced in the halo region that would require it.

Thank you for the comments! We have added the 'complete' argument in the recent push.

lharris4 · 2025-05-29T15:11:34Z

@gaokun227 can you take another look to see if your comments have been addressed? Thanks.

lharris4

@zhup01 thank you for your continued work on the TKE code. This is an important advance for FV3 and for our broader community.

I have a couple of mostly stylistic comments. The code is quite well-written already. With a few changes your good code can be excellent code, ready for our users and for future development.

lharris4 · 2025-05-29T15:20:45Z

model/fv_dynamics.F90

-                        q_split, u, v, w, delz, hydrostatic, pt, delp, q,             &
+                        q_split, u, v, w, delz, hydrostatic,                          &
+!The following variable is for SA-3D-TKE
+                        sa3dtke_dyco,                                                 &


I would suggest not passing sa3dtke_dyco as a separate flag to the high-level fv_dynamics() routine, but instead reading it from flagstruct% when a calculation is needed.

Thanks, Samuel is working on it!

Thank you very much! We have remove the sa3dtke_dyco separate flag and reading directly from flagstruct%

Thank you for making this change. When we go to build the other drivers we will add the datatype argument to the other fv_dynamics() invocations.

lharris4 · 2025-05-29T15:22:02Z

model/fv_arrays.F90

  end type fv_nest_type
+
+  !3D-SA-TKE (kyf) (modify for data structure)
+  type sa3dtke_type


Thank you for defining this useful datatype! It will be a good foundation for future development of 3D turbulence schemes.

lharris4 · 2025-05-29T15:27:44Z

model/dyn_core.F90

    logical used
    real :: split_timestep_bc

+!The following is for SA-3D-TKE


This 3D array tke is not used; later you use the recommended q(:,:,:,sgs_tke) instead. So you can delete tke.

Thanks! We have remove this line in the current push.

lharris4 · 2025-05-29T15:30:43Z

model/dyn_core.F90


+!The following is for SA-3D-TKE
+    real:: tke(bd%isd:bd%ied,bd%jsd:bd%jed,npz)
+    integer :: ntke


In FV3 (and FMS in general) the convention for tracer indices is to use the same name as the tracer field; so I would define sgs_tke instead of ntke.

Thanks! We have switched ntke into sgs_tke in the current push.

lharris4 · 2025-05-29T15:31:30Z

model/dyn_core.F90

+! get du/dz, dv/dz and dw/dz
+
+! KGao: use Lucas's method as in compute_dudz()
+!$OMP parallel do default(none) shared(npz,is,ie,js,je,ua,va,w,  &


Good use of OpenMP!

lharris4 · 2025-05-29T15:31:57Z

model/dyn_core.F90

+    real:: dkdx(bd%isd:bd%ied,bd%jsd:bd%jed,npz)
+    real:: dkdy(bd%isd:bd%ied,bd%jsd:bd%jed,npz)
+
+    integer :: ntke


Again, I recommend calling this index sgs_tke.

lharris4 · 2025-05-29T15:39:34Z

model/sw_core.F90

        do i=is,ie+1
+!The following is for SA-3D-TKE
+          if(sa3dtke_dyco) then
+           damp2 = abs(dt)*dku3d_h(i,j) 


Thanks also for remembering abs(dt) as this allows us to do our forward-backwards initialization.

gaokun227 · 2025-05-29T17:07:50Z

@gaokun227 can you take another look to see if your comments have been addressed? Thanks.

Hi Lucas, my comments are general comments, which are related to the handling the grid staggering and can be addressed in future PR. This PR is consistent with the version of code used in Ping's npj paper and I think it is good enough.

lharris4

Looks good to me. Thank you for addressing our concerns.

XiaqiongZhou-NOAA

Great work, thanks for contributing!

laurenchilutti · 2025-06-12T16:57:44Z

@XiaqiongZhou-NOAA @gaokun227 I know you have commented on this PR, but could you officially approve this PR? Thanks!

laurenchilutti · 2025-06-12T16:58:41Z

@BinLiu-NOAA Could you please merge in the latest dev/emc branch into your branch and resolve conflicts? Thank you!

…3dtke_gfspbl

BinLiu-NOAA · 2025-06-13T15:50:51Z

@BinLiu-NOAA Could you please merge in the latest dev/emc branch into your branch and resolve conflicts? Thank you!

Synced with the latest dev/emc branch now as of 06/13/2025.

@samuelkyfung

…And it will be passed in only if the sa3dtke_dyco is ".true.". This fixes debug build and run issues when sa3dtke_dyco is ".false." (default). From @samuelkyfung and @BinLiu-NOAA.

zhup01 · 2025-06-27T02:23:53Z

@zhup01 we have a couple of ways of handling arrays that are only used with specific configurations. Here, we declare nonhydrostatic arrays to be full-size only when the nonhydrostatic dynamics is active, and change the dummy array shapes as necessary:

https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1612-L1626 https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_dynamics.F90#L116

Alternately, you can put arrays in a data structure and allocate the arrays only when needed; then the data structure can be passed to routines regardless of whether the arrays in the structure are allocated or not:

https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1023 https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1568-L1579

Lucas, thank you so much for helping us! We will try it.

zhup01 · 2025-06-27T02:44:37Z

@zhup01 we have a couple of ways of handling arrays that are only used with specific configurations. Here, we declare nonhydrostatic arrays to be full-size only when the nonhydrostatic dynamics is active, and change the dummy array shapes as necessary:
https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1612-L1626 https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_dynamics.F90#L116
Alternately, you can put arrays in a data structure and allocate the arrays only when needed; then the data structure can be passed to routines regardless of whether the arrays in the structure are allocated or not:
https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1023 https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/blob/main/model/fv_arrays.F90#L1568-L1579

Lucas, thank you so much for helping us! We will try it.

Lucas, Samuel has already created sa3dtke_type data structure to hold all deform_1, deform_2, deform_3, dku3d_h, and dku3d_e variables into one big collection. Also, he has added the flagstruct%sa3dtke_dyco in order to allocate memory only if the switch is on. Thanks.

jkbk2004 · 2025-06-27T20:22:05Z

All tests are done ok at ufs-community/ufs-weather-model#2734. This pr can be merged. @bensonr @laurenchilutti @lharris4 @thomas-robinson can you merge this pr?

3D-TKE EMDF GFS PBL scheme related changes from FIU (Ping Zhu, Ping.Z…

7bdc2c6

…hu@noaa.gov, and Kwun Y. Fung).

BinLiu-NOAA changed the title ~~Code changes in the FV3 dycore to support 3DTKE EMDF PBL physic scheme~~ Changes in the FV3 dycore to support 3DTKE EMDF PBL physic scheme May 9, 2025

BinLiu-NOAA changed the title ~~Changes in the FV3 dycore to support 3DTKE EMDF PBL physic scheme~~ Changes in the FV3 dycore to support 3DTKE EDMF PBL physic scheme May 10, 2025

Update submodules fv3/atmos_cubed_sphere for 3dtke data structure

c50e339

BinLiu-NOAA marked this pull request as ready for review May 28, 2025 18:18

laurenchilutti requested review from XiaqiongZhou-NOAA, bensonr and lharris4 May 28, 2025 19:12

bensonr reviewed May 29, 2025

View reviewed changes

driver/UFS/atmosphere.F90 Outdated Show resolved Hide resolved

model/dyn_core.F90 Outdated Show resolved Hide resolved

model/dyn_core.F90 Outdated Show resolved Hide resolved

bensonr reviewed May 29, 2025

View reviewed changes

laurenchilutti requested a review from gaokun227 May 29, 2025 15:15

lharris4 suggested changes May 29, 2025

View reviewed changes

Update based on stylistic comments and mpi use in dyn_core

bd0e414

lharris4 approved these changes Jun 3, 2025

View reviewed changes

laurenchilutti requested a review from bensonr June 9, 2025 18:45

bensonr approved these changes Jun 10, 2025

View reviewed changes

XiaqiongZhou-NOAA reviewed Jun 12, 2025

View reviewed changes

XiaqiongZhou-NOAA approved these changes Jun 12, 2025

View reviewed changes

BinLiu-NOAA mentioned this pull request May 9, 2025

Bring in the scale-aware 3DTKE EDMF GFS PBL related changes ufs-community/ufs-weather-model#2732

Closed

gaokun227 approved these changes Jun 13, 2025

View reviewed changes

Merge remote-tracking branch 'remotes/upstream/dev/emc' into feature/…

219f701

…3dtke_gfspbl

Change dku3d_h to be an optional input variable for d_sw subroutine. …

ec21a2a

…And it will be passed in only if the sa3dtke_dyco is ".true.". This fixes debug build and run issues when sa3dtke_dyco is ".false." (default). From @samuelkyfung and @BinLiu-NOAA.

laurenchilutti merged commit 4f1a5ef into NOAA-GFDL:dev/emc Jun 30, 2025

gaokun227 mentioned this pull request Jul 22, 2025

3DTKE EDMF GFS PBL related updates #387

Merged

6 tasks

		Atm(mygrid)%sa3dtke_var%dku3d_h(i,j,k) = IPD_Tbd%dku3d_h(im,k1)
		Atm(mygrid)%sa3dtke_var%dku3d_e(i,j,k) = IPD_Tbd%dku3d_e(im,k1)

		call mpp_update_domains(q(:,:,:,ntke), domain)
		call mpp_update_domains(sa3dtke_var%dku3d_e(:,:,:), domain) ! update dku3d_e at halo

Changes in the FV3 dycore to support 3DTKE EDMF PBL physic scheme #382

Changes in the FV3 dycore to support 3DTKE EDMF PBL physic scheme #382

Uh oh!

Conversation

BinLiu-NOAA commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lharris4 commented May 9, 2025

Uh oh!

bensonr commented May 9, 2025

Uh oh!

zhup01 commented May 10, 2025

Uh oh!

lharris4 commented May 12, 2025

Uh oh!

zhup01 commented May 12, 2025

Uh oh!

gaokun227 commented May 12, 2025

Uh oh!

zhup01 commented May 12, 2025

Uh oh!

BinLiu-NOAA commented May 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lharris4 commented May 29, 2025

Uh oh!

lharris4 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaokun227 commented May 29, 2025

Uh oh!

lharris4 left a comment

Choose a reason for hiding this comment

Uh oh!

XiaqiongZhou-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

laurenchilutti commented Jun 12, 2025

Uh oh!

laurenchilutti commented Jun 12, 2025

Uh oh!

BinLiu-NOAA commented Jun 13, 2025

BinLiu-NOAA commented May 9, 2025 •

edited

Loading