Extend diag_mediator to allow the piecemeal posting of diagnostics

[ashao]

Some quantities in MOM6 are calculated in subroutines which expect slices of the model's 2d or 3d arrays. Diagnosing these quantities can be challenging because the usual post_data calls expect whole arrays. To solve this problem, the changes here introduce a dynamic buffer that can grow as needed over the course of an simulation.

This buffer keeps track of the diagnostic IDs and the index in the buffer. When a slot in the buffer is no longer needed, for example if the whole array has been computed and posted, the slot is marked as "available" for overwriting. The buffer is only allowed to grow if all the currently allocated slots are in use.

Any computational cost associated with growing this buffer will only happen in the first few steps of the model as post_data for requested diagnostics is called for the first time in that run.

This new piecemeal posting of a diagnostic was implemented for Kd_ePBL. The new diagnostic Kd_ePBL_col_by_col is the same as the the original field in the Baltic_OM4_05 case.

@adcroft @marshallward: Two points in particular that I'd like your feedback on:

• The underlying buffer is public, but I cannot think of a way to make it private and still allow a post_data call to be done without an extra copy. I have included a store subroutine that could be used if we made it private, but otherwise those should be deleted.
• Each time we post the data in a piecemeal fashion, the slot index is computed. We could put in some logic so that we store it once per loop at the beginning of the section where the diagnostic is being computed/posted. I am not sure that it would be worth the effort/complexity for what I am assuming would be a marginal performance gain.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 200 lines in your changes missing coverage. Please review.

Please upload report for BASE (dev/gfdl@40a59f7). Learn more about missing BASE report.
Report is 2 commits behind head on dev/gfdl.

Files with missing lines	Patch %	Lines
src/framework/MOM_diag_buffers.F90	0.00%	181 Missing ⚠️
src/framework/MOM_diag_mediator.F90	0.00%	19 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             dev/gfdl     #809   +/-   ##
===========================================
  Coverage            ?   21.81%           
===========================================
  Files               ?      137           
  Lines               ?    33019           
  Branches            ?     5850           
===========================================
  Hits                ?     7202           
  Misses              ?    25260           
  Partials            ?      557           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Hallberg-NOAA]

I think that this capability should be extended to include the ability to initialize a diagnostic buffer to land values to avoid the need to call the piecemeal posting for all columns, even over land.

Also, this PR should be revised to adhere to the guidance in the MOM6 style guide at https://github.com/NOAA-GFDL/MOM6/wiki/Code-style-guide, as described in specific comments.

[ashao]

@Hallberg-NOAA Thank you for the comments. I have added descriptions of the units (see the inline response) and made the if-statements conform the to the style guide. Additionally, the fill value for the arrays can now be specified; for the purposes of the piecemeal posting of the diagnostics these are set to the DIAG_MISSING_VALUE specified in MOM_input.

[Hallberg-NOAA]

With this latest set of changes, all of my concerns have been addressed, and I think that this PR is now ready to be merged in.

[Hallberg-NOAA]

Please modify the syntax of this code so that it compiles and works correctly with all 3 of the intel, gnu and nvidia compilers.

[Hallberg-NOAA]

Unfortunately we are now getting segmentation faults with all three of the intel, gnu and nvidia compilers when we try to run the pipeline testing with this PR. The GFDL regression testing at https://gitlab.gfdl.noaa.gov/ogrp/mom6ci/MOM6/-/pipelines/28215 (and in particular the intel run job at
https://gitlab.gfdl.noaa.gov/ogrp/mom6ci/MOM6/-/jobs/163349 )
seems to indicate that the problem in various test cases arises with the allocate() statement in grow_3d() on line 225 of MOM_diag_buffers.F90, or perhaps with the allocated(this%ids) test on line 105 in find_buffer_slot(), or on the reference to this%ids(:) on line 125 in grow_ids(). I suspect that these are arising from calls with an unallocated diag_buffer object.

This did pass our TC testing, but I triggered the test again a few hours later and confirmed that these problems are not arising because of a temporary problem with the Gaea computer. @ashao, if you get the chance to revisit this, that would be appreciated.

[ashao]

Just to clarify is this a new regression for this branch, or had this branch not previously been run on the internal pipeline before?

[Hallberg-NOAA]

I went back and looked at the original regression testing that we did back in late February at https://gitlab.gfdl.noaa.gov/ogrp/mom6ci/MOM6/-/pipelines/26532. That version did not compile with the Nvidia compiler, but it did compile with gnu and intel. When it ran, it was also getting segmentation faults and bus errors. We focused on the compile failures, but the run-time errors were also problematic, and we should have pointed these out at the same time. The failed tests include everything using ePBL, the simplest of which would be something like ocean_only/single_column/EPBL.

[ashao]

@Hallberg-NOAA: Thanks! That will definitely save some time in the debugging.