Add tracing instrumentation

[DusanJovic-NOAA]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

This PR adds a simple tracing module and updates some sub components' nuopc drivers to produce a trace file which can be used to identify performance issues.
The tracing module is not built and used by default. It can be enabled by setting a build option `-DUFS_TRACING=ON'

Commit Message:

* UFSWM - Add tracing instrumentation 
  * CICE - Add tracing instrumentation
  * CMEPS - Add tracing instrumentation
  * FV3 - Add tracing instrumentation
  * MOM6 - Add tracing instrumentation
  * WW3 - Add tracing instrumentation

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Add simple tracing instrumentation #2883

Sub component Pull Requests:

• CICE: Add tracing instrumentation to nuopc driver NOAA-EMC/CICE#103
• CMEPS: Add tracing instrumentation to nuopc driver NOAA-EMC/CMEPS#151
• FV3: Add tracing instrumentation to nuopc driver NOAA-EMC/ufsatm#1009
• MOM6: Add tracing instrumentation to nuopc driver  NOAA-EMC/MOM6#162
• WW3: Add tracing instrumentation to nuopc driver  NOAA-EMC/WW3#1494

UFSWM Blocking Dependencies:

• None

Documentation:

• This PR requires a documentation update, and the WM User's Guide has been updated based on the changes in this PR.
• This PR requires a documentation update, and a WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
• No documentation update is required for this PR (please explain).

---

Changes

Regression Test Changes (Please commit test_changes.list):

• No Baseline Changes.

Input data Changes:

• None.

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[DeniseWorthen]

@DusanJovic-NOAA This tracing will always be "on"; did you think about making it optional? I think each component would then need to retrieve an attribute (? do_tracing or something) and then only if maintask && do_tracing would the calls into ufs_trace be made.

EDIT: I see now that the PR includes the information that you need to compile w/ UFS_TRACING=ON.

[DusanJovic-NOAA]

@DusanJovic-NOAA This tracing will always be "on"; did you think about making it optional? I think each component would then need to retrieve an attribute (? do_tracing or something) and then only if maintask && do_tracing would the calls into ufs_trace be made.

EDIT: I see now that the PR includes the information that you need to compile w/ UFS_TRACING=ON.

Yes. Tracing will not be always "on". By default, it's "off". You turn it on by setting -DUFS_TRACING=ON. It's a build time option, not a run time. It must be a built time, because when components are used outside the UFS, the tracing subroutine is not going to be available, all tracing calls must be ifdef-ed out.  

[gspetro-NOAA]

@DusanJovic-NOAA I see this PR is open, but the PR template is not fully filled out. Do you have an estimate of when it will be ready/what remains to be done before we schedule it?

[DusanJovic-NOAA]

@DusanJovic-NOAA Please sync with develop; we're ready to begin processing this PR.

Done.

[gspetro-NOAA]

Heads up that there's a delay for the Hercules tests. RTs seemed to be hanging with a "launch failed requeued held" reason listed. I recloned and reran tests last night, and they hung around the same point (~250/320 remaining). This morning, while I was looking into reasons that might be, they restarted again and seem to be running fine, so now I'm just waiting for jobs to complete.

[JessicaMeixner-NOAA]

WW3 hash: NOAA-EMC/WW3@ae7faa5

[gspetro-NOAA]

@DusanJovic-NOAA Please revert .gitmodules and update the hashes:

• CICE: def9c53
• CMEPS: 5ec03ea
• UFSATM: 3c47d5b
• MOM6: e5a7888
• WW3: ae7faa5