Add Email Notification System for scrontab-Launched Rocoto Workflows

[AntonMFernando-NOAA]

Description

• Implements automated email notifications when jobs fail in scrontab-launched Rocoto experiments

  Resolves Notify users when scrontab-based experiments fail #3486

✓ Immediate Awareness: Users are notified promptly when jobs fail
✓ Reduced Alert Fatigue: Only NEW failures trigger emails
✓ Actionable Information: Includes job details and error log paths
✓ Platform Compatible: Gaea

• Email Formatting: Provides actionable information:
  01/20/26 14:30:15 UTC :: experiment.xml :: Cycle 2021122100, Task gfs_atmos_prod, jobid=12345, in state DEAD, ran for 3600 seconds, try=1 (of 2) Error log: /path/to/comroot/experiment/logs/2021122100/gfs_atmos_prod.log

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? NO (If YES, please indicate to which system(s))
  • GFS
  • GEFS
  • SFS
  • GCAFS
• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
  • EMC verif-global
  • GDAS
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

• ✓ Gaea (scrontab with special mail flags)
• ✓ Hera (standard crontab)

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

shellcheck

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 56 in e0a5db9

done <<< "$NEW_FAILURES"

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 62 in e0a5db9

if [[ -n "$EMAIL" ]] && command -v mail &> /dev/null; then

---

[shellcheck] _{reported by reviewdog 🐶}
Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead. SC2002

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 65 in e0a5db9

cat "$MSGFILE" | mail -r "${FROM_EMAIL}" -v -s "[{pslot}] Workflow Job Failures Detected" "${EMAIL}" 2>&1

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 65 in e0a5db9

cat "$MSGFILE" | mail -r "${FROM_EMAIL}" -v -s "[{pslot}] Workflow Job Failures Detected" "${EMAIL}" 2>&1

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 68 in e0a5db9

rm -f "$MSGFILE"

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 72 in e0a5db9

echo "$FAILED_JOBS" > "$LOCKFILE"

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 72 in e0a5db9

echo "$FAILED_JOBS" > "$LOCKFILE"

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 75 in e0a5db9

[[ -f "$LOCKFILE" ]] && rm -f "$LOCKFILE"

---

[shellcheck] _{reported by reviewdog 🐶}
Prefer putting braces around variable references even when not strictly required. SC2250

global-workflow/dev/workflow/rocoto/rocoto_scron_template.sh

Line 75 in e0a5db9

[[ -f "$LOCKFILE" ]] && rm -f "$LOCKFILE"

[DavidHuber-NOAA]

Looks good!

[DavidHuber-NOAA]

I repeated testing on both Ursa and Gaea and the email was set properly in both cases. Emails were sent on Gaea following a manually triggered failure. Merging.