Add VIIRS obs processing to dev/gcafs.v1 by CoryMartin-NOAA · Pull Request #4473 · NOAA-EMC/global-workflow

CoryMartin-NOAA · 2026-01-26T14:28:52Z

Description

For GCAFSv1 instead of using ObsForge for upstream observation processing, we will include the VIIRS AOD preprocessing from DCOM directly in the GCAFS workflow. This PR adds the scripts borrowed from ObsForge into the dev branch.

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES/NO (If YES, please indicate to which system(s))
- GFS
- GEFS
- SFS
- GCAFS
Is this a breaking change (a change in existing functionality)? YES/NO
Does this change require a documentation update? YES/NO
Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Checklist

…noaa/global-workflow into feature/gcafs-ecflow

…oaa/global-workflow into feature/more-ecflow

CoryMartin-NOAA · 2026-02-02T14:29:16Z

dev/jobs/JGLOBAL_OFFLINE_ATMOS_ANALYSIS

+#     COMIN_OBSPROC:COM_OBSPROC_TMPL \
+#     COMIN_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL
+export COMIN_OBSPROC=/lfs/h1/ops/prod/com/obsproc/v1.2/gdas.${PDY}/${cyc}/atmos/
+export COMIN_ATMOS_ANALYSIS=/lfs/h1/ops/prod/com/gfs/v16.3/gdas.${PDY}/${cyc}/atmos/


@aerorahul @DavidHuber-NOAA while I think the OBSPROC one can be rectified by run_envir=nco this COMIN_ATMOS_ANALYSIS is a good example I think of why this declare_from_tmpl may not be ideal

Or is the convention here that this should be COMINgfs?

global-workflow/dev/parm/config/gfs/config.com

Lines 38 to 48 in fb82cf0

if [[ "${RUN_ENVIR:-emc}" == "nco" ]]; then

COM_OBSPROC_TMPL=$(compath.py "${envir}/obsproc/${obsproc_ver}")'/${RUN}.${YMD}/${HH}/atmos'

COM_OBSFORGE_TMPL=$(compath.py "${envir}/obsforge/${obsforge_ver}")'/${RUN}.${YMD}/${HH}/'

COM_RTOFS_TMPL=$(compath.py "${envir}/rtofs/${rtofs_ver}")

COM_TCVITAL_TMPL=$(compath.py "${envir}/gfs/${gfs_ver}")'/${RUN}.${YMD}/${HH}/atmos'

else

COM_OBSPROC_TMPL='${DMPDIR}/${RUN}${DUMP_SUFFIX}.${YMD}/${HH}/atmos'

COM_OBSFORGE_TMPL='${IODADIR}/${RUN}${DUMP_SUFFIX}.${YMD}/${HH}'

COM_RTOFS_TMPL='${DMPDIR}'

COM_TCVITAL_TMPL='${DMPDIR}/${RUN}.${YMD}/${HH}/atmos'

fi

ush/jjob_header.sh

Co-authored-by: David Huber <[email protected]>

aerorahul · 2026-01-26T15:54:35Z

ush/python/pyobsforge/task/run_nc2ioda.py

+        result = subprocess.run([nc2ioda_exe, nc2ioda_yaml],
+                                cwd=task_config['DATA'],
+                                capture_output=True,
+                                text=True)


Use wxflow.executable

ush/python/pyobsforge/task/run_nc2ioda.py

aerorahul · 2026-01-26T15:56:31Z

ush/python/pyobsforge/task/aero_prepobs.py

+        logger.info("Copying ioda files to destination COMROOT directory")
+        yyyymmdd = self.task_config['PDY'].strftime('%Y%m%d')
+
+        comout = os.path.join(self.task_config['COMROOT'],


COMIN/COMOUT should be declared in the j-jobs

aerorahul · 2026-01-26T16:00:00Z

ush/python/pyobsforge/task/aero_prepobs.py

+        """
+        """
+        for platform in self.task_config.platforms:
+            print(f"========= platform: {platform}")


use logging here and elsewhere

aerorahul · 2026-02-02T14:43:18Z

dev/jobs/JGLOBAL_OFFLINE_ATMOS_ANALYSIS

+#     COMIN_OBSPROC:COM_OBSPROC_TMPL \
+#     COMIN_ATMOS_ANALYSIS:COM_ATMOS_ANALYSIS_TMPL
+export COMIN_OBSPROC=/lfs/h1/ops/prod/com/obsproc/v1.2/gdas.${PDY}/${cyc}/atmos/
+export COMIN_ATMOS_ANALYSIS=/lfs/h1/ops/prod/com/gfs/v16.3/gdas.${PDY}/${cyc}/atmos/


global-workflow/dev/parm/config/gfs/config.com

Lines 38 to 48 in fb82cf0

if [[ "${RUN_ENVIR:-emc}" == "nco" ]]; then

COM_OBSPROC_TMPL=$(compath.py "${envir}/obsproc/${obsproc_ver}")'/${RUN}.${YMD}/${HH}/atmos'

COM_OBSFORGE_TMPL=$(compath.py "${envir}/obsforge/${obsforge_ver}")'/${RUN}.${YMD}/${HH}/'

COM_RTOFS_TMPL=$(compath.py "${envir}/rtofs/${rtofs_ver}")

COM_TCVITAL_TMPL=$(compath.py "${envir}/gfs/${gfs_ver}")'/${RUN}.${YMD}/${HH}/atmos'

else

COM_OBSPROC_TMPL='${DMPDIR}/${RUN}${DUMP_SUFFIX}.${YMD}/${HH}/atmos'

COM_OBSFORGE_TMPL='${IODADIR}/${RUN}${DUMP_SUFFIX}.${YMD}/${HH}'

COM_RTOFS_TMPL='${DMPDIR}'

COM_TCVITAL_TMPL='${DMPDIR}/${RUN}.${YMD}/${HH}/atmos'

fi

CoryMartin-NOAA · 2026-02-02T14:49:46Z

Thanks @aerorahul I'll make those suggested changes later today. What do you suggest for the COMIN_ATMOS_ANALYSIS? Rename it to COMINgfs? Just define it manually and do not use the tmpl function? Something else?

aerorahul · 2026-02-02T21:01:47Z

Thanks @aerorahul I'll make those suggested changes later today. What do you suggest for the COMIN_ATMOS_ANALYSIS? Rename it to COMINgfs? Just define it manually and do not use the tmpl function? Something else?

We need a COMINgfs_ATMOS_ANALYSIS variable that makes it clear that the atmos analysis comes from GFS.
In the parm/config/gcafs/config.com, I think we should add a variable COM_GFS_TMPL=$(compath.py "${envir}/gfs/${gfs_ver}")'/${RUN}.${YMD}/${HH}' that should be used to create COMINgfs.
Once that is created, COMINgfs_ATMOS_ANALYSIS=$COMINgfs/analysis/atmos.

…oaa/global-workflow into feature/more-ecflow

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fixes for prep_emissions NRT * correct homegcafs --------- Co-authored-by: bbakernoaa <[email protected]>

…LYSIS

CoryMartin-NOAA and others added 30 commits November 14, 2025 14:49

start of an ecflow def

aa7db2e

touch a lot of empty ecflow scripts

d370bba

load modules in ecflow scripts

dbb6ceb

add prepare obs placeholder

e84b9f3

Add gcdas post and other cycles to suite definition file

564fc2e

End of day commit

279030f

add gcdas post

48b8553

Add in gcafs cycles

4c3ed57

fix bugs

c9d5fd8

Merge remote-tracking branch 'origin/develop' into feature/gcafs-ecflow

82ec40c

checkout release branch of gdas

4171f5c

add obs processing stuff to workflow; still needs work

560774d

update hash

8dba0c2

update to try on cactus

187b63b

fix link_workflow

913581b

Add expdir

5383ed6

Merge branch 'NOAA-EMC:develop' into release/gcafs.v0.0.1

ab5f856

Merge branch 'NOAA-EMC:develop' into feature/gcafs-ecflow

b1a9845

add some things

87e333f

Merge branch 'feature/gcafs-ecflow' into release/gcafs.v0.0.1

20d1501

Add EOL

27455f5

More ecflow changes

40f8bd0

more ecflow

f724aaa

Merge branch 'NOAA-EMC:develop' into feature/gcafs-ecflow

03fa9ed

More ecflow changes

0b89f40

more ecflow

fc1624e

Merge branch 'feature/gcafs-ecflow' of https://github.com/corymartin-…

c3a9fa4

…noaa/global-workflow into feature/gcafs-ecflow

Merge branch 'develop' into release/gcafs.v0.0.1

45014cb

remove script with annoying code norms

e440de0

fix issues in suite def file

4b8013d

CoryMartin-NOAA and others added 6 commits January 28, 2026 18:23

rename j-job

f07a7b2

rename back to gdas.cd

464dc14

gcdas back to gdas

54ff3d2

update hash

676c44c

Merge branch 'feature/more-ecflow' of https://github.com/corymartin-n…

80478e1

…oaa/global-workflow into feature/more-ecflow

update setup script

c7863e5

DavidNew-NOAA approved these changes Jan 30, 2026

View reviewed changes

CoryMartin-NOAA added 6 commits January 30, 2026 15:27

more small changes

b5bc5b7

some aod prep obs changes

231d08c

fix regex

b7c96de

changes for more jobs to run

4046113

fix style

5cc3b53

actually fix style

a488fe1

CoryMartin-NOAA commented Feb 2, 2026

View reviewed changes

ush/jjob_header.sh Outdated Show resolved Hide resolved

DavidHuber-NOAA reviewed Feb 2, 2026

View reviewed changes

ush/jjob_header.sh Outdated Show resolved Hide resolved

Apply suggestions from code review

0f20d5c

Co-authored-by: David Huber <[email protected]>

aerorahul reviewed Feb 2, 2026

View reviewed changes

CoryMartin-NOAA and others added 9 commits February 2, 2026 21:03

change to using wxflow executable

ed62a68

Merge branch 'feature/more-ecflow' of https://github.com/corymartin-n…

0e88c37

…oaa/global-workflow into feature/more-ecflow

Addressed comments, STILL NEEDS TESTED, USER BEWARE

27e4f4f

Update dev/jobs/JGLOBAL_OFFLINE_ATMOS_ANALYSIS

9b94a19

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

fixed a typo

bbc0b75

fixes for prep_emissions NRT (#12)

bad8284

* fixes for prep_emissions NRT * correct homegcafs --------- Co-authored-by: bbakernoaa <[email protected]>

Fix typo in logger usage

076bc71

Update GFS template path in config.com

a9d52b5

Update environment variable declarations in JGLOBAL_OFFLINE_ATMOS_ANA…

21cb379

…LYSIS

CoryMartin-NOAA merged commit 8f1c321 into NOAA-EMC:dev/gcafs.v1 Feb 5, 2026
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add VIIRS obs processing to dev/gcafs.v1#4473

Add VIIRS obs processing to dev/gcafs.v1#4473
CoryMartin-NOAA merged 101 commits intoNOAA-EMC:dev/gcafs.v1from
CoryMartin-NOAA:feature/more-ecflow

CoryMartin-NOAA commented Jan 26, 2026

Uh oh!

CoryMartin-NOAA Feb 2, 2026

Uh oh!

CoryMartin-NOAA Feb 2, 2026

Uh oh!

aerorahul Feb 2, 2026

Uh oh!

Uh oh!

Uh oh!

aerorahul Jan 26, 2026

Uh oh!

Uh oh!

aerorahul Jan 26, 2026

Uh oh!

aerorahul Jan 26, 2026

Uh oh!

aerorahul Feb 2, 2026

Uh oh!

CoryMartin-NOAA commented Feb 2, 2026

Uh oh!

aerorahul commented Feb 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

	if [[ "${RUN_ENVIR:-emc}" == "nco" ]]; then
	COM_OBSPROC_TMPL=$(compath.py "${envir}/obsproc/${obsproc_ver}")'/${RUN}.${YMD}/${HH}/atmos'
	COM_OBSFORGE_TMPL=$(compath.py "${envir}/obsforge/${obsforge_ver}")'/${RUN}.${YMD}/${HH}/'
	COM_RTOFS_TMPL=$(compath.py "${envir}/rtofs/${rtofs_ver}")
	COM_TCVITAL_TMPL=$(compath.py "${envir}/gfs/${gfs_ver}")'/${RUN}.${YMD}/${HH}/atmos'
	else
	COM_OBSPROC_TMPL='${DMPDIR}/${RUN}${DUMP_SUFFIX}.${YMD}/${HH}/atmos'
	COM_OBSFORGE_TMPL='${IODADIR}/${RUN}${DUMP_SUFFIX}.${YMD}/${HH}'
	COM_RTOFS_TMPL='${DMPDIR}'
	COM_TCVITAL_TMPL='${DMPDIR}/${RUN}.${YMD}/${HH}/atmos'
	fi

Conversation

CoryMartin-NOAA commented Jan 26, 2026

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

CoryMartin-NOAA Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

CoryMartin-NOAA Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

aerorahul Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aerorahul Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aerorahul Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

aerorahul Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

aerorahul Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

CoryMartin-NOAA commented Feb 2, 2026

Uh oh!

aerorahul commented Feb 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants