Skip to content

add logic to include avhrr in radstat tarball#4337

Merged
DavidHuber-NOAA merged 5 commits intoNOAA-EMC:developfrom
RussTreadon-NOAA:bugfix/avhrr
Dec 16, 2025
Merged

add logic to include avhrr in radstat tarball#4337
DavidHuber-NOAA merged 5 commits intoNOAA-EMC:developfrom
RussTreadon-NOAA:bugfix/avhrr

Conversation

@RussTreadon-NOAA
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA commented Dec 15, 2025 •
edited by DavidHuber-NOAA
Loading

Description

This PR adds logic to exglobal_diag.sh to include avhrr in the radstat tarball.

Resolves #4330

Type of change

  • Bug fix (fixes something broken)

Change characteristics

  • Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES
    • GFS
    • GEFS
    • SFS
    • GCAFS

I don't know if cycled GEFS, SFS, or GCAFS monitor or assimilate avhrr radiance data. If any of these networks monitor or assimilate avhrr radiance data and execute exglobal_diag.sh, the PR may alter the contents of the radstat tarball.

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

Test using g-w CI C96C48_hybatmDA

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners

@RussTreadon-NOAA RussTreadon-NOAA self-assigned this Dec 15, 2025
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Dec 15, 2025

WCOSS2 (Cactus) g-w CI C96C48_hybatmDA

Run g-w CI C96C48_hybatmDA twice on Cactus. The first run, control, did not include updates to

  • dev/scripts/exglobal_diag.sh from this PR
  • dev/scripts/exglobal_enkf_update.sh from PR #4332

The second run, update, includes updates to both these scripts from their respective PRs.

Below is a comparison of output from each run.

exglobal_diag.sh update
The control and update gdas_anal and enkfgdas_eobs processed avhrr data

gdas gsistat

o-g 01 rad   metop-b      avhrr           2053331        13052         7898    1633.0       1633.0      0.20676      0.20676    
o-g 01 rad   n18          avhrr           2100626        17319         9254    2002.5       2002.5      0.21639      0.21639    
o-g 01 rad   n19          avhrr           1645190        14250         7559    1850.9       1850.9      0.24486      0.24486

enkfgdas gsistat

o-g 01 rad   metop-b      avhrr           2053331         6533         3392    813.12       813.12      0.23972      0.23972    
o-g 01 rad   n18          avhrr           2100626         8583         4024    1077.9       1077.9      0.26786      0.26786    
o-g 01 rad   n19          avhrr           1645190         7170         3356    852.74       852.74      0.25409      0.25409

The control gdas and enkfgdas radstat do not contain avhrr data
gdas.t00z.radstat.tar

russ.treadon@clogin04> tar -tvf gdas.t00z.radstat.tar | wc
     42     252    3886
russ.treadon@clogin04:> tar -tvf gdas.t00z.radstat.tar | grep avhrr
russ.treadon@clogin04:>

enkfgdas.t00z.radstat_ensmean.tar

russ.treadon@clogin04:> tar -tvf enkfgdas.t00z.radstat_ensmean.tar  | wc
     17     102    1660
russ.treadon@clogin04:> tar -tvf enkfgdas.t00z.radstat_ensmean.tar  | grep avhrr
russ.treadon@clogin04:>

The update tarballs contain avhrr data
gdas.t00z.radstat.tar

russ.treadon@clogin05:> tar -tvf gdas.t00z.radstat.tar | wc
     48     288    4440
russ.treadon@clogin05:> tar -tvf gdas.t00z.radstat.tar | grep avhrr
-rw-r--r-- russ.treadon/emc  777920 2025-12-15 18:46 diag_avhrr_metop-b_ges.2021122100.nc4.gz
-rw-r--r-- russ.treadon/emc  887230 2025-12-15 18:46 diag_avhrr_n18_ges.2021122100.nc4.gz
-rw-r--r-- russ.treadon/emc  726606 2025-12-15 18:46 diag_avhrr_n19_ges.2021122100.nc4.gz
-rw-r--r-- russ.treadon/emc    784625 2025-12-15 18:46 diag_avhrr_metop-b_anl.2021122100.nc4.gz
-rw-r--r-- russ.treadon/emc    894026 2025-12-15 18:46 diag_avhrr_n18_anl.2021122100.nc4.gz
-rw-r--r-- russ.treadon/emc    732274 2025-12-15 18:46 diag_avhrr_n19_anl.2021122100.nc4.gz

enkfgdas.t00z.radstat_ensmean.tar

russ.treadon@clogin05:> tar -tvf enkfgdas.t00z.radstat_ensmean.tar | wc
     20     120    1955
russ.treadon@clogin05:> tar -tvf enkfgdas.t00z.radstat_ensmean.tar | grep avhrr
-rw-r--r-- russ.treadon/emc  10336047 2025-12-15 18:55 diag_avhrr_metop-b_ges.2021122100_ensmean.nc4
-rw-r--r-- russ.treadon/emc  11484599 2025-12-15 18:55 diag_avhrr_n18_ges.2021122100_ensmean.nc4
-rw-r--r-- russ.treadon/emc   9648546 2025-12-15 18:55 diag_avhrr_n19_ges.2021122100_ensmean.nc4

exglobal_enkf_update.sh update
The control enkfgdas.t00z.enkfstat.txt has the following assimilated data types

nid001019.cactus.wcoss2.ncep.noaa.gov 2:    54         amsua_metop-b  num_obs_tot=     71264
nid001019.cactus.wcoss2.ncep.noaa.gov 2:    66         amsua_metop-c  num_obs_tot=    243633
nid001019.cactus.wcoss2.ncep.noaa.gov 2:     2             amsua_n18  num_obs_tot=     26721
nid001019.cactus.wcoss2.ncep.noaa.gov 2:     3             amsua_n19  num_obs_tot=     48936
nid001019.cactus.wcoss2.ncep.noaa.gov 2:    60              atms_n20  num_obs_tot=    205977
nid001019.cactus.wcoss2.ncep.noaa.gov 2:    59              atms_npp  num_obs_tot=    138484
nid001019.cactus.wcoss2.ncep.noaa.gov 1:     5              omi_aura  nread=      1359  nkeep=     1359  num_obs_tot=      1359
nid001019.cactus.wcoss2.ncep.noaa.gov 1:    12            ompsnp_n20  nread=     29172  nkeep=    27839  num_obs_tot=     34187
nid001019.cactus.wcoss2.ncep.noaa.gov 1:    11           ompstc8_n20  nread=      2797  nkeep=     2796  num_obs_tot=      6348
nid001019.cactus.wcoss2.ncep.noaa.gov 1:    10           ompstc8_npp  nread=      2193  nkeep=     2193  num_obs_tot=      3552

avhrr data along with many other radiances (brightness temperature) observations are not assimilated.

The update enkfgdas.t00z.enkfstat.txt has the following assimilated data types

nid003715.cactus.wcoss2.ncep.noaa.gov 2:    69               abi_g16  num_obs_tot=    991315
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    68         ahi_himawari8  num_obs_tot=    989241
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    54         amsua_metop-b  num_obs_tot=    142555
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    66         amsua_metop-c  num_obs_tot=    979012
nid003715.cactus.wcoss2.ncep.noaa.gov 2:     1             amsua_n15  num_obs_tot=     33168
nid003715.cactus.wcoss2.ncep.noaa.gov 2:     2             amsua_n18  num_obs_tot=     59889
nid003715.cactus.wcoss2.ncep.noaa.gov 2:     3             amsua_n19  num_obs_tot=     82104
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    60              atms_n20  num_obs_tot=    729123
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    59              atms_npp  num_obs_tot=    661630
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    58         avhrr_metop-b  num_obs_tot=    594410
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    20             avhrr_n18  num_obs_tot=     93349
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    22             avhrr_n19  num_obs_tot=     96705
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    63          cris-fsr_n20  num_obs_tot=    941356
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    57          iasi_metop-b  num_obs_tot=    591018
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    71          iasi_metop-c  num_obs_tot=   1439830
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    56           mhs_metop-b  num_obs_tot=    149263
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    67           mhs_metop-c  num_obs_tot=    984967
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    13               mhs_n19  num_obs_tot=     89325
nid003715.cactus.wcoss2.ncep.noaa.gov 1:     5              omi_aura  nread=      1359  nkeep=     1359  num_obs_tot=      1359
nid003715.cactus.wcoss2.ncep.noaa.gov 1:    12            ompsnp_n20  nread=     29172  nkeep=    27839  num_obs_tot=     34187
nid003715.cactus.wcoss2.ncep.noaa.gov 1:    11           ompstc8_n20  nread=      2797  nkeep=     2796  num_obs_tot=      6348
nid003715.cactus.wcoss2.ncep.noaa.gov 1:    10           ompstc8_npp  nread=      2193  nkeep=     2193  num_obs_tot=      3552
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    50            seviri_m08  num_obs_tot=    120227
nid003715.cactus.wcoss2.ncep.noaa.gov 2:    25             ssmis_f17  num_obs_tot=    118271

All radiances, including avhrr, are assimilated.

The control and update eupd assimilate the same number of conventional observations and ozone observations. It's only the number of radiances which differ. The update assimilates much more radiance data.
control

nid001019.cactus.wcoss2.ncep.noaa.gov 0:  nobs_conv, nobs_oz, nobs_sat =      2342844       34187      243633
 total diag nobs_conv, nobs_oz, nobs_sat =      1937949       35521      399904

update

nid003715.cactus.wcoss2.ncep.noaa.gov 0:  nobs_conv, nobs_oz, nobs_sat =      2342844       34187     1439830
 total diag nobs_conv, nobs_oz, nobs_sat =      1937949       35521     2621534

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Dec 15, 2025

C96C48_hybatmDA tests indicate that the modified dev/scripts/exglobal_diag.sh in this PR works as intended. This PR is ready for review.

@RussTreadon-NOAA RussTreadon-NOAA marked this pull request as ready for review December 15, 2025 19:21
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Dec 16, 2025

Thanks you @CoryMartin-NOAA for your review and approval.

@aerorahul , @DavidHuber-NOAA , @jswhit , @CatherineThomas-NOAA : any suggestions I should incorporate? If not, I'll launch g-w CI on a NOAA RDHPCS machine.

@DavidHuber-NOAA DavidHuber-NOAA added the GFS Change This PR, if merged, will change results for the GFS. label Dec 16, 2025
DavidHuber-NOAA
DavidHuber-NOAA approved these changes Dec 16, 2025
View reviewed changes
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks @RussTreadon-NOAA.

@DavidHuber-NOAA
Copy link
Contributor

DavidHuber-NOAA commented Dec 16, 2025

@RussTreadon-NOAA I am happy to launch automated tests on Ursa or Hera unless you would like to monitor the tests. Which would you prefer?

@CatherineThomas-NOAA I will wait for your review before merging.

@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Dec 16, 2025

@DavidHuber-NOAA : Thank you for offering to launch automated tests for this PR. I'll stand down on running the tests myself.

@emcbot emcbot added CI-Ursa-Ready **CM use only** PR is ready for CI testing on Ursa CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa and removed CI-Ursa-Ready **CM use only** PR is ready for CI testing on Ursa labels Dec 16, 2025
CatherineThomas-NOAA
CatherineThomas-NOAA approved these changes Dec 16, 2025
View reviewed changes
Copy link
Contributor

@CatherineThomas-NOAA CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good to me and @RussTreadon-NOAA's test results show that it works as intended. Approved.

@emcbot emcbot added CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully and removed CI-Ursa-Building **Bot use only** CI testing is cloning/building on Ursa CI-Ursa-Running **Bot use only** CI testing on Ursa for this PR is in-progress labels Dec 16, 2025
@RussTreadon-NOAA
Copy link
Contributor Author

RussTreadon-NOAA commented Dec 16, 2025

Check

Spot check gdas radstat and enkfgdas radstat_ensmean tarballs from Ursa g-w CI and Gaea C6 g-w CI run for PR #4306. The Gaea C6 run does not have the avhrr fix. gdas and enkfgdas radstat tarballs from the Ursa test contain avhrr/

Below is an example of this check for C96C48_hybatmDA.

Ursa(ufe02):/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4337_a05fd274_6387/RUNTESTS/COMROOT$ tar -tvf ./C96C48_hybatmDA_a05fd274-6387/enkfgdas.20211221/06/ensstat/analysis/atmos/enkfgdas.t06z.radstat_ensmean.tar |wc
     20     120    1975
Ursa(ufe02):/scratch3/NCEPDEV/global/role.glopara/GFS_CI_CD/URSA/BUILDS/GITLAB/pr_cases_4337_a05fd274_6387/RUNTESTS/COMROOT$ tar -tvf ./C96C48_hybatmDA_a05fd274-6387/enkfgdas.20211221/06/ensstat/analysis/atmos/enkfgdas.t06z.radstat_ensmean.tar |grep avhrr
-rw-rw-r-- role.glopara/stmp  10314810 2025-12-16 17:08 diag_avhrr_metop-b_ges.2021122106_ensmean.nc4
-rw-rw-r-- role.glopara/stmp  10357289 2025-12-16 17:08 diag_avhrr_n18_ges.2021122106_ensmean.nc4
-rw-rw-r-- role.glopara/stmp  11252838 2025-12-16 17:08 diag_avhrr_n19_ges.2021122106_ensmean.nc4

The Ursa radstat tarball contains avhrr files

The Gaea C6 radstat tarball does not contain avhrr

Gaea(gaea67):/gpfs/f6/ira-sti/scratch/Russ.Treadon/stmp2/COMROOT$ tar -tvf ./C96C48_hybatmDA_pr4306/enkfgdas.20211221/06/ensstat/analysis/atmos/enkfgdas.t06z.radstat_ensmean.tar |wc
     17     102    1728
Gaea(gaea67):/gpfs/f6/ira-sti/scratch/Russ.Treadon/stmp2/COMROOT$ tar -tvf ./C96C48_hybatmDA_pr4306/enkfgdas.20211221/06/ensstat/analysis/atmos/enkfgdas.t06z.radstat_ensmean.tar |grep avhrr
Gaea(gaea67):/gpfs/f6/ira-sti/scratch/Russ.Treadon/stmp2/COMROOT$

@DavidHuber-NOAA DavidHuber-NOAA merged commit 7f875e8 into NOAA-EMC:develop Dec 16, 2025
6 checks passed
@RussTreadon-NOAA RussTreadon-NOAA deleted the bugfix/avhrr branch December 17, 2025 11:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CoryMartin-NOAA CoryMartin-NOAA CoryMartin-NOAA approved these changes

@CatherineThomas-NOAA CatherineThomas-NOAA CatherineThomas-NOAA approved these changes

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA approved these changes

@aerorahul aerorahul Awaiting requested review from aerorahul

Assignees

@RussTreadon-NOAA RussTreadon-NOAA

Labels

CI-Ursa-Passed **Bot use only** CI testing on Ursa for this PR has completed successfully GFS Change This PR, if merged, will change results for the GFS.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

avhrr data not in gdas and enkfgdas radstat tarballs

5 participants

@RussTreadon-NOAA @DavidHuber-NOAA @CoryMartin-NOAA @CatherineThomas-NOAA @emcbot