Refactor cleanup task and add checks for gempak files

[DavidHuber-NOAA]

Description

This adds more specific checks for gempak files and makes an attempt at making the cleanup job less disk-intensive by running a single find -f command.

Resolves #4271
Resolves #4166

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

Will test on an extended test.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidHuber-NOAA]

FYI @CatherineThomas-NOAA @JessicaMeixner-NOAA @aerorahul. I will test this on an extended case with spoofed older COM structures.

[emcbot]

C48mx500_3DVarAOWCDA FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_da19eb69-6066

Error Log Files:

/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_gfs_flux_1p00.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_gfs_netcdfb.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_gfsa.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_ocean_grib2.log

View Error Logs: (gfs_arch_tar_gfs_flux_1p00.log) (gfs_arch_tar_gfs_netcdfb.log) (gfs_arch_tar_gfsa.log) (gfs_arch_tar_ocean_grib2.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

[DavidHuber-NOAA]

@CatherineThomas-NOAA @aerorahul Could you please review this PR?

[ChristopherHill-NOAA]

The latest test of the workflow - specifically modifying GEMPAK data file cleanup through case C96_atm3DVar_extended_gempak3 - provided the desired reduction in the retention of past cycle GEMPAK data files while retaining those GEMPAK files needed by JGFS_ATMOS_GEMPAK_META, thereby providing resolution to #4166.

With DO_GEMPAK=YES, DO_BUFRSND=YES, and DO_ARCHCOM=YES each set in config.base with the test case, and with no errors resulting after changes to gfs_bfr2gpk.sh and gfs_sndp.sh, there appears to be resolution to #4271.

[aerorahul]

thanks for addressing the comments. lgtm.

[DavidHuber-NOAA]

Launching CI on WCOSS2.

[CatherineThomas-NOAA]

Looks great. Thanks @DavidHuber-NOAA!

[DavidHuber-NOAA]

All tests passed on WCOSS2 including archiving for C48_S2SW_extended and C96_atm3DVar_extended. Merging.