Fix METplus dependency matrix

[DavidHuber-NOAA]

Description

This fixes the METplus dependencies by checking backwards for up to (GFS_INTERVAL / assim_freq) cycles for completed gfs_arch_vrfy jobs. It also does a separate check for the last cycle.

Resolves #4281

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

Generated XMLs for several different cases.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added

[aerorahul]

Changes look good to me. Just one question. Approve pending CI

[JessicaMeixner-NOAA]

@DavidHuber-NOAA -

Our xml that we had no changes on for 12 hour gfs interval looks like:

                <dependency>
                        <or>
                                <taskdep task="gfs_arch_vrfy"/>
                                <and>
                                        <not><taskvalid task="gfs_arch_vrfy"/></not>
                                        <taskdep task="gfs_arch_vrfy" cycle_offset="00:00:00"/>
                                </and>
                                <and>
                                        <not><taskvalid task="gfs_arch_vrfy"/></not>
                                        <not><cycleexistdep cycle_offset="-6:00:00"/></not>
                                        <taskdep task="gfs_arch_vrfy" cycle_offset="00:00:00"/>
                                </and>
                        </or>
                </dependency>

We know that the first <taskdep task="gfs_arch_vrfy" cycle_offset="00:00:00"/> should be <taskdep task="gfs_arch_vrfy" cycle_offset="-6:00:00"/>

Using this branch, I made a xml file for the same stream and the metp dependency is now:

                <dependency>
                        <or>
                                <taskdep task="gfs_arch_vrfy"/>
                                <and>
                                        <not><taskvalid task="gfs_arch_vrfy"/></not>
                                        <or>
                                                <taskdep task="gfs_arch_vrfy" cycle_offset="-6:00:00"/>
                                        </or>
                                </and>
                                <and>
                                        <not><taskvalid task="gfs_arch_vrfy"/></not>
                                        <taskdep task="gfs_arch_vrfy" cycle_offset="-18:00:00"/>
                                        <not><cycleexistdep cycle_offset="-6:00:00"/></not>
                                        <not><cycleexistdep cycle_offset="-12:00:00"/></not>
                                </and>
                        </or>
                </dependency>

I know there's an extra <or> in the first <and> and the second set of <and>s are different.

[DavidHuber-NOAA]

@JessicaMeixner-NOAA Breaking this down:

We know that the first <taskdep task="gfs_arch_vrfy" cycle_offset="00:00:00"/> should be <taskdep task="gfs_arch_vrfy" cycle_offset="-6:00:00"/>

Using this branch, I made a xml file for the same stream and the metp dependency is now:

                <dependency>
                        <or>
                                <taskdep task="gfs_arch_vrfy"/>
                                <and>
                                        <not><taskvalid task="gfs_arch_vrfy"/></not>
                                        <or>
                                                <taskdep task="gfs_arch_vrfy" cycle_offset="-6:00:00"/>
                                        </or>
                                </and>

^^Since the METplus jobs run on the 18z cycle and the forecasts run on 00z and 12z, this checks that the last forecast of the cycle completed (i.e. 12z). The inner <and> <or> block would expand further if ever someone were to run with INTERVAL_GFS=18 (which seems highly unlikely) to

                                     <and>
                                         <not><taskvalid task="gfs_arch_vrfy"/></not>
                                         <or>
                                                 <taskdep task="gfs_arch_vrfy" cycle_offset="-6:00:00"/>
                                                 <taskdep task="gfs_arch_vrfy" cycle_offset="-12:00:00"/>
                                         </or>
                                     </and>

                                <and>
                                        <not><taskvalid task="gfs_arch_vrfy"/></not>
                                        <taskdep task="gfs_arch_vrfy" cycle_offset="-18:00:00"/>
                                        <not><cycleexistdep cycle_offset="-6:00:00"/></not>
                                        <not><cycleexistdep cycle_offset="-12:00:00"/></not>
                                </and>
                        </or>
                </dependency>

^^ This will only be triggered by the very last cycle. Since this retro case ends on a 00z cycle and the METplus job only runs on 18z, it is looking back 18 hours AND verifying the previous 2 cycles do not exist, which would only happen on the last cycle.

Do you agree with this logic?

[JessicaMeixner-NOAA]

Thank you for the explanations @DavidHuber-NOAA - I did not fully understand the logic before and thanks for making these updates!!!

[DavidHuber-NOAA]

Launching CI on WCOSS2.

[DavidHuber-NOAA]

All tests passed on WCOSS2.

[DavidHuber-NOAA]

Will merge after final approvals.

[aerorahul]

thanks for the fix.