Allow user to specify exposures or tiles to resubmit

[akremin]

Overview

This makes changes to desi_resubmit_queue_failures and desispec.workflow.processing.update_and_recursively_submit() to allow the user to specify either exposure ID's or tile ID's that they want to resubmit failures for. The code warns the user that any dependencies that failed will also be resubmitted but only submits the specified jobs and any required dependencies. If a specified job was successful, the script skips that entry. The command line script accepts a comma-separated list of exposures or tiles. If both are specified it raises an error. If none are specified it resubmits all failures on the night, as it did prior to this code change.

Testing

I ran 5 tests and confirmed that it did what I expected in all 5 circumstances. Logs for all 5 tests can be found in: /global/cfs/cdirs/desi/users/kremin/PRs/specify_resub.

Test 1: desi_resubmit_queue_failures -n 20250215 -e "279598,279654,279655" --dry-run-level=3 > test_exps_2resub_1good.log

• This specifies 3 science exposures on 20250215. 2 needed resubmission while 1 did not. It correctly resubmits the tilenight and cumulative jobs for just the 2 exposures that had previously failed.

Test 2: desi_resubmit_queue_failures -n 20250215 -e "279654,279655,279672" --dry-run-level=3 > test_exps_3resub.log

• This specifies 3 science exposures on 20250215. All 3 needed resubmission. It correctly resubmits the tilenight and cumulative jobs for just the 3 exposures that had previously failed, and nothing else.

Test 3: desi_resubmit_queue_failures -n 20250215 -t "83565,83566,83571" --dry-run-level=3 > test_tiles_3resub.log

• This specifies 3 science tiles on 20250215. All 3 needed resubmission. It correctly resubmits the tilenight and cumulative jobs for just the 3 tiles that had previously failed, and nothing else.

Test 4: desi_resubmit_queue_failures -n 20250215 -t "83565,83566,40457" --dry-run-level=3 > test_tiles_2resub_1good.log

• This specifies 3 science tiles on 20250215. 2 needed resubmission, while one did not. It correctly resubmits the tilenight and cumulative jobs for just the 2 tiles that had previously failed, and nothing else.

Test 5: desi_resubmit_queue_failures -n 20250215 --dry-run-level=3 > test_all_resub.log

• This doesn't specify any specific exposures or tiles, so it submits all 5 failing tiles on that night, including both tilenight and cumulative jobs for each.

[coveralls]

coverage: 38.991% (+0.002%) from 38.989%
when pulling 97173ef on specify_resub
into 0852688 on main.

[sbailey]

Looks good, no requests. Will merge now.