Skip to content

Add Archive workflow#231

Draft
zebbecker wants to merge 59 commits intostagingfrom
checkpoints
Draft

Add Archive workflow#231
zebbecker wants to merge 59 commits intostagingfrom
checkpoints

Conversation

@zebbecker
Copy link
Collaborator

@zebbecker zebbecker commented Aug 29, 2025

Adds a new DPS algorithm called eis-feds-archive.

This takes as input a single string, run_name. This tells DPS to look in FEDSinput/run_definitions/{run_name}/run_config.yaml for a YAML file that contains all input parameters and, optionally, overrides to the default environment settings.

The eis-feds-archive job uses DPS to complete the run defined in the run_config YAML, potentially as a series of chained subjobs.

Features:

  • Saves a checkpoint allfires and allpixels file every N days, where N = settings.ARCHIVE_RUN_JOB_SIZE
  • Splits large runs into smaller chunks and automatically handles submitting those chunks, in order, to DPS
  • Copies logs to output directory
  • Copies environment and platform metadata to output directory
  • Supports stateless retries that pick up where previous runs failed without needing to change any input parameters

@zebbecker zebbecker marked this pull request as draft September 2, 2025 18:22
@zebbecker zebbecker mentioned this pull request Sep 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant