Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More robust handling of NERSC outages #28

Open
weaverba137 opened this issue Dec 19, 2024 · 0 comments
Open

More robust handling of NERSC outages #28

weaverba137 opened this issue Dec 19, 2024 · 0 comments
Assignees

Comments

@weaverba137
Copy link
Member

During NERSC outages, sometimes a night is only partially processed or not processed at all. At minimum we need a more visible warning when this does happen, and, potentially add automation to recover missing nights and exposures.

This could get complicated though, for example if the previous summary file has to always be read prior to processing the individual exposures.

There is also an issue with when any recovery would be run. If recovery is to be fully automated, do we wait for the next day, i.e. the next time the scron job runs?

@weaverba137 weaverba137 self-assigned this Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant