Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make task failure more elegant #57

Closed
ktran9891 opened this issue Jun 28, 2018 · 1 comment
Closed

Make task failure more elegant #57

ktran9891 opened this issue Jun 28, 2018 · 1 comment

Comments

@ktran9891
Copy link
Collaborator

Issue:
When tasks fail, they do so inelegantly. In other words: We ignore them. And when our Cron goes back to do it again, it fails over and over at the same task. This was fine before, but these tasks start to snowball and bog down our workflow. This manifests into us trying to perform feedbacks on the same hundreds of adsorption sites with many failing over and over again for things like QHulls.

Current state:
If a task fails, we see the next-nearest-dependency failure.

Desired state:
It would be great if our logs simply ignored dependency errors for tasks that are still going through FireWorks. It would also be great to get the root non-dependency-error[s] to pop up upon trying to execute a task and then have that task emailed to us. Then we could try to fix it or give up/flag it as "broken" somehow, and then GASpy would know to start ignoring that task.

@ktran9891
Copy link
Collaborator Author

The recent refactoring addressed this. The new Luigi version actually addresses most of this, as it gives a much more useful traceback and log file now. And the QHull errors are now gone, too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant