Whenever a transient failure on an evaluation run happens the platform needs to analyse and retry the evaluation run if possible. We need to spike to understand in what conditions the retry makes sense. E.g. If retrying the eval run can put the agent on top it make sense to retry.
Right now an evaluation run can only be executed once. We need to expand this concept to allow for retries, capturing the results of each individual retry. We also need to explore ways of triggering this retry, platform retry vs validator retry.
Whenever a transient failure on an evaluation run happens the platform needs to analyse and retry the evaluation run if possible. We need to spike to understand in what conditions the retry makes sense. E.g. If retrying the eval run can put the agent on top it make sense to retry.
Right now an evaluation run can only be executed once. We need to expand this concept to allow for retries, capturing the results of each individual retry. We also need to explore ways of triggering this retry, platform retry vs validator retry.