Question: How to get activity retry information in middleware?

[LockTar]

Hi,

I've created a middleware for my durable function. This is working fine.

The middleware is creating/updating a shadow database of all the activities that are running with a custom status.
When an activity succeeds the status is completed. When there is an exception status is failed. When it's in progress the status is Running. Simple.

Now the question.
When an activity throws an exception, we sometimes have a retry policy for this activity.
How can I get the information from the retry policy? Reason:
When an activity is throwing an exception, my status is now set to failed. But when there is a retry (1 or multiple) I would like to still have the status to Running. Because functionally, nothing has changed.
So, I need the information:

1. Does this activity have a retry policy configured?
2. If it has a policy, how do I determine if this was the last "try" because then I need to set the status to failed.

Thanks!

[cgillum]

I assume you're asking about Functions middleware in the .NET Isolated worker.

The middleware pipeline doesn't know whether an activity is or is going to being retried. Only the orchestrator that scheduled the activity knows this. If you need middleware to take some action based on whether or not an activity is being retried as part of a retry policy, you'd need to use a custom retry handler which somehow provides this context. For example, including it as a field in the activity input.

[LockTar]

Thank you for the quick reply @cgillum.
Sorry that I wasn't clear. Yes, I'm talking about the Functions middleware in the .NET Isolated worker.

My middleware is now running for orchestrators and activities.
I understand that a retry for an activity is configured in the orchestrator.

So, when I use the Automatic retry on failure feature the middleware has no way to determine if this the first call, a retry and then of course if it's the last retry?
A bit strange because it should be stored somewhere in the (Azure storage/SQL Server) provider. Maybe I can add this with a PR if you point me in the right direction with the repository name where this is implemented??

If you need middleware to take some action based on whether or not an activity is being retried as part of a retry policy, you'd need to use a custom retry handler which somehow provides this context. For example, including it as a field in the activity input.

Your solution is then to use a custom retry handler
In example:

TaskOptions retryOptions = TaskOptions.FromRetryHandler(retryContext =>
{
    // Don't retry anything that derives from ApplicationException
    if (retryContext.LastFailure.IsCausedBy<ApplicationException>())
    {
        return false;
    }

    // TODO: Fill context:
    // - With the total number of attempts
    // - With the attempt number
    // - With a boolean indicating if this is the last attempt

    // Quit after N attempts
    return retryContext.LastAttemptNumber < 3;
});

try
{
    await context.CallActivityAsync("FlakeyActivity", options: retryOptions);
}
catch (TaskFailedException)
{
    // Case when the retry handler returns false...
}

But how can we add those values then? Hopefully not through the custom status? I don't really see an option for this...
I don't this is the correct way:

retryContext.OrchestrationContext.SetCustomStatus(new
{
    TotalAttempts = 3,
    AttemptNumber = retryContext.LastAttemptNumber,
    IsLastAttempt = retryContext.LastAttemptNumber == 3
});

[LockTar]

@cgillum do you have any ideas about this?